ClickHouse performance comparison

Tested Commits ^?

Reference	Patched
ClickHouse local version 25.6.1.2286 (official build).	commit 14872a3176293fc242e45360efa1fbb74acbc65b Author: Daniil Ivanik <61067749+divanik@users.noreply.github.com> Date: Sat Jun 7 16:36:23 2025 +0200 Merge 075e9eb09b752a1071fe908ee0f14d862dbb9a52 into ba10709d43407fee47aa7cf7b87f1fe18e1242f5

Reference

Patched

ClickHouse local version 25.6.1.2286 (official build).

commit 14872a3176293fc242e45360efa1fbb74acbc65b
Author: Daniil Ivanik <61067749+divanik@users.noreply.github.com>
Date:   Sat Jun 7 16:36:23 2025 +0200

    Merge 075e9eb09b752a1071fe908ee0f14d862dbb9a52 into ba10709d43407fee47aa7cf7b87f1fe18e1242f5

Error Summary ^?

Description
The query no. 2 of test 'set_index' has slowed down
The query no. 3 of test 'distinct_combinator' has slowed down
The query no. 21 of test 'int_parsing' has slowed down

Changes in Performance ^?

Old, s	New, s	Ratio of speedup (-) or slowdown (+)	Relative difference (new − old) / old	p < 0.01 threshold	Test	#	Query
0.02	0.024	+1.209x	0.203	0.051	set_index	2	SELECT count() FROM test_in WHERE -toInt64(a) IN (SELECT toInt64(rand(1)) FROM numbers(200000)) settings max_rows_to_read=200001, read_overflow_mode='break'
0.19	0.209	+1.101x	0.101	0.101	distinct_combinator	3	SELECT countIf(DISTINCT number % 10, number % 5 = 2) FROM numbers(100000000)
0.31	0.342	+1.101x	0.101	0.101	int_parsing	21	SELECT count() FROM hits_100m_single WHERE NOT ignore(toString(IsMobile)) SETTINGS max_threads = 1
0.21	0.23	+1.091x	0.091	0.091	distinct_combinator	2	SELECT sumIf(DISTINCT number % 10, number % 5 = 2) FROM numbers(100000000)
1.736	1.586	-1.095x	-0.087	0.086	order_by_single_nullable	1	select b from t_opt_orderby order by b
0.034	0.037	+1.088x	0.086	0.06	set_index	0	SELECT count() FROM test_in WHERE a IN (SELECT rand(1) FROM numbers(200000)) SETTINGS max_rows_to_read = 200001, read_overflow_mode = 'break'
0.068	0.073	+1.068x	0.067	0.05	count_from_formats	38	SELECT count() FROM table_Values group by _file, _path FORMAT Null
0.005	0.006	+1.057x	0.061	0.033	merge_tree_many_partitions	0	SELECT count() FROM bad_partitions

Unstable Queries ^?

Old, s	New, s	Relative difference (new - old)/old	p < 0.01 threshold	Test	#	Query
1.286	1.225	-0.048	1.443	number_formatting_formats	5	INSERT INTO table_JSONEachRow SELECT number FROM numbers(10000000) SETTINGS engine_file_truncate_on_insert = 1
0.074	0.131	0.769	0.803	map_update	2	WITH (range(100), range(100))::Map(String, UInt64) AS m1, (range(30), range(30))::Map(String, UInt64) AS m2 SELECT count() FROM numbers(50000) WHERE NOT ignore(mapUpdate(materialize(m1), materialize(m2)))
0.388	0.505	0.3	0.621	position_empty_needle	0	select position(materialize('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'), '') from numbers(100000000) format Null
0.414	0.475	0.148	0.568	position_empty_needle	2	select positionCaseInsensitive(materialize('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'), '') from numbers(100000000) format Null
0.454	0.618	0.362	0.501	uniq_with_key	4	SELECT uniqExact(a) FROM t_10000 group by b
0.107	0.106	-0.015	0.442	map_update	3	WITH (range(100), range(100))::Map(String, UInt64) AS m1, (range(30), range(30))::Map(String, UInt64) AS m2 SELECT count() FROM numbers(50000) WHERE NOT ignore(mapUpdate(materialize(m1), m2))
0.661	0.534	-0.191	0.383	position_empty_needle	1	select position(materialize('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'), '', 10) from numbers(100000000) format Null
0.992	0.946	-0.046	0.375	prefetch_in_aggregation	12	select number from numbers_mt(1e8) group by number format Null
0.23	0.29	0.263	0.368	polymorphic_parts_m	1	INSERT INTO hits_compact(UserID) SELECT rand() FROM numbers(10000)
1.428	1.5	0.049	0.364	number_formatting_formats	2	INSERT INTO table_Values SELECT number FROM numbers(10000000) SETTINGS engine_file_truncate_on_insert = 1
0.506	0.434	-0.143	0.303	position_empty_needle	3	select positionCaseInsensitive(materialize('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'), '', 10) from numbers(100000000) format Null
0.098	0.12	0.227	0.302	asof	2	SELECT COUNT(*) AS count FROM ( SELECT (number % 1000) AS visitor_id, number AS id FROM system.numbers LIMIT 1000000 ) AS sessions ASOF LEFT JOIN ( SELECT (number % 1000) AS visitor_id, number AS starting_session_id FROM system.numbers LIMIT 1000000 ) AS visitors ON visitors.visitor_id = sessions.visitor_id AND visitors.starting_session_id < sessions.id
0.376	0.423	0.124	0.276	position_empty_needle	6	select positionCaseInsensitiveUTF8(materialize('xẞyyaa1ẞ1yzẞXẞẞ1ẞẞ1bctest'), '') from numbers(100000000) format Null
0.399	0.43	0.078	0.271	norm_distance	5	SELECT sum(dist) FROM (SELECT LinfNorm(v) AS dist FROM vecs_Float32)
0.134	0.128	-0.046	0.263	string_join	2	SELECT 1 FROM strings AS l ANY LEFT JOIN strings AS r USING (short) FORMAT Null
0.209	0.208	-0.008	0.26	norm_distance	1	SELECT sum(dist) FROM (SELECT L2Norm(v) AS dist FROM vecs_BFloat16)
0.516	0.426	-0.174	0.241	string_join	1	SELECT 1 FROM hits_10m_words AS l ANY LEFT JOIN hits_10m_words AS r USING (word) FORMAT Null settings join_algorithm='parallel_hash'
0.177	0.174	-0.022	0.24	insert_select_squashing_dynamic	2	INSERT INTO dynamic_squash_performance_1 SELECT * FROM src_dynamic_squash_performance_1
0.028	0.031	0.103	0.238	has_all	8	SELECT hasAll(set, subset) FROM test_table_large_Int8 FORMAT Null
0.126	0.147	0.16	0.237	string_set	1	SELECT 1 FROM strings WHERE short IN (SELECT short FROM strings) FORMAT Null
0.144	0.145	0.006	0.235	join_used_flags	0	SELECT l.i64, r.i64, l.i32, r.i32 FROM test_join_used_flags l RIGHT JOIN test_join_used_flags r USING i64 format Null
0.224	0.226	0.006	0.228	general_purpose_hashes	60	SELECT count() from zeros(1000000) where not ignore(metroHash64(materialize('Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris sollicitudin nisi ac erat mollis dapibus. Maecenas leo purus, bibendum eu erat eget, iaculis molestie tortor. Phasellus maximus odio nec mauris ultrices dictum. Morbi efficitur nisl eget congue mollis. Vestibulum pharetra diam vitae urna interdum, eget ultricies justo sollicitudin. Nunc sit amet purus id leo tempus dignissim. Donec ac lacus ut orci tempus scelerisque quis ultricies nibh. Nullam lobortis, erat ac ullamcorper interdum, odio nisl elementum quam, ut malesuada massa nunc eget quam. Nam suscipit neque quis sapien ultricies imperdiet. Maecenas augue libero, finibus tristique sagittis et, semper nec arcu. Morbi non tortor ultrices, sollicitudin justo sed, accumsan ligula. Nullam at ipsum in nibh auctor ullamcorper. Nullam laoreet neque id lorem condimentum tincidunt. Nullam vel orci nibh. Ut sit amet sem faucibus, fringilla orci at, lacin...(60)
0.163	0.199	0.22	0.226	logical_functions_medium	19	SELECT count() FROM test_logical_functions._4_1_Nullable_Mixed WHERE NOT ignore(xor(x1,x2,x3,x4))
0.04	0.038	-0.062	0.224	order_by_decimals	0	SELECT toInt32(number) AS n FROM numbers(10000000) ORDER BY n DESC FORMAT Null
0.064	0.076	0.18	0.222	range	0	SELECT range(number % 100) FROM numbers(10000000) FORMAT Null

Test Performance Changes ^?

Test	Ratio of speedup (-) or slowdown (+)	Queries	Total not OK	Changed perf	Unstable
Total	+1.005x	558	46	8	38
order_by_single_nullable	-1.096x	2	2	1	1
string_join	-1.078x	4	2	0	2
encodeXMLComponent	-1.046x	1	0	0	0
lightweight_delete	-1.04x	1	0	0	0
materialized_view_parallel_insert	-1.038x	1	1	0	1
push_down_limit	-1.029x	3	1	0	1
set_disable_skip_index	-1.026x	1	0	0	0
array_fill	-1.026x	6	1	0	1
json_type	-1.023x	3	0	0	0
generate_table_function	-1.023x	10	0	0	0
order_by_decimals	-1.017x	5	1	0	1
aggregation_overflow	-1.017x	3	0	0	0
columns_hashing	-1.016x	5	0	0	0
group_array_moving_sum	-1.016x	10	0	0	0
norm_distance	-1.014x	10	5	0	5
trim_whitespace	-1.012x	6	0	0	0
if_transform_strings_to_enum	-1.012x	4	0	0	0
constant_column_search	-1.011x	10	1	0	1
coalesce	-1.01x	1	0	0	0
hashjoin_with_large_output	-1.009x	3	0	0	0
leftpad	-1.008x	7	0	0	0
point_in_polygon_const_3d	-1.007x	1	0	0	0
read_hits_with_aio	-1.007x	4	0	0	0
jit_small_requests	-1.006x	3	0	0	0
constant_column_comparison	-1.006x	10	0	0	0
insert_values_with_expressions	-1.006x	2	0	0	0
parallel_mv	-1.006x	1	0	0	0
jit_sort	-1.006x	10	0	0	0
merge_table_streams	-1.005x	1	0	0	0
basename	-1.005x	2	0	0	0
number_formatting_formats	-1.005x	10	2	0	2
concat_hits	-1.005x	10	1	0	1
prefetch_in_aggregation	-1.004x	10	1	0	1
logical_functions_medium	-1.003x	10	2	0	2
general_purpose_hashes	-1x	10	1	0	1
insert_select_squashing_dynamic	+1.002x	4	1	0	1
round_methods	+1.002x	10	1	0	1
optimize_sorting_for_input_stream	+1.005x	10	0	0	0
uniq_without_key	+1.005x	10	0	0	0
parallel_hash_build_phase	+1.005x	1	0	0	0
trivial_insert_select_from_files	+1.006x	9	0	0	0
join_used_flags	+1.006x	1	1	0	1
group_array_sorted	+1.007x	10	0	0	0
low_cardinality_insert	+1.007x	1	0	0	0
arrow_format	+1.007x	3	0	0	0
column_array_filter	+1.008x	8	0	0	0
point_in_polygon_3d_huge_multipolygon	+1.008x	1	0	0	0
empty_string_deserialization	+1.009x	1	0	0	0
logical_functions_small	+1.009x	10	0	0	0
int_parsing	+1.009x	10	1	1	0
IPv6	+1.009x	2	0	0	0
one_thousand_joins	+1.01x	1	0	0	0
hierarchical_dictionaries	+1.01x	10	0	0	0
IPv4	+1.011x	4	0	0	0
prepare_hash_before_merge	+1.011x	4	0	0	0
count_from_formats	+1.012x	10	1	1	0
injective_functions_inside_uniq	+1.012x	2	0	0	0
ip_trie	+1.016x	4	0	0	0
lot_of_subcolumns	+1.017x	1	0	0	0
lazyMaterialization	+1.02x	2	0	0	0
dotProduct	+1.022x	4	1	0	1
has_all	+1.023x	10	1	0	1
removing_group_by_keys	+1.024x	2	0	0	0
query_interpretation_join	+1.024x	1	0	0	0
materialized_view_deduplication	+1.026x	1	0	0	0
set_index	+1.028x	10	2	2	0
sparse_grams	+1.031x	2	0	0	0
dict_join	+1.034x	2	0	0	0
position_empty_needle	+1.034x	8	6	0	6
compact_part_subcolumns	+1.037x	2	0	0	0
string_set	+1.038x	7	1	0	1
distinct_combinator	+1.046x	4	2	2	0
insert_sequential_and_background_merges	+1.047x	1	0	0	0
uniq_with_key	+1.056x	7	1	0	1
merge_tree_many_partitions	+1.057x	1	1	1	0
asof	+1.066x	4	1	0	1
merge_tree_simple_select	+1.091x	1	0	0	0
polymorphic_parts_m	+1.093x	3	1	0	1
range	+1.098x	2	1	0	1
map_update	+1.181x	4	2	0	2

Test Times ^?

Test	Wall clock time, entire test, s	Total client time for measured query runs, s	Queries	Longest query, total for measured runs, s	Average query wall clock time, s	Shortest query, total for measured runs, s
Total	4002.481	2566.064	558	22.087	7.173	1.185
lightweight_delete	47.965	2.018	1	2.018	47.965	2.018
insert_sequential_and_background_merges	34.249	18.379	1	18.379	34.249	18.379
lot_of_subcolumns	32.672	2.002	1	2.002	32.672	2.002
polymorphic_parts_m	84.338	40.098	3	18.752	28.113	3.693
order_by_single_nullable	46.288	38.218	2	20.055	23.144	18.163
uniq_with_key	161.826	55.506	7	10.958	23.118	6.15
one_thousand_joins	22.785	18.42	1	18.42	22.785	18.42
avg_serialization	20.792	17.28	1	17.28	20.792	17.28
final_big_column	41.185	4.249	2	2.229	20.593	2.02
merge_tree_many_partitions	20.235	2.008	1	2.008	20.235	2.008
jit_sort	199.945	149.56	10	18.564	19.995	10.83
parallel_hash_build_phase	18.086	13.96	1	13.96	18.086	13.96
group_array_moving_sum	171.796	113.457	10	18.317	17.18	5.784
count_from_formats	171.48	20.961	10	2.263	17.148	2.003
coalesce	16.288	13.81	1	13.81	16.288	13.81
compact_part_subcolumns	31.757	4.248	2	2.145	15.879	2.102
set_index	154.195	20.911	10	2.52	15.419	2.004
insert_values_with_expressions	30.018	22.401	2	19.146	15.009	3.255
number_formatting_formats	137.885	130.083	10	22.087	13.789	2.03
prefetch_in_aggregation	137.878	93.612	10	19.73	13.788	2.007
materialized_view_parallel_insert	11.731	8.712	1	8.712	11.731	8.712
point_in_polygon_3d_huge_multipolygon	11.092	9.137	1	9.137	11.092	9.137
position_empty_needle	87.833	75.538	8	18.587	10.979	5.932
constant_column_search	104.203	90.384	10	17.333	10.42	2.445
basename	20.276	16.074	2	8.415	10.138	7.659
read_hits_with_aio	40.155	33.515	4	18.116	10.039	2.058
hierarchical_dictionaries	98.641	84.721	10	15.057	9.864	2.012
dotProduct	37.411	24.993	4	8.684	9.353	4.049
uniq_without_key	90.866	31.769	10	7.843	9.087	2.035
empty_string_deserialization	9.05	3.734	1	3.734	9.05	3.734
parallel_mv	8.952	6.61	1	6.61	8.952	6.61
trivial_insert_select_from_files	80.514	48.37	9	7.723	8.946	4.383
columns_hashing	42.673	36.599	5	14.625	8.535	2.098
right	16.206	13.669	2	7.232	8.103	6.437
has_all	80.635	25.691	10	5.284	8.064	2.005
distinct_combinator	32.068	10.011	4	3.084	8.017	2.004
norm_distance	78.296	58.445	10	10.153	7.83	3.18
int_parsing	76.541	56.591	10	10.412	7.654	2.012
if_string_hits	30.254	26.009	4	13.172	7.564	2.032
set_disable_skip_index	7.308	2.045	1	2.045	7.308	2.045
group_array_sorted	71.787	20.458	10	2.105	7.179	2.02
trim_whitespace	42.848	15.676	6	3.346	7.141	2.013
if_transform_strings_to_enum	27.904	23.791	4	8.65	6.976	2.247
aggregation_overflow	20.791	17.645	3	7.621	6.93	2.672
string_join	27.577	19.44	4	7.298	6.894	2.165
base64	60.982	52.14	9	11.926	6.776	2.055
constant_column_comparison	67.746	48.981	10	7.481	6.775	2.082
function_calculation_after_sorting_and_limit	20.286	17.107	3	12.956	6.762	2.038
inserts_arrays_lowcardinality	12.698	9.444	2	7.249	6.349	2.196
ip_trie	25.359	8.806	4	2.515	6.34	2.031
general_purpose_hashes	61.861	53.575	10	8.84	6.186	2.036
uniq	60.319	51.46	10	8.692	6.032	2.017
push_down_limit	17.777	15.336	3	9.694	5.926	2.027
concat_hits	56.463	48.758	10	9.54	5.646	2.026
generate_table_function	54.364	47.073	10	11.015	5.436	2.036
uniq_with_key_dist	35.906	26.967	7	4.457	5.129	2.873
array_fill	30.442	24.069	6	9.679	5.074	2.079
low_cardinality_insert	5.062	3.936	1	3.936	5.062	3.936
removing_group_by_keys	9.726	7.657	2	3.946	4.863	3.711
leftpad	33.689	29.379	7	6.311	4.813	2.025
insert_select_squashing_dynamic	18.16	13.919	4	6.461	4.54	2.206
string_set	31.418	22.836	7	5.035	4.488	2.075
hashjoin_with_large_output	12.34	10.175	3	3.494	4.113	3.238
asof	16.002	13.272	4	4.7	4	2.119
merge_tree_simple_select	3.871	1.232	1	1.232	3.871	1.232
simple_join_query	15.112	8.444	4	2.123	3.778	2.097
logical_functions_medium	37.379	23.131	10	4.245	3.738	2.019
uniqExactIf	18.638	15.482	5	5.68	3.728	2.009
injective_functions_inside_uniq	7.323	4.215	2	2.108	3.661	2.107
encodeXMLComponent	3.63	2.193	1	2.193	3.63	2.193
lazyMaterialization	7.232	5.424	2	3.017	3.616	2.408
flat_dictionary	35.539	30.275	10	5.586	3.554	2.004
async_remote_read	10.488	8.62	3	4.455	3.496	2.061
optimize_sorting_for_input_stream	33.614	30.554	10	9.309	3.361	2.005
insert_sparse_column	3.335	2.16	1	2.16	3.335	2.16
json_type	9.961	8.051	3	3.786	3.32	2.031
float_formatting	32.875	28.818	10	5.358	3.288	2.028
prepare_hash_before_merge	12.841	8.789	4	2.255	3.21	2.129
optimize_window_funnel	3.176	2.185	1	2.185	3.176	2.185
jit_small_requests	9.52	7.793	3	3.661	3.173	2.065
explain_ast	3.017	2.169	1	2.169	3.017	2.169
join_used_flags	2.954	2.094	1	2.094	2.954	2.094
h3	2.916	2.142	1	2.142	2.916	2.142
point_in_polygon_const_3d	2.902	2.099	1	2.099	2.902	2.099
IPv6	5.782	4.584	2	2.49	2.891	2.094
math	28.57	25.217	10	4.503	2.857	2.011
merge_table_streams	2.831	2.031	1	2.031	2.831	2.031
encrypt_decrypt_empty_string	28.262	23.007	10	2.562	2.826	2.015
dict_join	5.517	4.199	2	2.179	2.758	2.02
query_interpretation_join	2.755	2.042	1	2.042	2.755	2.042
column_array_filter	21.845	18.554	8	2.977	2.731	2.082
join_merge_filter_into_join	2.707	2.139	1	2.139	2.707	2.139
orc_filter_push_down	5.391	4.039	2	2.024	2.696	2.014
replaceRegexp_fallback	10.713	9.196	4	2.674	2.678	2.021
arrow_format	7.978	6.164	3	2.096	2.659	2.001
range	5.271	4.299	2	2.18	2.636	2.119
analyze_array_tuples	2.585	2.009	1	2.009	2.585	2.009
logical_functions_small	25.735	22.355	10	2.716	2.574	2.002
materialized_view_deduplication	2.56	2.01	1	2.01	2.56	2.01
sequence_match	25.294	21.005	10	2.218	2.529	2.021
order_by_decimals	12.584	10.674	5	2.225	2.517	2.043
writing_valid_utf8	7.55	6.279	3	2.141	2.517	2.028
functions_geo	15.095	13.167	6	2.649	2.516	2.04
map_update	9.856	8.25	4	2.177	2.464	2.004
round_methods	24.54	21.769	10	2.242	2.454	2.095
datetime64_conversion	4.881	4.093	2	2.073	2.441	2.02
bigint_formatting	9.613	8.389	4	2.129	2.403	2.052
tsv_csv_nullable_parsing	4.789	4.069	2	2.04	2.395	2.029
and_function	4.745	4.057	2	2.032	2.373	2.025
rewrite_aggregate_if	9.302	8.187	4	2.103	2.325	2.009
join_filter_pushdown_equivalent_sets	11.458	10.062	5	2.016	2.292	2.009
bit_operations_fixed_string	9.036	8.077	4	2.032	2.259	2.006
IPv4	9.028	8.116	4	2.068	2.257	2.009
count	6.745	5.297	3	2.063	2.248	1.185
non_function_col_short_circuit	8.906	8.066	4	2.019	2.227	2.013
arithmetic_operations_in_aggr_func	13.026	12.183	6	2.039	2.171	2.013
joins_in_memory	21.626	20.325	10	2.111	2.163	2.008
parse_illegal_datetime	7.866	7.268	4	2.008	1.966	1.62
sparse_grams	3.839	3.314	2	1.749	1.92	1.565

Metric Changes ^?

Metric	Old median value	New median value	Relative difference	Times difference
MemoryShared	96743424	260145152	1.689	2.689
MemoryResidentWithoutPageCache	1353502720	1501634560	0.109	1.109
MemoryResident	1353502720	1501634560	0.109	1.109
MemoryVirtual	58064134144	63393464320	0.092	1.092
jemalloc.retained	48476590080	52029415424	0.073	1.073
MemoryDataAndStack	57003692032	60458749952	0.061	1.061
jemalloc.arenas.all.dirty_purged	166398181	156355444	-0.06	1.064
TotalRowsOfMergeTreeTablesSystem	267995	237716	-0.113	1.127
MaxPartCountForPartition	8	7	-0.125	1.143
KeeperOpenFileDescriptorCount	157	133	-0.153	1.18

ClickHouse performance comparison

Tested Commits ?

Error Summary ?

Changes in Performance ?

Unstable Queries ?

Test Performance Changes ?

Test Times ?

Metric Changes ?

Tested Commits ^?

Error Summary ^?

Changes in Performance ^?

Unstable Queries ^?

Test Performance Changes ^?

Test Times ^?

Metric Changes ^?