thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 19:32:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,500 Commits 9,319 Branches 1,783 Tags 2.1 GiB
dcbd3b9c26
Commit Graph

15578 Commits

Author SHA1 Message Date
kssenii
c4bad255b7 Fix 2023-10-18 14:37:33 +02:00
Igor Nikonov
01fa15c33f Less copying while handling annoucement 2023-10-18 12:20:14 +00:00
Duc Canh Le
2d638d6470 rocksdb: not creating block during writing 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-10-18 09:43:54 +00:00
Amos Bird
602f01f651
Reuse granule during skip index reading 2023-10-18 14:40:34 +08:00
Igor Nikonov
135e4c8412 Merge remote-tracking branch 'origin/master' into pr-progress-bar 2023-10-17 20:42:39 +00:00
Igor Nikonov
fb3d3a37ba Add progress bar for in order coordinator 2023-10-17 20:09:39 +00:00
Alexey Milovidov
79eccfb642
Merge pull request #55678 from azat/fix-OR-resubmit 
Fix filtering by virtual columns with OR filter in query (resubmit)
 2023-10-17 20:33:19 +02:00
avogar
323486f9e8 Add tests 2023-10-17 18:10:47 +00:00
kssenii
eb4519b9d6 Fix 2023-10-17 19:02:04 +02:00
Kseniia Sumarokova
7cedfeff12
Merge pull request #55145 from kssenii/materialized-pg-allow-unique-identifier 
Allow to have several MaterializedPostgreSQL tables following the same Postgres table
 2023-10-17 18:47:32 +02:00
Han Fei
4b23142775
Merge branch 'master' into hanfei/statistic 2023-10-17 16:40:31 +02:00
vdimir
c657e500ac
Analyzer: fix block stucture mismatch in matview with engine distributed 2023-10-17 14:02:30 +00:00
vdimir
1d2ac48261
Merge pull request #55681 from ClickHouse/vdimir/fix_analyzer_materialized_distributed_join 
Fix StorageMaterializedView::isRemote
 2023-10-17 16:00:15 +02:00
alesapin
abe6df6fbc
Merge pull request #55645 from ClickHouse/restore_top_level_parts 
Restore only top-level parts instead of unexpected
 2023-10-17 15:45:21 +02:00
kssenii
c549083e2c Fix 2023-10-17 14:23:59 +02:00
avogar
032d82e004 Fixes 2023-10-17 12:12:10 +00:00
Alexander Gololobov
11a62bc751
Merge pull request #55621 from ClickHouse/merge-52352 
Merging #52352
 2023-10-17 13:58:53 +02:00
avogar
527e96db3f Fix 2023-10-17 11:43:51 +00:00
avogar
4892618aa9 Update with master, fix comments 2023-10-17 11:42:19 +00:00
avogar
bb46aa4d57 Clean uo 2023-10-17 11:07:41 +00:00
avogar
f87938735a Fixes and improvements for Iceberg storage 2023-10-17 11:07:37 +00:00
Duc Canh Le
f20d055e1e do not re-create serialization for each row 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-10-17 10:12:27 +00:00
kssenii
42ed249954 Fix build 2023-10-17 12:03:49 +02:00
Robert Schulze
96a82df75a
Merge pull request #55616 from HarryLeeIBM/hlee-fts-max-rows 
Inverted index: Replace setting density with `max_rows_in_postings_list`
 2023-10-17 11:36:46 +02:00
Raúl Marín
3eaf752284 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-17 11:36:39 +02:00
Kseniia Sumarokova
550a658236
Merge pull request #55144 from kssenii/fix-iceberg-metadata-parsing 
Fix storage Iceberg files retrieval
 2023-10-17 11:31:07 +02:00
kssenii
4464c86895 Merge remote-tracking branch 'origin/master' into s3-queue-fixes 2023-10-17 11:16:52 +02:00
kssenii
8744cd9c0d Fix 2023-10-17 11:15:53 +02:00
Igor Nikonov
4b36761f89
Merge branch 'master' into pr-progress-bar 2023-10-17 10:36:32 +02:00
Alexey Milovidov
de8215f5dd Merge remote-tracking branch 'origin/master' into morton-utils 2023-10-17 04:55:45 +02:00
HarryLeeIBM
fb8cd298d1 Code refactoring according to code review 2023-10-16 17:52:39 -07:00
Michael Kolupaev
ce7eca0615
DWARF input format (#55450) 
* Add ReadBufferFromFileBase::isRegularLocalFile()

* DWARF input format

* Review comments

* Changed things around ENABLE_EMBEDDED_COMPILER build setting

* Added 'ranges' column

* no-msan no-ubsan
 2023-10-16 17:00:07 -07:00
Ilya Golshtein
786183ee24 merge_row_policy: get rid of RequiredSourceColumnsVisitor 2023-10-16 22:16:53 +00:00
Ilya Golshtein
7f991bf6e7 merge_row_policy: a comment 2023-10-16 21:49:16 +00:00
robot-clickhouse
89eed73fb9
Merge pull request #55519 from azat/zk-retry-fix 
Avoid retrying keeper operations on INSERT after table shutdown
 2023-10-16 23:32:13 +02:00
Ilya Golshtein
ea95a49e86 merge_row_policy: some changes per code review 2023-10-16 21:22:43 +00:00
Kseniia Sumarokova
96c518be5b
Merge branch 'master' into s3-queue-fixes 2023-10-16 22:19:13 +02:00
Kseniia Sumarokova
3e9a1cf69f
Merge branch 'master' into fix-iceberg-metadata-parsing 2023-10-16 22:18:52 +02:00
Kseniia Sumarokova
c31299d86c
Merge branch 'master' into materialized-pg-allow-unique-identifier 2023-10-16 22:18:23 +02:00
robot-ch-test-poll1
aca98cf725
Merge pull request #53774 from helifu/master13 
Display reloaded values of server settings
 2023-10-16 21:17:17 +02:00
Kruglov Pavel
cbdb62d389
Merge pull request #55666 from ClickHouse/Avogar-patch-1 
Fix progress bar for s3 and azure Cluster functions with url without globs
 2023-10-16 18:25:00 +02:00
kssenii
0a6a4b3894 Review fixes 2023-10-16 18:15:17 +02:00
Kseniia Sumarokova
8f3974d289
00002_log_and_exception_messages_formatting ... 2023-10-16 17:30:02 +02:00
vdimir
c9a2a4922d
Fix StorageMaterializedView::isRemote 2023-10-16 15:18:47 +00:00
Igor Nikonov
6abd1a6462
Merge branch 'master' into zk-retry-fix 2023-10-16 16:56:07 +02:00
Azat Khuzhin
96f2a46a66 Fix filtering by virtual columns with OR filter in query 
The problem with the initial implementation #52653 was:
- OR can have multiple arguments
- It simply not correct to assume that if there are two arguments this is OK.
  Consider the following example:

    "WHERE (column_not_from_partition_by = 1) OR false OR false"

  Will be converted to:

    "WHERE false OR false"

And it will simply read nothing.

Yes, we could apply some optimization for bool, but this will not always
work, since to optimize things like "0 = 1" we need to execute it.

And the only way to make handle this correctly (with ability to ignore
some commands during filtering) is to make is_constant() function return
has it use something from the input block, so that we can be sure, that
we have some sensible, and not just "false".

Plus we cannot simply ignore the difference of the input and output
arguments of handling OR, we need to add always-true (1/true) if the
size is different, since otherwise it could break invariants (see
comment in the code).

This includes (but not limited to):
- _part* filtering for MergeTree
- _path/_file for various File/HDFS/... engines
- _table for Merge
- ...

P.S. analyzer does not have this bug, since it execute expression as
whole, and this is what filterBlockWithQuery() should do actually
instead, but this will be a more complex patch.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit b107712e0c)
 2023-10-16 15:42:10 +02:00
Raúl Marín
f799f5d7a1 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-16 15:07:11 +02:00
Igor Nikonov
fdd7bfbb83 Merge remote-tracking branch 'origin/master' into pr-progress-bar 2023-10-16 12:59:00 +00:00
alesapin
9a925a0802 Fix typo 2023-10-16 13:35:20 +02:00
Raúl Marín
478f635d60
Merge pull request #53792 from ClickHouse/revert-53782-revert-52762-planner-prepare-filters-for-analysis 
Revert "Revert "Planner prepare filters for analysis""
 2023-10-16 13:21:54 +02:00
helifu
dc0f8ed793 Add a new column 'is_hot_reloadable' to indicate whether hot loading is supported 2023-10-16 19:10:35 +08:00
serxa
0e17da22eb Merge branch 'master' into async-loader-integration 2023-10-16 11:08:29 +00:00
alesapin
31dc46a889 Better test 2023-10-16 13:04:45 +02:00
Kruglov Pavel
6dcc756948
Fix for azure 2023-10-16 12:38:55 +02:00
Kruglov Pavel
836e35b6c4
Fix progress bar for s3 and azure Cluster functions with url without globs 2023-10-16 12:38:10 +02:00
alesapin
dbebdc3dca Fix comment 2023-10-16 11:05:59 +02:00
Antonio Andelic
6c543c730b
Merge pull request #55657 from ClickHouse/revert-55418-fix-OR 
Revert "Fix filtering by virtual columns with OR filter in query"
 2023-10-16 10:51:06 +02:00
alesapin
2d9812b9e9 Fix headers 2023-10-16 10:43:25 +02:00
alesapin
692fcd8422 Merge remote-tracking branch 'origin/master' into repro_49887 2023-10-16 10:39:16 +02:00
alesapin
697c1e3bb0
Merge pull request #55604 from ClickHouse/evillique-partition-parameters 
Implement query paramenters support in partition expression
 2023-10-16 10:26:02 +02:00
alesapin
26a8aa441c Better shutdown handling 2023-10-16 10:24:54 +02:00
Antonio Andelic
fb901c24a1
Revert "Fix filtering by virtual columns with OR filter in query" 2023-10-16 09:45:49 +02:00
Alexey Milovidov
f5aabbecfc Add a comment 2023-10-16 02:24:34 +02:00
Alexey Milovidov
26c0c0b86d More tests 2023-10-16 02:19:24 +02:00
Alexey Milovidov
8e8471f5db Fix error 2023-10-16 00:25:00 +02:00
Alexey Milovidov
724f04e6f6 Allow conditions on part of arguments 2023-10-15 22:21:27 +02:00
Alexey Milovidov
c932a5e0a1 Allow conditions on part of arguments 2023-10-15 22:21:10 +02:00
Alexey Milovidov
0bf6fd3671 Merge remote-tracking branch 'origin/master' into morton-utils 2023-10-15 21:28:00 +02:00
alesapin
bdafe17e9a
Update src/Storages/MergeTree/ActiveDataPartSet.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2023-10-15 20:17:55 +02:00
alesapin
13ca296497
Update src/Storages/MergeTree/ActiveDataPartSet.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2023-10-15 20:17:50 +02:00
alesapin
6ac790e918
Update src/Storages/MergeTree/ActiveDataPartSet.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2023-10-15 20:17:46 +02:00
alesapin
8a865cdfb2
Merge pull request #55646 from ClickHouse/fix_some_typos 
Fix some typos
 2023-10-15 18:28:33 +02:00
Alexey Milovidov
0f89b4aad6
Merge pull request #55640 from ClickHouse/preparation-key-condition-2 
KeyCondition: small changes
 2023-10-15 18:23:53 +02:00
Kseniia Sumarokova
2015a0fc17
Merge branch 'master' into fix-iceberg-metadata-parsing 2023-10-15 18:03:53 +02:00
kssenii
f3056b1643 Add a test 2023-10-15 18:02:34 +02:00
alesapin
3b02748cb6 Fix some typos 2023-10-15 15:43:02 +02:00
alesapin
41479b21e6 Restore only top-level parts instead of unexpected 2023-10-15 15:33:33 +02:00
kssenii
0ddee6ad6c Fix 2023-10-15 14:44:57 +02:00
Kseniia Sumarokova
b26115eabe
Merge branch 'master' into materialized-pg-allow-unique-identifier 2023-10-15 14:17:32 +02:00
Alexander Gololobov
395a54915d
Fix totalBytes() 2023-10-15 10:40:37 +02:00
Alexey Milovidov
42f3751bd5 Minor changes 2023-10-15 06:33:59 +02:00
Alexey Milovidov
17cbc12831
Update KeyCondition.cpp 2023-10-15 07:20:04 +03:00
Alexey Milovidov
22b06e61f2 Add a comment 2023-10-15 06:00:28 +02:00
Alexey Milovidov
569f42d1d9 It might be working 2023-10-15 05:36:53 +02:00
Alexey Milovidov
daaa3db809 Add space-filling curves 2023-10-15 04:37:10 +02:00
Alexey Milovidov
438811ea8c Add space-filling curves 2023-10-15 04:34:26 +02:00
Alexey Milovidov
1ef7bc41d4 Fix garbage 2023-10-15 00:07:28 +02:00
Alexey Milovidov
7764dedb8e Something strange 2023-10-14 22:44:44 +02:00
Alexey Milovidov
25efd188d5 KeyCondition: small changes 2023-10-14 22:17:36 +02:00
Alexey Milovidov
50ee412b19
Merge pull request #55625 from ClickHouse/preparation-key-condition 
KeyCondition: preparation
 2023-10-14 22:13:39 +02:00
Alexey Milovidov
bafabf1a56 Fix typo in comment 2023-10-14 19:31:46 +02:00
Alexey Milovidov
d6984a2819 Preparation 2023-10-14 04:46:22 +02:00
Alexey Milovidov
a904ee970f KeyCondition: preparation 2023-10-14 04:32:25 +02:00
Alexey Milovidov
bbd67d262a Merging #52352 2023-10-14 02:52:53 +02:00
Alexey Milovidov
769ed2e19d
Merge branch 'master' into fix-OR 2023-10-14 02:25:21 +02:00
alesapin
cc92359225 Fxi 2023-10-13 19:46:53 +02:00
HarryLeeIBM
25545d504d Replace setting density to max_rows_in postings_list for full text search 2023-10-13 10:31:21 -07:00
Kseniia Sumarokova
310e922b40
Merge branch 'master' into materialized-pg-allow-unique-identifier 2023-10-13 19:14:50 +02:00
alesapin
daa8dbbbae Make as flexible as it can be 2023-10-13 18:51:32 +02:00
alesapin
f80dd06f52 Fix tuple parser 2023-10-13 17:36:38 +02:00
alesapin
78575720d1 Simpler and better 2023-10-13 16:52:04 +02:00
alesapin
47d48d81e0 Implement feature 2023-10-13 16:22:18 +02:00
kssenii
6fde98a33f Minor improvement 2023-10-13 15:56:05 +02:00
kssenii
44ea61e57f Improve shutdown 2023-10-13 15:23:36 +02:00
kssenii
b6b124f5a9 Usability improvement 2023-10-13 14:16:56 +02:00
kssenii
a9c0c20cad Update documentation 2023-10-13 13:21:50 +02:00
kssenii
bfe174f71b Fix test 2023-10-13 12:57:57 +02:00
kssenii
d64b990712 Merge remote-tracking branch 'origin/master' into s3-queue-fixes 2023-10-13 12:13:56 +02:00
Raúl Marín
500dab9569 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-13 10:12:57 +02:00
Jihyuk Bok
3c6ddb5e15
Add backward compatibility on the obsoleted storage configuration (#55557) 
* add backward compatibility on the deleted storage configuration

* update the reference file
 2023-10-13 00:37:55 +02:00
Igor Nikonov
95d5011fca Fix style 2023-10-12 21:27:24 +00:00
Igor Nikonov
0c8e49f519 Update total rows 
+ calculate total rows based on rows in ranges to read
+ update progress after all ranges to read were announced
 2023-10-12 20:34:26 +00:00
Nikita Mikhaylov
81bbb0fce1
Improve merge selecting algorithm for big partitions (#55526) 
Improved overall resilience for ClickHouse in case of many parts within partition (more than 1000). It might reduce the number of TOO_MANY_PARTS errors.
 2023-10-12 17:00:25 +02:00
Nikita Mikhaylov
a2ee30707d
Better error message in case when merge selecting task failed (#55554) 2023-10-12 16:59:03 +02:00
kssenii
1917a882b7 Better 2023-10-12 16:36:19 +02:00
Han Fei
42dc61abf5 Merge branch 'master' into hanfei/statistic 2023-10-12 16:19:45 +02:00
Robert Schulze
332a0cfa24
Merge pull request #55529 from ucasfl/invert 
Forbid create inverted index if setting not enabled
 2023-10-12 14:40:58 +02:00
vdimir
5bc6642b57
Merge pull request #55480 from myrrc/fix/s3-remove-stale-parts 
Remove existing moving/ dir if allow_remove_stale_moving_parts is off
 2023-10-12 12:00:30 +02:00
flynn
268ea30c41 Forbid create inverted index if setting not enabled 2023-10-12 03:50:55 +00:00
alesapin
6a9e19c29f Fxi distributed race condition 2023-10-11 19:46:32 +02:00
Azat Khuzhin
47ff13056f Avoid retrying keeper operations on INSERT after table shutdown 
Also note, that it should check shutdown_prepared_called, instead of
shutdown_called, since shutdown of the database/server firstly calls
flushAndPrepareForShutdown() for every storage. And if you have Buffer
table that writes to your ReplicatedMergeTree it will not be finished
until all retries will be made.

Otherwise it can take significantly long time to finish.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-11 18:25:43 +02:00
alesapin
9235408435 Merge branch 'partition-parameters' of github.com:evillique/ClickHouse into evillique-partition-parameters 2023-10-11 18:06:12 +02:00
alesapin
e2cde338fb better headers 2023-10-11 17:45:32 +02:00
alesapin
e364ed61a6 Proper fix 2023-10-11 17:42:44 +02:00
Robert Schulze
9d04d3c3ad
Merge remote-tracking branch 'rschu1ze/master' into better-use-mysql-types-in-show-columns 2023-10-11 15:04:03 +00:00
Han Fei
253b8efa58 do not modify column type when there is statistic 2023-10-11 16:57:40 +02:00
Raúl Marín
0b9bd809e7 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-11 16:50:22 +02:00
vdimir
d3f9145351
Update src/Storages/MergeTree/MergeTreePartsMover.cpp 2023-10-11 13:17:54 +02:00
vdimir
b634711303
Update src/Storages/MergeTree/MergeTreePartsMover.cpp 2023-10-11 13:14:01 +02:00
Robert Schulze
bd43b84bf8
Make use_mysql_types_in_show_columns affect only SHOW COLUMNS 2023-10-10 23:09:49 +00:00
Mike Kot
06f3b82cc9 remove moving/ dir if allow_remove_stale_ is off 2023-10-10 19:56:22 +00:00
Azat Khuzhin
b107712e0c Fix filtering by virtual columns with OR filter in query 
The problem with the initial implementation #52653 was:
- OR can have multiple arguments
- It simply not correct to assume that if there are two arguments this is OK.
  Consider the following example:

    "WHERE (column_not_from_partition_by = 1) OR false OR false"

  Will be converted to:

    "WHERE false OR false"

And it will simply read nothing.

Yes, we could apply some optimization for bool, but this will not always
work, since to optimize things like "0 = 1" we need to execute it.

And the only way to make handle this correctly (with ability to ignore
some commands during filtering) is to make is_constant() function return
has it use something from the input block, so that we can be sure, that
we have some sensible, and not just "false".

Plus we cannot simply ignore the difference of the input and output
arguments of handling OR, we need to add always-true (1/true) if the
size is different, since otherwise it could break invariants (see
comment in the code).

This includes (but not limited to):
- _part* filtering for MergeTree
- _path/_file for various File/HDFS/... engines
- _table for Merge
- ...

P.S. analyzer does not have this bug, since it execute expression as
whole, and this is what filterBlockWithQuery() should do actually
instead, but this will be a more complex patch.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-10 20:47:52 +02:00
alesapin
ab09aee754 Initial implementation 2023-10-10 19:59:48 +02:00
avogar
9ca92c1b91 Rename files 2023-10-10 16:25:51 +00:00
avogar
1afd655bbc Allow to save unparsed records and errors in RabbitMQ, NATS and FileLog engines 2023-10-10 16:21:33 +00:00
Raúl Marín
95d2063e91 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-10 17:29:45 +02:00
Nikita Mikhaylov
4456fe40f9
Remove the old code for projection analysis (#55112) 2023-10-10 17:13:32 +02:00
alesapin
666c690b4f
Merge pull request #55309 from ClickHouse/fix_redundant_check 
Fix bug with inability to drop detached partition in replicated merge tree on top of S3 without zero copy
 2023-10-10 11:48:00 +02:00
helifu
ca0c4ba147
Merge branch 'ClickHouse:master' into master13 2023-10-10 13:49:49 +08:00
robot-ch-test-poll4
8609cd25fc
Merge pull request #55392 from azat/dist-insert-threads-fix 
Fix missing thread accounting for insert_distributed_sync=1
 2023-10-10 03:18:16 +02:00
Alexey Milovidov
25d4e41b5e
Merge pull request #55377 from k-morozov/feature/add_internal_replication_to_system_clusters 
add internal_replication to system.clusters
 2023-10-10 01:18:01 +02:00
Alexey Milovidov
b50889a8ff
Merge pull request #55186 from ClickHouse/remove-in-memory-data-parts 
Remove the support for in memory data parts (part 2)
 2023-10-09 23:53:25 +02:00
Anton Popov
26a938c8cf
Merge pull request #55202 from canhld94/clickhouse_fix_too_many_marks 
Fix recalculation of skip indexes in ALTER UPDATE queries when table has adaptive granularity
 2023-10-09 19:24:59 +02:00
Azat Khuzhin
5d2a97106c Fix missing thread accounting for insert_distributed_sync=1 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-09 15:41:52 +02:00
pufit
a5bcf176be
Merge pull request #55108 from CurtizJ/add-field-to-system-parts 
Add fields `substreams` and `filenames` to `system.parts_columns`
 2023-10-09 15:11:56 +02:00
Igor Nikonov
0b4d76780f
Merge branch 'master' into revert-53782-revert-52762-planner-prepare-filters-for-analysis 2023-10-09 14:44:00 +02:00
Han Fei
fcb19b4f77 Merge branch 'master' into hanfei/statistic 2023-10-09 14:24:58 +02:00
Alexey Milovidov
e633f36ce9
Merge pull request #47663 from cangyin/fix-invalid-segment-id 
skip hardlinking inverted index files in mutation
 2023-10-09 13:31:46 +02:00
Alexey Milovidov
a02dedbb6a
Merge pull request #54989 from myrrc/patch-6 
Remove useless path from lockSharedData in StorageReplicatedMergeTree
 2023-10-09 13:28:06 +02:00
Alexey Milovidov
e840d3ce58
Merge pull request #55370 from azat/hdfs-leak 
Fix leak in StorageHDFS
 2023-10-09 13:16:45 +02:00
Konstantin Morozov
54edff684b add internal_replication 2023-10-09 10:32:22 +00:00
Azat Khuzhin
2ecd9ddabd Fix leak in StorageHDFS 
LSan report:

    Direct leak of 400 byte(s) in 5 object(s) allocated from:
        0 0x564c44d5308d in operator new[](unsigned long) (/usr/bin/clickhouse+0xbdf908d) (BuildId: 9bd2befe9c5f1d960238405d4a3fc9860273b81a)
        1 0x564c7b1b83f2 in hdfsGetPathInfo build_docker/./contrib/libhdfs3/src/client/Hdfs.cpp:1096:18
        2 0x564c670a8fed in DB::HDFSSource::URISIterator::Impl::next() build_docker/./src/Storages/HDFS/StorageHDFS.cpp:471:25
        3 0x564c670a54e9 in DB::HDFSSource::URISIterator::next() build_docker/./src/Storages/HDFS/StorageHDFS.cpp:510:19

CI: https://s3.amazonaws.com/clickhouse-test-reports/55261/de503f75dcbc5a4d0e7fbb2e6b08c2106d62848a/integration_tests__asan__[3_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-09 10:46:00 +02:00
Alexey Milovidov
808b78984f
Merge pull request #55356 from ClickHouse/better-exception-messages 
Better exception messages
 2023-10-09 02:42:26 +02:00
Alexey Milovidov
885e07cd0b
Update MutateTask.cpp 2023-10-09 02:33:46 +03:00
Alexey Milovidov
e3771f10d2 Merge branch 'master' into remove-in-memory-data-parts 2023-10-09 01:08:20 +02:00
Alexey Milovidov
e1dc80b5ab
Revert "Support SAMPLE BY for VIEW" 2023-10-09 00:08:46 +03:00
Alexey Milovidov
2aaddcce18 Better exception messages 2023-10-08 22:54:33 +02:00
Robert Schulze
282200ef50
Merge pull request #55215 from rschu1ze/more-mysql-compat 
Add `information_schema.columns.extra` and `information_schema.tables.table_rows`
 2023-10-08 12:21:34 +02:00
Kseniia Sumarokova
0af252ae31
Merge pull request #55297 from ClickHouse/kssenii-patch-6 
MaterializedPostgreSQL: remove back check
 2023-10-08 10:31:23 +02:00
Robert Schulze
f4bc58ea92
Merge remote-tracking branch 'rschu1ze/master' into more-mysql-compat 2023-10-08 07:49:37 +00:00
Robert Schulze
0450da9bac
Naming fixes 2023-10-07 20:12:10 +00:00
alesapin
eec64ba1d9 Fix bug with inability to drop detached partition in replicated merge tree on top of S3 without zero copy 2023-10-07 19:09:47 +02:00
Robert Schulze
2e8cac6e9c
Make use_mysql_types_in_show_columns independent from the connection 
Fixes: #55273
 2023-10-07 13:06:41 +00:00
Kseniia Sumarokova
f0747b7a6e
Update MaterializedPostgreSQLConsumer.cpp 2023-10-07 14:28:52 +02:00
Duc Canh Le
bea4a4eb85 Merge branch 'master' into clickhouse_fix_too_many_marks 
rebase and re-trigger CI
 2023-10-06 09:45:48 +00:00
Nikolai Kochetov
a9a3529025 Merge branch 'master' into planner-prepare-filters-for-analysis-2 2023-10-05 12:15:29 +00:00
Ilya Golshtein
3480243fa3 merge_row_policy: per code review 2023-10-05 08:23:53 +00:00
Ilya Golshtein
9a561cff82 merge_row_policy: cleanup, verbose short messages in 00002_log ... 2023-10-05 08:23:53 +00:00
Ilya Golshtein
57f1e56276 merge_row_policy: cleanup, comments, new tests 2023-10-05 08:23:52 +00:00
Ilya Golshtein
6161116bf2 merge_row_policy: stylecheck again 2023-10-05 08:23:52 +00:00
Ilya Golshtein
f89a538f31 merge_row_policy: stylecheck 2023-10-05 08:23:52 +00:00
Ilya Golshtein
b39a201cd0 merge_row_policy: refactored with ReadFromMerge::RowPolicyData 2023-10-05 08:23:52 +00:00
Ilya Golshtein
b57d8bc4a9 merge_row_policy: works again with adding missed columns 2023-10-05 08:23:52 +00:00
Ilya Golshtein
f4c77c1f18 merge_row_policy: namesDifference, try to handle nonselected columns 2023-10-05 08:23:52 +00:00
Ilya Golshtein
e2ddf40cfd merge_row_policy: further cleanup 2023-10-05 08:23:52 +00:00
Ilya Golshtein
da5f607242 merge_row_policy: cleanup, remove some debug output 2023-10-05 08:23:52 +00:00
Ilya Golshtein
978a535849 merge_row_policy: addFilter(() together with FilterTransform 2023-10-05 08:23:52 +00:00
Ilya Golshtein
a19cd8089b merge_row_policy: add_filter() actually works 2023-10-05 08:23:52 +00:00
Ilya Golshtein
c457fa727e merge_row_policy: addFilter() instead of FilterTransform 2023-10-05 08:23:52 +00:00
Ilya Golshtein
359cd4d32a merge_row_policy: style fixes and back to storage->read 2023-10-05 08:23:52 +00:00
Ilya Golshtein
657c39c79e merge_row_policy: row_policy_storage_merge test 2023-10-05 08:23:52 +00:00
Ilya Golshtein
b6f682dc3f merge_row_policy: FilterTransform 2023-10-05 08:23:52 +00:00
Ilya Golshtein
eebdff472e merge_row_policy: original behavior restored + extra debug 2023-10-05 08:23:52 +00:00
Ilya Golshtein
2acc4c223d merge_row_policy - extra debug 2023-10-05 08:23:52 +00:00
Ilya Golshtein
0464b4fd06 merge_row_policy: with QueryProcessingStage::Complete 2023-10-05 08:23:52 +00:00
Ilya Golshtein
1e31859936 merge_row_policy: tiny cleanup 2023-10-05 08:23:52 +00:00
Ilya Golshtein
020d76a383 merge_row_policy: initial 2023-10-05 08:23:52 +00:00
Robert Schulze
53683f56c0
Merge remote-tracking branch 'rschu1ze/master' into more-mysql-compat 2023-10-04 12:45:10 +00:00
Nikolai Kochetov
c5100cd9b1 Fixing build 2023-10-04 09:01:29 +00:00
Robert Schulze
16a864171d
Merge pull request #55210 from HarryLeeIBM/hlee-fts-query 
Speed up buildup of inverted index using absl::flat_hash_map
 2023-10-04 10:01:35 +02:00
Han Fei
ba878fb43a support rename 2023-10-04 00:58:26 +02:00
Nikolai Kochetov
d944b59902 Merge branch 'master' into planner-prepare-filters-for-analysis-2 2023-10-03 14:28:16 +00:00
Nikolai Kochetov
3b757d60a2 Remove commented code. 2023-10-03 14:22:20 +00:00
Robert Schulze
907e792a93
Add information_schema.columns.extras 
Fixes: #55183
 2023-10-03 13:11:03 +00:00
Robert Schulze
2aefa760c3
Add information_schema.tables.table_rows 
Fixes: #55182
 2023-10-03 12:24:10 +00:00
robot-ch-test-poll
e06f72ee55
Merge pull request #54052 from VanDarkholme7/missing_subcolumns 
Support missed Object(Nullable(json)) subcolumns in query.
 2023-10-03 10:32:51 +02:00
HarryLeeIBM
9e1208a793 Speed up inverted index building using flat_hash_map 2023-10-02 15:20:35 -07:00
Duc Canh Le
57e902d375 more detail logging for exception while loading marks 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-10-02 09:29:56 +00:00
Alexey Milovidov
5ebd8ef41d Remove Meilisearch 2023-09-30 06:02:38 +02:00
Alexey Milovidov
6e87fa3f67 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-in-memory-data-parts 2023-09-30 05:12:29 +02:00
Alexey Milovidov
de8e068da7
Merge pull request #55184 from ClickHouse/remove-garbage-broken-detached-parts 
Remove the removal of broken detached parts
 2023-09-30 06:10:40 +03:00
Alexey Milovidov
db9377fc59
Merge pull request #55172 from amosbird/fix-55106 
Fix partition pruning of extra columns in set.
 2023-09-30 04:17:14 +03:00
Alexey Milovidov
6de0250650 Continue removing in-memory data parts 2023-09-30 03:10:59 +02:00
Alexey Milovidov
d8b5713023 Remove the removal of broken detached parts 2023-09-30 01:47:44 +02:00
robot-ch-test-poll4
4ef46d1153
Merge pull request #55130 from ClickHouse/improve-message-predicate 
Improve error message in MergePredicate
 2023-09-29 23:29:41 +02:00
robot-ch-test-poll3
f7874c1855
Merge pull request #55143 from hanfei1991/hanfei/fix-empty-retries 
check if block is empty after async insert retries
 2023-09-29 23:29:21 +02:00
Amos Bird
5595d2d20b
Fix partition pruning of extra columns in set. 2023-09-30 02:32:31 +08:00
Robert Schulze
fabd167af6
Merge pull request #55128 from rschu1ze/disable-error-update 
Suppress error statistics update for internal exceptions
 2023-09-29 18:37:12 +02:00
Han Fei
8afc21f7af refine 2023-09-29 18:20:16 +02:00
kssenii
d1f6888119 Allow unique consumer identifier 2023-09-29 16:17:02 +02:00
kssenii
1ef21bab3d Fix data race 2023-09-29 16:02:51 +02:00
Nikita Mikhaylov
043000b088 Better 2023-09-29 15:59:54 +02:00
kssenii
a7f389a760 Fix 2023-09-29 15:14:25 +02:00
Han Fei
e57217ea9c check if block is empty after async insert retries 2023-09-29 15:14:16 +02:00
Nikita Mikhaylov
804ae4c58b Better 2023-09-29 11:16:24 +00:00
Robert Schulze
c1b95e8b3c
Suppress error statistics update for internal exceptions 2023-09-29 10:58:44 +00:00
Robert Schulze
bb4d141132
Merge pull request #55069 from HarryLeeIBM/hlee-ftsearch 
Add optimization for roaring bitmaps in full text search
 2023-09-29 11:16:16 +02:00
Alexey Milovidov
95d573f70d Update autogenerated version to 23.10.1.1 and contributors 2023-09-29 08:52:43 +02:00
Anton Popov
5789db24a7 add fields 'substreams' and 'filenames' to system.parts_columns 2023-09-28 16:59:01 +00:00
Han Fei
d6c1c0e805 refine tests and support drop columns with statistic 2023-09-28 18:35:18 +02:00
kssenii
d644992192 Fxi 2023-09-28 16:25:04 +02:00
kssenii
4a7922507b Minor changes 2023-09-28 16:18:00 +02:00
kssenii
7d91ba109a Update doc 2023-09-28 13:58:59 +02:00
kssenii
1749874e7b Fxi 2023-09-28 13:51:07 +02:00
Igor Nikonov
1983b4cfd1
Merge pull request #55026 from ClickHouse/insert-quorum-wo-keeper-retries 
Fix: insert quorum w/o keeper retries
 2023-09-28 11:05:21 +02:00
Amos Bird
51f603bdfc
Rename as suggested. 2023-09-28 15:11:56 +08:00
Amos Bird
056b2693a3
Reorganize combinators into dedicated folder 2023-09-28 15:11:55 +08:00
Han Fei
18f6fd1e5f more comments 2023-09-28 07:31:48 +02:00
Alexey Milovidov
aacb3471b9
Update StorageReplicatedMergeTree.cpp 2023-09-28 02:41:50 +03:00
kssenii
57cfb88cb6 Fix typos check, fix build wihtout s3 2023-09-27 22:22:49 +02:00
HarryLeeIBM
71adb052ee Add optimization for roaring bitmaps in full text search 2023-09-27 12:23:00 -07:00
János Benjamin Antal
c911c8daf4
Prevent attaching parts from tables with different projections or indices (#55062) 
* Prevent attaching parts from tables with different projections or indices

* Update docs

* Add tests
 2023-09-27 15:02:53 -04:00
kssenii
c579f5b321 Fix style check 2023-09-27 20:10:05 +02:00
kssenii
42783894b7 Add comments 2023-09-27 20:07:19 +02:00
kssenii
e0ff76a7f9 Fix 2023-09-27 18:44:53 +02:00
Han Fei
6ce77b07f7 refine exception messages 2023-09-27 17:59:53 +02:00
kssenii
f753b91a3b Better maintenance of processing node 2023-09-27 17:17:52 +02:00
Nikita Taranov
86f4f91573
Merge branch 'master' into fix_key_analysis_merge_engine 2023-09-27 16:13:14 +02:00
kssenii
6b191a1afe Better 2023-09-27 14:54:31 +02:00
Robert Schulze
2d750e9e2f
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-reenable-checks 2023-09-27 12:15:29 +00:00
Robert Schulze
e9c3032f83
Merge pull request #55037 from rschu1ze/inf_sch_tables 
Implement information_schema.tables.data_length
 2023-09-27 14:09:38 +02:00
kssenii
d77452c561 Fix style check 2023-09-27 12:06:41 +02:00
kssenii
6c778d4b12 Fix build without s3 2023-09-27 12:05:13 +02:00
Igor Nikonov
98bf455726 Merge remote-tracking branch 'origin/master' into insert-quorum-wo-keeper-retries 2023-09-27 09:02:39 +00:00
Han Fei
76b9d43dc4 make code beautiful 2023-09-27 11:01:53 +02:00
Robert Schulze
3e0bc6fe42
Disable '-clang-analyzer-unix.Malloc' for now 2023-09-27 08:51:17 +00:00
Robert Schulze
3a1663a460
Merge pull request #55035 from rschu1ze/is_deterministic_resubmit 
Added field "is_deterministic" to system.functions
 2023-09-27 10:13:18 +02:00
Michael Kolupaev
23bc286280
Print more information about one logical error in MergeTreeDataWriter (#55001) 2023-09-26 13:23:26 -07:00
Han Fei
6c4474d3d1 refine code structure 2023-09-26 22:22:11 +02:00
Robert Schulze
e22de584e9
information_schema.tables.data_length 2023-09-26 20:08:54 +00:00
Han Fei
4e7745f757 Merge branch 'master' into hanfei/statistic 2023-09-26 21:43:53 +02:00
Robert Schulze
dcdc8bf0e9
Added field "is_deterministic" to system.functions 2023-09-26 19:13:23 +00:00
Robert Schulze
cde10fe7b5
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-reenable-checks 2023-09-26 18:59:41 +00:00
Robert Schulze
ab2b5cd5ca
More fixes 2023-09-26 18:55:10 +00:00
Robert Schulze
6634043a9f
Merge pull request #54837 from rschu1ze/obsolete-mt-settings 
Mark obsolete settings in system.merge_tree_settings as is_obsolete = 1
 2023-09-26 19:55:05 +02:00
Han Fei
8cb9fe0154 refine code 2023-09-26 19:16:01 +02:00
George Gamezardashvili
0ce30ab6d5
SSH keys authentication (#41109) 
Added new type of authentication based on SSH keys. It works only for Native TCP protocol.

Co-authored-by: Nikita Mikhaylov <nikitamikhaylov@clickhouse.com>
Co-authored-by: Robert Schulze <robert@clickhouse.com>
 2023-09-26 17:50:19 +02:00
Igor Nikonov
54f25409ef Test with fail point 2023-09-26 15:19:04 +00:00
kssenii
14b09d3cdc Add caching 2023-09-26 16:23:24 +02:00
Alexander Tokmakov
f39cb17881
Revert "Added field "is_deterministic" to system.functions" 2023-09-26 15:01:05 +02:00
Igor Nikonov
fd7862cd10 Remove keeper retries when waiting for insert quorum 2023-09-26 12:13:32 +00:00
Anton Popov
6cd32eb7b5
Merge pull request #50612 from CurtizJ/long-column-names 
Allow to replace long file names to hashes
 2023-09-26 13:05:33 +02:00
Robert Schulze
e4c081652d
Suppress more checks 2023-09-26 11:01:35 +00:00
Robert Schulze
d10a61444c
Merge remote-tracking branch 'rschu1ze/master' into obsolete-mt-settings 2023-09-26 09:55:47 +00:00
Robert Schulze
e954ffe939
Cleanups 2023-09-26 09:51:48 +00:00
Nikita Taranov
595027b1c6 better 2023-09-26 11:51:34 +02:00
Robert Schulze
9fff447716
Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
Robert Schulze
fa40732395
Merge pull request #54766 from rschu1ze/is_deterministic 
Added field "is_deterministic" to system.functions
 2023-09-26 11:05:03 +02:00
János Benjamin Antal
036aa9f756
Merge pull request #54869 from usurai/ceil 
Update ceilRowsToCompleteGranules to use binary search.
 2023-09-26 09:46:20 +02:00
VanDarkholme7
9d74502488
Merge branch 'ClickHouse:master' into missing_subcolumns 2023-09-26 13:48:16 +08:00
VanDarkholme7
7dda1646fd fix 2023-09-26 02:32:05 +00:00
Nikolay Degterinsky
987906680f
Revert "Fix NATS high cpu usage" 2023-09-26 03:36:14 +02:00
Nikita Taranov
694792fa5d fix for analyzer 2023-09-25 23:58:29 +02:00
Nikita Taranov
8d19566c7d impl 2023-09-25 23:58:20 +02:00
kssenii
4d78dbacfe Fix style check, fix build without s3 2023-09-25 23:38:16 +02:00
Kseniia Sumarokova
5057a31bac
Merge pull request #54844 from kssenii/materialized-postrgesql-safer-parts 
Fix "Invalid number of rows in Chunk" in MaterializedPostgreSQL
 2023-09-25 23:37:12 +02:00
kssenii
251b0165ca Better 2023-09-25 22:55:41 +02:00
kssenii
3b54b6be88 Add system tables 2023-09-25 20:04:15 +02:00
Robert Schulze
6f3d7fcc86
Better lint 2023-09-25 17:11:32 +00:00
Robert Schulze
0c490465d4
Stabilize tests 2023-09-25 17:07:48 +00:00
Robert Schulze
d170304359
Parallelize Annoy index creation + minor test fix 2023-09-25 17:07:23 +00:00
Robert Schulze
5233adecb0
Merge remote-tracking branch 'rschu1ze/master' into is_deterministic 2023-09-25 16:27:50 +00:00
Kruglov Pavel
af8b1e4978
Merge pull request #54855 from Avogar/obsolete-format-settings 
Move obsolete format settings to separate section
 2023-09-25 18:00:31 +02:00
Mike Kot
e2dd78fdd9
Remove useless path from lockSharedData in StorageReplicatedMergeTree 2023-09-25 16:56:09 +02:00
Anton Popov
0ebc22e851
Merge branch 'master' into long-column-names 2023-09-25 16:55:14 +02:00
usurai
330bcff239 Update ceilRowsToCompleteGranules to use binary search. 
Replace the linear method in ceilRowsToCompleteGranules with a binary search.
 2023-09-25 22:52:00 +08:00
kssenii
e4256eb268 Better 2023-09-25 13:44:45 +02:00
kssenii
47050c2605 Merge remote-tracking branch 'upstream/master' into s3-queue-fixes 2023-09-25 13:42:52 +02:00
Azat Khuzhin
7a761730fa Respect max_block_size while reading form system.stack_trace 
Reading from system.stack_trace can be very slow, especially if you read
all columns (there are some optimizations to avoid doing some actions),
that requires:
- reading /proc/self/task
- reading /proc/self/task/*/comm
- sending signal to each of the thread (to read query_id and trace_id)

And even if you have some conditions (for which there are some
optimizations, i.e. not sending signal to the thread which is the
slowest operation under load) it will still send signals to every thread
and wait for result.

This patch implements reading from system.stack_trace with respect to
max_block_size, that will make queries with lower max_block_size and
LIMIT works significantly faster.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-24 14:01:24 +02:00
Azat Khuzhin
01d22805ef Remove indent for anonymous namespace 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-24 14:01:24 +02:00
Azat Khuzhin
e0bca06a5d Add some logging for preparing rows for system.stack_trace 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-24 14:00:43 +02:00
robot-clickhouse
320e4c47f3
Merge pull request #54943 from amosbird/fix-54941 
Rebuild minmax_count_projection when partition key gets modified
 2023-09-23 22:11:52 +02:00
robot-clickhouse-ci-2
2a26557391
Merge pull request #54399 from vahpetr/fix-nats-high-cpu-usage 
Fix NATS high cpu usage
 2023-09-23 21:36:21 +02:00
Sergei Trifonov
c1489d52b7
Merge pull request #54618 from ClickHouse/io-scheduling-throttler 
add IO scheduling node to limit bandwidth
 2023-09-23 21:26:17 +02:00
Alexey Milovidov
e89a8e4d13
Merge pull request #53492 from ClickHouse/fix-prewhere-lost-column 
Fix: moved to prewhere condition actions can lose column
 2023-09-23 22:24:00 +03:00
serxa
44acb53ad3 Merge branch 'master' into io-scheduling-throttler 2023-09-23 10:51:18 +00:00
Sergei Trifonov
4032fc17bb
Merge pull request #54780 from ucasfl/comment 
update comment
 2023-09-23 12:48:48 +02:00
Amos Bird
81804ce9ba
Rebuild minmax_count_projection when partition key gets modified 2023-09-23 17:59:06 +08:00
robot-ch-test-poll2
3c93a939a2
Merge pull request #54936 from ClickHouse/pufit/s3-yet-another-num-streams-adjustment 
Set a minimum limit of `num_streams` in StorageS3
 2023-09-23 02:59:50 +02:00
robot-clickhouse-ci-2
d98234dc9d
Merge pull request #54803 from Avogar/ephemeral-columns-from-files 
Forbid special columns for file/s3/url/... storages, fix insert into ephemeral columns from files
 2023-09-22 23:24:42 +02:00
Alexey Milovidov
783d1873b2
Merge pull request #54839 from rschu1ze/vector-search-test-refactoring 
Refactor and split up vector search tests
 2023-09-23 00:10:47 +03:00
robot-ch-test-poll3
1334202af4
Merge pull request #54863 from zvonand/zvonand-refactor-globs 
Refactor and simplify multi-directory globs
 2023-09-22 23:09:05 +02:00
pufit
13eee0e950 Set a minimum limit of num_streams in StorageS3 2023-09-22 14:13:20 -04:00
Nikolai Kochetov
cf40723179
Merge pull request #54895 from amosbird/allow-nullable-primary-keys-for-projections 
Always allow nullable keys for projections
 2023-09-22 19:09:24 +02:00
Alexander Tokmakov
21ffbd88de
Merge pull request #54859 from ClickHouse/fix_zero_copy_locks 
Fix zero copy locks with hardlinks
 2023-09-22 18:27:03 +02:00
Kruglov Pavel
4d675dbad4
Merge pull request #54825 from azat/fix-virtual-columns-filtering 
Fix filtering parts with indexHint for non analyzer (resubmit)
 2023-09-22 18:20:16 +02:00
Dmitry Novik
0f8afdbbe7 Merge remote-tracking branch 'origin/master' into fix-prewhere-lost-column 2023-09-22 14:50:05 +00:00
Alexander Tokmakov
f54c3c6a9f
Update StorageReplicatedMergeTree.cpp 2023-09-22 15:07:55 +02:00
Robert Schulze
e9c0dbd33e
Merge remote-tracking branch 'rschu1ze/master' into tableau-online-information-schema 2023-09-22 11:03:06 +00:00
Robert Schulze
9a91f97ac4
Merge pull request #53831 from rschu1ze/clang-17 
Switch CI to Clang 17
 2023-09-22 12:28:35 +02:00
Kseniia Sumarokova
163961115b
Merge pull request #54842 from kssenii/storage-rabbitmq-minor-change 
Add some logging to StorageRabbitMQ
 2023-09-22 11:51:27 +02:00
zvonand
2eb63649bd fix old workflows 2023-09-22 00:30:46 +02:00
Robert Schulze
877e4f3aab
Merge remote-tracking branch 'rschu1ze/master' into clang-17 2023-09-21 20:21:12 +00:00
pufit
99c1d76604 Fix division by zero in StorageS3 2023-09-21 15:55:42 -04:00
Robert Schulze
a46e0deb85
Resolve bad caching of INCBIN'd external data 2023-09-21 17:41:21 +00:00
Michael Kolupaev
be0c512329
Fix virtual columns having incorrect values after ORDER BY (#54811) 
Fix virtual columns having incorrect values after ORDER BY
 2023-09-21 10:14:28 -07:00
Michael Kolupaev
9af9b4a085
Enable connection pooling for s3 table function (#54812) 
Enable connection pooling for s3 table function
 2023-09-21 09:27:20 -07:00
Amos Bird
973cd5e972
Always allow nullable keys for projections 2023-09-21 22:46:16 +08:00
Robert Schulze
5209bd2d51
Merge remote-tracking branch 'rschu1ze/master' into clang-17 2023-09-21 14:45:55 +00:00
zvonand
e2b40749a6 the same for HDFS 2023-09-21 16:45:47 +02:00
Robert Schulze
f5137dd0b4
More clang-tidy fixes 2023-09-21 14:40:57 +00:00
zvonand
bc5f141a6c fix {..} for File 2023-09-21 16:40:38 +02:00
Anton Popov
4d7ecf39ac check file collisions on all alters 2023-09-21 14:16:03 +00:00
slvrtrn
2b8a263355 Change referential_constraints to an empty table 2023-09-21 16:11:08 +02:00
pufit
bd387f6d2c
Merge pull request #54815 from ClickHouse/pufit/optimal-num-streams-for-s3-storage 
Adjusting `num_streams` by expected work in StorageS3
 2023-09-21 08:41:39 -04:00
slvrtrn
e49ddbb678 Merge remote-tracking branch 'origin' into tableau-online-information-schema 2023-09-21 14:25:04 +02:00
Igor Nikonov
d260b7abe4
Merge pull request #54519 from ClickHouse/parallel-replicas-not-enough-replicas 
Disable parallel replicas on shards with not enough nodes
 2023-09-21 14:07:25 +02:00
Robert Schulze
44b7fa1030
Merge remote-tracking branch 'rschu1ze/master' into is_deterministic 2023-09-21 09:29:44 +00:00
Robert Schulze
c2a593baf9
Split tests 2023-09-21 09:17:38 +00:00
Vitaly Baranov
bf4662aca7
Merge pull request #53336 from azat/backups/s3-throttling 
Fix throttling of BACKUPs from/to S3 (in case native copy was not used) and in some other places as well
 2023-09-21 10:42:28 +02:00
zvonand
a05bb020d4 do all the same for HDFS + remove setting 2023-09-21 01:50:41 +02:00
pufit
4a2f7976f0 Resolve PR issues 2023-09-20 19:43:02 -04:00
zvonand
5f67788c03 update docs and comments 2023-09-21 00:16:17 +02:00
zvonand
9eccb12183 small refactor of storageFile globs 2023-09-20 23:59:10 +02:00
Robert Schulze
e95f9ee814
Merge remote-tracking branch 'rschu1ze/master' into is_deterministic 2023-09-20 21:17:57 +00:00
Alexander Tokmakov
f8062d77e1 fix rare data loss 2023-09-20 22:38:25 +02:00
Alexander Tokmakov
ec80068611 add a test for no such key 2023-09-20 22:37:24 +02:00
Alexander Tokmakov
18acc0c19d
Merge pull request #54641 from ClickHouse/followup_54550 
Follow-up to #54550
 2023-09-20 19:44:56 +02:00
pufit
20105958a8 add reserve 2023-09-20 13:37:06 -04:00
pufit
71c7e3c81e Add logging, fix thread name length 2023-09-20 13:33:25 -04:00
Azat Khuzhin
c706101891 Fix throttling of BACKUPs from/to S3 (in case native copy was not used) 
In some cases native copy is not possible, and such requests should be
throttled.

v0: copyS3FileNativeWithFallback
v2: revert v0 and pass write_settings
v3: pass read_settings to copyFile()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-20 18:28:43 +02:00
avogar
3e08800cb5 Forbid special columns for file/s3/url/... storages, fix insert into ephemeral columns from files 2023-09-20 16:25:55 +00:00
avogar
54cee196bd Move obsolete format settings to separate section 2023-09-20 16:00:28 +00:00
Robert Schulze
2c91e52da1
Merge pull request #54600 from rschu1ze/fix-annoy-crash 
Annoy/Usearch index: Fix LOGICAL_ERROR during build-up with default values
 2023-09-20 17:05:16 +02:00
pufit
729c8aa29f fix glob iterator estimated objects 2023-09-20 10:41:47 -04:00
kssenii
5d66189d87 Fix 2023-09-20 16:39:24 +02:00
kssenii
3ef2c37b92 Add some logging to StorageRabbitMQ 2023-09-20 15:54:11 +02:00
pufit
a1c98bc8c9 fix build 2023-09-20 09:50:35 -04:00
Igor Nikonov
2d446fc1ca Merge remote-tracking branch 'origin/master' into parallel-replicas-not-enough-replicas 2023-09-20 13:40:08 +00:00
slvrtrn
fb0ba86489 Merge remote-tracking branch 'origin' into tableau-online-information-schema 2023-09-20 15:14:40 +02:00
Robert Schulze
a4eeed50b6
Mark obsolete settings in system.merge_tree_settings as is_obsolete = 1 2023-09-20 12:55:53 +00:00
Azat Khuzhin
9c3fb64106 Check type of the filter expressions while filtering by virtual columns 
This should fix filtering by virtual columns with non-uint8 types, i.e.
queries like:

    SELECT * FROM data WHERE 1.0

Fixes: 02346_full_text_search
Fixes: 00990_hasToken_and_tokenbf
v2: move check out from is_const to filterBlockWithQuery(), since in is_const there is no way to validate sets.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-20 13:36:25 +02:00
flynn
dfa95ecc2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into comment 2023-09-20 11:16:33 +00:00
Nikolai Kochetov
0bc41bab74
Merge pull request #54544 from kitaisreal/key-condition-in-function-different-types 
KeyCondition IN function support different types
 2023-09-20 12:54:01 +02:00
alesapin
2ff36b6e0c
Merge pull request #54768 from ClickHouse/better_exception_in_check_data_part 
Better exception message in checkDataPart
 2023-09-20 12:04:46 +02:00
Robert Schulze
f3bf8388b6
Incorporate feedback 2023-09-20 10:00:00 +00:00
SmitaRKulkarni
d8adf05de2
Added a new column _block_number (#47532) 
Added a new virtual column _block_number which is persisted on merges when allow_experimental_block_number_column is enabled
 2023-09-20 11:31:12 +02:00
Azat Khuzhin
d9a634eb0f Fix filtering parts with indexHint for non analyzer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit ffa82e9297)
 2023-09-20 11:29:35 +02:00
pufit
34aecc0bf3 Adjusting num_streams by expected work in StorageS3 2023-09-19 23:05:48 -04:00
slvrtrn
636fc506aa Replace information_schema view columns aliases with uppercase 2023-09-19 23:42:40 +02:00
Igor Nikonov
e28f4e819c
Merge branch 'master' into parallel-replicas-not-enough-replicas 2023-09-19 23:05:24 +02:00
Azat Khuzhin
c439c4bca2
Revert "Fix filtering parts with indexHint for non analyzer" 2023-09-19 21:39:21 +02:00
Han Fei
eb86c113e5
Merge pull request #54184 from priera/fix_mv_insertion_when_duplicated_parts 
Inserting only non-duplicate chunks in MV
 2023-09-19 20:41:45 +02:00
Kruglov Pavel
d555fdffa5
Merge pull request #54449 from azat/parts-prune-indexHint 
Fix filtering parts with indexHint for non analyzer
 2023-09-19 19:15:41 +02:00
Han Fei
17243b84ab Merge branch 'master' into hanfei/statistic 2023-09-19 17:09:17 +02:00
Han Fei
83d1b44a3e validate 2023-09-19 17:08:15 +02:00
slvrtrn
e4e9727572 Update information_schema docs 2023-09-19 14:14:17 +02:00
alesapin
9dcc41cf83 Fxi 2023-09-19 13:47:32 +02:00
alesapin
28b71163b4 Merge branch 'master' into better_exception_in_check_data_part 2023-09-19 13:45:56 +02:00
alesapin
7b1cf9d912 Revert "Better check for exceptions in checkData of non replicated merge tree" 
This reverts commit 363ee7cae5.
 2023-09-19 13:45:34 +02:00
alesapin
363ee7cae5 Better check for exceptions in checkData of non replicated merge tree 2023-09-19 13:41:55 +02:00
priera
b92e37d2df Fixing PR comments 2023-09-19 12:15:01 +02:00
priera
94d737264c inserting non-duplicate chunks 2023-09-19 12:15:00 +02:00
serxa
311db94640 default max_burst equals to max_speed as in Throttler 2023-09-19 10:10:21 +00:00
Maksim Kita
c7ddbab9bc Updated implementation 2023-09-19 13:06:19 +03:00
flynn
c3d12ce0e1 update comment 2023-09-19 08:18:37 +00:00
Anton Popov
72f2bfd166 fix build 2023-09-19 05:35:57 +00:00
VanDarkholme7
783b48b730
Merge branch 'ClickHouse:master' into missing_subcolumns 2023-09-19 10:33:06 +08:00
Anton Popov
624782bd8f Merge remote-tracking branch 'upstream/master' into HEAD 2023-09-18 23:13:20 +00:00
Anton Popov
d94c1b1ec9 add check for collisions in filenames 2023-09-18 22:27:04 +00:00
slvrtrn
d91a656dd4 Revert schemata.sql format 2023-09-18 22:33:16 +02:00
slvrtrn
ad7b707658 Update information_schema.tables sql and docs 2023-09-18 22:30:27 +02:00
Robert Schulze
097217628e
Merge remote-tracking branch 'rschu1ze/master' into is_deterministic 2023-09-18 20:17:17 +00:00
slvrtrn
9a0d07c682 Update information_schema docs 2023-09-18 22:14:53 +02:00
slvrtrn
54f902f0b7 Adjust information_schema for Tableau Online 2023-09-18 19:50:05 +02:00
Sema Checherinda
e7550523c8
Merge pull request #54651 from CheSema/limit_backoff_timeout 
limit the delay before next try in S3
 2023-09-18 19:41:26 +02:00
Robert Schulze
4373ed0f98
Added field "is_deterministic" to system.functions 2023-09-18 17:27:53 +00:00
Alexander Sapin
326eaa9bc5 Better exception message in checkDataPart 2023-09-18 18:26:53 +02:00
Igor Nikonov
047d214436 Merge remote-tracking branch 'origin/master' into parallel-replicas-not-enough-replicas 2023-09-18 15:29:56 +00:00
Alexey Milovidov
e4d47539ab
Merge pull request #54477 from azat/view-sampling 
Support SAMPLE BY for VIEW
 2023-09-18 14:00:52 +03:00
Kruglov Pavel
9c888ea42b
Merge pull request #53549 from Avogar/group-by-constant-keys 
Optimize group by constant keys
 2023-09-18 12:12:40 +02:00
Yakov Olkhovskiy
711876dfa8
Merge pull request #54696 from ClickHouse/fix-match-keycondition 
Fix: avoid using regex match, possibly containing alternation, as a key condition.
 2023-09-17 22:45:13 -04:00
Robert Schulze
de4f22e20a
Typo 2023-09-17 15:27:30 +00:00
Robert Schulze
9f009cccd5
Incorporate review feedback 2023-09-17 15:22:51 +00:00
Robert Schulze
f5e8028bb1
Merge pull request #54642 from rschu1ze/broken-re2st 
Remove broken lockless variant of re2
 2023-09-17 15:30:57 +02:00
robot-clickhouse-ci-1
a878d104d7
Merge pull request #52839 from zvonand/zvonand-globs-small-fix 
Fix directory permissions for multi-directory globs. Follow-up #50559
 2023-09-16 20:52:30 +02:00
robot-ch-test-poll1
a2ed5c2cae
Merge pull request #54174 from ClickHouse/vdimir/fix_hostname_const 
Fix hostname and co result constness in new analyzer
 2023-09-16 19:37:08 +02:00
VanDarkholme7
cf8c614fed Fix bug on DELETE and UPDATE with json subcolumns 2023-09-16 13:04:19 +00:00
robot-ch-test-poll4
5fb8e46967
Merge pull request #54635 from Avogar/cluster-func-names-with-spaces 
Fix schema inference for *Cluster functions for column names with spaces
 2023-09-15 20:36:33 +02:00
Yakov Olkhovskiy
f52e87f474 fix 2023-09-15 18:30:26 +00:00
Yakov Olkhovskiy
c2d46286e7 we can't process regex with alternation here - bypass 2023-09-15 18:21:38 +00:00
Bhavna Jindal
dc99de3c44 Log engine Mark file to read and write in little Endian for s390x compatibility 2023-09-15 08:59:40 -07:00
Maksim Kita
29e4352c17 Updated implementation 2023-09-15 17:04:24 +03:00
Maksim Kita
7a7697059c Fixed tests 2023-09-15 17:04:24 +03:00
Maksim Kita
32cad222bf KeyCondition IN function support different types 2023-09-15 17:04:24 +03:00
Kruglov Pavel
2075f9c667
Merge branch 'master' into group-by-constant-keys 2023-09-15 15:10:08 +02:00
kssenii
6846fe3c58 Fxi 2023-09-15 14:21:08 +02:00
Andrey Zvonov
ffbed02f14
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-09-15 14:18:07 +03:00
avogar
e647346b36 Better 2023-09-15 11:11:04 +00:00
Kruglov Pavel
1bc29696be
Merge pull request #54576 from HarryLeeIBM/hlee-s390x-rand-gen 
Fix UInt256 and IPv4 random data generation on s390x
 2023-09-15 11:42:47 +02:00
Robert Schulze
945179be46
Annoy: Fix LOGICAL_ERROR with default values #52258 2023-09-14 19:23:32 +00:00
Robert Schulze
c422a8f0dc
Cosmetics 2023-09-14 18:05:20 +00:00
Robert Schulze
f0eadd4b22
Merge pull request #54103 from ClickHouse/ustweaks 
Small usearch index improvements: metrics and configurable internal data type
 2023-09-14 19:58:53 +02:00
Sema Checherinda
d9e15c00c9 limit the delay before next try in S3 2023-09-14 19:45:07 +02:00
vdimir
1aa18e0eb6
Analyzer: Remove constants from header in StorageDistributed 2023-09-14 16:44:18 +00:00
kssenii
342755d35e Better 2023-09-14 18:41:31 +02:00
Robert Schulze
7b378dbad3
Remove broken lockless variant of re2 2023-09-14 16:40:42 +00:00
Alexander Tokmakov
52de64c2bd remove unneeded code 2023-09-14 18:30:12 +02:00
kssenii
afcb0b2b9a Merge remote-tracking branch 'upstream/master' into s3-queue-fixes 2023-09-14 17:16:29 +02:00
Mikhail f. Shiryaev
bf962a8674
Merge pull request #54517 from HarryLeeIBM/hlee-s390x-jemalloc-bins 
Fix endian issue in jemalloc_bins for s390x
 2023-09-14 15:43:50 +02:00
avogar
f240243c86 Fix schema inference for *Cluster functions for column names with spaces 2023-09-14 13:34:11 +00:00
Sergei Trifonov
12c71be9ca
Merge branch 'master' into async-loader-integration 2023-09-14 15:29:07 +02:00
Michael Kolupaev
8997464867
Small usearch index improvements: metrics and f16 2023-09-14 11:24:47 +00:00
serxa
09d4bf6930 add IO scheduling node for bandwidth throttling 2023-09-14 08:51:47 +00:00
Andrey Zvonov
ff16ddbbb3
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-09-14 11:31:49 +03:00
robot-clickhouse-ci-2
e111d4abd3
Merge pull request #54610 from CurtizJ/fix-after-refactoring 
Fix reading of virtual columns in reverse order
 2023-09-14 03:38:27 +02:00
Anton Popov
df2d1ed638 fix reading of virtual columns in reverse order 2023-09-13 21:09:49 +00:00
Tim Windelschmidt
7979090333 use env to find bash 2023-09-13 18:57:11 +02:00
Robert Schulze
59cded9d84
Merge pull request #54563 from rschu1ze/issue-54541 
Fix AST fuzzer crash in MergeTreeIndex{FullText|Inverted}
 2023-09-13 16:29:06 +02:00
Alexander Tokmakov
a365557d9e
Merge pull request #54550 from ClickHouse/fix_zero_copy_garbage 
Fix zero copy garbage
 2023-09-13 15:02:05 +02:00
HarryLeeIBM
735039673a Fix UInt256 and IPv4 random data generation on s390x 2023-09-13 04:55:02 -07:00
alesapin
e8055ba509
Merge pull request #54549 from ClickHouse/fix_redundant_checksums_doesnt_match_error 
Fix rare case of CHECKSUM_DOESNT_MATCH error
 2023-09-13 13:02:50 +02:00
Robert Schulze
8c7a256632
Fix full text index fix 2023-09-13 07:57:48 +00:00
Robert Schulze
e69c1c2f9a
Fix inverted index fix 2023-09-13 07:56:35 +00:00
Han Fei
6970411c2e fix wide part and fuzzer 2023-09-13 08:26:25 +02:00
Robert Schulze
28068f11e1
Fix #54541 2023-09-12 21:23:23 +00:00
alesapin
18ab92eb4f Fix style 2023-09-12 21:57:46 +02:00
alesapin
e17a824fec Fxi 2023-09-12 21:42:32 +02:00
alesapin
6ec65d6be7 Make function slightly more readable 2023-09-12 21:41:05 +02:00
Alexander Tokmakov
46fae52c29 fix zero copy garbage 2023-09-12 20:01:46 +02:00
alesapin
f3729da64b Fix rare case of CHECKSUM_DOESNT_MATCH error 2023-09-12 19:55:56 +02:00
Igor Nikonov
7b3f32b95a
Merge pull request #54520 from ClickHouse/pr-cleanup 
Parallel replicas: cleanup unused params
 2023-09-12 19:48:18 +02:00
Han Fei
430a4fda9c fix some tests 2023-09-12 18:30:55 +02:00
robot-clickhouse
1c8ee76ba2
Merge pull request #54513 from Avogar/formats-with-names-no-header 
Fix possible parsing error in WithNames formats with disabled input_format_with_names_use_header
 2023-09-12 17:58:03 +02:00
Anton Popov
38e0689b3b
Merge pull request #54512 from CurtizJ/fix-use-after-free 
Fix use-after-free in `MergeTreePrefetchedReadPool`
 2023-09-12 16:51:12 +02:00
Dmitry Novik
7edc0a30a3
Merge pull request #54335 from arenadata/ADQM-1109 
Added peak_threads_usage to query_log table
 2023-09-12 16:03:27 +02:00
Dmitry Novik
f730a003ea
Merge pull request #54467 from ClickHouse/drop-lost-part-log 
Add a log message on replicated table drop
 2023-09-12 15:59:18 +02:00
Igor Nikonov
1287f68745 Handle clusterAllReplicas/remote cases to avoid unnecessary logging 2023-09-12 12:52:29 +00:00
Igor Nikonov
3acc26001a Merge remote-tracking branch 'origin/master' into pr-cleanup 2023-09-12 10:38:06 +00:00
Alexey Milovidov
e9daff6c2f
Merge pull request #54506 from azat/system.detached_parts-modification_time 
Add modification_time into system.detached_parts
 2023-09-12 04:53:59 +03:00
Alexey Milovidov
44a9dbd2b7
Update StorageSystemDetachedParts.cpp 2023-09-12 04:53:51 +03:00
Sema Checherinda
0ea0799118
Merge pull request #54430 from CheSema/remove-recursive-with-projection 
reproduce and fix the bug in removeSharedRecursive
 2023-09-12 00:18:55 +02:00
Igor Nikonov
d5ea047ab8 Parallel replicas: cleanup unused params 2023-09-11 21:52:40 +00:00
HarryLeeIBM
26b17f88d5 Add comment for the fix 2023-09-11 13:52:58 -07:00
HarryLeeIBM
0a81389359 Fix endian issue in jemalloc_bins for s390x 2023-09-11 12:14:01 -07:00
Azat Khuzhin
8cd6153c83 Add modification_time into system.detached_parts 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-11 20:29:55 +02:00
Dmitry Novik
b12d1219f2 Merge remote-tracking branch 'origin/master' into drop-lost-part-log 2023-09-11 17:05:05 +00:00
Dmitry Novik
b69a90d61f Change log message level 2023-09-11 14:57:25 +00:00
avogar
2d8f33bfa2 Fix parsing error in WithNames formats while reading subset of columns with disabled input_format_with_names_use_header 2023-09-11 14:55:37 +00:00
Anton Popov
ec37e93461 fix use-after-free in MergeTreePrefetchedReadPool 2023-09-11 14:55:21 +00:00
Nikolai Kochetov
903c966cc8
Merge branch 'master' into planner-prepare-filters-for-analysis-2 2023-09-11 16:14:03 +02:00
Han Fei
57b5f3ca78 fix 2023-09-11 07:24:36 +02:00
Sergei Trifonov
08bad4d440
Merge pull request #47009 from ClickHouse/disks-and-io-sheduler 2023-09-11 07:22:14 +02:00
robot-ch-test-poll
6f3374d684
Merge pull request #54481 from ucasfl/hints 
Refactor IHints
 2023-09-10 15:46:36 +02:00
Azat Khuzhin
5535c7462d Support SAMPLE BY for VIEW 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-10 08:24:46 +02:00
flynn
6ec783a746 Refactor IHints 2023-09-10 03:48:12 +00:00
Han Fei
f9abf16441 fix fast tests 2023-09-10 02:02:01 +02:00
Igor Nikonov
d5ffff93a7
Merge pull request #54468 from ClickHouse/distributed-cleanup-select-info 
Cleanup: unnecessary SelectQueryInfo usage around distributed
 2023-09-09 21:47:36 +02:00
Sema Checherinda
1298dba0c6 fix the bug, add useful logs 2023-09-09 15:54:28 +02:00
Igor Nikonov
2fdc700da2 Style 2023-09-09 13:33:18 +00:00
Igor Nikonov
a12a375a1d Style check 2023-09-09 12:51:34 +00:00
Igor Nikonov
fff2a2d81c Fix style 2023-09-09 08:29:33 +00:00
Igor Nikonov
5470a5b60f Fix style 2023-09-09 06:58:18 +00:00
Mingliang Pan
4d05b57c21
Merge branch 'ClickHouse:master' into master 2023-09-09 11:12:45 +08:00
Igor Nikonov
7a396139df Cleanup: unnecessary SelectQueryInfo usage around distributed 2023-09-08 21:53:38 +00:00
Dmitry Novik
da69a9fda0 Add a log message on replicated table drop 2023-09-08 21:41:42 +00:00
Azat Khuzhin
ffa82e9297 Fix filtering parts with indexHint for non analyzer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-09-08 18:17:06 +02:00
Andrey Zvonov
ff4586e663
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-09-08 18:17:06 +02:00
kssenii
220a67eca7 Better 2023-09-08 16:38:05 +02:00
Anton Popov
1aa34c16e3 Merge remote-tracking branch 'upstream/master' into HEAD 2023-09-08 14:00:45 +00:00
Han Fei
ddcb64f39f update docs and refine statements 2023-09-08 02:27:17 +02:00
kssenii
7338b560a8 S3Queue fixes 2023-09-07 19:17:21 +02:00
Anton Popov
67b27643ed fix creash in case of exception in prefetched pool 2023-09-07 16:56:13 +00:00
Petr Vasilev
d0884d3267 Merge branch 'master' of https://github.com/vahpetr/ClickHouse into fix-nats-high-cpu-usage 2023-09-07 18:48:01 +03:00
Petr Vasilev
cb969aa54a Remove empty line 2023-09-07 18:47:31 +03:00
Petr Vasilev
4ef241445f Remove received_mutex and received_cv 2023-09-07 18:44:18 +03:00
Antonio Andelic
fae0e13baa Properly remove added interserver endpoint 2023-09-07 14:34:36 +00:00
robot-ch-test-poll2
f28ad1e136
Merge pull request #54368 from ClickHouse/fix_replacing_final_error 
Fix logical error in vertical merge + replacing merge tree + optimize cleanup
 2023-09-07 15:45:43 +02:00
serxa
a2f8f6bf29 working on review comments 2023-09-07 13:21:43 +00:00
Petr Vasilev
c0333263a0 Rollback do_reschedule flag 2023-09-07 15:07:21 +03:00
Petr Vasilev
9c61ceb051 Fix nats high cpu usage 2023-09-07 13:59:32 +03:00
zvonand
c36a3fbd39 renamed setting 2023-09-07 12:00:25 +03:00
Alexey Milovidov
1b30f91e7a
Merge pull request #51304 from ClickHouse/alter-table-add-comment 
Add support for ALTER TABLE MODIFY COMMENT
 2023-09-07 07:30:57 +03:00
Alexey Gerasimchuck
c2f4dc0f14 Implemented peak_threads_usage 2023-09-07 01:32:39 +00:00
serxa
77697211cd add docs for system.scheduler + small fix 2023-09-06 19:30:39 +00:00
SmitaRKulkarni
f0146c0ff0
Increase retries for test_merge_tree_azure_blob_storage (#54069) 
* Increase max_single_download_retries & max_single_read_retries for test_merge_tree_azure_blob_storage

* Updated logs to log exception leading to detaching part

* Updated to catch RequestFailedException azure exception as its the base exception

* Updated isRetryableExcepion to take std::exception_ptr as input

* Fixed style check fails

* Fix build issue & moved NetException & TimeoutException to retryable exceptions

* Fixed style check with catch
 2023-09-06 19:57:59 +02:00
Alexander Sapin
4368c00052 Fix logical error in vertical merge + replacing merge tree + optimize cleanup 2023-09-06 18:43:56 +02:00
Kruglov Pavel
5b29640b22
Merge branch 'master' into group-by-constant-keys 2023-09-06 18:06:51 +02:00
robot-ch-test-poll4
8c71582316
Merge pull request #54185 from ClickHouse/analyzer-partition-id-pruning 
Fix partition id pruning for analyzer.
 2023-09-06 16:53:35 +02:00
robot-ch-test-poll1
9e50e8ff20
Merge pull request #54193 from ClickHouse/fix_replace_partition_with_zero_copy 
Fix REPLACE/MOVE PARTITION with zero-copy replication
 2023-09-06 16:04:25 +02:00
Alexander Sapin
a176c38876 Fix tests with database replicated 2023-09-06 14:17:46 +02:00
Alexander Sapin
518d4b4357 Merge branch 'master' into alter-table-add-comment 2023-09-06 12:34:04 +02:00
Anton Popov
401496aa1f Apply suggestions from code review 
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
 2023-09-06 10:27:46 +00:00
robot-ch-test-poll1
d9746b5ea1
Merge pull request #54336 from arenadata/ADQM-1157 
Used assert_cast instead of dynamic_cast in ExternalDataSourceCache
 2023-09-06 08:28:21 +02:00
robot-clickhouse-ci-1
02339a1f22
Merge pull request #54326 from ClickHouse/fix_segfault_system_zookeeper 
Fix segfault in system.zookeeper
 2023-09-06 02:20:50 +02:00
Alexey Gerasimchuck
f9edf01280 used assert_cast instead of dynamic_cast 2023-09-05 22:57:40 +00:00
robot-clickhouse-ci-1
7403ee689f
Merge pull request #53877 from vitlibar/add-restore-setting-allow-broken-parts 
Add restore setting "restore_broken_parts_as_detached"
 2023-09-05 21:28:08 +02:00
Kruglov Pavel
de801ce563
Merge pull request #54293 from ClickHouse/pasha-returned-from-vacation 
Code improvement for reading from archives
 2023-09-05 21:09:09 +02:00
Alexander Tokmakov
a8489c21da fix segfault in system.zookeeper 2023-09-05 20:23:16 +02:00
robot-ch-test-poll
2e7edf4413
Merge pull request #53638 from arenadata/ADQM-987 
Add `backup_log` system table
 2023-09-05 17:03:41 +02:00
Antonio Andelic
88930a335c Apply comments 2023-09-05 12:32:07 +00:00
Anton Popov
5da7133c94
Merge pull request #53735 from priera/fix_zk_error_when_doing_replace 
37737 fixed internal error when replacing with byte-equal parts
 2023-09-05 12:51:14 +02:00
alesapin
095bb37bc4
Merge pull request #54244 from bigo-sg/hdfs-config 
Minor improvement, load from config
 2023-09-05 12:36:38 +02:00
Kruglov Pavel
fd42f0bac0
Merge pull request #54271 from Avogar/better-count-and-progress-archives 
Fix using count from cache and improve progress bar for reading from archives
 2023-09-05 12:12:39 +02:00
Antonio Andelic
a51c333c5f Improve code for reading from archives 2023-09-05 08:37:02 +00:00
Victor Krasnov
132ab6067b Merge branch 'master' into ADQM-987 2023-09-05 10:01:44 +03:00
Alexey Milovidov
a59c694e4f
Merge pull request #54225 from Hexta/recreate-secondary-index-system-data-size 
Fix system.data_skipping_indices for recreated indices
 2023-09-05 01:13:09 +03:00
Alexey Milovidov
1a458d524e
Merge pull request #54260 from ClickHouse/vdimir/enable_allow_remove_stale_moving_parts 
Enable allow_remove_stale_moving_parts by default
 2023-09-05 01:11:22 +03:00
robot-clickhouse
926c5636dd
Merge pull request #42599 from ClickHouse/build-fuzzer-protocol 
libFuzzer: add CI fuzzers build, add tcp protocol fuzzer, fix other fuzzers.
 2023-09-04 22:41:54 +02:00
avogar
e9456082c1 Fix using count from cache and improve progress bar for reading from archives 2023-09-04 17:33:59 +00:00
Vitaly Baranov
8060407c70 Add backup setting "check_parts" and restore setting "restore_broken_parts_as_detached". 2023-09-04 18:54:03 +02:00
vdimir
f2ff693a39
Enable allow_remove_stale_moving_parts by default 2023-09-04 15:41:22 +00:00
Alexander Tokmakov
dfcc6f78b3 fix 2023-09-04 16:51:25 +02:00
serxa
aa8e9c3cbc fix style 2023-09-04 14:18:54 +00:00
Han Fei
f60dad0598 Merge branch 'master' into hanfei/statistic 2023-09-04 16:17:07 +02:00
Alexander Tokmakov
ea996b0a00 Merge branch 'master' into fix_replace_partition_with_zero_copy 2023-09-04 15:55:51 +02:00
Han Fei
2330a28f7b support estimate by stat 2023-09-04 15:51:00 +02:00
Sergei Trifonov
258da91b5f
Merge branch 'master' into disks-and-io-sheduler 2023-09-04 15:14:28 +02:00
serxa
b5233efc20 fix system table and scheduler config reloading 2023-09-04 12:35:17 +00:00
Victor Krasnov
78760639d2 Merge branch 'master' into ADQM-987 2023-09-04 13:59:20 +03:00
zhanglistar
a2fd3a4de8 remove unnecessary check for HDFS load config 2023-09-04 17:52:43 +08:00
Alexey Milovidov
864e66cbf7 Fix TOCTOU 2023-09-04 04:06:24 +02:00
Alexey Milovidov
e618dc993e Fix bad code in the system.filesystem_cache 2023-09-04 03:53:18 +02:00
serxa
8fdd118e0c fix build 2023-09-03 22:40:00 +00:00
serxa
adcab639a4 add system.scheduler table for introspection 2023-09-03 21:48:33 +00:00
Alexey Milovidov
86223699be
Merge pull request #54200 from ClickHouse/fix-ipv6-bloom 
Fix: allow IPv6 for bloom filter
 2023-09-03 23:08:51 +03:00
Artur Malchanau
04d0b0c09a
Fix system.data_skipping_indices for recreated indices 
Fix secondary index size in system.data_skipping_indices table in the
case of dropped and added again index.
 2023-09-03 20:36:55 +03:00
Kseniia Sumarokova
b3319f7908
Minor changes (#54171) 2023-09-03 15:47:52 +02:00
robot-ch-test-poll2
776f232ec0
Merge pull request #54215 from ClickHouse/remove-metadata-cache 
Deprecate metadata cache, part 2
 2023-09-03 07:17:26 +02:00
Alexey Milovidov
7cb4d52d92 Deprecate metadata cache, part 2 2023-09-03 03:56:11 +02:00
Alexey Milovidov
282636fd74 S3Queue is experimental 2023-09-03 00:00:37 +02:00
Michael Kolupaev
8505b3c940
Merge pull request #54134 from kssenii/refactor-reading-from-remote-fs 
Small refactoring for read from object storage
 2023-09-02 11:13:03 -07:00
Yakov Olkhovskiy
a3216c78ef allow IPv6 for bloom filter 2023-09-01 22:44:05 +00:00
Alexander Tokmakov
59d28f3787 fix 2023-09-01 22:40:08 +02:00
Sergei Trifonov
aac5b0386a
Update src/Storages/System/StorageSystemAsyncLoader.h 2023-09-01 20:40:38 +02:00
Alexander Tokmakov
bba5a1d480 update zero copy locks correctly 2023-09-01 20:10:23 +02:00
Sergei Trifonov
cf5ea46713
Merge pull request #53338 from azat/throttling-fixes 
Fix IO throttling during copying whole directories
 2023-09-01 18:29:15 +02:00
Alexey Milovidov
f8c2ea986d Update autogenerated version to 23.9.1.1 and contributors 2023-09-01 17:46:49 +02:00
Nikolai Kochetov
46087fd3e5 Refactor. 2023-09-01 15:45:08 +00:00
Victor Krasnov
52cce27e81 Merge branch 'master' into ADQM-987 2023-09-01 18:00:20 +03:00
Nikolai Kochetov
4cbfca78e4 Fixing 01748_partition_id_pruning 2023-09-01 14:29:31 +00:00
Yakov Olkhovskiy
361b21b416 fix fuzzers, cmake refactor, add target fuzzers 2023-09-01 14:20:50 +00:00
robot-clickhouse-ci-2
ae4f6c9683
Merge pull request #54137 from kssenii/s3-queue-better-default-path 
Better path for storage `S3Queue` metadata
 2023-09-01 16:01:38 +02:00
Anton Popov
033dba4772
Merge pull request #54126 from ClickHouse/CurtizJ-patch-8 
Small improvements in `getAlterMutationCommandsForPart`
 2023-09-01 14:10:45 +02:00
kssenii
9eb1dfcd12 Refactor buffers reading from object storage 2023-09-01 14:03:07 +02:00
Nikolai Kochetov
291bd76f1d
Merge pull request #54133 from ClickHouse/fixing-01086_odbc_roundtrip-with-analyzer 
Fixing 01086_odbc_roundtrip with analyzer.
 2023-09-01 12:54:50 +02:00
Kseniia Sumarokova
c51b6ca60c
Update StorageS3Queue.cpp 2023-08-31 18:29:06 +02:00
kssenii
da185fc8b1 Better paths for s3 queue metadata 2023-08-31 18:16:07 +02:00
Nikolai Kochetov
dc062856b0 Fixing 01086_odbc_roundtrip with analyzer. 2023-08-31 15:23:27 +00:00
Anton Popov
af70fdfcfa mute noisy log message 2023-08-31 15:05:35 +00:00
Igor Nikonov
c3caead859
Merge pull request #52688 from Algunenano/better_zk_retries_on_insert 
Improve insert retries on keeper session expiration
 2023-08-31 17:00:02 +02:00
Dmitry Novik
5266511281 Merge remote-tracking branch 'origin/master' into revert-53782-revert-52762-planner-prepare-filters-for-analysis 2023-08-31 14:40:53 +00:00
pufit
1bc6b40a12
Merge pull request #35102 from tonickkozlov/bloom-filter-index/deduplicate 
Prune bloom filter indices based on data cardinality
 2023-08-31 10:04:19 -04:00
Anton Kozlov
af21620ab5
Update src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h 
Co-authored-by: pufit <pufit@yandex.ru>
 2023-08-31 14:06:21 +01:00
Anton Kozlov
d3123da824
Update src/Storages/MergeTree/MergeTreeIndexAggregatorBloomFilter.cpp 
Co-authored-by: pufit <pufit@yandex.ru>
 2023-08-31 14:05:57 +01:00
Anton Kozlov
3f6ef6b5d9
Update src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h 
Co-authored-by: pufit <pufit@yandex.ru>
 2023-08-31 14:05:41 +01:00
Anton Kozlov
e6895688d7
Update src/Storages/MergeTree/MergeTreeIndexAggregatorBloomFilter.cpp 
Co-authored-by: pufit <pufit@yandex.ru>
 2023-08-31 14:05:28 +01:00
Anton Popov
579ec699bf Merge remote-tracking branch 'upstream/master' into HEAD 2023-08-31 12:59:59 +00:00
Anton Popov
77111a229d fix clang-tidy 2023-08-31 12:59:21 +00:00
Raúl Marín
4703c463aa Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-31 14:37:02 +02:00
Andrey Zvonov
83405a8808
Merge branch 'master' into zvonand-globs-small-fix 2023-08-31 08:42:48 +02:00
Antonio Andelic
0148e15aee
Merge pull request #53880 from ClickHouse/archive-improvements-2 
Improve schema inference for archives
 2023-08-30 21:11:34 +02:00
Alexander Tokmakov
83c5e2fba6
Revert "Fix bug on mutations with subcolumns of type JSON in predicates of UPDATE and DELETE queries." 2023-08-30 12:56:17 +02:00
Antonio Andelic
f406019413 Apply PR comments 2023-08-30 09:26:01 +00:00
Antonio Andelic
ddb58217d4 Merge branch 'master' into archive-improvements-2 2023-08-30 07:43:25 +00:00
Antonio Andelic
36fb7cfbd1
Merge pull request #54012 from ClickHouse/refactor-async-insert-with-dedup 
Refactor logic around async insert with deduplication
 2023-08-30 09:37:55 +02:00
Azat Khuzhin
51e8b16916 Fix IO throttling during copying whole directories 
v2: pass local context during ALTER TABLE MOVE PARTITION TO VOLUME/DISK as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-29 23:13:46 +02:00
Nikita Mikhaylov
97efd14e90
Merge pull request #54009 from ClickHouse/change-istorage-interface 
Change IStorage interface by random walk, no goal in particular
 2023-08-29 22:22:53 +02:00
Sergei Trifonov
802579f3f1
Merge pull request #49618 from ClickHouse/concurrency-control-controllable 
Make concurrency control controllable
 2023-08-29 19:44:51 +02:00
Anton Popov
8197580746 fix tests and better interface 2023-08-29 17:23:52 +00:00
Antonio Andelic
e96c665be7 Refactor logic around async insert with dedup 2023-08-29 14:48:43 +00:00
Alexander Tokmakov
17f3754193
Merge pull request #53851 from ClickHouse/distributed_no_flush_on_shutdown 
Follow-up to #53501
 2023-08-29 16:34:22 +02:00
Nikita Mikhaylov
96cabcae3a Change interface 2023-08-29 16:26:48 +02:00
Anton Popov
7bc5627912 fix typos 2023-08-29 14:21:58 +00:00
Anton Popov
02391eafe1 refactoring of reading from MergeTree tables 2023-08-29 13:40:12 +00:00
Anton Popov
6ce4a82a04
Merge pull request #53677 from VanDarkholme7/mutations_subcolumns 
Fix bug on mutations with subcolumns of type JSON in predicates of UPDATE and DELETE queries.
 2023-08-29 15:08:00 +02:00
Antonio Andelic
a0dd350e5e Fix assert 2023-08-29 09:59:16 +00:00
Raúl Marín
45e8185f9b Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-08-29 11:17:34 +02:00
Andrey Zvonov
6195390c77
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-08-29 13:46:09 +05:00
János Benjamin Antal
a1a45ee905
Merge pull request #53766 from ilejn/exceptions_kafka_consumers 
system.kafka_consumers exception info improvements
 2023-08-29 10:22:26 +02:00
Antonio Andelic
8708ebd7fb Correctly check files 2023-08-29 07:40:08 +00:00
Sergei Trifonov
c10e42ca05
Merge branch 'master' into async-loader-integration 2023-08-28 20:20:31 +02:00
Anton Popov
03f5d3feb4 fix checkDataPart 2023-08-28 14:59:03 +00:00
Raúl Marín
174f7a4e63 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-28 16:27:45 +02:00
Alexander Tokmakov
d41eca1dcc rename new method 2023-08-28 16:01:00 +02:00
Antonio Andelic
9b99f25d75 Improve schema inference 2023-08-28 13:11:52 +00:00
Robert Schulze
4c78c08847
Add test 2023-08-28 13:02:24 +00:00
Davit Vardanyan
9190fb2ba3 Fix: Deserialize 2023-08-28 16:18:12 +04:00
Andrey Zvonov
c9b0db9c61
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-08-28 17:15:33 +05:00
Andrey Zvonov
2447eb027c
Update StorageHDFS.cpp 2023-08-28 15:12:44 +03:00
Igor Nikonov
a2686af496
Merge pull request #53862 from ClickHouse/cleanup-parallel-replicas-0 
Remove unused parallel replicas coordinator in query info
 2023-08-28 13:30:23 +02:00
Antonio Andelic
5a0c2ca108 Merge branch 'master' into archive-improvements-2 2023-08-28 08:34:42 +00:00
VanDarkholme7
bac257743b
Merge branch 'ClickHouse:master' into mutations_subcolumns 2023-08-28 14:10:31 +08:00
Igor Nikonov
7368bf65b3 Remove unused parallel replicas coordinator in query info 2023-08-27 22:12:39 +00:00
Victor Krasnov
87c90767d9 Merge branch 'master' into ADQM-987 2023-08-27 09:14:42 +03:00
alesapin
0e601b856f
Merge pull request #53751 from Algunenano/clang18 
Support clang-18 (Wmissing-field-initializers)
 2023-08-25 23:27:06 +02:00
Andrey Zvonov
911bf94fbd
Merge branch 'master' into zvonand-globs-small-fix 2023-08-25 23:50:07 +05:00
Victor Krasnov
921f0c8b5d Merge branch 'master' into ADQM-987 2023-08-25 21:20:23 +03:00
Alexander Tokmakov
32efbe77d1
Merge pull request #53811 from ClickHouse/fix_02443_2 
Fix a bug in attach partition
 2023-08-25 20:16:53 +03:00
Alexander Tokmakov
9ab545e28c do not wait for flush on shutdown 2023-08-25 19:09:10 +02:00
Alexander Gololobov
684d5aab88
Merge pull request #53797 from ClickHouse/add_profile_event_for_lock 
Add event for tracking data parts lock
 2023-08-25 18:51:49 +02:00
Kseniia Sumarokova
34ac113af6
Merge pull request #53121 from kssenii/fix-cache-logical-error 
Fix possible error from cache "Read unexpected size"
 2023-08-25 12:16:24 +02:00
Nikolai Kochetov
4e4620250d Fixing clang tidy. 2023-08-25 09:51:52 +00:00
Kruglov Pavel
d80be4673b
Merge pull request #53692 from Avogar/cache-count 
Cache number of rows in files for count in file/s3/url/hdfs/azure functions
 2023-08-25 09:08:52 +02:00
Antonio Andelic
f0e800092f
Merge pull request #53796 from ClickHouse/fix-filtered-url-paths 
Fix reading from `url` with all filtered paths
 2023-08-25 08:49:33 +02:00
Yakov Olkhovskiy
415a993c91 fix fuzzers build 2023-08-24 23:22:39 +00:00
alesapin
3a289534cb Fxi 2023-08-24 20:48:10 +02:00
Nikolai Kochetov
506c5667a7 Remove comment. 2023-08-24 18:21:41 +00:00
Alexander Tokmakov
cbbb81f5cc
Merge pull request #53405 from canhld94/limit_concurent_background_optimize 
Limit number of concurrent background partition optimize
 2023-08-24 20:49:38 +03:00
Alexander Tokmakov
75d32bfe77
Merge pull request #53750 from ClickHouse/Add_retries_for_azure_retryable_errors 
Add retries for azure retryable errors
 2023-08-24 20:26:42 +03:00
Alexander Tokmakov
e1d135dc01 fix another bug in attach partition 2023-08-24 19:10:25 +02:00
Nikolai Kochetov
9737dbca75 Fix window functions. 2023-08-24 16:37:27 +00:00
Raúl Marín
6490213207 Merge remote-tracking branch 'blessed/master' into clang18 2023-08-24 17:35:26 +02:00
Kruglov Pavel
44db5fa992
Merge branch 'master' into cache-count 2023-08-24 17:21:18 +02:00
avogar
c4f8113a78 Fix tests 2023-08-24 15:05:45 +00:00
Robert Schulze
45d924c62c
Merge pull request #53616 from wzb5212/skip_idx_bug_fix 
Fix number of dropped granules in EXPLAIN PLAN index=1
 2023-08-24 16:46:47 +02:00
Han Fei
10c622fa08 Merge branch 'master' into hanfei/statistic 2023-08-24 15:56:06 +02:00
Antonio Andelic
6eb400c6ae Fix reading from url with all filtered paths 2023-08-24 13:37:11 +00:00
alesapin
415b3826e0 Add event for tracking data parts lock 2023-08-24 15:28:38 +02:00
Ilya Golshtein
f58b4a812d exceptions_kafka_consumers: code review suggestions are addressed 2023-08-24 13:10:21 +00:00
Arthur Passos
2bade7db08
Add global proxy setting (#51749) 
* initial impl

* fix env ut

* move ut directory

* make sure no null proxy resolver is returned by ProxyConfigurationResolverProvider

* minor adjustment

* add a few tests, still incomplete

* add proxy support for url table function

* use proxy for select from url as well

* remove optional from return type, just returns empty config

* fix style

* style

* black

* ohg boy

* rm in progress file

* god pls don't let me kill anyone

* ...

* add use_aws guards

* remove hard coded s3 proxy resolver

* add concurrency-mt-unsafe

* aa

* black

* add logging back

* revert change

* imrpove code a bit

* helper functions and separate tests

* for some reason, this env test is not working..

* formatting

* :)

* clangtidy

* lint

* revert some stupid things

* small test adjusmtments

* simplify tests

* rename test

* remove extra line

* freaking style change

* simplify a bit

* fix segfault & remove an extra call

* tightly couple proxy provider with context..

* remove useless include

* rename config prefix parameter

* simplify provider a bit

* organize provider a bit

* add a few comments

* comment out proxy env tests

* fix nullptr in unit tests

* make sure old storage proxy config is properly covered without global context instance

* move a few functions from class to anonymous namespace

* fix no fallback for specific storage conf

* change API to accept http method instead of bool

* implement http/https distinction in listresolver, any still not implemented

* implement http/https distinction in remote resolver

* progress on code, improve tests and add url function working test

* use protcol instead of method for http and https

* small fix

* few more adjustments

* fix style

* black

* move enum to proxyconfiguration

* wip

* fix build

* fix ut

* delete atomicroundrobin class

* remove stale include

* add some tests.. need to spend some more time on the design..

* change design a bit

* progress

* use existing context for tests

* rename aux function and fix ut

* ..

* rename test

* try to simplify tests a bit

* simplify tests a bit more

* attempt to fix tests, accept more than one remote resolver

* use proper log id

* try waiting for resolver

* proper wait logic

* black

* empty

* address a few comments

* refactor tests

* remove old tests

* baclk

* use RAII to set/unset env

* black

* clang tidy

* fix env proxy not respecting any

* use log trace

* fix wrong logic in getRemoteREsolver

* fix wrong logic in getRemoteREsolver

* fix test

* remove unwanted code

* remove ClientConfigurationperRequest and auxilary classes

* remove unwanted code

* remove adapter test

* few adjustments and add test for s3 storage conf  with new proxy settings

* black

* use chassert for context

* Add getenv comment
 2023-08-24 16:07:26 +03:00
Antonio Andelic
8e1d38d377 Merge branch 'master' into archive-improvements-2 2023-08-24 13:03:36 +00:00
Han Fei
d80fd7c74c small improvement 2023-08-24 14:20:34 +02:00
Han Fei
c29261e5b4 support estimate by t-digest 2023-08-24 14:10:22 +02:00
Nikolai Kochetov
cb851fcee0 Cleanup. 2023-08-24 11:07:17 +00:00
Nikolai Kochetov
33b8b93d1b Re-implement getOptimizedQueryProcessingStage with analyzer. 2023-08-24 11:07:17 +00:00
Nikolai Kochetov
fc90a1a0bd Fix some skip_unused_shards tests. 2023-08-24 11:07:17 +00:00
Nikolai Kochetov
26e0ad8d72 Re-impl evaluateExpressionOverConstantCondition [part 1] 2023-08-24 11:07:17 +00:00
Nikolai Kochetov
93ab53163a
Revert "Revert "Planner prepare filters for analysis"" 2023-08-24 12:32:56 +02:00
Raúl Marín
1c98568df9 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-24 12:22:42 +02:00
Robert Schulze
8c9fade58c
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-24 10:17:34 +00:00
Kseniia Sumarokova
a4f39fe134
Merge branch 'master' into fix-cache-logical-error 2023-08-24 12:00:18 +02:00
Alexander Tokmakov
011c7f7064
Revert "Planner prepare filters for analysis" 2023-08-24 12:52:03 +03:00
Kseniia Sumarokova
187706558a
Merge pull request #51622 from kssenii/system-sync-cache 
Add `system sync filesystem cache` command
 2023-08-24 11:42:32 +02:00
Ilya Golshtein
901240eede exceptions_kafka_consumers: clang-tidy improvement 2023-08-24 08:44:08 +00:00
helifu
adfdba4b83 Display reloaded values of server settings 2023-08-24 12:32:30 +08:00
Ilya Golshtein
ec0c12960a exceptions_kafka_consumers: stylecheck 2023-08-23 23:10:25 +00:00
Ilya Golshtein
e7213ef5b9 exceptions_kafka_consumers: new test, MV exceptions propagated 2023-08-23 22:46:04 +00:00
Kruglov Pavel
685ffc5856
Fix style 2023-08-23 22:51:14 +02:00
Kruglov Pavel
f7e1abd774
Merge branch 'master' into cache-count 2023-08-23 22:31:49 +02:00
Kruglov Pavel
063c986680
Merge pull request #53637 from Avogar/fast-count-from-files 
Optimize count from files in most input formats
 2023-08-23 22:17:29 +02:00
avogar
93ba73715a Merge branch 'master' of github.com:ClickHouse/ClickHouse into group-by-constant-keys 2023-08-23 18:47:42 +00:00
avogar
894513f6cd Fix tests 2023-08-23 18:43:08 +00:00
Nikolai Kochetov
3a4a7d70a2
Merge pull request #52762 from kitaisreal/planner-prepare-filters-for-analysis 
Planner prepare filters for analysis
 2023-08-23 18:59:43 +02:00
Anton Kozlov
a5b6a91b54 [bloom filter] Write hashes for bloom filter into a hash set for optimal sizing 2023-08-23 14:17:55 +00:00
Victor Krasnov
8b2342c205 Minor code cleanup: remove some redundant includes of InterpreterAlterQuery.h 2023-08-23 14:16:36 +00:00
Raúl Marín
93dac0c880 Support clang-18 (Wmissing-field-initializers) 2023-08-23 15:53:45 +02:00
Smita Kulkarni
262c39329b Updated to retry loading parts for azure TransportException 2023-08-23 15:50:11 +02:00
Smita Kulkarni
1a074ecf93 Updated to not mark part as broken for azure TransportException in loadDataPart 2023-08-23 15:33:01 +02:00
Smita Kulkarni
7fdf90ccf1 Added azure TransportException to retryable errors 2023-08-23 15:21:37 +02:00
avogar
68e3af56d4 Address comments 2023-08-23 13:19:15 +00:00
Kruglov Pavel
592fa77987
Merge branch 'master' into cache-count 2023-08-23 15:18:02 +02:00
Kruglov Pavel
7e362a2110
Merge branch 'master' into fast-count-from-files 2023-08-23 15:13:20 +02:00
Robert Schulze
bea651e828
Merge pull request #53657 from rschu1ze/cache-improvements 
Expose more settings from caches
 2023-08-23 14:58:45 +02:00
Robert Schulze
a093f519fc
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-23 12:37:46 +00:00
Robert Schulze
5dec6f19f3
Merge pull request #51325 from xinhuitian/fix-annoy-index-update 
Do not reset Annoy index during build-up with > 1 mark
 2023-08-23 14:31:06 +02:00
robot-ch-test-poll1
c22ffa6195
Merge pull request #53529 from Avogar/filter-files-all-table-functions 
Use filter by file/path before reading in url/file/hdfs table functins
 2023-08-23 14:21:23 +02:00
Robert Schulze
761268f86a
Merge remote-tracking branch 'rschu1ze/master' into skip_idx_bug_fix 2023-08-23 10:55:04 +00:00
Kruglov Pavel
e193aec583
Merge branch 'master' into fast-count-from-files 2023-08-23 12:15:34 +02:00
kssenii
325e2d3172 Merge remote-tracking branch 'upstream/master' into system-sync-cache 2023-08-23 11:44:19 +02:00
priera
fd51839b97 basic implementation 2023-08-23 11:19:01 +02:00
Antonio Andelic
83d4b819f3 Better support for reading from archives 2023-08-23 08:10:30 +00:00
xinhuitian
920762e66d fix USearch 2023-08-23 10:54:56 +08:00
Tian Xinhui
7cf057e716
Merge branch 'ClickHouse:master' into fix-annoy-index-update 2023-08-23 10:33:14 +08:00
pufit
926533306c
Merge pull request #53668 from ClickHouse/pufit/fix_s3_threads 
Limiting the number of parsing threads for the S3 source
 2023-08-22 19:09:33 -04:00
Kruglov Pavel
a83a868acd
Update src/Storages/HDFS/StorageHDFS.cpp 
Co-authored-by: Dmitry Novik <mrnovikd@gmail.com>
 2023-08-22 19:08:04 +02:00
Robert Schulze
d5a7277a59
Better parameter name 2023-08-22 15:43:13 +00:00
pufit
e42da9411b Fix variables names 2023-08-22 11:23:10 -04:00
Raúl Marín
abfd48bcd2 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-22 17:01:22 +02:00
Nikolai Kochetov
5ee9b56040 Do not change the snapshot after at reading from MT 2023-08-22 13:50:51 +00:00
Victor Krasnov
258f9927b4 Merge branch 'master' into ADQM-987 2023-08-22 16:49:32 +03:00
Kruglov Pavel
de960f3c35
Fix style 2023-08-22 15:22:17 +02:00
Kruglov Pavel
a34c9c3ebc
Fix build 2023-08-22 15:21:54 +02:00
Kruglov Pavel
67c5c0203b
Merge branch 'master' into fast-count-from-files 2023-08-22 15:03:48 +02:00
avogar
89b402f4e2 Fix 2023-08-22 13:01:07 +00:00
avogar
7f9e81d504 Clean up 2023-08-22 12:55:00 +00:00
alesapin
b4aaeb1a82 Fix bad log line format 2023-08-22 14:53:35 +02:00
Kruglov Pavel
c0bdd0e00b
Merge branch 'master' into cache-count 2023-08-22 14:42:22 +02:00
avogar
b4145aeddc Cache number of rows in files for count in file/s3/url/hdfs/azure functions 2023-08-22 11:59:59 +00:00
Robert Schulze
f2fb7365e6
Add settings for 1. size_ratio (all caches) and 2. policy (index caches) 
1. is an expert-level setting, default is 0.5, applies only to SLRU.

Also, I noticed that we expose cache policy settings for the mark and
the uncompresed cache but not for the index mark and the index
uncompressed cache. Changed that as well, it simplifies the code a bit.
 2023-08-22 11:04:29 +00:00
Ilya Golshtein
7cd8237a42 exceptions_kafka_consumers: initial 2023-08-22 10:26:52 +00:00
Robert Schulze
acaa2eee58
Merge pull request #53556 from ClibMouse/feature/big-endian-UUID-correction 
Change Big Endian-UUID to work the same as Little Endian-UUID
 2023-08-22 11:57:44 +02:00
alesapin
1a5a9fca87
Merge pull request #53605 from irenjj/feat_53538 
Add column 'name' to system.clusters as an alias to cluster.
 2023-08-22 11:34:21 +02:00
VanDarkholme7
2638cc69d7 fix bug on DELETE and UPDATE with JSON subcolumns 2023-08-22 09:26:17 +00:00
Tian Xinhui
76016d9593
Merge branch 'master' into fix-annoy-index-update 2023-08-22 16:58:29 +08:00
pufit
9d454d9afc Merge branch 'master' into pufit/fix_s3_threads 
# Conflicts:
#	src/Storages/StorageS3.cpp
#	src/Storages/StorageS3.h
#	src/Storages/StorageURL.cpp
#	src/Storages/StorageURL.h
 2023-08-21 21:32:15 -04:00
pufit
98a701e2c1 Limiting number of parsing threads for S3 source 2023-08-21 21:21:03 -04:00
Robert Schulze
a20ceef398
Merge pull request #53447 from davvard/master 
USearch HNSW index for ANN search
 2023-08-21 23:34:26 +02:00
Michael Kolupaev
2f4d433e69 Parquet filter pushdown 2023-08-21 14:15:52 -07:00
Nikolai Kochetov
4222ac528d Merge branch 'master' into planner-prepare-filters-for-analysis 2023-08-21 16:48:33 +00:00
Kruglov Pavel
aebbf00a5a
Fix 2023-08-21 17:47:30 +02:00
Anton Popov
3e9a182555 better interfaces for getting of stream name in part 2023-08-21 13:52:25 +00:00
Kruglov Pavel
a2f81bf4ea
Fix typo 2023-08-21 15:20:44 +02:00
Kruglov Pavel
dfdd26ee71
Fix build 2023-08-21 15:04:03 +02:00
Kruglov Pavel
88aee95122
Merge branch 'master' into fast-count-from-files 2023-08-21 14:46:33 +02:00
avogar
584bd57f5c Clean up 2023-08-21 12:43:11 +00:00
avogar
47304bf7aa Optimize count from files in most input formats 2023-08-21 12:30:52 +00:00
Kruglov Pavel
9d9bdde0e3
Merge branch 'master' into filter-files-all-table-functions 2023-08-21 14:26:59 +02:00
Kruglov Pavel
9869196021
Update VirtualColumnUtils.cpp 2023-08-21 14:22:48 +02:00
Antonio Andelic
ed15babddd
Merge pull request #53198 from ClickHouse/improve-archive-reading 
Improve reading from archives
 2023-08-21 14:11:25 +02:00
Victor Krasnov
a7813260c0 Merge branch 'master' into ADQM-987 2023-08-21 14:45:13 +03:00
Kruglov Pavel
21413636ac
Remove debug logging 2023-08-21 13:30:08 +02:00
Robert Schulze
2c330516b8
Another compile fix 2023-08-21 11:12:39 +00:00
Duc Canh Le
2c07065f81 increase the default value 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-08-21 11:04:24 +00:00
Raúl Marín
f2d3c9c281 Merge remote-tracking branch 'blessed/master' into better_zk_retries_on_insert 2023-08-21 12:14:31 +02:00
Robert Schulze
066ec559ad
Do not reset Annoy index during build-up with > 1 mark 2023-08-21 10:05:43 +00:00
王智博
8bb0f18019 bad size of mark skip idx 2023-08-20 22:57:53 +08:00
Kseniia Sumarokova
404fb0588a
Merge branch 'master' into fix-cache-logical-error 2023-08-20 16:28:13 +02:00
irenjj
a9b65ee929 Add Alias for system.clusters. 2023-08-20 14:47:07 +08:00
irenjj
71459c6d65 Revert 2023-08-20 14:35:59 +08:00
irenjj
e64a1fc26d Add column 'name' to system.clusters as an alias to cluster. 2023-08-20 11:52:14 +08:00
Igor Nikonov
a69ea02418 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-19 21:43:25 +00:00
Kseniia Sumarokova
aeebedfa0d
Merge branch 'master' into system-sync-cache 2023-08-19 13:32:48 +04:00
Kseniia Sumarokova
c0b8d7eddf
Merge pull request #50999 from ilejn/system_kafka_consumers 
system.kafka_consumers table to monitor kafka consumers
 2023-08-19 13:31:18 +04:00
kssenii
247abe0a2a Merge remote-tracking branch 'upstream/master' into system-sync-cache 2023-08-19 08:43:53 +04:00
Alexey Milovidov
8520a7baca
Merge pull request #53559 from azat/skip-index-set-fix 
Fix possible UB in Set skipping index for functions with incorrect args
 2023-08-19 02:06:18 +03:00
Alexey Milovidov
6fb24fe74d
Merge pull request #53560 from azat/skip-index-inverted-fix-ub 
Fix possible UB in inverted indexes (experimental feature)
 2023-08-19 02:06:04 +03:00
Igor Nikonov
aebda10f7c Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-18 23:00:58 +00:00
Alexey Milovidov
7ad8ea78bb
Merge pull request #48990 from incfly/partition 
Add partition to the system part_log and merge table.
 2023-08-18 21:38:53 +03:00
avogar
8e445b5461 Fixes 2023-08-18 17:49:40 +00:00
Azat Khuzhin
c78b94bed6 Fix possible UB in inverted indexes (experimental feature) 
It is possible to have incorrect types there in case of index contains
functions, add a proper check after value had been changed.

<details>

<summary>MSan report</summary>

```
==182==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55658547c59e in DB::MergeTreeConditionInverted::traverseASTEquals() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:551:26
    1 0x556585475566 in DB::MergeTreeConditionInverted::traverseAtomAST() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:438:21
    2 0x55658547e73e in DB::MergeTreeConditionInverted::MergeTreeConditionInverted()::$_1::operator()(DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:228:73
    9 0x556585489e35 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:252:14
    10 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    11 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    12 0x5565854804e6 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::RPNBuilder(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::Block, std::__1::shared_ptr<DB::PreparedSets>, std::__1::function<bool (DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&)> const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:218:9
    13 0x55658546fb87 in DB::MergeTreeConditionInverted::MergeTreeConditionInverted(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>, DB::Block const&, DB::GinFilterParameters const&, DB::ITokenExtractor const*) build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:223:28
    19 0x55658547ceda in DB::MergeTreeIndexInverted::createIndexCondition(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:716:12
    20 0x556587125734 in DB::buildIndexes(std::__1::optional<DB::ReadFromMergeTree::Indexes>&, std::__1::shared_ptr<DB::ActionsDAG>, DB::MergeTreeData const&, std::__1::shared_ptr<DB::Context const> const&, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&) build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1292:48
    21 0x556587121aaa in DB::ReadFromMergeTree::applyFilters() build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1305:5
    22 0x55658726cf4f in DB::QueryPlanOptimizations::optimizeTreeThirdPass(DB::QueryPlan::Node&, std::__1::list<DB::QueryPlan::Node, std::__1::allocator<DB::QueryPlan::Node>>&) build_docker/./src/Processors/QueryPlan/Optimizations/optimizeTree.cpp:201:38
    23 0x5565870bc489 in DB::QueryPlan::optimize(DB::QueryPlanOptimizationSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:485:5
    24 0x5565870b919a in DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:161:5
    25 0x55658206385a in DB::InterpreterSelectWithUnionQuery::execute() build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379:31
    26 0x556582c15c50 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:751:40
    27 0x556582c09681 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1173:30
    28 0x5565860e7bd7 in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:421:24
    29 0x556586126e9e in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2057:9
```

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-18 18:15:17 +02:00
Azat Khuzhin
1e2b0c8dc4 Fix possible UB in Set skipping index for functions with incorrect args 
Usually functions are validated before index analysis, however it is
not the case for parallel replicas, and it requires additional checks
before interpreting function arguments.

<details>

<summary>stack trace</summary>

```
==172==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55cf82aedd6c in DB::ASTFunction* typeid_cast<DB::ASTFunction*, DB::IAST>(DB::IAST*) build_docker/./src/Common/typeid_cast.h:42:73
    1 0x55cf82aedd6c in DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, false>::value(DB::IAST*) build_docker/./src/Common/TypePromotion.h:38:43
    2 0x55cf82aedd6c in std::__1::invoke_result<decltype(&CastHelper<DB::ASTFunction, false>::value), DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, std::is_reference_v<DB::ASTFunction>>, DB::IAST*>::type DB::TypePromotion<DB::IAST>::as<DB::ASTFunction>() build_docker/./src/>
    3 0x55cf82aedd6c in DB::MergeTreeIndexConditionSet::operatorFromAST(std::__1::shared_ptr<DB::IAST>&) build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:602:25
    4 0x55cf82ae5bc1 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:547:9
    5 0x55cf82ae5de6 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:552:13
    6 0x55cf82ae06db in DB::MergeTreeIndexConditionSet::MergeTreeIndexConditionSet() build_docker/.>
    ...
    12 0x55cf82aef09c in DB::MergeTreeIndexSet::createIndexCondition() const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:703:12
    13 0x55cf84951cd4 in DB::buildIndexes()
    14 0x55cf84955ed3 in DB::ReadFromMergeTree::selectRangesToReadImpl()
    15 0x55cf8494caef in DB::ReadFromMergeTree::selectRangesToRead()
    16 0x55cf82a409a9 in DB::MergeTreeDataSelectExecutor::estimateNumMarksToRead()
    17 0x55cf827f728d in DB::MergeTreeData::canUseParallelReplicasBasedOnPKAnalysis()
    18 0x55cf827f627e in DB::MergeTreeData::getQueryProcessingStage()
    19 0x55cf7f2f4969 in DB::InterpreterSelectQuery::getSampleBlockImpl() build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:937:31
    20 0x55cf7f2daa00 in DB::InterpreterSelectQuery::InterpreterSelectQuery()
    24 0x55cf7f520b98 in DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter()
    25 0x55cf7f51b6cd in DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery()
    27 0x55cf7f1d4ea9 in DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterFactory.cpp:162:16
    28 0x55cf8012e485 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1032:31
    29 0x55cf80121bc1 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1229:30
    30 0x55cf8389295f in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:424:24
    31 0x55cf838d7dfb in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2050:9
```

</details>

CI: https://s3.amazonaws.com/clickhouse-test-reports/53214/d99b10c340909ed4ee2e6edf0921e8a2f8561b0d/fuzzer_astfuzzermsan/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-18 18:02:46 +02:00
Austin Kothig
6b42975d33
Change BE-UUID to work the same as LE-UUID. Included high and low getters to provide cleaner code when accessing undertype. 2023-08-18 08:19:46 -07:00
Yakov Olkhovskiy
84cb358c14
Merge pull request #43210 from amosbird/storage_executable_stderr 
Add different reactions when external command stderr has data
 2023-08-18 11:17:46 -04:00
Alexander Tokmakov
a6684f7a37
Merge pull request #53484 from azat/rmt-queue-cleanup 
Remove outdated code in ReplicatedMergeTreeQueue::initialize()
 2023-08-18 18:11:33 +03:00
avogar
2d4167c612 FRemove unneded changes 2023-08-18 14:50:07 +00:00
avogar
aac5131859 Fix for no_more_keys 2023-08-18 13:21:16 +00:00
avogar
7fa72f5cdf Optimize group by constant keys 2023-08-18 12:47:26 +00:00
Alexander Gololobov
50061b9cb4
Merge pull request #53452 from ClickHouse/such_much_watch 
Deduplicate same watch callbacks registered multiple times
 2023-08-18 14:43:03 +02:00
Alexander Tokmakov
0bd378afd9
Merge pull request #53381 from bakwc/bakwc-patch-2 
Fixed system.data_skipping_indices for MaterializedMySQL
 2023-08-18 14:22:40 +03:00
Duc Canh Le
2d353f8c43 Merge branch 'master' into limit_concurent_background_optimize 2023-08-18 09:25:27 +00:00
Amos Bird
b6de70b042
StorageExecutable check_exit_code default to false 2023-08-18 15:38:47 +08:00
Amos Bird
c43bf153f5
Refactor 2023-08-18 15:38:46 +08:00
Amos Bird
dd0c71b32a
Add error_exit_reaction 2023-08-18 15:38:46 +08:00
Amos Bird
476f3cedc1
Various reactions when executable stderr has data 2023-08-18 15:38:45 +08:00
Alexey Milovidov
a882bf1bb8
Merge pull request #53495 from ClickHouse/i_hate_materialized_views 
Fix a race condition between RESTART REPLICAS and DROP DATABASE
 2023-08-18 06:40:04 +03:00
Igor Nikonov
06c98956b3 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-17 20:42:50 +00:00
Jianfei Hu
0c81546960 remove the comments. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-17 20:13:33 +00:00
pufit
a52249872e
Merge pull request #53313 from jorisgio/joris/max_threads_for_indexes 
add max_threads_for_indexes settings
 2023-08-17 13:59:53 -04:00
avogar
60b0b88d50 Clean up 2023-08-17 16:59:57 +00:00
avogar
4c32097df3 Use filter by file/path before reading in url/file/hdfs table functions, reduce code duplication 2023-08-17 16:54:43 +00:00
Dmitry Novik
1bd478dba2 Merge remote-tracking branch 'origin/master' into fix-prewhere-lost-column 2023-08-17 16:00:35 +00:00
Alexander Gololobov
54e2be4e32 Use WatchCallbackPtr 2023-08-17 17:32:04 +02:00
Dmitry Novik
01cb96d39a Leave only necessary conditions in output 2023-08-17 15:27:24 +00:00
Alexander Tokmakov
89c365803f
Merge pull request #53424 from ClickHouse/exception_message_patterns6 
Less exceptions with runtime format string
 2023-08-17 17:58:04 +03:00
Igor Nikonov
3e9ae1751e Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-17 14:45:37 +00:00
Filipp Ozinov
ce87451b66
Merge branch 'ClickHouse:master' into bakwc-patch-2 2023-08-17 18:37:00 +04:00
Anton Popov
0865c0794e Merge remote-tracking branch 'upstream/master' into HEAD 2023-08-17 14:08:59 +00:00
Alexander Tokmakov
72b52250ea
Merge pull request #53501 from azat/dist-flush_on_detach 
Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
 2023-08-17 16:56:04 +03:00
Azat Khuzhin
fcb66379d1 Remove outdated code in ReplicatedMergeTreeQueue::initialize() 
Initially added in #28817
But got cleaned up in #41981

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-17 13:45:26 +02:00
Robert Schulze
ec7daed52f
Merge pull request #50276 from ClibMouse/feature/mergetree-checksum-big-endian-support 
MergeTree & SipHash checksum big-endian support
 2023-08-17 13:27:18 +02:00
Robert Schulze
42644bfbf5
Rename file to MergeTreeIndexUSearch.h/cpp 2023-08-17 09:54:12 +00:00
Robert Schulze
f420006e09
Make the code consistent with Annoy index and fix build 2023-08-17 09:42:13 +00:00
Robert Schulze
456a02da1b
Merge remote-tracking branch 'rschu1ze/master' 2023-08-17 09:30:27 +00:00
Robert Schulze
1215a23de6
Merge pull request #53422 from rschu1ze/consistency-fixes-for-caches 
Refactorings for configuration of in-memory caches
 2023-08-17 11:08:32 +02:00
Mingliang Pan
2db0b4f52a
Update StorageEmbeddedRocksDB.cpp 
add drop function to close file before drop table
 2023-08-17 15:14:14 +08:00
Mingliang Pan
349f3b9c02
Update StorageEmbeddedRocksDB.h 
add drop method to close file before drop table
 2023-08-17 15:12:42 +08:00
Azat Khuzhin
17ca2661a1 Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown 
Sometimes you can have tons of data there, i.e. few TiBs, and sending
them on server shutdown does not looks sane (maybe there is a bug and
you need to update/restart to fix flushing).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-17 08:58:06 +02:00
Igor Nikonov
49f7b4acd6 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-16 23:31:49 +00:00
Igor Nikonov
21ef1f1d1c Remove unnecessary Scalars usage for cluster_for_parallel_replicas 2023-08-16 23:24:01 +00:00
Alexander Tokmakov
d7272410bd the worst fix for a deadlock ever 2023-08-17 00:42:51 +02:00
Robert Schulze
8a58e0df3e
Merge remote-tracking branch 'rschu1ze/master' into consistency-fixes-for-caches 2023-08-16 21:26:04 +00:00
Robert Schulze
353854520c
Merge remote-tracking branch 'rschu1ze/master' into factorize-constants 2023-08-16 21:24:21 +00:00
Yakov Olkhovskiy
154b3a1ae4 preserve previous steps output 2023-08-16 21:17:34 +00:00
Alexander Tokmakov
69bc4f5139 Merge branch 'master' into exception_message_patterns6 2023-08-16 22:58:24 +02:00
Jianfei Hu
fcad273724 remove the comments. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-16 17:49:28 +00:00
Jianfei Hu
1e600351d9 try again the if value.empty() in the beginning. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-16 17:49:28 +00:00
Jianfei Hu
49502ac0ce revert unrelated style change. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-16 17:49:28 +00:00
Jianfei Hu
02b7bd94eb suspect some root cause of value[0]. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-16 17:49:28 +00:00
Jianfei Hu
7227cc6d2b try parts[0] instead of front(). 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-16 17:49:28 +00:00
Jianfei Hu
afaaef6875 wip checking whether system.merge cause sigsev. 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-16 17:49:28 +00:00
Alexey Milovidov
5e3a69b4c4 Update MergeList.cpp 2023-08-16 17:49:28 +00:00
Alexey Milovidov
bc77aab2e8 Update MergeList.h 2023-08-16 17:49:28 +00:00
Alexey Milovidov
7376b7ec8c Update MergeList.h 2023-08-16 17:49:28 +00:00
Alexey Milovidov
664bcfc8b3 Update MergeList.cpp 2023-08-16 17:49:28 +00:00
Alexey Milovidov
ecd1fb56b4 Update MergeList.cpp 2023-08-16 17:49:28 +00:00
Jianfei Hu
bd4df60df6 fix merges 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-16 17:49:28 +00:00
Jianfei Hu
59a81b82bc wip the merge partition 
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
 2023-08-16 17:49:28 +00:00
Robert Schulze
5a87a09c84
Tweak exception handling 2023-08-16 17:16:52 +00:00
Andrey Zvonov
eb9c837d20
Merge branch 'ClickHouse:master' into zvonand-globs-small-fix 2023-08-16 19:22:00 +03:00
Davit Vardanyan
a0ec274766 Refactor: Naming 2023-08-16 17:30:56 +04:00
Davit Vardanyan
620733dd46 Fix: Exception handling 2023-08-16 17:30:29 +04:00
Robert Schulze
ea69eb095c
Merge remote-tracking branch 'ClickHouse/master' into usearch 2023-08-16 08:07:57 +00:00
Alexander Tokmakov
b32bdd1b0a
Merge pull request #53164 from ClickHouse/fixes_for_detached_parts 
Fixes for detach/attach partition and broken detached parts cleanup
 2023-08-16 00:09:13 +03:00
Robert Schulze
56a6d20137
Merge remote-tracking branch 'rschu1ze/master' into factorize-constants 2023-08-15 18:21:25 +00:00
Igor Nikonov
49134711dc Polishing 2023-08-15 17:37:01 +00:00
Robert Schulze
f558048bea
Merge remote-tracking branch 'rschu1ze/master' into consistency-fixes-for-caches 2023-08-15 17:12:52 +00:00
Igor Nikonov
9825f8c76e Remove debug logging 2023-08-15 17:03:32 +00:00
kssenii
2d74472ffb Merge remote-tracking branch 'upstream/master' into system-sync-cache 2023-08-15 20:31:26 +04:00
zvonand
0efaa5ab45 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-globs-small-fix 2023-08-15 21:20:52 +05:00