thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-19 14:11:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
107,025 Commits 9,138 Branches 1,778 Tags 2 GiB
4c89327c9f
Commit Graph

36740 Commits

Author SHA1 Message Date
Saulius Valatka
8fa9a99ba1 implement pending request queue to prevent CQ overflows 2023-01-28 21:54:44 +02:00
Saulius Valatka
ac2c921bdf add initial io_uring support 2023-01-28 21:54:44 +02:00
Saulius Valatka
fb45ed038f add liburing to build 2023-01-28 21:54:44 +02:00
Pradeep Chhetri
8156a6761f Set compression method and level for backup writer 
Signed-off-by: Pradeep Chhetri <pradeepchhetri4444@gmail.com>
 2023-01-28 21:49:59 +08:00
freedomDR
5836a3f153 refactor coder 2023-01-28 04:27:11 +00:00
freedomDR
379850b6c4 add system sync file cache 2023-01-28 04:27:10 +00:00
Alexey Milovidov
bc2f454522
Merge branch 'master' into block-non-float-gorilla-v2 2023-01-28 03:30:12 +03:00
Alexey Milovidov
9b1b247f9b
Merge pull request #45585 from kitaisreal/low-cardinality-insert-fix 
LowCardinality insert fix
 2023-01-28 03:26:40 +03:00
Alexey Milovidov
657d9f69e7
Merge pull request #45686 from amosbird/issue_45590 
Fix key description when encountering duplicate primary keys
 2023-01-28 02:22:42 +03:00
Joanna Hulboj
df030a56f0 FIXUP: Remove use of legacy exceptions api 2023-01-27 21:27:09 +00:00
Azat Khuzhin
ffd5617614 Improve logging for broken disks 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-27 21:11:10 +01:00
Azat Khuzhin
6b42b66257 Simply filesystem helpers to check is-readable/writable/executable 
Use one system call - faccessat(), over multiple system calls.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-27 21:11:10 +01:00
Kruglov Pavel
96700abbe1
Merge pull request #45678 from azat/formats/json-parse-tupels 
Add ability to ignore unknown keys in JSON object for named tuples
 2023-01-27 21:11:05 +01:00
Joanna Hulboj
9a559b5475 FIXUP: More comments about shuffle 2023-01-27 20:07:56 +00:00
Joanna Hulboj
8791a44e01 FIXUP: Added info and test for materialized array 2023-01-27 20:07:56 +00:00
Joanna Hulboj
3b472eb2dd FIXUP: Darwin compilation issue 2023-01-27 20:07:56 +00:00
Joanna Hulboj
e1d281206f Clamp the limit for arrayPartialShuffle 2023-01-27 20:07:56 +00:00
Joanna Hulboj
b0ba8c02be FIXUP: Darwin compilation issue 2023-01-27 20:07:55 +00:00
Joanna Hulboj
d378e453c1 FIXUP: fix in arrayPartialShuffle 2023-01-27 20:07:55 +00:00
Joanna Hulboj
a8b78abc54 Added arrayPartialShuffle function 2023-01-27 20:07:55 +00:00
Joanna Hulboj
31eb936457 Added Fisher-Yates shuffle and partial-shuffle 2023-01-27 20:07:55 +00:00
Joanna Hulboj
09789b027f FIXUP: PR comments 2023-01-27 20:07:55 +00:00
Joanna Hulboj
a65b2cf861 FIXUP: Simplify logic by using permute function 2023-01-27 20:07:55 +00:00
Joanna Hulboj
c09a4dd132 FIXUP: Docs into code 2023-01-27 20:07:55 +00:00
Joanna Hulboj
8d8d1bb887 FIXUP: make clang-tidy happier 2023-01-27 20:07:55 +00:00
Joanna Hulboj
01624e2f23 FIXUP: style 2023-01-27 20:07:55 +00:00
Joanna Hulboj
2355780737 Minor formatting 2023-01-27 20:07:55 +00:00
Joanna Hulboj
4adb6288eb Add arrayShuffle function 2023-01-27 20:07:55 +00:00
Igor Nikonov
300f78df96
Merge pull request #45567 from ClickHouse/enable-remove-redundant-sorting 
Enable query_plan_remove_redundant_sorting optimization by default
 2023-01-27 19:14:36 +01:00
Vitaly Baranov
af74c008c4 Use one request to implement S3ObjectStorage::getObjectMetadata instead of two ones. 2023-01-27 18:42:37 +01:00
kssenii
9c4a97bbd1 Merge remote-tracking branch 'upstream/master' into storage-rabbitmq-fix-flush 2023-01-27 17:17:09 +01:00
Anton Popov
d8fe2bcbaa
Merge branch 'master' into randomize-mt-settings 2023-01-27 16:03:00 +01:00
Vitaly Baranov
d02b255b55 Add new setting "allow_head_object_request" to S3::RequestSettings. 2023-01-27 15:10:34 +01:00
Vitaly Baranov
f9282cfcac Make version_id an optional parameter again. 2023-01-27 15:10:19 +01:00
Vitaly Baranov
aea9ccdb60 Pass request settings to S3::getObjectInfo(). 2023-01-27 15:10:09 +01:00
Vitaly Baranov
a8304525ed Move getObjectInfo() to a separate header. 2023-01-27 15:09:38 +01:00
Nikolai Kochetov
313947e0b0
Merge branch 'master' into issue_45590 2023-01-27 15:08:28 +01:00
Anton Popov
f10e82355e
Merge pull request #45630 from CurtizJ/fix-performance-columns-description 
Fix performance of short queries with `Array` columns
 2023-01-27 14:59:55 +01:00
Alexander Tokmakov
ec0738ef31
Merge pull request #45692 from ClickHouse/tavplubix-patch-3 
Fix build
 2023-01-27 16:39:56 +03:00
Robert Schulze
83593810ff
Make check work with Nullable / Array / Tuple 2023-01-27 13:02:54 +00:00
Alexander Tokmakov
f375b1246f
Merge pull request #45363 from ClickHouse/fix_hang_during_drop_in_zero_copy_replication 
Fix no shared id during drop for the fourth time
 2023-01-27 15:59:27 +03:00
Nikolai Kochetov
f6ad6296ba Restore. 2023-01-27 12:45:41 +00:00
Alexander Tokmakov
1494262a38
Update DataPartStorageOnDiskFull.cpp 2023-01-27 15:42:41 +03:00
Nikolai Kochetov
b716e2d754 A bit cleaner fix. 2023-01-27 12:37:23 +00:00
alesapin
716d25c33f
Merge pull request #45619 from CurtizJ/refactoring-merge-tree-parts 
Refactoring of code near merge tree parts
 2023-01-27 13:36:33 +01:00
Alexander Tokmakov
c366806c3e
Merge pull request #45527 from ClickHouse/exception_message_patterns4 
Better formatting for exception messages 2
 2023-01-27 15:31:52 +03:00
Salvatore Mesoraca
59c17a7719
functions: hashing: add sipHash{64,128}Keyed 2023-01-27 13:00:54 +01:00
Salvatore Mesoraca
31e4d042f8
function: hashing: add generic support for keyed algos 2023-01-27 13:00:53 +01:00
Salvatore Mesoraca
2e72e27206
common: siphash: add support for custom keys 2023-01-27 13:00:53 +01:00
Robert Schulze
e30c38cab3
Fix up some leftovers 2023-01-27 10:43:43 +00:00
Maksim Kita
8e1e136d0c
Merge pull request #42976 from kitaisreal/analyzer-settings-push-down 
Analyzer SETTINGS push down
 2023-01-27 12:52:23 +03:00
Azat Khuzhin
1a8437f2c9 Add ability to ignore unknown keys in JSON object for named tuples 
This can be useful in case your input JSON is complex, while you need
only few fields in it.

This behaviour is controlled by the
input_format_json_ignore_unknown_keys_in_named_tuple setting name, that
is turned OFF by default.

This will, almost, allow to parse gharchive dataset without jq. "almost"
because of two things:
- Tuple cannot be Nullable, so such keys with Tuple type in ClickHouse
  cannot be `null` in JSON
- You cannot use dot.dot notation to extract columns for file() engine,
  only tupleElement()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-27 10:01:08 +01:00
Amos Bird
716516f84a
Fix test 2023-01-27 16:58:41 +08:00
Amos Bird
fd60954081
Fix key description on duplicate primary keys 
This can happen in projections. See
https://github.com/ClickHouse/ClickHouse/issues/45590 for details.
 2023-01-27 13:01:36 +08:00
Alexander Tokmakov
7af3c6d162 fix 2023-01-27 01:22:29 +01:00
Alexey Milovidov
5b257ab806
Merge pull request #45233 from ClickHouse/improve_week_day 
Revert "Revert "Improve week day""
 2023-01-27 02:44:17 +03:00
Alexey Milovidov
f401449bd9
Merge pull request #45661 from ClickHouse/generate-random-lowcardinality 
Allow LowCardinality in GenerateRandom
 2023-01-27 02:14:38 +03:00
Robert Schulze
032cdb986e
Merge pull request #45615 from ClickHouse/fix-gorilla 
Fix codec pipeline Delta (or DoubleDelta) + Gorilla
 2023-01-27 00:04:39 +01:00
Robert Schulze
7d97fab2dc
Untangle query result cache settings 
This PR modularizes the configuration options of the query result cache
(QRC).

1. Add a feature toggle "allow_experimental_query_result_cache = false"
   which controls whether the experimental QRC can be used at all. This
   configuration option will be removed once the QRC becomes
   non-experimental.

2. Modularize the existing two configuration settings
   "enable_experimental_query_result_cache" and
   "enable_experimental_query_result_cache_passive_usage" into three
   orthogonal settings:

   - "use_query_result_cache = false" controls whether a query (or all
     queries of the session) utilize the query result cache

   - settings "use_query_result_cache_passive_usage = true" and
     "use_query_result_cache_active_usage = true" control the way a
     query (or all queries of the session) utilize the query result
     cache: "passive" use is reading, "active" use is writing.
 2023-01-26 21:17:48 +00:00
Igor Nikonov
211edc054e Fix check for sum*(Float) 2023-01-26 19:26:53 +00:00
Alexander Tokmakov
68a60ad4f6 fix 2023-01-26 20:20:56 +01:00
Alexander Tokmakov
061026c307 fix 2023-01-26 19:01:29 +01:00
Vitaly Baranov
5b469eb91f
Merge pull request #45603 from vitlibar/fix-wiping-sensitive-info 
Fix wiping sensitive info in logs
 2023-01-26 15:44:24 +01:00
Alexey Milovidov
d942a4ddef Allow LowCardinality in GenerateRandom 2023-01-26 15:35:20 +01:00
Alexander Tokmakov
d7c697ee38 fix 2023-01-26 15:24:39 +01:00
Alexander Tokmakov
067b1f5f13 Merge branch 'master' into exception_message_patterns4 2023-01-26 15:20:58 +01:00
Igor Nikonov
ba92705ea1 Fix duplicated setting in changes history 2023-01-26 14:20:44 +00:00
Alexander Tokmakov
f676e9eaff
Merge pull request #45613 from azat/replicated-improve-log 
Improve "at least part X is missing" error message
 2023-01-26 16:48:18 +03:00
Anton Popov
92fa39798b Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-26 13:29:23 +00:00
Anton Popov
b58b73b0e7
Merge pull request #45529 from CurtizJ/fix-storage-s3-race 
Try to fix test `test_storage_s3/test.py::test_wrong_s3_syntax` (race in `StorageS3`)
 2023-01-26 14:21:32 +01:00
Igor Nikonov
105d72e299 Fix: sum*() with Nullable(Float) 2023-01-26 13:17:06 +00:00
Anton Popov
87af86e9a3 fix projections 2023-01-26 13:11:31 +00:00
Alexander Tokmakov
14db798191 fix 2023-01-26 13:56:16 +01:00
Maksim Kita
a19346b59a OptimizeGroupByFunctionKeysPass style fixes 2023-01-26 12:48:12 +01:00
Maksim Kita
32129ca7dc Fixed build 2023-01-26 12:47:50 +01:00
Maksim Kita
6d3a5241e8 QueryTreePassManager refactor passes to use QueryNode, UnionNode scope context 2023-01-26 12:47:50 +01:00
Maksim Kita
8e6593f367 Fixed code review issues 2023-01-26 12:46:58 +01:00
Maksim Kita
b538fae749 Analyzer SETTINGS push down 2023-01-26 12:46:58 +01:00
Maksim Kita
7a8a8dcd2f
Merge pull request #45415 from ClickHouse/redundant-group-by-keys 
Remove redundant group by keys with constants
 2023-01-26 14:45:25 +03:00
Anton Popov
b52cf17fa6
Merge pull request #45612 from ClickHouse/CurtizJ-patch-7 
Remove useless code
 2023-01-26 12:38:52 +01:00
Antonio Andelic
ebfe7f9991 Fix style 2023-01-26 11:18:07 +00:00
Maksim Kita
ca1ec2f1d5
Merge pull request #45518 from ClickHouse/column-source-check 
Make ColumnNode::isEqualImpl more strict
 2023-01-26 14:16:57 +03:00
Robert Schulze
227b8676cd
Tiny improvements around the Gorilla/Delta codecs 2023-01-26 11:11:24 +00:00
Robert Schulze
9c48ac79c8
Fix issue #45195 2023-01-26 10:44:26 +00:00
alesapin
84af5eb58b Fxi 2023-01-26 11:42:53 +01:00
alesapin
d5036f2ebc Merge branch 'fix_hang_during_drop_in_zero_copy_replication' of github.com:ClickHouse/ClickHouse into fix_hang_during_drop_in_zero_copy_replication 2023-01-26 11:42:30 +01:00
alesapin
1f61abce46 One useful logging line 2023-01-26 11:41:43 +01:00
Antonio Andelic
45cc45c149 Merge branch 'master' into keeper-use-fallocate 2023-01-26 10:10:20 +00:00
Alexander Tokmakov
a584ad0eb1 forbid runtime strings 2023-01-26 10:52:47 +01:00
Kseniia Sumarokova
297516a084
Merge pull request #45614 from kssenii/rename-some-setting 
Rename setting `max_query_cache_size`
 2023-01-26 10:46:03 +01:00
Antonio Andelic
1186718658 Add overallocate size setting 2023-01-26 09:44:37 +00:00
Antonio Andelic
6fe9e9a67f
Merge pull request #45617 from ClickHouse/fix-bfngram-index 
Correctly check types when using N-gram bloom filter index
 2023-01-26 09:05:56 +01:00
robot-clickhouse-ci-1
73c62ae390
Merge pull request #45610 from azat/is_detached-fix 
Extend assertion in buildPushingToViewsChain() to respect is_detached
 2023-01-26 01:15:55 +01:00
Alexander Tokmakov
9b670946db Merge branch 'master' into exception_message_patterns5 2023-01-26 00:41:32 +01:00
Mikhail f. Shiryaev
120c31d57f
Merge pull request #45634 from ClickHouse/23.2-prepare 
Update version after release
 2023-01-26 00:10:53 +01:00
Mikhail f. Shiryaev
53d89844be
Update version to 23.2.1.1 2023-01-25 23:57:29 +01:00
Igor Nikonov
e1302e1a32 Fix: sum*() with Floats 
sum(Float) depends on order, do not remove sorting in such case
 2023-01-25 22:39:28 +00:00
Anton Popov
79ea6a1c9c fix performance of short queries 2023-01-25 22:20:01 +00:00
Alexander Tokmakov
92c62e2de2
Update src/Storages/StorageReplicatedMergeTree.cpp 2023-01-26 00:28:46 +03:00
Han Fei
0221b50028
Merge pull request #45601 from hanfei1991/hanfei/async-cache-conflict 
fix race of async-insert-blockids-cache
 2023-01-25 22:28:42 +01:00
Alexander Tokmakov
3744fa2c63 format more messages 2023-01-25 21:16:42 +01:00
Anton Popov
0f46c1ecc6 fix build 2023-01-25 20:13:54 +00:00
sichenzhao
243ac52259
Added two metrics about memory usage in cgroup to asynchronous metrics (#45301) 2023-01-25 20:32:17 +01:00
alesapin
d0428a9b42 Fxi 2023-01-25 20:30:19 +01:00
Kruglov Pavel
cd3d3fdd68
Merge pull request #45483 from Avogar/fix-hdfs-cluster 
Fix schema inference from insertion table in hdfsCluster
 2023-01-25 20:24:51 +01:00
Azat Khuzhin
00c341850f Improve "at least part X is missing" error message 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-25 19:53:45 +01:00
Anton Popov
a455083b87 Merge remote-tracking branch 'upstream/master' into refactoring-merge-tree-parts 2023-01-25 18:01:12 +00:00
Kruglov Pavel
83a212a128
Merge branch 'master' into more-fsync-fixes 2023-01-25 18:40:41 +01:00
Anton Popov
8e3698c91f refactoring of code near merge tree parts 2023-01-25 17:34:09 +00:00
Antonio Andelic
dfaed11c83 Merge branch 'master' into keeper-use-fallocate 2023-01-25 16:45:34 +00:00
Antonio Andelic
7fa906487c Fix ngram index with null 2023-01-25 16:40:04 +00:00
Igor Nikonov
2b5206c223 Merge remote-tracking branch 'origin/master' into enable-remove-redundant-sorting 2023-01-25 16:32:14 +00:00
Igor Nikonov
fc23ff6ad2 Add entity into setting changes history for compatibility mode 2023-01-25 16:28:54 +00:00
Antonio Andelic
7bfdfa4bf7 Preallocate more space 2023-01-25 16:26:14 +00:00
Anton Popov
d9aabb48c3 remove useless code 2023-01-25 16:11:15 +00:00
kssenii
29de87e145 Rename setting 2023-01-25 16:36:36 +01:00
Nikolay Degterinsky
6b2f3de293
Merge pull request #45512 from evillique/fix-msan-build 
Fix MSan build once again (too heavy translation units)
 2023-01-25 16:11:21 +01:00
Alexander Tokmakov
ae795d87b2 fix 2023-01-25 16:06:40 +01:00
Azat Khuzhin
71286e779e Extend assertion in buildPushingToViewsChain() to respect is_detached 
Follow-up for: #45566 (@tavplubix)
Refs: #43161 (@AlfVII)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-25 15:42:06 +01:00
Vitaly Baranov
5b24094405 Remove unnecessary ifs. 2023-01-25 15:08:58 +01:00
Vitaly Baranov
9014023625 Fix wiping sensitive info in INSERT INTO FUNCTION. 2023-01-25 14:50:49 +01:00
alesapin
55af44ca2f Better fix 2023-01-25 14:40:39 +01:00
Alexander Tokmakov
b9e5c586e6
Merge pull request #45566 from ClickHouse/resubmit_45493 
Resubmit "Fix possible in-use table after DETACH"
 2023-01-25 16:39:55 +03:00
Alexander Tokmakov
6eb557b2ba Merge branch 'master' into exception_message_patterns4 2023-01-25 13:49:17 +01:00
Robert Schulze
59528cfca0
Merge pull request #45460 from ClickHouse/inv-index-cleanup 
Cleanup of inverted index
 2023-01-25 13:23:38 +01:00
Vitaly Baranov
32efe92199
Merge pull request #45487 from vitlibar/use-new-copy-s3-functions-in-s3-obj-storage 
Use new copy s3 functions in S3ObjectStorage
 2023-01-25 13:22:04 +01:00
alesapin
96d8546b1c Merge branch 'master' into fix_hang_during_drop_in_zero_copy_replication 2023-01-25 13:07:14 +01:00
Dmitry Novik
f71c213aff
Merge branch 'master' into column-source-check 2023-01-25 12:53:36 +01:00
Han Fei
8c4bbcd0fc fix race of async-insert-blockids-cache 2023-01-25 12:41:01 +01:00
Antonio Andelic
7421d8f872 Merge branch 'master' into keeper-use-fallocate 2023-01-25 11:36:47 +00:00
Kseniia Sumarokova
5da5327729
Merge pull request #45062 from kssenii/fix-sqlite-data-race-in-init-db 
Fix data race in openSQLiteDB
 2023-01-25 12:32:34 +01:00
Vladimir C
cfefaebe8f
Merge pull request #45558 from ClickHouse/vdimir/remove-ast-fuse 2023-01-25 11:25:36 +01:00
Sergei Trifonov
0d1ea05ff6
Merge pull request #45007 from ClickHouse/cancellable-mutex-integration 
Fast shared mutex integration
 2023-01-25 11:15:46 +01:00
SmitaRKulkarni
b0bbe7518a
Merge pull request #45497 from ClickHouse/45486_Fix_flaky_test_for_disallowing_concurrent_backups_restores 
Updated backup/restore status when concurrent backups & restores are not allowed
 2023-01-25 08:37:24 +01:00
Bharat Nallan
2ef8fcb318
Merge branch 'master' into ncb/odbc-connection-pool-fixes 2023-01-24 21:27:20 -08:00
Nikolay Degterinsky
97aef55a7f Fix typo 2023-01-24 23:00:02 +00:00
Nikolay Degterinsky
d8d85d9bbd Merge remote-tracking branch 'upstream/master' into fix-msan-build 2023-01-24 22:57:47 +00:00
Kseniia Sumarokova
2b8b1ad5d4
Additional check in MergeTreeReadPool (#45515) 
* Check ranges

* Check equality just in case

* Check under ndebug
 2023-01-24 23:56:26 +01:00
Nikolay Degterinsky
fb6838b043 Review suggestions 2023-01-24 22:54:01 +00:00
Alexander Tokmakov
dceeae995e Merge branch 'master' into exception_message_patterns4 2023-01-24 23:22:38 +01:00
Alexander Tokmakov
d1baa7300c reformat ParsingException 2023-01-24 23:21:29 +01:00
Robert Schulze
7f010d2b61
Merge pull request #45522 from HarryLeeIBM/hlee-s390x-transform 
Fix endian issue in transform function for s390x
 2023-01-24 22:12:25 +01:00
Maksim Kita
32b4bbf26c LowCardinality insert fix 2023-01-24 21:16:17 +01:00
Vitaly Baranov
3fc366e747 Make retries in copyS3File() more correct. 2023-01-24 18:05:40 +01:00
Alexander Tokmakov
6621834e37 fix 2023-01-24 17:10:54 +01:00
Alexander Tokmakov
dd57215934 Merge branch 'master' into exception_message_patterns4 2023-01-24 17:03:12 +01:00
avogar
7eeb2a0bc7 Change comment 2023-01-24 15:46:32 +00:00
avogar
159f49266e Don't infer Dates from 8 digit numbers 2023-01-24 15:45:27 +00:00
Kruglov Pavel
689fbea759
Merge pull request #45478 from Avogar/fix-arrow-abort 
Fix possible aborts in arrow lib
 2023-01-24 16:37:44 +01:00
Smita Kulkarni
6be7d1c24a Addressed review comments and renamed function to hasConcurrentBackups/Restores - Updated backup/restore status when concurrent backups & restores are not allowed 2023-01-24 16:20:12 +01:00
avogar
07ad750366 Fix tests 2023-01-24 15:08:19 +00:00
vdimir
0e09d0a00c
Remove FuseSumCountAggregatesVisitor 2023-01-24 14:58:26 +00:00
Vitaly Baranov
7d72b28bfe Use ProfileEvents::S3CopyObject again. 2023-01-24 15:54:15 +01:00
Kseniia Sumarokova
dd18dd2953
Merge pull request #45341 from ClibMouse/feature/non-throwing-hastoken 
Introduce non-throwing variants of hasToken
 2023-01-24 15:37:36 +01:00
serxa
51da43d6cf fix try_shared_lock() in SharedMutex and CancelableSharedMutex 2023-01-24 14:36:07 +00:00
Alexander Tokmakov
31e16c4b4d fix 2023-01-24 15:29:19 +01:00
Kruglov Pavel
4bd3f0e5ef
Merge pull request #44953 from Avogar/tsv-csv-detect-header 
Detect header in CSV/TSV/CustomSeparated files automatically
 2023-01-24 15:13:52 +01:00
Alexander Tokmakov
6ecae8388e Merge branch 'master' into exception_message_patterns4 2023-01-24 14:42:36 +01:00
Igor Nikonov
41b94b4954 Enable query_plan_remove_redundant_sorting optimization by default 2023-01-24 13:38:21 +00:00
Alexander Tokmakov
cd8c3bbcc5 fix 2023-01-24 14:36:30 +01:00
Igor Nikonov
683b1b7dbb
Merge pull request #45420 from ClickHouse/revert-45414-revert-43905-igor/remove_redundant_order_by 
Revert "Revert "Remove redundant sorting""
 2023-01-24 14:31:28 +01:00
Alexander Tokmakov
42a976fe3d Revert "Revert "Merge pull request #45493 from azat/fix-detach"" 
This reverts commit 9dc4f02d15.
 2023-01-24 14:15:51 +01:00
Azat Khuzhin
54ef885a61 Fsync all small files at once after mutation 
Everything else is handled in MergedBlockOutputStream::finalizePart()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Azat Khuzhin
4b775ccdd5 Do fsync all files at once for fetched parts to decrease latency 
For filessystems like ext4, fsync of one file will handle all operations
before, so this can be pretty time consuming.

And in case of you write multiple files in a loop, and at the end of
each iteration sync each file, then during writing of this file there
can be other operations in journal, and hence more work for fsync.

Let's call fsync for all files at once instead, like
MergedBlockOutputStream does.

Hope that keeping all file buffers till the end, will not cause troubles
(buffering and so forth).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Azat Khuzhin
b370e3faa5 Move fsync inside transaction callback in DataPartStorageOnDisk::rename() 
Otherwise, it is useless.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Vladimir C
5c845d1ab9
Merge branch 'master' into ch_canh_fix_sparkbar 2023-01-24 13:51:34 +01:00
SmitaRKulkarni
d387835774
Merge branch 'master' into 45486_Fix_flaky_test_for_disallowing_concurrent_backups_restores 2023-01-24 13:49:32 +01:00
Anton Popov
5c0307bc6a fix race in StorageURL and StorageHDFS 2023-01-24 12:34:43 +00:00
Vladimir C
f818e3070d
Merge pull request #45391 from ClickHouse/vdimir/join_prewhere_bug_44062 2023-01-24 13:23:31 +01:00
Alexander Tokmakov
c25e5ff8de
Merge pull request #45545 from ClickHouse/revert_45439 
Revert "Merge pull request #45493 from azat/fix-detach"
 2023-01-24 15:21:33 +03:00
Nikolai Kochetov
e4c0608a01
Merge pull request #45150 from ClickHouse/fix-ip-function-mod 
Binary arithmetic for IPv4
 2023-01-24 13:11:02 +01:00
Vladimir C
0403f801a8
Merge pull request #44238 from ClickHouse/vdimir/grace_join_mem 2023-01-24 13:02:11 +01:00
Maksim Kita
2f1092e6d2
Merge pull request #45425 from ClickHouse/fix-ipv6-parser 
Fix ipv6 parser
 2023-01-24 14:59:47 +03:00
Igor Nikonov
7108189b45 Merge remote-tracking branch 'origin/master' into revert-45414-revert-43905-igor/remove_redundant_order_by 2023-01-24 11:46:24 +00:00
Igor Nikonov
ba5fff2a8a Disable the optimization to avoid sort.xml perf test fail in other PRs 2023-01-24 11:42:56 +00:00
Alexander Tokmakov
9dc4f02d15 Revert "Merge pull request #45493 from azat/fix-detach" 
This reverts commit a182a6b851, reversing
changes made to c47a29a089.
 2023-01-24 12:20:37 +01:00
Robert Schulze
574cab5d7e
Remove transitory parameter 2023-01-24 11:05:29 +00:00
Robert Schulze
aaf7653108
Merge remote-tracking branch 'origin/master' into block-non-float-gorilla-v2 2023-01-24 10:14:10 +00:00
Robert Schulze
9ff2bfcbf5
Merge remote-tracking branch 'origin/master' into inv-index-cleanup 
src/Interpreters/GinFilter.cpp
	src/Interpreters/InterpreterCreateQuery.cpp
	src/Storages/MergeTree/MergeTreeData.cpp
	src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp
	src/Storages/MergeTree/MergeTreeIndexInverted.cpp
 2023-01-24 10:09:42 +00:00
Alexey Milovidov
dda6a69f9a
Merge branch 'master' into tavplubix-patch-1 2023-01-24 13:04:25 +03:00
Alexey Milovidov
54bc501d39
Update StorageReplicatedMergeTree.cpp 2023-01-24 13:03:28 +03:00
Alexey Milovidov
851f6bf910
Merge pull request #45448 from azat/fix-query-hang 
Fix possible (likely distributed) query hung
 2023-01-24 13:00:37 +03:00
Smita Kulkarni
642f9ca549 Merge branch 'master' into 45486_Fix_flaky_test_for_disallowing_concurrent_backups_restores 2023-01-24 09:37:28 +01:00
Smita Kulkarni
9ae5ac2388 Moved concurrency checks inside functions - Updated backup/restore status when concurrent backups & restores are not allowed 2023-01-24 09:31:51 +01:00
avogar
3c5a81f9f9 Add unit test for recursive checkpoints 2023-01-24 01:20:13 +00:00
Anton Popov
8a1ea62aec fix race on cancelation of query in StorageS3 2023-01-24 01:12:01 +00:00
Alexander Tokmakov
9d13ca4a16 fix 2023-01-24 01:49:16 +01:00
ltrk2
ee5818f27b Fix the build with ENABLE_VECTORSCAN disabled 2023-01-23 16:43:03 -08:00
Alexander Tokmakov
6ddd76c5a0 fix 2023-01-24 01:30:26 +01:00
Alexander Tokmakov
c6910f39b9 fix 2023-01-24 01:11:58 +01:00
Alexander Tokmakov
414693feb2 fixes 2023-01-24 00:46:03 +01:00
Kruglov Pavel
cd1cd904a7
Merge branch 'master' into tsv-csv-detect-header 2023-01-23 23:49:56 +01:00
ltrk2
9710b8643c Refine the solution 2023-01-23 14:27:48 -08:00
Alexander Tokmakov
bb4c8e169f check number of parameters in format string 2023-01-23 23:16:16 +01:00
Anton Popov
b987ed5037
Merge pull request #45499 from CurtizJ/add-fields-system-formats 
Add fields to table `system.formats`
 2023-01-23 23:08:35 +01:00
Nikita Taranov
95d17d59fb
impl (#45289) 2023-01-23 22:55:52 +01:00
Alexander Tokmakov
3f6594f4c6 forbid old ctor of Exception 2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Alexey Milovidov
62a8de34cc
Merge pull request #44811 from azat/build/glibc2.36-fix 
Fix ASan builds for glibc 2.36+
 2023-01-23 23:57:20 +03:00
Kruglov Pavel
478a552a0a
Merge branch 'master' into tsv-csv-detect-header 2023-01-23 21:47:17 +01:00
HarryLeeIBM
61f55ebb8a Fix endian issue in transform function for s390x 2023-01-23 11:54:35 -08:00
Robert Schulze
7adec3d8a4
Update src/Storages/MergeTree/MergeTreeIndexInverted.cpp 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-01-23 20:53:25 +01:00
Robert Schulze
d8af5bc2ef
Update src/Storages/MergeTree/MergeTreeIndexInverted.cpp 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-01-23 20:53:17 +01:00
Robert Schulze
f2f520485f
Update src/Interpreters/GinFilter.h 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-01-23 20:49:25 +01:00
Robert Schulze
0ab7ae6c17
Merge pull request #43797 from ClickHouse/query-result-cache 
Query result cache [experimental]
 2023-01-23 19:54:35 +01:00
Vladimir C
121adf309d
Merge branch 'master' into vdimir/grace_join_mem 2023-01-23 19:39:03 +01:00
vdimir
5be2b31e67
Review fixes 2023-01-23 18:09:26 +00:00
Maksim Kita
7b48c75e82
Merge pull request #45485 from kitaisreal/remove-unnecessary-get-total-row-count-function-calls 
Remove unnecessary getTotalRowCount function calls
 2023-01-23 21:02:51 +03:00
Dmitry Novik
796b362e2d Make ColumnNode::isEqualImpl more strict 2023-01-23 17:33:57 +00:00
avogar
73538117eb Fix and add a test 2023-01-23 16:24:00 +00:00
AVMusorin
82f194fbc6
added mutex for increaseProcessedSize 2023-01-23 17:15:50 +01:00
Kruglov Pavel
af2c1bac6a
Fix typo 2023-01-23 17:13:16 +01:00
Anton Popov
08b2e524c7 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-23 15:34:53 +00:00
Kseniia Sumarokova
8ec39ab6f3
Merge pull request #45442 from kssenii/minor-changes 
Minor improvements around reading from remote
 2023-01-23 15:46:30 +01:00
Nikolay Degterinsky
f9960361db Fix MSan build 2023-01-23 14:38:07 +00:00
Alexander Tokmakov
a182a6b851
Merge pull request #45493 from azat/fix-detach 
Fix possible in-use table after DETACH
 2023-01-23 17:36:45 +03:00
Kruglov Pavel
84200be7d2
Better comment 2023-01-23 15:31:07 +01:00
Kruglov Pavel
3b53b195be
Fix special build 2023-01-23 15:27:36 +01:00
Kruglov Pavel
fd7e04a96d
Merge pull request #45446 from kssenii/fix-assertion-in-cache-over-local 
Fix Buffer's offsets mismatch logical error in stress test
 2023-01-23 14:38:25 +01:00
Kseniia Sumarokova
16ffed2c0e
Merge pull request #45502 from azat/dist/fix-race-on-create 
Fix a race between Distributed table creation and INSERT into it
 2023-01-23 14:38:11 +01:00
Anton Popov
61b0c4da38
Merge pull request #45498 from CurtizJ/fix-race-parallel-parsing 
Fix abort in destructor of `ParallelParsingInputFormat`
 2023-01-23 14:12:53 +01:00
Aleksandr
2caeed901b
Merge branch 'master' into fix_backup_restore_num_files 2023-01-23 13:43:20 +01:00
vdimir
dac86d48d2
fixing join data was released 2023-01-23 12:34:36 +00:00
Sergei Trifonov
0fbfa17863
Merge branch 'master' into cancellable-mutex-integration 2023-01-23 12:44:09 +01:00
Azat Khuzhin
51019bc9f3 Fix a race between Distributed table creation and INSERT into it 
Initializing queues for pending on-disk files for async INSERT cannot be
done after table had been attached and visible to user, since it
initializes the per-table counter, that is used during INSERT.

Now there is a window, when this counter is not initialized and it will
start from the beginning, and this could lead to CANNOT_LINK error:

    Destination file /data/clickhouse/data/urls_v1/urls_in/shard6_replica1/13129817.bin is already exist and have different inode

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-23 09:55:43 +01:00
Smita Kulkarni
383af7f3a6 Moved settings inside backups section - Updated backup/restore status when concurrent backups & restores are not allowed 2023-01-23 08:59:44 +01:00
Anton Popov
8a39337c7c add fields to table system.formats 2023-01-23 02:04:31 +00:00
Anton Popov
f181254fb0 fix race in destructor of ParallelParsingInputFormat 2023-01-23 01:18:58 +00:00
kssenii
dfac0bb42d Fix stress test 2023-01-22 20:31:49 +01:00
Kseniia Sumarokova
c3d545b64a
Merge pull request #45489 from azat/build/queue-fwd-decl 
Forward declaration of ConcurrentBoundedQueue in ThreadStatus
 2023-01-22 20:23:05 +01:00
Smita Kulkarni
310ae62d90 Updated backup/restore status when concurrent backups & restores are not allowed 
Implementation:
* Moved concurrent backup/restore check inside try-catch block which sets the status so that other nodes in cluster are aware of failures.
* Renamed backup_uuid to restore_uuid in RestoreSettings.
Testing:
* Updated test test_backup_and_restore_on_cluster/test_disallow_concurrency to check for specific backup/restore id.
 2023-01-22 19:01:09 +01:00
Robert Schulze
4ece499f19
Fix build 2023-01-22 12:26:03 +00:00
Robert Schulze
340f406553
Merge branch 'master' into query-result-cache 2023-01-22 13:21:36 +01:00
Robert Schulze
531e64dc39
Merge branch 'master' into inv-index-cleanup 2023-01-22 13:20:38 +01:00
Azat Khuzhin
ca8af3d18c Fix possible in-use table after DETACH 
Right now in case of DETACH/ATTACH there can be a window when after the
table had been DETACH'ed someone will still use it, the common example
here is MVs handling.

It happens because TableExclusiveLockHolder does not guards the
shard_ptr of the IStorage, and so if someone holds it, then it can use
it. So if ATTACH will be done for this table then, you can have multiple
instances of it.

It is not possible for DROP, because before using a table, you should
lock it and after table had been DROP'ed you cannot lock it anymore.

So let's do the same for DETACH.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-22 13:18:56 +01:00
Alexander Gololobov
91fb1eab8c
Merge pull request #45373 from ClickHouse/throttler-metrics 
Add detailed profile events for throttling
 2023-01-22 11:20:45 +01:00
Alexander Tokmakov
1174eaa132
Merge pull request #45492 from azat/revert/dist/async-INSERT-metrics 
Revert "Merge pull request #44922 from azat/dist/async-INSERT-metrics"
 2023-01-22 00:45:10 +03:00
Azat Khuzhin
a55798626a Revert "Merge pull request #44922 from azat/dist/async-INSERT-metrics" 
There are the following problems with this patch:
- Looses files on exception
- Existing current_batch.txt on startup leads to ENOENT error and hung
  of distributed sends without ATTACH/DETACH
- Race between creating the queue for sending at table startup and
  INSERT, if it had been created from INSERT, then it will not be
  initialized from disk

They were addressed in #45491, but it makes code more cmoplex and plus
since, likely, the release is comming, it is better to revert the
change.

This reverts commit 94604f71b7, reversing
changes made to 80f6a45376.
 2023-01-21 22:42:00 +01:00
Robert Schulze
430203de79
Fix typo 2023-01-21 19:46:18 +00:00
Robert Schulze
97d1bed114
Merge branch 'master' into improve_week_day 2023-01-21 20:40:33 +01:00
Maksim Kita
19f1bae5ed
Merge pull request #45254 from kitaisreal/planner-small-fixes 
Planner small fixes
 2023-01-21 19:54:17 +03:00
Maksim Kita
6594aa8f1e
Merge pull request #44234 from kitaisreal/analyzer-expired-context-crash-fix 
Analyzer expired Context crash fix
 2023-01-21 18:49:50 +03:00
Azat Khuzhin
2a8f116c18 Forward declaration of ConcurrentBoundedQueue in ThreadStatus 
ThreadStatus is the header that recomplies almost all ClickHouse
modules.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-21 16:02:09 +01:00
Vitaly Baranov
5ceb64accc Use new copy s3 functions in S3ObjectStorage. 2023-01-21 15:47:58 +01:00
Aleksandr
206eb4d446
removed unused if statement for increaseProcessedSize 
Co-authored-by: Vitaly Baranov <vitlibar@hotmail.com>
 2023-01-21 15:07:25 +01:00
Vitaly Baranov
f0fda580d0
Merge pull request #45188 from vitlibar/backup-to-s3-memory-optimization 
Optimize memory consumption during backup to S3
 2023-01-21 12:37:35 +01:00
Maksim Kita
5146087db1 Fix style 2023-01-21 12:19:09 +01:00
Nikolai Kochetov
b877c484d2
Merge pull request #45481 from ClickHouse/fix-deadlock-with-allow_asynchronous_read_from_io_pool_for_merge_tree 
Fix possible deadlock with allow_asynchronous_read_from_io_pool_for_merge_tree in case of exception from ThreadPool::schedule
 2023-01-21 12:05:34 +01:00
Nikolai Kochetov
ec1e2436cc
Merge pull request #45450 from ClickHouse/fix-disabled-two-level-agg 
Fix disabled two-level aggregation from HTTP
 2023-01-21 12:01:59 +01:00
Sema Checherinda
962894afc8
Merge pull request #44909 from CheSema/intersect-prev-part 
Do not merge over a gap with outdated undeleted parts
 2023-01-21 11:51:21 +01:00