thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
109,109 Commits 9,338 Branches 1,783 Tags 2.1 GiB
2d8060d843
Commit Graph

5370 Commits

Author SHA1 Message Date
Anton Popov
92fa39798b Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-26 13:29:23 +00:00
Anton Popov
87af86e9a3 fix projections 2023-01-26 13:11:31 +00:00
Alexander Tokmakov
14db798191 fix 2023-01-26 13:56:16 +01:00
Anton Popov
b52cf17fa6
Merge pull request #45612 from ClickHouse/CurtizJ-patch-7 
Remove useless code
 2023-01-26 12:38:52 +01:00
Smita Kulkarni
8c885ffeae Updated checking of SYSTEM SYNC REPLICA 
Implementation:
* Updated to wait for current last entry to be processed (after pulling shared log) instead of queue size becoming 0.
* Updated Subscriber to notify both queue size and removed log_entry_id.
 2023-01-26 11:40:09 +01:00
Alexander Tokmakov
a584ad0eb1 forbid runtime strings 2023-01-26 10:52:47 +01:00
Antonio Andelic
6fe9e9a67f
Merge pull request #45617 from ClickHouse/fix-bfngram-index 
Correctly check types when using N-gram bloom filter index
 2023-01-26 09:05:56 +01:00
Alexander Tokmakov
9b670946db Merge branch 'master' into exception_message_patterns5 2023-01-26 00:41:32 +01:00
Anton Popov
79ea6a1c9c fix performance of short queries 2023-01-25 22:20:01 +00:00
Han Fei
0221b50028
Merge pull request #45601 from hanfei1991/hanfei/async-cache-conflict 
fix race of async-insert-blockids-cache
 2023-01-25 22:28:42 +01:00
Alexander Tokmakov
3744fa2c63 format more messages 2023-01-25 21:16:42 +01:00
Anton Popov
0f46c1ecc6 fix build 2023-01-25 20:13:54 +00:00
Azat Khuzhin
00c341850f Improve "at least part X is missing" error message 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-25 19:53:45 +01:00
Anton Popov
a455083b87 Merge remote-tracking branch 'upstream/master' into refactoring-merge-tree-parts 2023-01-25 18:01:12 +00:00
Kruglov Pavel
83a212a128
Merge branch 'master' into more-fsync-fixes 2023-01-25 18:40:41 +01:00
Anton Popov
8e3698c91f refactoring of code near merge tree parts 2023-01-25 17:34:09 +00:00
Antonio Andelic
7fa906487c Fix ngram index with null 2023-01-25 16:40:04 +00:00
Anton Popov
d9aabb48c3 remove useless code 2023-01-25 16:11:15 +00:00
Alexander Tokmakov
6eb557b2ba Merge branch 'master' into exception_message_patterns4 2023-01-25 13:49:17 +01:00
Robert Schulze
59528cfca0
Merge pull request #45460 from ClickHouse/inv-index-cleanup 
Cleanup of inverted index
 2023-01-25 13:23:38 +01:00
Han Fei
8c4bbcd0fc fix race of async-insert-blockids-cache 2023-01-25 12:41:01 +01:00
Sergei Trifonov
0d1ea05ff6
Merge pull request #45007 from ClickHouse/cancellable-mutex-integration 
Fast shared mutex integration
 2023-01-25 11:15:46 +01:00
Kseniia Sumarokova
2b8b1ad5d4
Additional check in MergeTreeReadPool (#45515) 
* Check ranges

* Check equality just in case

* Check under ndebug
 2023-01-24 23:56:26 +01:00
Azat Khuzhin
54ef885a61 Fsync all small files at once after mutation 
Everything else is handled in MergedBlockOutputStream::finalizePart()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Azat Khuzhin
4b775ccdd5 Do fsync all files at once for fetched parts to decrease latency 
For filessystems like ext4, fsync of one file will handle all operations
before, so this can be pretty time consuming.

And in case of you write multiple files in a loop, and at the end of
each iteration sync each file, then during writing of this file there
can be other operations in journal, and hence more work for fsync.

Let's call fsync for all files at once instead, like
MergedBlockOutputStream does.

Hope that keeping all file buffers till the end, will not cause troubles
(buffering and so forth).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Azat Khuzhin
b370e3faa5 Move fsync inside transaction callback in DataPartStorageOnDisk::rename() 
Otherwise, it is useless.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Robert Schulze
9ff2bfcbf5
Merge remote-tracking branch 'origin/master' into inv-index-cleanup 
src/Interpreters/GinFilter.cpp
	src/Interpreters/InterpreterCreateQuery.cpp
	src/Storages/MergeTree/MergeTreeData.cpp
	src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp
	src/Storages/MergeTree/MergeTreeIndexInverted.cpp
 2023-01-24 10:09:42 +00:00
Alexander Tokmakov
bb4c8e169f check number of parameters in format string 2023-01-23 23:16:16 +01:00
Alexander Tokmakov
3f6594f4c6 forbid old ctor of Exception 2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Robert Schulze
7adec3d8a4
Update src/Storages/MergeTree/MergeTreeIndexInverted.cpp 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-01-23 20:53:25 +01:00
Robert Schulze
d8af5bc2ef
Update src/Storages/MergeTree/MergeTreeIndexInverted.cpp 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-01-23 20:53:17 +01:00
Maksim Kita
7b48c75e82
Merge pull request #45485 from kitaisreal/remove-unnecessary-get-total-row-count-function-calls 
Remove unnecessary getTotalRowCount function calls
 2023-01-23 21:02:51 +03:00
Anton Popov
08b2e524c7 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-23 15:34:53 +00:00
Kseniia Sumarokova
8ec39ab6f3
Merge pull request #45442 from kssenii/minor-changes 
Minor improvements around reading from remote
 2023-01-23 15:46:30 +01:00
Sergei Trifonov
0fbfa17863
Merge branch 'master' into cancellable-mutex-integration 2023-01-23 12:44:09 +01:00
Robert Schulze
531e64dc39
Merge branch 'master' into inv-index-cleanup 2023-01-22 13:20:38 +01:00
Nikolai Kochetov
b877c484d2
Merge pull request #45481 from ClickHouse/fix-deadlock-with-allow_asynchronous_read_from_io_pool_for_merge_tree 
Fix possible deadlock with allow_asynchronous_read_from_io_pool_for_merge_tree in case of exception from ThreadPool::schedule
 2023-01-21 12:05:34 +01:00
Sema Checherinda
962894afc8
Merge pull request #44909 from CheSema/intersect-prev-part 
Do not merge over a gap with outdated undeleted parts
 2023-01-21 11:51:21 +01:00
Maksim Kita
47385a19e7 Remove unnecessary getTotalRowCount function calls 2023-01-21 11:27:07 +01:00
Nikolai Kochetov
dcd84c152a Fix possible deadlock with allow_asynchronous_read_from_io_pool_for_merge_tree in case of exception from ThreadPool::schedule 2023-01-20 18:57:47 +00:00
Alexander Tokmakov
910d6dc0ce
Merge pull request #45342 from ClickHouse/exception_message_patterns 
Save message format strings for DB::Exception
 2023-01-20 18:46:52 +03:00
Robert Schulze
1a966a9590
Fix bad comparison 2023-01-20 13:05:06 +00:00
kssenii
8d20af8127 Fix 2023-01-20 13:34:23 +01:00
Robert Schulze
bfc3b4f5ca
Suffix "GinFilter" --> "Inverted" 2023-01-20 12:02:35 +00:00
Robert Schulze
0738b2499c
Use GinFilters typedef where possible 2023-01-20 11:52:04 +00:00
Robert Schulze
0b77f07f67
Remove superfluous check (the same is checked in MergeTreeIndices.cpp) 2023-01-20 11:50:35 +00:00
Robert Schulze
d2c830ec39
Cosmetics 2023-01-20 11:49:08 +00:00
Robert Schulze
72973076c9
Rename MergeTreeIndexGin.h/cpp to MergeTreeIndexInverted.h/cpp 2023-01-20 11:42:36 +00:00
Robert Schulze
1ef2704539
Cosmetics 2023-01-20 11:39:23 +00:00
Robert Schulze
463cc843de
"segment file" --> "segment metadata file" 2023-01-20 11:26:22 +00:00
Robert Schulze
58df3953bb
Move some code around (no other changes) 2023-01-20 11:24:23 +00:00
Robert Schulze
3267ac2787
Prefix more typedefs in DB namespace with "Gin" 2023-01-20 11:19:07 +00:00
Robert Schulze
919b67f117
Cosmetics 2023-01-20 11:15:28 +00:00
Robert Schulze
98e117dca6
SegmentDictionary --> GinSegmentDictionary, also move typedef 2023-01-20 11:09:49 +00:00
Robert Schulze
908fa83f72
Move some typedefs around 2023-01-20 11:08:19 +00:00
Robert Schulze
44618927f9
Inline two short methods + uppercase 2023-01-20 11:04:35 +00:00
Robert Schulze
5c3cc5283f
"term dictionary" --> "dictionary" 2023-01-20 10:53:41 +00:00
Robert Schulze
be936b257c
Make version enum private 2023-01-20 10:48:43 +00:00
Robert Schulze
0653f86de9
Various cosmetic cleanups 2023-01-20 10:45:35 +00:00
Frank Chen
f867c4c206 Add an overriden fromCurrentException to simply the code 2023-01-20 16:43:10 +08:00
Frank Chen
38799eb852 Merge two methods as one 2023-01-20 16:35:28 +08:00
Frank Chen
be8ceeb4bb Save stack trace in part_log 2023-01-20 16:22:40 +08:00
Frank Chen
1fe9bde594 Set replica name to zero copy exclusive lock 2023-01-20 15:48:33 +08:00
Frank Chen
fb8452aa49 Add last_exception_time to replication_queue 2023-01-20 15:20:57 +08:00
kssenii
4ce8950712 Minor changes 2023-01-19 17:53:10 +01:00
larryluogit
52ae33dba7
Merge branch 'master' into ftsearch 2023-01-19 11:34:11 -05:00
Sema Checherinda
ae1dfb9ce5
Update src/Storages/MergeTree/MergeTreeData.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-01-18 15:21:11 +01:00
Han Fei
e51123c9b0 fix data race 2023-01-18 13:11:07 +01:00
Sergei Trifonov
c443c1ece0
Merge branch 'master' into hanfei/async-cache 2023-01-18 00:19:49 +01:00
Alexander Tokmakov
5cd90c1a3e Merge branch 'master' into exception_message_patterns 2023-01-17 20:04:04 +01:00
Alexander Tokmakov
72e8615bec formatting of some exception messages 2023-01-17 20:03:56 +01:00
Alexander Tokmakov
8b13b85ea0
Merge pull request #44543 from ClickHouse/text_log_add_pattern 
Add a column with a message pattern to text_log
 2023-01-17 20:19:32 +03:00
Han Fei
8a74238fe0 improve 2023-01-17 15:47:52 +01:00
HarryLeeIBM
e7add8218f Addressed more review comments and ClangTidy errors 2023-01-17 06:29:13 -08:00
Alexander Tokmakov
522686f78b less empty patterns 2023-01-17 01:19:44 +01:00
Alexander Tokmakov
870cfcc36a less fmt::runtime usages 2023-01-17 00:11:59 +01:00
Sema Checherinda
90fa1ecd49 make that old_parts_lifetime=0 deletes files instantly at drop/truncate 2023-01-16 16:47:12 +01:00
Sema Checherinda
c51f4d7be1 do not merge over a gap with uotdate parts, delete empty parts with respect to old_parts_lifetime 2023-01-16 16:47:11 +01:00
Sema Checherinda
25e16388d7 better message in MergeTreeDataMergerMutator when parts intersect 2023-01-16 16:47:11 +01:00
Anton Popov
6863cd152f
Merge pull request #42181 from CurtizJ/optimize-loading-parts 
Do not load inactive parts at startup
 2023-01-16 14:38:50 +01:00
Han Fei
30a798182a Merge branch 'master' into hanfei/async-cache 2023-01-16 14:07:36 +01:00
Han Fei
5617f7f616 address comments 2023-01-15 22:51:10 +01:00
Han Fei
701dc88d6f
Merge branch 'master' into hanfei/support-advance-dedup 2023-01-15 19:46:28 +01:00
Han Fei
c859f8dbe5
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2023-01-15 19:46:16 +01:00
Han Fei
bb2c0914e9
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2023-01-15 19:46:09 +01:00
Alexander Tokmakov
d857d62a03 remove another set of macros 2023-01-13 20:34:31 +01:00
Alexander Tokmakov
2d7773fccc Merge branch 'master' into text_log_add_pattern 2023-01-13 20:33:46 +01:00
Han Fei
ed49ebf01a update setting explain 2023-01-13 20:26:08 +01:00
Han Fei
2fb2f503e3 Update src/Storages/MergeTree/MergeTreeSettings.h 
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
 2023-01-13 20:20:08 +01:00
Han Fei
9e99c7e116 Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
 2023-01-13 20:19:13 +01:00
Han Fei
a258a39eb1 Merge branch 'master' into hanfei/async-cache 2023-01-13 20:17:58 +01:00
Anton Popov
487de70d01 fix locking at loading outdated data parts 2023-01-13 17:05:32 +00:00
Anton Popov
71188c22ee fix race on 'relative_data_path' 2023-01-13 12:19:41 +00:00
Alexander Tokmakov
51d94314d6
Merge pull request #45235 from ClickHouse/more_verbose_logs_about_replication_log_entries 
More verbose logs about replication log entries
 2023-01-13 15:05:21 +03:00
Alexander Tokmakov
d9b8cf9c04 more verbose logs about replication log entries 2023-01-12 20:32:39 +01:00
serxa
693489a8ad review fixes 2023-01-12 15:51:04 +00:00
Sergei Trifonov
12d8543578
Merge branch 'master' into cancellable-mutex-integration 2023-01-12 16:03:49 +01:00
Arthur Passos
301b684933 remove trialing whitespace? 2023-01-12 10:00:18 -03:00
Anton Popov
ca80240d84 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-12 12:13:45 +00:00
Arthur Passos
1b3c7766f8
Merge branch 'master' into add_select_final_mt_setting 2023-01-12 08:31:04 -03:00
Anton Popov
8bdf63f8e5
Merge pull request #45191 from ClickHouse/fix-ip-part-name 
Fix part ID generation for IP types for backward compatibility
 2023-01-12 12:15:06 +01:00
Igor Nikonov
edbcb24da0
Merge pull request #44954 from ClickHouse/fix-insert-delay 
Fix: insert delay calculation
 2023-01-12 09:43:00 +01:00
HarryLeeIBM
6d576cf157 Added more documentation 2023-01-11 13:40:20 -08:00
Yakov Olkhovskiy
9c4ea5a16b fix part ID generation for IP types for backward compatibility 2023-01-11 18:51:37 +00:00
Alexander Tokmakov
0fba3d6d81
Merge pull request #44468 from ClickHouse/fix_get_part_name 
Fix incorrect usages of `getPartName()`
 2023-01-11 16:24:02 +03:00
Igor Nikonov
1d002e45a9 Fix test & review comments 2023-01-11 12:49:01 +00:00
Nikolai Kochetov
5e7a6ac619
Merge pull request #45122 from ClickHouse/revert-45121-revert-44653-custom-reading-for-mutation 
Revert "Revert "Custom reading for mutation""
 2023-01-11 12:37:32 +01:00
Nikolai Kochetov
d0c1079242
Merge pull request #45133 from ClickHouse/fix-additinal-tables-filter-with-count-projection 
Fix additional_table_filters with minmax/count projection.
 2023-01-11 12:25:40 +01:00
Anton Popov
85734bb321 fix build 2023-01-11 08:32:33 +00:00
Anton Popov
6c4f53bdca Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-11 01:00:57 +00:00
HarryLeeIBM
150f9b48a4 Fix functional test for full text search 2023-01-10 14:35:06 -08:00
Igor Nikonov
543221b8b9 Merge remote-tracking branch 'origin/master' into fix-insert-delay 2023-01-10 22:15:41 +00:00
Igor Nikonov
e75df5e76b Test 
+ fix some review comments
 2023-01-10 22:14:54 +00:00
Alexander Tokmakov
881b17492f Merge branch 'master' into fix_get_part_name 2023-01-10 21:39:35 +01:00
Alexander Tokmakov
255881a2ff
Merge pull request #44466 from ClickHouse/parallel_parts_cleanup_with_zero_copy 
Parallel parts cleanup with zero copy replication
 2023-01-10 22:44:39 +03:00
Ilya Yatsishin
0efceae6ce
Merge pull request #44917 from azat/mt-fix-log-message 
Fix logging message in MergeTreeDataMergerMutator (about merged parts)
 2023-01-10 19:49:31 +01:00
Nikolai Kochetov
9a81f27fb2 Fix additional_table_filters with minmax/count projection. 2023-01-10 17:32:26 +00:00
HarryLeeIBM
bc48b17817 Addressed more review comments. 2023-01-10 08:26:27 -08:00
Nikolai Kochetov
4673b3fe1d
Revert "Revert "Custom reading for mutation"" 2023-01-10 16:31:01 +01:00
Alexander Tokmakov
c8ec130be4
Revert "Custom reading for mutation" 2023-01-10 17:51:30 +03:00
Han Fei
bef6168c05 support cache for async inserts block ids 2023-01-10 13:19:12 +01:00
Anton Popov
b9bf92fa5d check for network errors while loading parts 2023-01-10 11:55:08 +00:00
Nikolai Kochetov
11418963c0
Merge pull request #44653 from ClickHouse/custom-reading-for-mutation 
Custom reading for mutation
 2023-01-10 12:16:24 +01:00
alesapin
616e37ba43
Merge pull request #44847 from ClickHouse/fix_rare_race 
Fix rare race which can lead to queue hang
 2023-01-10 12:09:10 +01:00
Nikolai Kochetov
3e430b1b6e Add comments. Remove unused flag from Context. 2023-01-09 19:52:11 +00:00
Anton Popov
1e5d6e44a5 fix deadlock at shutdown 2023-01-09 13:07:46 +00:00
Sergei Trifonov
81d2ea30ba
Merge branch 'master' into cancellable-mutex-integration 2023-01-07 19:37:46 +01:00
Anton Popov
1f32ffedf8
Merge pull request #43221 from ClickHouse/refactoring-ip-types 
Replace domain IP types (IPv4, IPv6) with native
 2023-01-07 12:01:21 +01:00
Alexey Milovidov
9fcd111724
Merge pull request #45003 from amosbird/issue_44979 
Fix minmax_count_projection with _partition_value
 2023-01-07 03:36:56 +03:00
serxa
15bb127b01 replace every std::shared_mutex with DB::FastSharedMutex 2023-01-06 23:35:38 +00:00
Amos Bird
200d40c53d
Fix minmax_count_projection with _partition_value 
Check if virtual column `_partition_value` can be used when building
minmax_count_projection block.
 2023-01-07 05:38:28 +08:00
Dmitry Novik
9d44c7a737
Merge pull request #44541 from ClickHouse/cleanup-resolved-function 
Followup #43761
 2023-01-06 18:09:32 +01:00
kssenii
8fde606768 Resolve some review comments 2023-01-06 16:10:00 +01:00
kssenii
2a030c1dc0 Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2023-01-06 15:58:27 +01:00
alesapin
da83e97b4c Fix startup issue 2023-01-06 15:36:53 +01:00
Anton Popov
b25f875674
Merge pull request #44875 from ClickHouse/fix-another-one-cannot-read-all-data-for-lc-dict-error 
Fix right offset for reading LowCardinality dictionary from remote fs
 2023-01-06 15:24:36 +01:00
alesapin
b20f07b2cc Merge branch 'master' into fix_rare_race 2023-01-06 14:20:08 +01:00
Anton Popov
fcdbc8123d better loading of outdated parts 2023-01-06 11:59:54 +00:00
Alexey Milovidov
b88be9df8c
Merge pull request #44956 from ClickHouse/fix_read_result_consistency_check_on_exception 
Do not check read result consistency when unwinding
 2023-01-06 06:22:42 +03:00
Alexey Milovidov
a205b41c56
Merge pull request #44947 from Algunenano/background_pool_size_logical 
Change error code on invalid background_pool_size config
 2023-01-06 03:19:40 +03:00
Alexander Gololobov
1b31b8bb91 Do not check read result consistency when unwinding 2023-01-06 00:26:08 +01:00
Igor Nikonov
a67afdff6a Fix: insert delay calculation 2023-01-05 22:43:41 +00:00
Dmitry Novik
ebead611b6
Merge branch 'master' into cleanup-resolved-function 2023-01-05 19:27:08 +01:00
Raúl Marín
b0715ec5eb Change error code on invalid background_pool_size config 2023-01-05 18:47:02 +01:00
Anton Popov
ed00399eb6 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-05 14:12:02 +00:00
Larry Luo
f2bc5bb335 Added initializer 2023-01-05 05:41:29 -08:00
Igor Nikonov
2a105d1862 Fix stupid mistake - max() -> min() 2023-01-05 12:38:27 +00:00
Igor Nikonov
db46267a95 Fast fix: force upper bound for time to delay INSERT 2023-01-05 12:38:27 +00:00
Larry Luo
fd0c6cccc8 Fixed style error 2023-01-04 20:08:28 -08:00
Larry Luo
7a4d499cb1 Added density to control Inverted index size 2023-01-04 19:42:45 -08:00
Azat Khuzhin
d06ac2d468 Fix logging message in MergeTreeDataMergerMutator (about merged parts) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-04 19:07:01 +01:00
Yakov Olkhovskiy
7a5a36cbed
Merge branch 'master' into refactoring-ip-types 2023-01-04 11:11:06 -05:00
alesapin
2cd7247619 Merge branch 'master' into fix_rare_race 2023-01-04 15:06:20 +01:00
Kruglov Pavel
9056925c7d
Merge pull request #44874 from Algunenano/time_fix 
Do not throw DB::Exception when folders do not exist
 2023-01-04 01:17:57 +01:00
Nikolai Kochetov
c51f5e6fac Fix typo. 2023-01-03 18:41:37 +00:00
Nikolai Kochetov
da26f62a9b Fix right offset for reading LowCardinality dictionary from remote fs in case if right mark was in the middle of compressed block. 2023-01-03 18:19:51 +00:00
Dmitry Novik
a84318af5f
Merge branch 'master' into cleanup-resolved-function 2023-01-03 17:52:29 +01:00
Raúl Marín
0785bf6b29 Do not throw DB::Exception when folders do not exist 2023-01-03 17:05:54 +01:00
Nikolai Kochetov
6b1a697b12
Merge pull request #44850 from ClickHouse/respect-context-in-projection-optimization 
Save context created from projection analysis.
 2023-01-03 13:30:51 +01:00
Han Fei
7d7de3833b
Merge pull request #44651 from hanfei1991/hanfei/fix-async-insert-cleanup 
fix bug that async blocks cleanup not work
 2023-01-02 21:58:27 +01:00
Dmitry Novik
929300a721
Update MergeTreeData.cpp 2023-01-02 21:18:16 +01:00
Dmitry Novik
04fb410a40
Merge branch 'master' into parallel_parts_cleanup_with_zero_copy 2023-01-02 20:48:39 +01:00
Dmitry Novik
c646048af9
Merge pull request #44461 from ClickHouse/parts-removal-limit 
Add part removal limit for one CleanupThread iteration
 2023-01-02 20:04:19 +01:00
Alexey Milovidov
e855d3519a
Merge branch 'master' into refactoring-ip-types 2023-01-02 21:58:53 +03:00
Nikolai Kochetov
09325ff17e Store context created from projection analysis. 2023-01-02 17:00:48 +00:00
alesapin
6fc63878b4 Fix rare race which can lead to queue hang 2023-01-02 16:57:25 +01:00
Han Fei
eef0136415 add debug log 2023-01-02 11:59:44 +01:00
Larry Luo
7cc84c6743 refactor filter functions support 2022-12-31 18:05:07 -08:00
Larry Luo
71da1b1379 Updated inverted index to use ActionsDAG 2022-12-31 18:05:06 -08:00
Larry Luo
f0a2c06261 Fixed errors caused by merging. 2022-12-31 18:05:06 -08:00
HarryLeeIBM
d7a7de9601 Minor fixes for addressing reviews 2022-12-31 18:05:06 -08:00
HarryLeeIBM
6a5fef632d Added more tests and experimental flag, etc. 2022-12-31 18:05:01 -08:00
Harry-Lee
b43ddde50a Refactor full text search according to reviews 2022-12-31 18:03:14 -08:00
Larry Luo
00e4411740 Created FST namespace to hide FST details 2022-12-31 18:03:14 -08:00
Larry Luo
64f807f31a fixed an edge case that index column has no data 2022-12-31 18:03:14 -08:00
Larry Luo
da93210cc6 Updated MergeTreeConditionGinFilter due to SelectQueryInfo changes. 2022-12-31 18:03:14 -08:00
Larry Luo
a33232cb0a Clean up code 2022-12-31 18:03:14 -08:00
Larry Luo
7e5f311da9 Fix MemorySanitizer error 2022-12-31 18:03:14 -08:00
Larry Luo
0a47378e5f Added initializer for current segment 2022-12-31 18:03:14 -08:00
Larry Luo
514e17d037 Code clean up 2022-12-31 18:03:14 -08:00
Larry Luo
1d9bf9962c Code clean up 2022-12-31 18:03:14 -08:00
Larry Luo
657ce7c614 Cleanup code and tests 2022-12-31 18:03:13 -08:00
Larry Luo
3652740535 Code clean up 2022-12-31 18:03:13 -08:00
Larry Luo
0b63d6b5be Resolve conflicts and style errors. 2022-12-31 18:03:13 -08:00
HarryLeeIBM
bc8be368bb Fix coding styles 2022-12-31 18:03:13 -08:00
Larry Luo
91b79b26e3 Use DataPartStorage in GinIndexStore 2022-12-31 18:03:02 -08:00
Harry-Lee
4b29c02f35 Remove useless FUNCTION_LIKE 2022-12-31 18:00:17 -08:00
HarryLeeIBM
e58f0ef3d2 Remove roaring bmp cleanup code which can cause memory leak 2022-12-31 18:00:17 -08:00
HarryLeeIBM
0839ff60ee Initial Checkin of full text search 2022-12-31 18:00:17 -08:00
Alexey Milovidov
50bb7f90d8
Merge pull request #44809 from ClickHouse/fix-race-in-mergetreedatapart 
Fix race in `system.parts` and `system.parts_columns`
 2023-01-01 01:09:46 +03:00
Han Fei
a036c6bc01
Merge branch 'master' into hanfei/fix-async-insert-cleanup 2022-12-31 20:52:16 +01:00
Alexey Milovidov
3a5cea8281 Fix race in system.parts and system.parts_columns 2022-12-31 15:07:26 +01:00
Alexey Milovidov
26eb2f57e1 Fix race in system.parts and system.parts_columns 2022-12-31 15:06:57 +01:00
Azat Khuzhin
e7bf2cc959 Add a comment for lack of fsync in MergedBlockOutputStream::finalizePartOnDisk() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-12-30 22:39:13 +01:00
Azat Khuzhin
9fb6004c92 Add missing fsync for small files (ttl.txt, columns.txt, ...) in mutations 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-12-30 22:22:39 +01:00
Nikolai Kochetov
758f30cc17 Cleanup. 2022-12-30 17:53:46 +00:00
Han Fei
454f40ab17 fix test 2022-12-30 16:30:33 +01:00
Alexey Milovidov
13b76ad760
Merge pull request #44719 from ClickHouse/fix_attach_thread_deadlock 
Fix deadlock in attach thread
 2022-12-30 13:56:18 +03:00
Nikolay Degterinsky
dfe93b5d82
Merge pull request #42284 from Algunenano/perf_experiment 
Performance experiment
 2022-12-30 03:14:22 +01:00
Alexander Gololobov
fbae502499
Merge pull request #42126 from ClickHouse/fix_invalid_number_of_rows_in_chunk_with_prewhere 
Fixing "Invalid number of rows in Chunk" with lightweight deletes
 2022-12-29 21:46:12 +01:00
alesapin
5dfefb43a3 Fix deadlock in attach thread 2022-12-29 18:52:20 +01:00
Dmitry Novik
81c0db3b67 Reuse FunctionBase for IndexHint 2022-12-29 15:47:43 +00:00
Dmitry Novik
bd480f18c1
Merge branch 'master' into parts-removal-limit 2022-12-29 16:27:54 +01:00
Alexander Gololobov
a02a631d51 Cleanups based on code review 2022-12-29 15:00:42 +01:00
Alexander Gololobov
7df137e460 Replaced asserts with logical errors 2022-12-29 14:33:11 +01:00
alesapin
02c3e708af
Merge pull request #44694 from Algunenano/old_part_improvement 
Improve handling of old parts
 2022-12-29 13:14:52 +01:00
alesapin
b1ed48a977
Merge pull request #44628 from ClickHouse/CurtizJ-fix-restart-after-quorum-insert 
Fix restart after quorum insert
 2022-12-29 12:51:54 +01:00
Alexander Gololobov
059ec6f747 Cleanups 2022-12-29 01:22:47 +01:00
Alexey Milovidov
4311e6c147
Merge pull request #44683 from CurtizJ/fix-flaky-test-4 
Fix test `01130_in_memory_parts`
 2022-12-29 01:53:24 +03:00
Raúl Marín
32d1662503 Improve handling of old parts 2022-12-28 21:22:40 +01:00
Anton Popov
8d2788445b fix style check 2022-12-28 19:29:56 +00:00
Raúl Marín
e915ce1e95 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-12-28 20:15:43 +01:00
alesapin
697617548c Tiny improvement 2022-12-28 18:59:24 +01:00
Anton Popov
bc122d368a Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-28 17:45:04 +00:00
Nikolai Kochetov
763fcdc939
Update MergeTreeSequentialSource.cpp 2022-12-28 18:17:40 +01:00
Alexander Gololobov
10a058d138 More cleanups in the logic of applying current step filter and final filter 2022-12-28 18:07:36 +01:00
Alexander Gololobov
fd5d328fae Test accumulating filters ignoring prewhere_info->need_filter flag 2022-12-28 18:07:36 +01:00
Alexander Gololobov
a7adc0a91b Cleanups 2022-12-28 18:07:36 +01:00
Alexander Gololobov
b22711baa3 Reset need_filter flag when filter is applied 2022-12-28 18:07:36 +01:00
Alexander Gololobov
ada6422985 Restored old logic for filling _part_offset 2022-12-28 18:07:36 +01:00
Alexander Gololobov
4cebc6f3a4 Cleanups 2022-12-28 18:07:36 +01:00
Alexander Gololobov
13e457c754 Cleanups 2022-12-28 18:07:36 +01:00
Alexander Gololobov
a1cdad4815 Fix for reading _part_offset along with newly added column that has only default value 2022-12-28 18:07:35 +01:00
Alexander Gololobov
ac1549f6b3 Skip filtering if there are no rows after optimize() 2022-12-28 18:07:35 +01:00
Alexander Gololobov
f273f8712d Avoid filtering same column in block_before_prewhere if it is present in the result 2022-12-28 18:07:35 +01:00
Alexander Gololobov
f3646248c5 Avoid unneeded work if all rows were filtered 2022-12-28 18:07:35 +01:00
Alexander Gololobov
75152ddabb Apply filter only if needed 2022-12-28 18:07:35 +01:00
Alexander Gololobov
a18850458c Test applying current filter at each step 2022-12-28 18:07:35 +01:00
Alexander Gololobov
29b5c4af07 Test dirty intermediate changes 2022-12-28 18:07:35 +01:00
Alexander Gololobov
c561acb774 Properly handle low cardinality column as prewhere filter 2022-12-28 18:07:35 +01:00
Alexander Gololobov
aa276b230b Don't need to save filter and rows_per_granule from previous step 2022-12-28 18:07:35 +01:00
Alexander Gololobov
c4a01cbd5b Fix for propely cleaning rows_per_granule_original between prewhere steps 2022-12-28 18:07:35 +01:00
Alexander Gololobov
abbb58107c Fix for "out of bound" in ColumnVector::insertRangeFrom called from shrink() 2022-12-28 18:07:35 +01:00
Alexander Gololobov
79874e8733 Fix for "Invalid number of rows in Chunk" 2022-12-28 18:07:35 +01:00
Alexander Gololobov
bdf51545f7 Added FilterWithCachedCount class instead of caching counts in filter_bytes_map 2022-12-28 18:07:35 +01:00
Azat Khuzhin
31a88d4eae Fix PREWHERE with row-level filters (when row filter is always true/false) 
In case of row-level filters optimized out, i.e. converted to
always true/false, it is possible for MergeTreeRangeReader to reuse
incorrect statistics for the filter (countBytesInResultFilter()), and
because of this it simply does not apply other filters, since it assume
that this filter does not need to filter anything.

Fixes: #40956
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-12-28 18:07:35 +01:00
Alexander Gololobov
d44392b366 Checking the fix for "Invalid number of rows in Chunk" 2022-12-28 18:07:35 +01:00
Anton Popov
38f0cf9b10 fix test 01130_in_memory_parts 2022-12-28 16:40:31 +00:00
Dmitry Novik
6efd64a61c
Merge branch 'master' into parts-removal-limit 2022-12-28 17:34:48 +01:00
Han Fei
17358f5e98 clean headers 2022-12-28 17:29:07 +01:00
Han Fei
66111ef241 deduplicate async inserts in the same block earlier 2022-12-28 17:25:36 +01:00
Anton Popov
2adf0e9db3 do not wait loading parts at shutdown 2022-12-28 16:07:14 +00:00
alesapin
3a1e9f4a4d Fix style 2022-12-28 14:08:13 +01:00
Nikolai Kochetov
c203aa054a Use sequential source for mutations. 2022-12-27 20:28:25 +00:00
Han Fei
4d8513024c fix bug that async blocks cleanup not work 2022-12-27 20:12:09 +01:00
alesapin
3a7f26ff3a Split test into two 2022-12-27 16:15:23 +01:00
Raúl Marín
948ce8db93 Revert "Merge pull request #38953 from ClickHouse/add-allocation-ptr-to-trace-log" 
This reverts commit e99849d031, reversing
changes made to 7ad3ff8a9e.
 2022-12-27 14:28:13 +01:00
alesapin
560223ac19 Merge branch 'fix-restart-after-quorum-insert' of github.com:CurtizJ/ClickHouse into CurtizJ-fix-restart-after-quorum-insert 2022-12-27 13:48:55 +01:00
Alexey Milovidov
bfe895975e
Merge pull request #44575 from ClickHouse/minimal-changes 
Non-significant changes
 2022-12-27 15:34:27 +03:00
Alexey Milovidov
464a513f0e
Merge pull request #44615 from ClickHouse/projection-grouping-sets 
Disable projections in presense of any grouping sets, including WITH ROLLUP, WITH CUBE and WITH TOTALS
 2022-12-27 15:31:03 +03:00
Alexey Milovidov
1b21cc018e
Merge pull request #44342 from ClickHouse/add-test-29883 
Add a test for #29883
 2022-12-27 14:55:14 +03:00
Raúl Marín
fc1fa82a39
Merge branch 'master' into perf_experiment 2022-12-27 10:51:58 +01:00
Alexey Milovidov
593675a7c5 Disable projections in presense of any grouping sets, including WITH ROLLUP, WITH CUBE and WITH TOTALS 2022-12-26 22:31:41 +01:00
Igor Nikonov
ba88bc17d4 Fix typo 2022-12-26 21:01:00 +00:00
Igor Nikonov
3204f7353b Fix: handle exceptions from unlockSharedData() during error handling 
+ renamePartToTemporary() lambda to reused code
 2022-12-26 20:49:04 +00:00
Igor Nikonov
0d428318a9 Merge remote-tracking branch 'origin/master' into fix-restart-after-quorum-insert 2022-12-26 19:18:35 +00:00
Alexey Milovidov
52b306d544 Improve variable name 2022-12-26 18:00:18 +01:00
Alexey Milovidov
71850f6913 Check for bad index_granularity 2022-12-25 16:06:34 +01:00
Alexey Milovidov
f22fc35900 Minimal changes 2022-12-25 14:18:05 +01:00
Nikolai Kochetov
03352a15b6 Use sequential source for mutations. 2022-12-23 22:39:27 +00:00
Alexander Tokmakov
a0668af762
Update MergeTreeData.cpp 2022-12-23 23:06:37 +03:00
Alexander Tokmakov
b2af49a4cf Merge branch 'master' into parallel_parts_cleanup_with_zero_copy 2022-12-23 20:02:17 +01:00
Alexander Tokmakov
ca989e9212 less runtime format strings 2022-12-23 19:50:34 +01:00
Anton Popov
0722020cae Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-23 18:15:55 +00:00
alesapin
2fecf420cb
Merge pull request #44513 from ClickHouse/small_polishing 
Minor code polishing
 2022-12-23 13:47:26 +01:00
Vladimir C
7482ea54ab
Merge pull request #43972 from ClickHouse/vdimir/tmp-data-in-fs-cache-2 2022-12-23 11:59:27 +01:00
Alexander Tokmakov
6b6504f0d1 Merge branch 'master' into parallel_parts_cleanup_with_zero_copy 2022-12-23 02:04:43 +01:00
Dmitry Novik
6c23721255
Merge pull request #43761 from ClickHouse/refector-function-node 
Refactor FunctionNode
 2022-12-22 23:53:27 +01:00
alesapin
a32fab90d5
Merge pull request #44436 from ClickHouse/fix_bug_with_max_task_count_update 
Fix bug with merge/mutate pool size increase
 2022-12-22 23:48:54 +01:00
Dmitry Novik
d3502937c7 Update MergeTreeIndexSet 2022-12-22 22:17:18 +00:00
Dmitry Novik
cff882d506 Merge remote-tracking branch 'origin/master' into refector-function-node 2022-12-22 21:34:29 +00:00
alesapin
8d0ad4bbcd Strange why I have to supress in .cpp 2022-12-22 16:14:19 +01:00
alesapin
fa467b3398 Minor code polishing 2022-12-22 14:31:42 +01:00
Nikolai Kochetov
773de7a8a6
Merge pull request #44472 from CurtizJ/fix-mutations 
Fix mutations with setting `max_streams_for_merge_tree_reading`
 2022-12-22 14:05:20 +01:00
alesapin
049103c550
Merge pull request #44495 from azat/mergetree/static-storage-fix 
Do not try to remove WAL/move broken parts for static storage
 2022-12-22 12:27:08 +01:00
vdimir
88008726d8
Merge branch 'master' into vdimir/tmp-data-in-fs-cache-2 2022-12-22 10:58:39 +00:00
alesapin
380c6cebe4
Merge pull request #44393 from ClickHouse/small_improvement 
Lock table for share during startup for database ordinary
 2022-12-22 11:42:42 +01:00
alesapin
bf188c46a4
Merge branch 'master' into fix_bug_with_max_task_count_update 2022-12-22 11:40:08 +01:00
Maksim Kita
e5ebf20044
Merge pull request #44097 from kitaisreal/analyzer-support-set-index 
Analyzer support Set index
 2022-12-22 13:30:15 +03:00
vdimir
d30d25dbbe
Temporary files evict fs cache 2022-12-22 10:22:49 +00:00
Yakov Olkhovskiy
a8cb29da4b
Merge branch 'master' into refactoring-ip-types 2022-12-21 23:56:24 -05:00
alesapin
9b97cd98c8
Update src/Storages/MergeTree/MergeTreeDataMergerMutator.h 2022-12-21 23:27:17 +01:00
alesapin
97d43a6748 Add comment 2022-12-21 23:26:26 +01:00
alesapin
edcd0d95ca Revert "Remove explicit dependency on fast test to debug the issue" 
This reverts commit 2fe969a2e1.
 2022-12-21 23:25:50 +01:00
Azat Khuzhin
1985d321a6 Do not try to remove WAL for static storage 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-12-21 18:09:38 +01:00
Azat Khuzhin
e06c169b43 Do not try to move parts to detached for static storage 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-12-21 18:09:38 +01:00
alesapin
a4fbea2184 Relax synchronization to avoid deadlock 2022-12-21 15:59:28 +01:00
Alexander Tokmakov
4a38a03f68 fix 2022-12-21 15:42:25 +01:00
Anton Popov
4b7f1e0cf2 fix mutations with setting 'max_streams_for_merge_tree_reading' 2022-12-21 01:59:53 +00:00
Dmitry Novik
42812dfa08 Add logging 2022-12-21 00:14:01 +00:00
Alexey Milovidov
a6f50edde3
Merge branch 'master' into add-test-29883 2022-12-21 01:45:18 +03:00
Alexander Tokmakov
cdc3912743 fix incorrect usages of getPartName() 2022-12-20 22:44:27 +01:00
Alexander Tokmakov
5f5335876a
Merge branch 'master' into parallel_parts_cleanup_with_zero_copy 2022-12-21 00:35:21 +03:00
Alexander Tokmakov
3f851d5a09 parallel parts cleanup with zero copy replication 2022-12-20 22:04:26 +01:00
alesapin
2fe969a2e1 Remove explicit dependency on fast test to debug the issue 2022-12-20 22:00:57 +01:00
Dmitry Novik
2ec720c5c4 Add part removal limit for one CleanupThread iteration 2022-12-20 19:15:13 +00:00
Dmitry Novik
4793412887
Merge branch 'master' into refector-function-node 2022-12-20 18:26:19 +01:00
alesapin
f06b311abc It's really needed 2022-12-20 18:09:38 +01:00
alesapin
2818520572 Fix bug with merge/mutate pool size increase 2022-12-20 14:51:11 +01:00