thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
78,426 Commits 9,338 Branches 1,783 Tags 2.1 GiB
2f4e7e1d4e
Commit Graph

2725 Commits

Author SHA1 Message Date
kssenii
658b33717f A little nicer 2021-10-24 20:43:34 +03:00
Azat Khuzhin
f9b27608f5 Fix reading of columns if offset_in_compressed_file is the same for mark range 
Fixes: 00078_group_by_arrays
 2021-10-24 18:58:22 +03:00
Azat Khuzhin
93c34824c0 Fix explicit 0 size for createReadBufferFromFileBase() 2021-10-24 18:58:22 +03:00
Azat Khuzhin
4599e9191b Fix explicit 0 size for readFile() 2021-10-24 18:58:22 +03:00
Kseniia Sumarokova
153dd9a895
Update MergeTreeSettings.h 2021-10-22 16:27:01 +03:00
kssenii
39b9e9c258 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-22 15:40:41 +03:00
Azat Khuzhin
496aa43a9f Remove check_columns argument from MergeTree code (false was never passed) 2021-10-21 21:24:59 +03:00
kssenii
56f0560c46 Asserts and read till end option 2021-10-21 20:54:03 +03:00
Dmitry Novik
58ee783be4
Merge pull request #30293 from amosbird/projection-fix19 
Fix projections with JOIN alias columns
 2021-10-21 17:49:21 +03:00
Anton Popov
b1a74af44d
Merge pull request #30180 from CurtizJ/remove-sample-by 
Allow to remove `SAMPLE BY` expression
 2021-10-21 17:09:53 +03:00
kssenii
e4e157688d Review fixes 2021-10-21 01:57:43 +03:00
Nikita Mikhaylov
56869c0ec4
Merge pull request #30360 from nikitamikhaylov/allow-graphite-final 
Allow to use FINAL clause in SELECT from GraphiteMergeTree
 2021-10-20 19:57:44 +03:00
kssenii
0a90be8317 Remove write_final_mark setting 2021-10-20 14:13:04 +00:00
Amos Bird
2f615e9176
Use original_query for projection analysis 2021-10-20 21:30:15 +08:00
Nikolai Kochetov
8986de7e50
Merge pull request #30384 from amosbird/projection-fix21 
Fix crash when minmax_count projection is used with primary key in partition expr
 2021-10-20 16:17:13 +03:00
Kseniia Sumarokova
5324cc8359
Merge pull request #30282 from kssenii/fix-local-less-threads 
Less threads in clickhouse-local, fix Ok. printing
 2021-10-19 22:54:28 +03:00
Amos Bird
05d93796dc
Fix minmax_count projection with primary key in partition expr 2021-10-19 23:36:07 +08:00
Nikita Mikhaylov
75487be899 White list of storages that supports final 2021-10-19 12:16:33 +00:00
Nikita Mikhaylov
4fbd332bf1 Done 2021-10-19 11:53:55 +00:00
Nikita Mikhaylov
ea2a647724
Merge pull request #30192 from azat/granula-adjust-LOGICAL_ERROR-fix 
Fix reading from MergeTree with max_read_buffer_size=0
 2021-10-19 13:25:55 +03:00
kssenii
e53335bc6f Better way 2021-10-19 08:21:38 +00:00
Nikolai Kochetov
cc7a3ea9ff
Merge pull request #30322 from ClickHouse/update-mergine-sorted-memory-optimization 
Update memory optimisation for MergingSorted.
 2021-10-19 11:12:50 +03:00
kssenii
df5dc4e991 Adjust MergeTreeIndexReader 2021-10-19 10:51:45 +03:00
kssenii
49106f407f Fix some checks 2021-10-19 09:34:27 +03:00
Alexander Tokmakov
89fe606d15 try fix 'some fetches may stuck' 2021-10-18 23:16:02 +03:00
Nikita Mikhaylov
d004fd96ed
Merge pull request #30318 from azat/fix-shared-build 
Move SquashingTransform to Interpreters (to fix split build)
 2021-10-18 22:41:01 +03:00
Azat Khuzhin
e576fd17bd Do not allow reading to empty buffer in MergeTreeReaderStream 
Changelog:
- Verify that all rows was read in MergeTreeReaderWide
- Ignore some exceptions for Wide parts
- Take max_rows_to_read into account
- Verify that requested rows is not zero (otherwise it is too tricky)
- Simply verify that buffer is not empty
 2021-10-18 21:19:34 +03:00
Azat Khuzhin
cd4b33c8c9 Verify that all rows was read in MergeTreeReaderCompact 
v0: Use fmt-like style exception in MergeTreeReaderCompact
v2: Update the check
 2021-10-18 21:19:34 +03:00
Azat Khuzhin
41e6df0f4b Remove unused offset_columns from MergeTreeReaderWide::readRows() 2021-10-18 21:19:34 +03:00
kssenii
d5d4817350 Add disk proxies back, add setReadUntilPosition to ReadBuffer 2021-10-18 18:35:11 +03:00
Nikolai Kochetov
6e479b301a Update memory optimisation for MergingSorted. 2021-10-18 12:54:12 +03:00
Anton Popov
71b474793c
Merge pull request #30174 from CurtizJ/better-interfaces 
Better interfaces for `IDataType` and `ISerialization`
 2021-10-18 12:07:04 +03:00
kssenii
3d39ea8e37 Fix WHERE 2021-10-18 11:06:30 +03:00
Azat Khuzhin
6d4af3bac1 Move SquashingTransform to Interpreters (to fix split build) 
clickhouse_common_io requires clickhouse_core:

    ld.lld: error: undefined symbol: DB::blocksHaveEqualStructure(DB::Block const&, DB::Block const&)
    >>> referenced by SquashingTransform.cpp:92 (/src/ch/clickhouse/src/Common/SquashingTransform.cpp:92)
    >>>               src/CMakeFiles/clickhouse_common_io.dir/Common/SquashingTransform.cpp.o:(void DB::SquashingTransform::append<DB::Block&&>(DB::Block&&))
    >>> referenced by SquashingTransform.cpp:92 (/src/ch/clickhouse/src/Common/SquashingTransform.cpp:92)
    >>>               src/CMakeFiles/clickhouse_common_io.dir/Common/SquashingTransform.cpp.o:(void DB::SquashingTransform::append<DB::Block const&>(DB::Block const&))

while clickhouse_core requires clickhouse_common_io:

    "clickhouse_core" of type SHARED_LIBRARY
      depends on "roaring" (weak)
      depends on "clickhouse_common_io" (weak)
      depends on "clickhouse_common_config" (weak)
      depends on "clickhouse_common_zookeeper" (weak)
      depends on "clickhouse_dictionaries_embedded" (weak)
      depends on "clickhouse_parsers" (weak)

Follow-up for: #30247 (cc @KochetovNicolai)
 2021-10-18 10:28:36 +03:00
alexey-milovidov
7beffb04ad
Merge pull request #29495 from ClickHouse/improvement_for_strange_mutations 
Minor improvement for strange mutations
 2021-10-17 20:57:06 +03:00
Nikolai Kochetov
2a95d7fc87
Merge pull request #29918 from amosbird/projection-improve5 
Add primary key to minmax_count_projection
 2021-10-17 14:04:18 +03:00
Nikolai Kochetov
bfcbf5abe0 Merge branch 'master' into removing-data-streams-folder 2021-10-17 10:42:37 +03:00
alexey-milovidov
4f11cfa59d
Merge pull request #28760 from azat/mutator-forbid-cleaner 
Forbid cleaning of tmp directories that can be used by an active mutation/merge.
 2021-10-17 03:18:01 +03:00
kssenii
4a0bdd296d Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-16 18:35:48 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
alesapin
9418eda122
Merge pull request #30111 from ClickHouse/remove_trash 
Remove trash from MergeTreeReadPool
 2021-10-16 15:41:41 +03:00
Amos Bird
9ceb668a4d
Fix TSan 2021-10-16 18:13:41 +08:00
Amos Bird
77c4a5fa18
Better 2021-10-16 18:13:41 +08:00
Amos Bird
77a2022cf5
Add primary key to minmax_count_projection 2021-10-16 18:13:41 +08:00
Nikolai Kochetov
067eaadadd Merge branch 'master' into removing-data-streams-folder 2021-10-16 09:46:05 +03:00
Nikolai Kochetov
c668696047
Merge pull request #30171 from ClickHouse/remove-stream-interfaces 
Remove stream interfaces
 2021-10-16 09:34:01 +03:00
Anton Popov
d2ba3bcaa4
Merge pull request #28363 from amosbird/fastload2 
Speed up part loading for JBOD
 2021-10-16 02:37:32 +03:00
Azat Khuzhin
07e8b2b3c7 Do not try to remove temporary paths that is currently in written by merge/mutation 
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
 2021-10-16 00:43:52 +03:00
Azat Khuzhin
0b3bf43d61 Rewrite MergeTreeData::clearOldTemporaryDirectories() to use early continue 2021-10-16 00:43:51 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
mergify[bot]
477ea62257
Merge branch 'master' into remove-sample-by 2021-10-15 15:14:17 +00:00
kssenii
759f747080 Fix build 2021-10-15 17:12:34 +03:00
kssenii
881ae8617e Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-15 15:09:56 +03:00
kssenii
edfb1a5521 Better performance for smaller number of threads 2021-10-15 15:07:39 +03:00
Nikita Mikhaylov
d1138a8a25
Merge pull request #29691 from azat/fix-merge-mutate-memory-tracking 
Fix memory tracking for merges and mutations
 2021-10-15 13:53:52 +03:00
Nikolai Kochetov
ad8a344b46 Move TTL streams and algo 2021-10-15 13:11:57 +03:00
Maksim Kita
7beaeb8b13
Merge branch 'master' into indexcache 2021-10-15 12:59:09 +03:00
kssenii
3995506d37 Adjust range reader for remote fs reads 2021-10-15 11:36:26 +03:00
Azat Khuzhin
2a7a1d8df5 Avoid losing any allocations context from merges/mutations 2021-10-15 01:43:28 +03:00
Azat Khuzhin
8a209a78d7 Set query_id for mutations/merges 
This will allow to distinguish allocations in trace_log.
 2021-10-15 01:43:28 +03:00
Azat Khuzhin
8cc45bea7b Avoid accounting memory from another mutation/merge 
Before this path it was possible that one merge/mutation may account
memory from another, due to ThreadStatus::untracked_memory.

And this cause flakiness of 01200_mutations_memory_consumption.
 2021-10-15 01:43:28 +03:00
Azat Khuzhin
fd38cbb0df Fix memory tracking for merges and mutations (by destroying earlier) 
It fixes only some tiny allocations, and so it should not affect any
huge mutations/merges.

And plus, this should not be a real fix, since peak_memory_usage is
obtained before even destrying this objects, and destroing objects will
unlikely update peak memory usage (although it is possible).

v0: do this in dtors
v2: do this earlier
 2021-10-15 01:43:27 +03:00
Azat Khuzhin
e4b39ca99c Remove unused Block member from MutateTask 
v0: Fix excessive memory usage in MutateTask
    This is relevant only for 01200_mutations_memory_consumption test, since
    this is just a block.
v2: just remove unused Block member, since other part had been fixed in #29768
 2021-10-15 01:43:27 +03:00
mergify[bot]
92dd528c78
Merge branch 'master' into remove_trash 2021-10-14 14:14:07 +00:00
Anton Popov
476d7a411f allow to remove SAMPLE BY expression 2021-10-14 17:05:20 +03:00
Anton Popov
c2d376d0d6
Merge branch 'master' into better-interfaces 2021-10-14 14:33:36 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Nikolai Kochetov
3d3e143c29
Merge pull request #30001 from ClickHouse/remove-streams-from-formats 
Remove streams from formats.
 2021-10-14 12:39:07 +03:00
Anton Popov
92413aed68 better interfaces for IDataType and ISerialization 2021-10-14 05:36:49 +03:00
Nikolai Kochetov
2957971ee3 Remove some last streams. 2021-10-13 21:22:02 +03:00
alesapin
1801ea4955 Merge branch 'master' into remove_trash 2021-10-13 18:10:31 +03:00
alesapin
d2dfbb5ab6 Remove trash from MergeTreeReadPool 2021-10-13 15:25:44 +03:00
Amos Bird
23602f4607
Speed up part loading for JBOD 2021-10-13 12:33:54 +08:00
Kseniia Sumarokova
bb15b57753
Merge pull request #30016 from ucasfl/sample-by-tuple 
Fix  crash of sample by tuple()
 2021-10-12 19:46:21 +03:00
Nikolai Kochetov
c36bb9206e
Merge pull request #29879 from amosbird/queryidholderfix 
Fix potential leak of query_id_holder
 2021-10-12 15:42:02 +03:00
Nikolai Kochetov
077aba4a97
Merge pull request #29520 from amosbird/projection-improve4 
Get rid of naming limitation of projections.
 2021-10-12 14:25:29 +03:00
Maksim Kita
91a48e9651
Merge pull request #29896 from kitaisreal/system-data-skipping-indices-added-size 
System table data skipping indices added size
 2021-10-12 12:16:57 +03:00
Maksim Kita
c15b67c182 Fix naming 2021-10-12 11:42:24 +03:00
alesapin
5ac0d36392
Merge pull request #30005 from ClickHouse/more_strict_check_for_intersecting_parts 
More strict check for intersecting parts
 2021-10-12 11:38:09 +03:00
feng lv
b5d69d599e fix sample by tuple() 
add tests

fix

fix

fix
 2021-10-12 05:19:32 +00:00
alexey-milovidov
f1b550cdaf
Merge pull request #29974 from ClickHouse/minimized-toolchain 
Experiment with minimized toolchain (2)
 2021-10-11 22:04:01 +03:00
Alexander Tokmakov
706e2b6b88 more strict check for intersecting parts 2021-10-11 21:42:46 +03:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
Alexey Milovidov
2370a8d6da Fix PVS-Studio 2021-10-11 14:28:46 +03:00
Amos Bird
72bccaa501
Fix path name 2021-10-11 19:12:08 +08:00
Maksim Kita
b0d887a0fe Added tests 2021-10-11 14:00:10 +03:00
Amos Bird
83717b7c3b
Get rid of naming limitation of projections. 2021-10-11 18:32:17 +08:00
Maksim Kita
ce0c41e1ad Fixed tests 2021-10-11 11:39:50 +03:00
Maksim Kita
2d069acc22 System table data skipping indices added size 2021-10-11 11:39:50 +03:00
Amos Bird
ebfb013ea1
Fix potential leak of query_id_holder 2021-10-11 15:56:00 +08:00
kssenii
448d63a0fc Range read with upper bound offset 2021-10-11 01:53:51 +03:00
kssenii
aa4b797808 Turn off tasks stealing for remote disk 2021-10-10 23:22:58 +03:00
Azat Khuzhin
6c33eaee32 Fix filtering by tuple (some conditions was lost during analyzing) 
Fixes: #29281
Fixes: test_cluster_copier/test_three_nodes.py::test
 2021-10-10 22:37:52 +03:00
Nikolai Kochetov
a95c28ec4b
Merge pull request #29898 from ClickHouse/remove-native-stream 
Remove some more streams.
 2021-10-10 21:01:16 +03:00
tavplubix
fa1396a9dd
Merge pull request #29790 from ClickHouse/fix-bad-cast-in-parser-create 
Fix bad cast in ParserCreateQuery
 2021-10-08 23:53:38 +03:00
Nikolai Kochetov
c6bce1a4cf Update Native. 2021-10-08 20:21:19 +03:00
Nikolai Kochetov
340b53ef85 Remove some more streams. 2021-10-08 17:03:54 +03:00
Nikolai Kochetov
9eddee5517
Merge pull request #29768 from ClickHouse/remove-merging-streams 
Remove some merging streams
 2021-10-08 13:15:10 +03:00
Maksim Kita
717998a37f
Merge pull request #29825 from azat/memory_profiler_step-cleaner-api 
Make memory_profiler_step API cleaner
 2021-10-07 22:44:47 +03:00
tavplubix
de2fd9d762
Merge pull request #29812 from ClickHouse/better_diagnostic_for_optimize 
Better diagnostic for OPTIMIZE
 2021-10-07 22:22:25 +03:00
Kseniia Sumarokova
185d58a7cc
Merge pull request #29766 from ClickHouse/remove_redundant_seeks 
Less seeks in compressed buffers
 2021-10-07 17:04:50 +03:00
Alexander Tokmakov
7e2bc184ec fix another suspicious places, add test 2021-10-07 16:43:49 +03:00
tavplubix
754e038eec
Merge pull request #29803 from ClickHouse/intersects-previous-part-usability 
Improve usability of error messages when error is caused by sophisticated interventions
 2021-10-07 15:24:17 +03:00
Alexander Tokmakov
5365cc686f Merge branch 'master' into improvement_for_strange_mutations 2021-10-07 13:26:58 +03:00
Alexander Tokmakov
4119848a41 better diagnostic for OPTIMIZE 2021-10-07 13:08:10 +03:00
Vladimir C
0422939893
Merge pull request #29281 from lingtaolf/bugfix/tupleFilter 2021-10-07 12:24:26 +03:00
Azat Khuzhin
71cffbf521 Make memory_profiler_step API cleaner 
Right now to configure memory_profiler_step/total_memory_profiler_step
you need to call:

    MemoryTracker::setOrRaiseProfilerLimit()
    MemoryTracker::setProfilerStep()

But it is easy to forget about setOrRaiseProfilerLimit(), since there is
no even any comments about this.

So instead, make setOrRaiseProfilerLimit() private and call it from
setProfilerStep()
 2021-10-07 10:52:16 +03:00
Maksim Kita
4ec7311d4d
Merge pull request #29738 from kitaisreal/added-function-ngram 
Added function ngram
 2021-10-07 10:21:00 +03:00
Maksim Kita
b7b39da8cc Fixed build 2021-10-06 23:22:29 +03:00
Nikolai Kochetov
d0c6f11fcb More. 2021-10-06 20:59:27 +03:00
Alexey Milovidov
1455cfeddd Improve usability of error messages when error is caused by sophisticated interventions 2021-10-06 15:28:52 +03:00
Maksim Kita
33353be4ed Fixed tests 2021-10-06 15:12:07 +03:00
Nikolai Kochetov
6b619512aa Fix style. 2021-10-06 12:37:56 +03:00
Maksim Kita
90eb74c933 Fixed tests 2021-10-06 00:37:45 +03:00
mergify[bot]
ab55225c8f
Merge branch 'master' into remove-merging-streams 2021-10-05 14:16:34 +00:00
Nikolai Kochetov
96d070a5ba Fix tests. 2021-10-05 16:58:24 +03:00
mergify[bot]
3b1f49aac7
Merge branch 'master' into nv/parts-uuid-move-shard-kill 2021-10-05 12:36:02 +00:00
alesapin
9aab0eb0a2 Fixup 2021-10-05 13:41:09 +03:00
alesapin
8186339307 Less seeks in compressed buffers 2021-10-05 12:11:25 +03:00
Alexander Tokmakov
bb32432943 more fixes 2021-10-05 00:13:18 +03:00
Nikolai Kochetov
2001ebbf9d Fix build. 2021-10-04 21:52:31 +03:00
Maksim Kita
3fe9fdac96 Added function ngram 2021-10-04 18:53:17 +03:00
Azat Khuzhin
117e9e77c8 Apply max_untracked_memory/memory_profiler_{step,sample_probability} during mutate/merge 2021-10-03 17:39:07 +03:00
Maksim Kita
395119339f Merge branch 'master' into rename-common 2021-10-02 17:15:10 +03:00
alexey-milovidov
86d1ce63cf
Merge pull request #29649 from CurtizJ/merging-29376 
Merging #29376
 2021-10-02 12:02:42 +03:00
Alexey Milovidov
2ad5dbdfe8 Rename "common" to "base" 2021-10-02 11:30:38 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Anton Popov
bb92ff52d8
Update IMergeTreeDataPart.cpp 2021-10-02 00:50:19 +03:00
Anton Popov
d404efbf7e add test for freeze in-memory parts 2021-10-01 22:05:36 +03:00
Anton Popov
f0363c524d Merge branch 'master' of git://github.com/mo-avatar/ClickHouse into merging-29376 2021-10-01 20:29:33 +03:00
Maksim Kita
57666725f9
Merge pull request #29623 from kitaisreal/bloom-filter-indexes-map-support-fix-unsafe-identifier-cast 
BloomFilter indexes map data type support fix unsafe identifier cast
 2021-10-01 19:28:04 +03:00
Nikita Mikhaylov
b2f7b6fae8
Fix race between MergeTask and storage destruction (#29614) 2021-10-01 18:55:04 +03:00
vdimir
7444d64156
Fix for recursive tuples in MergeTreeWhereOptimizer 2021-10-01 17:01:03 +03:00
vdimir
23ce41e3bb
MergeTreeWhereOptimizer supports constant tuple in both sides of equals 2021-10-01 17:01:03 +03:00
vdimir
64ffacd9e0
MergeTreeWhereOptimizer::tryAnalyzeTuple, fix fuzz 2021-10-01 17:01:02 +03:00
vdimir
e326b12a37
Fix style in MergeTreeWhereOptimizer.cpp 2021-10-01 17:01:02 +03:00
凌涛
f660224a5f
add tests 2021-10-01 17:01:01 +03:00
凌涛
ea68944055
fixed index didn't work in tuple where 2021-10-01 17:01:01 +03:00
Maksim Kita
c793c8e7fa BloomFilter indexes map data type support fix unsafe identifier cast 2021-10-01 14:40:00 +03:00
Alexey Milovidov
4271f8c738 Merge remote-tracking branch 'origin/master' into yandex-to-clickhouse-in-configs 2021-10-01 01:33:58 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
Nikolai Kochetov
68f8b9d235 Update ColumnGathererStream 2021-09-29 20:45:01 +03:00
mergify[bot]
6a2ac6e871
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-29 13:29:08 +00:00
Maksim Kita
0279afa738
Merge pull request #29474 from kitaisreal/bloom-filter-indexes-updated-tests 
Bloom filter indexes updated tests
 2021-09-28 22:40:27 +03:00
Maksim Kita
9ecd0a3f4f
Merge pull request #29220 from ClickHouse/full-text-bloom-filter-index-map-data-type 
Merging #28511
 2021-09-28 22:40:17 +03:00
alexey-milovidov
1e6f9ac635
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-28 20:26:44 +03:00
mo-avatar
6c16ec83eb in memory part lost when freeze 2021-09-28 22:52:01 +08:00
Maksim Kita
f2402e1a10 Fixed tests 2021-09-28 12:52:07 +03:00
Maksim Kita
71fb9e6059 Bloom filter indexes updated tests 2021-09-28 12:38:27 +03:00
Azat Khuzhin
9a23b20965 Remove debug from MergeTreeIndexConditionBloomFilter::traverseASTEquals() 2021-09-28 10:28:28 +03:00