thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
61,541 Commits 9,338 Branches 1,783 Tags 2.1 GiB
1c501e7d97
Commit Graph

1639 Commits

Author SHA1 Message Date
Pavel Kruglov
a437ee4e31 Merge branch 'master' of github.com:ClickHouse/ClickHouse into select-final 2021-01-29 14:25:47 +03:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
Anton Popov
031132038b fix filtering by uint8 greater than 127 2021-01-29 10:39:18 +03:00
alexey-milovidov
fa7a795c3a
Merge pull request #19588 from ClickHouse/disable-checksums-on-read 
Allow to disable checksums on read
 2021-01-29 02:37:26 +03:00
alesapin
2881c830e3 Merge branch 'master' into fix_rare_bug_after_part_corruption 2021-01-28 23:16:52 +03:00
Denis Glazachev
009d1e3fc5 Merge branch 'master' into row-policy-with-prewhere 
* master: (247 commits)
  Update 01014_lazy_database_basic.sh
  Update 00459_group_array_insert_at.sql
  Same for ALTER
  Syntax and links
  Added test.
  Check where and prewhere identifiers exist.
  Remove redundant lsof
  Add lsof to fasttest
  Add new line to status
  Make integration odbc tests idempotent
  Minor code improvement in JOIN
  Arcadia does not have bitmaps
  Make Fuzzer more reliable
  Add missing lsof for fasttest docker image
  Update test
  The test most likely would not work in Arcadia
  Less parser depth
  Fix stack overflow in coroutine
  Fuzzer: better messages.
  Fixed build.
  ...
 2021-01-28 21:48:12 +04:00
alesapin
be1104d6c1 Fix typos 2021-01-28 19:07:47 +03:00
alesapin
c1d18fc8a2 Fix very rare case 2021-01-28 18:10:53 +03:00
alesapin
a7ca26fec4
Merge pull request #19708 from ClickHouse/fix_flaky_test_multiple_ttl 
Fix flaky test test_concurrent_ttl_merges
 2021-01-28 15:26:56 +03:00
alesapin
d14f4a525b Merge branch 'master' into fix_rare_bug_after_part_corruption 2021-01-28 11:09:41 +03:00
alesapin
402e031a22 Throw exception only in debug mode 2021-01-28 11:07:18 +03:00
Alexey Milovidov
065d22cf2b Merge branch 'master' into disable-checksums-on-read 2021-01-28 03:07:31 +03:00
Anton Popov
e5125b8c73 add comments and test for compatibility 2021-01-28 02:39:15 +03:00
alesapin
c74631c650 Fix test and add logical error 2021-01-27 21:54:05 +03:00
Anton Popov
a8f3078ce9 Merge remote-tracking branch 'upstream/master' into HEAD 2021-01-27 19:48:55 +03:00
alesapin
5622e6daa6 Fix rare max_number_of_merges_with_ttl_in_pool limit overrun for non-replicated MergeTree 2021-01-27 14:56:12 +03:00
alesapin
6fc39b10d3 Spelling 2021-01-27 13:11:48 +03:00
alesapin
01c8b9e1b1 Fix rare bug when some replicated operations (like mutation) cannot process some parts after data corruption 2021-01-27 13:07:18 +03:00
Alexey Milovidov
48b4d98b21 Amend 2021-01-27 04:48:41 +03:00
Alexey Milovidov
a6f5f40a65 Merge branch 'master' into disable-checksums-on-read 2021-01-27 04:09:00 +03:00
Anton Popov
c7070da85a better abstractions in disk interface 2021-01-26 17:49:35 +03:00
Denis Glazachev
0d1c9479f8 Fix compilation/linter 2021-01-26 18:39:12 +04:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
tavplubix
2d6a71fced
Merge pull request #19443 from ClickHouse/fix_virtual_parts_in_parts_to_do 
Addition to #15537
 2021-01-26 11:27:14 +03:00
Amos Bird
66fe97d8bd
Per MergeTree table query limit 2021-01-26 14:03:31 +08:00
Alexey Milovidov
8dfa933028 Amend 2021-01-25 23:48:10 +03:00
Alexey Milovidov
9ee5c1535e Allow to disable checksums on read 2021-01-25 23:29:04 +03:00
Denis Glazachev
11b53d3b9d Fix compilation/linter 2021-01-26 00:17:48 +04:00
Denis Glazachev
ff5ce1a5ae Fix compilation 2021-01-25 22:01:59 +04:00
Anton Popov
658f24dcff
Merge pull request #19358 from CurtizJ/fix-subcolumns 
Fix several cases, while reading subcolumns
 2021-01-25 20:26:07 +03:00
Denis Glazachev
4f6c880232 Pass and handle a chain of multiple prewhere infos 2021-01-25 18:31:59 +04:00
tavplubix
5f07bfb9f8
Update ReplicatedMergeTreeQueue.cpp 2021-01-25 16:15:47 +03:00
Alexander Tokmakov
3bd4d97353 Merge branch 'master' into database_replicated 2021-01-25 14:19:04 +03:00
tavplubix
a88a564aae
Update ReplicatedMergeTreeQueue.cpp 2021-01-25 12:51:06 +03:00
alexey-milovidov
73501102f3
Merge pull request #19528 from azat/merge_tree_min_for_concurrent_read-SIGSEGV 
Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX
 2021-01-25 00:18:42 +03:00
Azat Khuzhin
1c364b6ee3 Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX 
In case of 0 or too huge value it will try to read not existing marks
and got:

    Logical error: 'Trying to get non existing mark 11936128518282651045, while size is 2'.
 2021-01-24 14:39:57 +03:00
Alexey Milovidov
7d2108d4e9 Fix double whitespace 2021-01-23 02:57:35 +03:00
Alexander Tokmakov
963dadae54 fix wrong parts in parts_to_do 2021-01-22 19:07:19 +03:00
alexey-milovidov
9818bd319a
Merge pull request #19381 from azat/parts-types-metrics 
Add metrics for MergeTree parts (Wide/Compact/InMemory) types
 2021-01-22 17:24:50 +03:00
alexey-milovidov
4e11e7cfa9
Merge pull request #18772 from azat/optimize-memory-tracking-fix 
[RFC] Fix memory tracking for OPTIMIZE TABLE/merges
 2021-01-22 03:48:06 +03:00
Azat Khuzhin
cb951c2116 Add metrics for MergeTree parts types 
- PartsWide
- PartsCompact
- PartsInMemory
 2021-01-21 21:17:00 +03:00
Pavel Kruglov
900580af02 Add parallel select when there is one part with level>0 in select final 2021-01-21 20:34:50 +03:00
Anton Popov
ac3de63a71 fix several cases, while reading subcolumns 2021-01-21 15:34:11 +03:00
Anton Ivashkin
357d98eb36 Merge master 2021-01-20 12:23:03 +03:00
Anton Ivashkin
eba98b04b0 Zero copy replication over S3: Hybrid storage support 2021-01-18 19:16:45 +03:00
Alexander Tokmakov
7f97a11c84 Merge branch 'master' into database_replicated 2021-01-18 17:09:39 +03:00
Pavel Kovalenko
1e3a059f64 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskCacheWrapper.cpp
#	src/Disks/S3/DiskS3.cpp
 2021-01-18 13:39:49 +03:00
Alexey Milovidov
029302d766 Merge with master 2021-01-16 17:09:44 +03:00
Alexey Milovidov
24c8e53440 Merge branch 'master' into multiple-nested 2021-01-16 16:28:40 +03:00
alexey-milovidov
4efc7a7dc3
Update MergeTreeIOSettings.h 2021-01-16 13:07:58 +03:00
alexey-milovidov
2e2988e5d8
Merge pull request #19146 from azat/server-memory-limit-blocking 
MemoryTracker: Do not ignore server memory limits during blocking by default
 2021-01-16 11:09:19 +03:00
alesapin
67fd381034
Merge pull request #19123 from ClickHouse/additional_check_in_writer 
Fix max granules size in MergeTreeDataWriter
 2021-01-16 10:17:44 +03:00
alexey-milovidov
5f189c5756
Merge pull request #19122 from ClickHouse/data-part-better-code 
Add metrics for part number in MergeTree in ClickHouse
 2021-01-16 00:20:15 +03:00
alexey-milovidov
e09ec9fed5
Merge pull request #18373 from amosbird/fix-18364 
Fix 2-arg functions with constant in PK analysis
 2021-01-16 00:20:03 +03:00
Azat Khuzhin
61b2d0ce42 MemoryTracker: Do not ignore server memory limits during blocking by default 2021-01-15 22:46:58 +03:00
alesapin
184dbedb06 Fix stupid error 2021-01-15 21:50:30 +03:00
alexey-milovidov
b97beea22a
Merge pull request #19101 from ClickHouse/check_compression_codec_read 
Fix compression codec read for empty files
 2021-01-15 20:55:58 +03:00
alexey-milovidov
971ff2ee0a
Merge pull request #19086 from ClickHouse/faster-parts-removal 
Faster parts removal, more safe and efficient interface of IDisk
 2021-01-15 20:37:35 +03:00
alesapin
8ccaa6ede9 Additional check for huge granules in MergeTreeDataWriter 2021-01-15 15:40:37 +03:00
Alexey Milovidov
e238fd64ac Add part metrics 2021-01-15 15:28:53 +03:00
Alexey Milovidov
6a2a5e53ed Slightly better code of IMergeTreeDataPart #18955 2021-01-15 15:15:13 +03:00
alexey-milovidov
78fff6bc39
Merge branch 'master' into multiple-nested 2021-01-15 14:54:27 +03:00
Alexey Milovidov
4bae04d500 Merge branch 'master' into amosbird/fix-18364 2021-01-15 14:37:35 +03:00
alexey-milovidov
4a71971b43
Update KeyCondition.cpp 2021-01-15 14:36:07 +03:00
alexey-milovidov
8d58ce532a
Merge pull request #19064 from CurtizJ/restrict-modify-ttl 
Restrict MODIFY TTL for tables created in old syntax
 2021-01-15 14:09:47 +03:00
Alexey Milovidov
d553e46a06 Merge branch 'master' into faster-parts-removal 2021-01-15 13:25:20 +03:00
alesapin
d601faa669
Merge pull request #18935 from fastio/bugfix_attach_partition_does_not_reset_mutation 
Bugfix: attach partition should reset the mutation
 2021-01-15 12:21:16 +03:00
alesapin
e106df2ad0 Fix comment 2021-01-15 12:10:03 +03:00
alesapin
0662d6bd7d Fix compression codec read for empty files 2021-01-15 12:04:23 +03:00
Alexey Milovidov
8276a1c8d2 Faster parts removal, more safe and efficient interface of IDisk 2021-01-14 19:24:13 +03:00
Anton Popov
ac426c3da6 restrict MODIFY TTL for tables created in old syntax 2021-01-14 15:32:20 +03:00
Anton Popov
0e903552a0 fix TTLs with WHERE 2021-01-13 17:04:27 +03:00
ygrek
8f2a830d83
add zstd long range option (#17184) 
* add zstd long compression option

* tests: add zstd long read-write test

Co-authored-by: Joris Giovannangeli <joris.giovannangeli@ahrefs.com>
Co-authored-by: ip <igor@ahrefs.com>
 2021-01-13 16:22:59 +03:00
alesapin
73e536a074
Merge pull request #18928 from ClickHouse/more_checks_in_writer_wide 
More checks in merge tree writer wide
 2021-01-13 09:59:45 +03:00
Anton Popov
d7200ee2ed minor changes 2021-01-13 02:20:32 +03:00
Pavel Kovalenko
b09862b7b9 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-12 20:18:40 +03:00
Anton Popov
15ead18673 Merge remote-tracking branch 'upstream/master' into HEAD 2021-01-12 19:46:10 +03:00
Anton Popov
60b88986bf minor changes near TTL computation 2021-01-12 19:42:49 +03:00
Anton Popov
58b9ef5a10 fix TTL info serialization 2021-01-12 17:14:47 +03:00
alexey-milovidov
2c71b997de
Merge pull request #18464 from hexiaoting/dev_fp_as_pk 
Disallow floating point as partition key
 2021-01-12 13:08:41 +03:00
alesapin
f3e55183ad Better test and check 2021-01-12 11:46:31 +03:00
alesapin
5ae31b8068 Slightly relax check 2021-01-12 10:59:14 +03:00
Anton Popov
5822ee1f01 allow multiple rows TTL with WHERE expression 2021-01-12 02:07:21 +03:00
Azat Khuzhin
05608687d6 Get back memory tracking blocker in calculateAndSerializePrimaryIndex() 
But reduce scope, to avoid leaking too much memory, since there are old
values in last_block_index_columns.

The scope of the MemoryTracker::BlockerInThread has been increased in #8290
 2021-01-12 01:00:37 +03:00
Azat Khuzhin
82edbfb581 Account query memory limits and sampling for OPTIMIZE TABLE/merges 2021-01-12 00:49:41 +03:00
Azat Khuzhin
bd05d9db2f Fix memory tracking for OPTIMIZE TABLE queries 
Because of BlockerInThread in
MergeTreeDataPartWriterOnDisk::calculateAndSerializePrimaryIndex memory
was accounted incorrectly and grows constantly.

And IIUC there is no need in that blocker, since INSERT SELECT shares
the same thread group.
 2021-01-12 00:49:40 +03:00
Pavel Kovalenko
0856b2c514 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-11 20:37:08 +03:00
fastio
8dde70b937 Bugfix: attach partition should reset the mutation 2021-01-11 21:26:43 +08:00
alesapin
c5df8f324c More checks in writer wide 2021-01-11 15:03:00 +03:00
Anton Popov
36ae0e4d35 Merge remote-tracking branch 'upstream/master' into HEAD 2021-01-11 13:51:12 +03:00
Nikolai Kochetov
ee094ed7fd
Merge pull request #18896 from amosbird/keyconditionalias 
correct index analysis of WITH aliases
 2021-01-11 11:13:28 +03:00
alexey-milovidov
571e37188a
Merge pull request #18886 from ClickHouse/remove-useless-code-4 
Remove useless code
 2021-01-10 13:13:17 +03:00
Amos Bird
44758935df
correct index analysis of WITH aliases 2021-01-10 17:40:47 +08:00
Alexey Milovidov
c38dca155c Fix clang-tidy 2021-01-10 05:51:54 +03:00
Alexey Milovidov
76149947ef Remove useless headers 2021-01-10 05:48:57 +03:00
Alexey Milovidov
6eb5a5f4d9 Remove useless code 2021-01-10 03:28:59 +03:00
alexey-milovidov
b2577327f1
Merge pull request #18881 from ClickHouse/remove-useless-headers-2 
Remove useless headers
 2021-01-09 11:01:07 +03:00
alexey-milovidov
b60a37afc7
Merge pull request #18870 from azat/storage-features 
Storage features improvements
 2021-01-09 10:57:45 +03:00
Alexey Milovidov
8a57f8f42a Remove useless headers 2021-01-09 08:32:10 +03:00
alexey-milovidov
6dcb887eed
Merge pull request #18857 from amosbird/fixmaxpartitionremotesetting 
Correctly override default settings remotely
 2021-01-08 22:25:43 +03:00
Azat Khuzhin
714d5a067a Expose supports_parallel_insert via system.table_engines 2021-01-08 14:57:24 +03:00
Azat Khuzhin
64cb6405ac Drop IStorage::supportsSettings() (replaced with StorageFeatures::supports_settings) 2021-01-08 14:36:26 +03:00
Amos Bird
a3d19fa64d
Correctly override default settings remotely 2021-01-08 12:28:09 +08:00
Azat Khuzhin
b1f08f5c27 Rename FileSyncGuard to DirectorySyncGuard 2021-01-07 20:26:18 +03:00
Azat Khuzhin
513a824f30 Fix fsync_part_directory for parts renames 2021-01-07 19:30:25 +03:00
alexey-milovidov
a08db94343
Revert "Add metrics for part number in MergeTree in ClickHouse" 2021-01-07 16:40:52 +03:00
Alexey Milovidov
36e1361cf8 Miscellaneous 2021-01-07 15:29:34 +03:00
alexey-milovidov
f91626e7ff
Merge pull request #17838 from weeds085490/dev/add_metrics_for_parts 
Add metrics for part number in MergeTree in ClickHouse
 2021-01-07 15:27:04 +03:00
alexey-milovidov
13e4579052
Merge pull request #18712 from amosbird/limitpartition 
add max_partitions_to_read setting
 2021-01-06 21:29:58 +03:00
alexey-milovidov
5b98767652
Update MergeTreeSettings.h 2021-01-06 21:29:45 +03:00
weeds085490
5f5b86b485 Merge remote-tracking branch 'origin' into dev/add_metrics_for_parts 2021-01-06 17:32:45 +08:00
Amos Bird
0260953a47
better 2021-01-06 17:18:48 +08:00
Amos Bird
a157a5b3b3
add max_partitions_to_read setting 2021-01-04 12:40:48 +08:00
Alexey Milovidov
c59deb4bca Allow Replicated tables in Arcadia 2021-01-03 19:20:09 +03:00
Alexey Milovidov
4b3ae495d6 Merge branch 'master' into CurtizJ-multiple-nested 2021-01-02 00:25:16 +03:00
Nikita Mikhaylov
4165a58ce4
Merge pull request #11617 from nikitamikhaylov/parallel-parsing-input-format 
Parallel formatting
 2020-12-31 21:22:16 +04:00
alexey-milovidov
c2f0451e39
Merge pull request #18362 from ClickHouse/allow_low_cardinality_key_alter 
Add ability to change some types for primary key
 2020-12-31 15:27:52 +03:00
Amos Bird
f93e30bed6
Fix warning 2020-12-31 11:06:15 +08:00
alexey-milovidov
9bc571eacc
Update KeyCondition.cpp 2020-12-30 17:58:43 +03:00
Nikita Mikhailov
cc794b7a83 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-30 16:55:15 +03:00
Nikita Mikhailov
60b4a36c4a arcadia fix + live view fix + cleanup 2020-12-30 07:50:58 +03:00
Nikita Mikhailov
2b90b4e01d Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-29 22:47:06 +03:00
Yatsishin Ilya
9b4b2fa441 fix typo 2020-12-29 17:57:44 +03:00
Yatsishin Ilya
a25fa1996b fix style checks 2020-12-29 17:57:12 +03:00
hexiaoting
e6c9361559 Add a MergeTree setting: allow_floating_point_partition_key 2020-12-29 11:12:02 +08:00
Nikita Mikhailov
c0eef84c50 better 2020-12-29 04:15:26 +03:00
Nikita Mikhailov
d293ae5e8e style 2020-12-28 20:18:53 +03:00
Nikita Mikhailov
c3288c3fbf Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-28 15:09:37 +03:00
Anton Popov
11283e3d81 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-25 21:25:59 +03:00
alesapin
0950c16fb2 Fix unexpected change 2020-12-25 18:49:29 +03:00
alesapin
2c1d443944 Control alter sequential execution more correctly 2020-12-25 18:47:17 +03:00
Anton Popov
b60c00ba74 refactoring of TTL stream 2020-12-25 18:46:13 +03:00
hexiaoting
a47e177b3e Allow float point as partition key when attach table 2020-12-25 11:05:16 +08:00
alexey-milovidov
491f481713
Merge pull request #18481 from ClickHouse/disable_write_with_aio 
Disable write with AIO even for big merges
 2020-12-25 04:24:43 +03:00
alesapin
5d94b7eec0 Remove non working code 2020-12-24 19:30:53 +03:00
Nikita Mikhaylov
6818b24fe5
Merge pull request #18381 from CurtizJ/part-types-merge 
Restrict merges from wide to compact parts
 2020-12-24 17:47:14 +03:00
Anton Popov
42725b27c0 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-24 15:18:15 +03:00
hexiaoting
b3b08a035e Disallow floating point as partition key 2020-12-24 18:30:20 +08:00
Yatsishin Ilya
ce1e2b18a3 Fix codespell warnings. Split style checks. Update style checks docker 2020-12-24 13:17:52 +03:00
alexey-milovidov
c4bbb95b0f
Merge pull request #18405 from ClickHouse/add_test_for_compact_parts 
Add test for already working code
 2020-12-24 01:24:09 +03:00
alesapin
f3f5466d23
Merge pull request #18306 from amosbird/restrictsettings 
Extend mergetree setting alter restriction
 2020-12-23 23:09:59 +03:00
Anton Popov
84ec3930ba
Merge pull request #18223 from ClickHouse/delayed_last_mark_for_wide_parts 
Better adaptive granularity calculation for wide parts
 2020-12-23 19:37:25 +03:00
Anton Popov
57857dda63 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-23 15:16:43 +03:00
alesapin
fb1c6c4eb4
Update MergeTreeSettings.h 2020-12-23 15:03:55 +03:00
alesapin
54455b4740 Add test for already working code 2020-12-23 14:53:49 +03:00
Anton Popov
518cac6150 fix for review 2020-12-23 13:11:24 +03:00
Alexander Kuzmenkov
51b2329295
Merge branch 'master' into aku/window-prototype 2020-12-23 09:26:11 +03:00
Amos Bird
8b5714b2ac
Fix 2-arg functions with constant in PK analysis 2020-12-23 12:29:29 +08:00
Nikita Mikhailov
dcfbe782c6 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-23 05:20:22 +03:00
roverxu
e38ecee30e fix style 2020-12-23 09:20:45 +08:00
Anton Popov
78c4b9f2bc do not allow merge from wide to compact parts 2020-12-23 03:16:15 +03:00
Anton Popov
5101804e30
Merge pull request #18366 from CurtizJ/remove-unused-code 
Remove unused code
 2020-12-23 02:45:33 +03:00
Anton Popov
1be39fddac fix subcolumns with some storages 2020-12-22 19:42:37 +03:00
Anton Popov
b6ff6300b2 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-22 18:06:21 +03:00
Anton Popov
40b5ffe3b5 minor renames 2020-12-22 18:06:12 +03:00
Anton Popov
371050a16e remove unused code 2020-12-22 15:22:54 +03:00
徐炘
81f9623e2e
Merge branch 'master' into dev/add_metrics_for_parts 2020-12-22 19:04:28 +08:00
alesapin
35413635e7 Add ability to change some types for primary key 2020-12-22 13:41:12 +03:00
flynn
4e580f7b7c
Add asynchronous metrics on MergeTree tables data volume (#17639) 
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-12-22 13:34:35 +03:00
root
d6a69ee164 remove code 2020-12-22 18:12:28 +08:00
Amos Bird
7a81e0a0f5
Extend mergetree setting alter restriction 2020-12-21 18:45:39 +08:00
roverxu
f3fef24d30 fix deadlock 2020-12-21 16:30:27 +08:00
alesapin
07b5c03c54 Better adjustment of the last mark 2020-12-21 11:24:52 +03:00
Alexander Kuzmenkov
85c1bc1253 Merge remote-tracking branch 'origin/master' into tmp 2020-12-21 10:46:21 +03:00
roverxu
8e72b6d31b Merge remote-tracking branch 'origin' into dev/add_metrics_for_parts 2020-12-21 09:25:54 +08:00
alesapin
23eebb3f90 Merge branch 'master' into delayed_last_mark_for_wide_parts 2020-12-20 23:45:44 +03:00
alesapin
0041c293ab Fix gcc-10 build 2020-12-20 11:01:39 +03:00
Alexey Milovidov
9be5fa9ef2 Merge branch 'master' into Enmk-Optimize_deduplicate 2020-12-20 09:57:10 +03:00
alesapin
ca6708e594 Fix check and granule adjust for non adaptive parts 2020-12-19 21:59:20 +03:00
Vasily Nemkov
e5ec81f7cd Single quotes around column names 2020-12-19 20:58:23 +02:00
alesapin
b2d53d802d Fix for non-adaptive granularity 2020-12-18 23:32:52 +03:00
Anton Popov
25ddd78976 fix nested 2020-12-18 23:09:34 +03:00
Nikolai Kochetov
d7fc426458
Merge pull request #18216 from ClickHouse/fix-18137 
Fix 18137
 2020-12-18 18:49:02 +03:00
alesapin
cd598034cd Fix add tests 2020-12-18 17:44:31 +03:00
alesapin
c1e7d5d3c1 Working code 2020-12-18 16:49:45 +03:00
Anton Popov
009efeff40 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-18 16:14:09 +03:00
Anton Popov
31e6956ac8 fix alters of nested 2020-12-18 15:29:57 +03:00
Vasily Nemkov
e166aae3f9 Using CSV-like strings for list of columns to deduplicate by instead of JSON-like notation. 2020-12-18 13:44:56 +02:00
alesapin
26637bd7ee
Merge pull request #17943 from ClickHouse/try_rewrite_writer 
Small simplification of MergeTreeDataWriter
 2020-12-18 12:41:16 +03:00
Nikolai Kochetov
eef58c85f0 Put thread back to set if it had stolen all tasks of backoff thread. 2020-12-18 11:33:55 +03:00
Alexander Kuzmenkov
5e19eaf2f0 Merge remote-tracking branch 'origin/master' into tmp 2020-12-18 03:49:59 +03:00
Nikita Mikhailov
6a55e018e4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-18 02:25:32 +03:00
alexey-milovidov
6fc3ca8b7b
Merge pull request #16724 from kitaisreal/added-function-accurate-cast-or-null 
Added function accurateCastOrNull, allow different types inside IN subquery
 2020-12-18 00:45:27 +03:00
alesapin
8140fbd0fb
Merge pull request #18200 from ClickHouse/more_logs_during_quorum 
More logs during quorum insert
 2020-12-18 00:22:19 +03:00
alexey-milovidov
dd78c9dc75
Merge pull request #17566 from ClickHouse/relax-too-many-parts-1 
Relax "Too many parts" threshold, part 1
 2020-12-17 20:44:44 +03:00
Anton Popov
a42b00c9aa Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-17 20:43:23 +03:00
Anton Popov
6de8b05b7c improvements in subcolumns reading 2020-12-17 20:10:33 +03:00
alesapin
aecdd02be4 Better message 2020-12-17 19:14:42 +03:00
alesapin
f45993cd1e More logs during quorum insert 2020-12-17 19:13:01 +03:00
alesapin
911a1cd48f Suppress gdb warning 2020-12-17 13:25:31 +03:00
alesapin
b74e7accf1 Review fixes 2020-12-17 11:17:39 +03:00
alesapin
f2957d6f05 Merge branch 'master' into try_rewrite_writer 2020-12-17 10:40:25 +03:00
Alexey Milovidov
b25f126569 Added a test 2020-12-17 04:52:27 +03:00
Alexey Milovidov
9a017b77df Merge branch 'master' into relax-too-many-parts-1 2020-12-17 04:40:39 +03:00
Alexander Kuzmenkov
a1326414ff Merge remote-tracking branch 'origin/master' into tmp 2020-12-17 00:46:14 +03:00
Alexander Kuzmenkov
fef24c81e3 some grammar 2020-12-17 00:44:05 +03:00
Anton Ivashkin
0f0500ca0c Merge master 2020-12-16 18:31:13 +03:00
alesapin
849db47f8a Better exception messages 2020-12-16 17:31:17 +03:00
roverxu
1065ddef65 fix bug when drop data 2020-12-16 22:02:34 +08:00
Alexey Milovidov
37363ae89d Comment update 2020-12-16 09:43:16 +03:00
Alexey Milovidov
4bd5d3b662 Comment update 2020-12-16 06:36:43 +03:00
Alexey Milovidov
88c5031dfa Add a commit #16595 2020-12-16 06:27:28 +03:00
alesapin
1bd2f49fb5 Merge branch 'master' into try_rewrite_writer 2020-12-15 19:05:04 +03:00
Nikolai Kochetov
6defcbb662
Merge branch 'master' into optimize-data-on-insert 2020-12-15 16:50:42 +03:00
alesapin
8670836573 Add some comments 2020-12-15 13:34:28 +03:00
alesapin
74c2211510 More clear 2020-12-15 12:54:48 +03:00
alesapin
a50187c995 Slightly better 2020-12-15 11:04:37 +03:00
Nikita Mikhaylov
5a47928431 save 2020-12-15 00:56:47 +03:00
Nikita Mikhaylov
f40f3ced2a fix JSONEachRowArray 2020-12-15 00:56:46 +03:00
Nikita Mikhaylov
e0addac6fc save changes 2020-12-15 00:56:46 +03:00
Maksim Kita
18dc118298 Fixed compile issues 2020-12-14 22:12:15 +03:00
Maksim Kita
0464859cfe Updated usage of different types during IN query 
1. Added accurateCast function.
2. Use accurateCast in Set during execute.
3. Added accurateCast tests.
4. Updated select_in_different_types tests.
 2020-12-14 22:12:15 +03:00
Maksim Kita
f4b8e8ef99 Allow different types inside IN subquery 2020-12-14 22:12:15 +03:00
alesapin
34d1ad1aca Fix writer compact 2020-12-14 17:27:39 +03:00
alesapin
9e98cdad5a Accident changes 2020-12-14 16:05:07 +03:00
alesapin
47e3f619e5 Remove redundant code 2020-12-14 16:01:01 +03:00
alesapin
a87fb3fae0 Fix style 2020-12-14 15:52:15 +03:00
alesapin
303c12d2e4 Slightly better 2020-12-14 15:51:14 +03:00
alesapin
6fb23da851 Refactoring 2020-12-14 15:03:49 +03:00
alesapin
70be03aa7d Working compact parts 2020-12-14 14:36:15 +03:00
alesapin
fbcf85cb42 Something working for wide parts 2020-12-14 14:06:02 +03:00
alesapin
12b6a54a24 Merge branch 'master' into try_rewrite_writer 2020-12-14 10:30:01 +03:00
alesapin
fd695c9ff4 Dive into debug 2020-12-14 10:28:42 +03:00
Azat Khuzhin
5b3ab48861 More forward declaration for generic headers 
The following headers are pretty generic, so use forward declaration as
much as possible:
- Context.h
- Settings.h
- ConnectionTimeouts.h
(Also this shows that some missing some includes -- this has been fixed)

And split ConnectionTimeouts.h into ConnectionTimeoutsContext.h (since
module part cannot be added for it, due to recursive build dependencies
that will be introduced)

Also remove Settings from the RemoteBlockInputStream/RemoteQueryExecutor
and just pass the context, since settings was passed only in speicifc
places, that can allow making a copy of Context (i.e. Copier).

Approx results (How much units will be recompiled after changing file X?):

- ConnectionTimeouts.h
  - mainline: 100

- Context.h:
  - mainline: ~800
  - patched:  415

- Settings.h:
  - mainline: 900-1K
  - patched:  440 (most of them because of the Context.h)
 2020-12-12 17:43:10 +03:00
alesapin
a24a76828e Fix 2020-12-11 18:18:51 +03:00
alesapin
4b83707c63 Compileable code for some reason 2020-12-11 18:00:58 +03:00
alesapin
57c30c4164 Garbage 2020-12-11 16:20:19 +03:00
alesapin
7ffd76a605 Fix bug 2020-12-11 11:41:02 +03:00
alesapin
cbba73672e Merge branch 'master' into try_rewrite_writer 2020-12-11 11:09:47 +03:00
Alexander Kuzmenkov
d757f32300 Remove some redundant includes to speed up build 2020-12-11 02:58:11 +03:00
alexey-milovidov
c35c52be87
Merge pull request #17867 from ucasFL/mergetree-setting 
Add settings `min_compress_block_size` and `max_compress_block_size` to MergeTreeSettings
 2020-12-10 22:54:05 +03:00
alexey-milovidov
f1ee1ef16a
Update MergeTreeSettings.h 2020-12-10 22:50:54 +03:00
alesapin
1cb06bd975 Split one method 2020-12-10 19:29:10 +03:00
alesapin
90fa9a2073 Merge branch 'master' into try_rewrite_writer 2020-12-10 18:36:10 +03:00
alesapin
f3d1dc360c Fix style 2020-12-10 12:24:52 +03:00
alesapin
0704b478ee Less constuctors 2020-12-10 12:22:43 +03:00
alesapin
2787b615b0 Make interface as simple as possible 2020-12-10 11:57:52 +03:00
alesapin
fef65b0cbd
Merge pull request #17919 from ClickHouse/additional_size_check 
Add additional columns size check for MergeTree in debug mode
 2020-12-09 22:40:06 +03:00
alesapin
7979b3655c Better formating 2020-12-09 21:22:07 +03:00
alesapin
3b88c077ed Mechanical refactoring 2020-12-09 21:19:49 +03:00
alesapin
af73db93e6 Mechanical refactoring 2020-12-09 21:10:09 +03:00
Pavel Kruglov
c8c543ca94 Add throwing exceptions 2020-12-09 18:07:58 +03:00
alesapin
b307e545a9 Fix check 2020-12-09 14:46:04 +03:00
alesapin
0f4056fd95 Add additional size check in debug mode 2020-12-09 14:23:37 +03:00
alesapin
7783ddb3af
Revert "Fix index granularity calculation on block borders" 2020-12-09 14:05:48 +03:00
Vasily Nemkov
59fc301344 Fixed test to be less flaky 
Also logging expanded list of columns passed from `DEDUPLICATE BY` to actual deduplication routines.
 2020-12-08 19:44:34 +03:00
Kruglov Pavel
e19eb6f17a
Merge branch 'master' into optimize-data-on-insert 2020-12-08 15:57:46 +03:00
Vasily Nemkov
8c5daf0925 Fixed building tests with GCC-10 
Also a minor cleanup of the test code.
 2020-12-08 13:15:18 +03:00
Anton Popov
06d5b87bc9 fix nested and subcolumns 2020-12-07 22:02:26 +03:00
Anton Popov
d7aad3bf79 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-07 19:13:01 +03:00
Vasily Nemkov
168155eeec Minor: cleanup 2020-12-07 18:07:40 +03:00
Vasily Nemkov
dbdc018ab8 Fixed and refined unite-test 
* no more undefined values for attributes in ReplicatedMergeTreeLogEntry
* validation of string serialization format
 2020-12-07 17:38:25 +03:00
Anton Popov
66e0add2ba fix nested 2020-12-07 16:35:12 +03:00
feng lv
50b64ba1f8 fix 2020-12-07 13:18:31 +00:00
feng lv
d7b30f74d1 add min-max_compress_block_size to mergetree settings 2020-12-07 12:47:51 +00:00
Vasily Nemkov
70ea507dae OPTIMIZE DEDUPLICATE BY columns 
Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on.

Following syntax variants are now supported:

OPTIMIZE TABLE table DEDUPLICATE; -- the old one
OPTIMIZE TABLE table DEDUPLICATE BY *;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT (colX, colY);
OPTIMIZE TABLE table DEDUPLICATE BY col1,col2,col3;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex');
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT (colX, colY);

Note that * behaves just like in SELECT: MATERIALIZED, and ALIAS columns are not used for expansion.
Also, it is an error to specify empty list of columns, or write an expression that results in an empty list of columns, or deduplicate by an ALIAS column.
Column transformers other than EXCEPT are not supported.
 2020-12-07 09:44:07 +03:00
feng lv
a634e4ed80 fix incorrect initialize MergeTreeWriterSettings 2020-12-06 08:57:02 +00:00
alesapin
0eec52b1fd
Merge pull request #17737 from ClickHouse/fix_segfault_in_distributed_out_stream 
Fix segfault when 'not enough space'
 2020-12-05 16:14:30 +03:00
Anton Popov
d1ef6a897e
Merge pull request #17802 from detailyang/patch-1 
hotfix:check in_memory_parts_enable_wal
 2020-12-05 03:05:38 +03:00
Anton Popov
60b0cbb1c1
Merge pull request #15939 from Avogar/optimize_final_optimization 
Optimize final optimization
 2020-12-05 02:26:27 +03:00
Pavel Kruglov
4b58528b9e Rename getPos to getRow, change mergeBlock, pass setting instead of context 2020-12-04 19:25:30 +03:00
Pavel Kruglov
905ba78adc Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-12-04 18:56:46 +03:00
detailyang
e7a151fd1e
hotfix:check in_memory_parts_enable_wal 2020-12-04 23:14:27 +08:00
alesapin
42544a9833 Merge branch 'master' into fix_segfault_in_distributed_out_stream 2020-12-04 17:41:10 +03:00
Pavel Kruglov
9dbced0474 Pass setting instead of context 2020-12-04 17:01:59 +03:00
roverxu
2a6c1463a1 Add metrics for part number in MergeTree 2020-12-04 11:42:58 +08:00
Ivan
315ff4f0d9
ANTLR4 Grammar for ClickHouse and new parser (#11298) 2020-12-04 05:15:44 +03:00
Anton Popov
cd1917c7a6
Merge branch 'master' into optimize_final_optimization 2020-12-03 16:52:51 +03:00
Alexander Tokmakov
bfbf150c67 fix segfault when 'not enough space' 2020-12-02 17:49:43 +03:00
Alexander Tokmakov
19c8399eb0 Merge branch 'master' into database_replicated 2020-12-01 17:33:07 +03:00
Alexey Milovidov
6590fdfdd2 Fix logging 2020-11-30 11:44:01 +03:00
alesapin
25f40db2fb
Merge pull request #17499 from ClickHouse/concurrent_mutation_and_random_kill 
Fix kill mutation on concurrent alter queries
 2020-11-30 10:51:50 +03:00
Alexey Milovidov
2e6bedce91 Relax "Too many parts" threshold, part 1 2020-11-29 18:08:02 +03:00
alexey-milovidov
fabceebbce
Merge pull request #17145 from amosbird/cddt 
Fix unmatched type comparison in KeyCondition
 2020-11-29 14:29:35 +03:00
alexey-milovidov
f4a61ac3c3
Merge pull request #17527 from ucasFL/spelling 
fix spelling errors
 2020-11-29 13:45:42 +03:00
feng lv
7e3524caa1 fix spelling errors 2020-11-28 08:17:20 +00:00
alexey-milovidov
c189f6405f
Merge pull request #16767 from azat/optimize_trivial_count_query-fix 
Fix optimize_trivial_count_query with partition predicate
 2020-11-28 08:43:12 +03:00
Nikita Mikhaylov
6f3db7ff50
Merge pull request #17330 from CurtizJ/move-to-prewhere 
Allow to move conditions to prewhere with compact parts
 2020-11-28 02:02:42 +03:00
alesapin
6567796014 Fix kill mutation on concurrent alter queries 2020-11-27 18:46:52 +03:00
Alexander Tokmakov
9e3fd3c170 Merge branch 'master' into database_replicated 2020-11-27 17:08:34 +03:00
Anton Popov
1a4ed07a98 add some more comments 2020-11-27 15:47:27 +03:00
alexey-milovidov
dfae1efbbd
Merge pull request #17070 from fastio/master 
Support multiple ZooKeeper clusters
 2020-11-27 10:38:01 +03:00
Amos Bird
022ba2b0a9
Fix unmatched type comparison in KeyCondition 2020-11-26 16:15:50 +08:00
alesapin
960d077612 Better functions 2020-11-26 10:25:57 +03:00
alexey-milovidov
75a78e6c20
Update BackgroundJobsExecutor.h 2020-11-26 07:09:05 +03:00
Azat Khuzhin
0b47f4a9e9 Fix optimize_trivial_count_query with partition predicate 
Consider the following example:

    CREATE TABLE test(p DateTime, k int) ENGINE MergeTree PARTITION BY toDate(p) ORDER BY k;
    INSERT INTO test VALUES ('2020-09-01 00:01:02', 1), ('2020-09-01 20:01:03', 2), ('2020-09-02 00:01:03', 3);

- SELECT count() FROM test WHERE toDate(p) >= '2020-09-01' AND p <= '2020-09-01 00:00:00'
  In this case rpn will be (FUNCTION_IN_RANGE, FUNCTION_UNKNOWN (due to strict), FUNCTION_AND)
  and for optimize_trivial_count_query we cannot use index if there is at least one FUNCTION_UNKNOWN.
  since there is no post processing and return count() based on only the first predicate is wrong.

  Before this patch FUNCTION_UNKNOWN was allowed for optimize_trivial_count_query, and the result was wrong.

And two examples above just to show the difference, the behaviour hadn't been changed with this patch:

- SELECT * FROM test WHERE toDate(p) >= '2020-09-01' AND p <= '2020-09-01 00:00:00'
  In this case will be (FUNCTION_IN_RANGE, FUNCTION_IN_RANGE (due to non-strict), FUNCTION_AND)
  so it will prune everything out and nothing will be read.

- SELECT * FROM test WHERE toDate(p) >= '2020-09-01' AND toUnixTimestamp(p)%5==0
  In this case will be (FUNCTION_IN_RANGE, FUNCTION_UNKNOWN, FUNCTION_AND)
  and all, two, partitions will be scanned, but due to filtering later none of rows will be matched.
 2020-11-25 23:09:17 +03:00
tavplubix
6477251ea1
Update BackgroundJobsExecutor.h 2020-11-25 23:06:00 +03:00
Alexander Tokmakov
5ccf975a4a Merge branch 'master' into database_replicated 2020-11-24 14:38:02 +03:00
Anton Popov
852a08eacb allow to move conditions to prewhere with compact parts 2020-11-23 21:27:59 +03:00
Anton Popov
b384beb564 Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-23 17:46:51 +03:00
Amos Bird
172b7e9ed1
global in set index. 2020-11-23 22:05:08 +08:00
tavplubix
5cc9cb01cd
Merge pull request #16751 from amosbird/globalcontext 
Make global_context consistent.
 2020-11-22 18:46:17 +03:00
alesapin
9c8b0da382
Merge pull request #16033 from nvartolomei/nv/parts-uuid 
Add unique identifiers IMergeTreeDataPart structure
 2020-11-22 16:13:19 +03:00
Anton Popov
1f65478b49 implement nested with multiple nesting 2020-11-21 20:27:42 +03:00
Pavel Kruglov
623dc2df7a Update comment 2020-11-20 17:32:39 +03:00
Pavel Kruglov
ca3fe49a2a Make setting global 2020-11-20 17:29:13 +03:00
Nicolae Vartolomei
7c8bc1c04e Use JSON metadata in WAL 2020-11-20 13:49:17 +00:00
Nicolae Vartolomei
040aba9f85 Add uuid.txt to checksums for parts stored on disk 
We are breaking backwards compatibility anyway (but agted by a setting)
 2020-11-20 13:49:17 +00:00
Nicolae Vartolomei
94293ca3ce Assign UUIDs to parts only when configured to do so 
Avoid breaking backwards compatibility by default for now.
 2020-11-20 13:49:17 +00:00
Pavel Kruglov
e5588b94aa Fix style and tests 2020-11-20 13:38:53 +03:00