thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
65,045 Commits 9,328 Branches 1,783 Tags 2.1 GiB
9915bd0a8b
Commit Graph

1903 Commits

Author SHA1 Message Date
Azat Khuzhin
940ff6cb76 Do not use c-style cast for std::min(max_part_removal_threads) 2021-04-11 12:22:39 +03:00
Azat Khuzhin
6e71757814 Fix thread accounting for max_part_removal_threads 2021-04-11 11:43:30 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Azat Khuzhin
a6703c5ac2 Fix inactive_parts_to_throw_insert=0 with inactive_parts_to_delay_insert>0 2021-04-10 09:57:08 +03:00
Nikolai Kochetov
28ca191102
Merge pull request #22087 from ClickHouse/better-filter-push-down 
Better filter push down
 2021-04-09 10:22:17 +03:00
alesapin
4b0f973f93
Merge pull request #18215 from filimonov/quorum-dedup-issue 
Fix issue with quorum retries behaviour
 2021-04-09 10:19:50 +03:00
Anton Popov
1c28878f5d
Merge pull request #22726 from azat/mt-reverse-total-rows-approx-fix 
Fix approx total rows accounting for reverse reading from MergeTree
 2021-04-09 00:16:49 +03:00
Alexander Tokmakov
3422bd1742 check parts visibility for select 2021-04-08 20:20:45 +03:00
Nikolai Kochetov
794d7c89b6 Remove ReverseRowsStep 2021-04-08 16:18:17 +03:00
Nikolai Kochetov
febb187da0 Added test, 2021-04-08 15:54:27 +03:00
Nikolai Kochetov
7ffbeac9df Add info about indexes to ReadFromMergeTree step. 2021-04-08 14:48:54 +03:00
alesapin
e762d02e37 More const 2021-04-08 13:38:40 +03:00
alesapin
e17444757b Fix rare case when quorum insert is not really quorum because of deduplication 2021-04-08 13:35:38 +03:00
Nikolai Kochetov
7c5a9133df Add index info to ReadFromStorageStep. 2021-04-08 11:19:04 +03:00
alesapin
9fd251eaaf
Merge pull request #22514 from ClickHouse/merge_tree_deduplication 
Non replicated merge tree deduplication
 2021-04-07 10:19:43 +03:00
Azat Khuzhin
60b000ed51 Fix approx total rows accounting for reverse reading from MergeTree 2021-04-06 21:13:51 +03:00
Azat Khuzhin
f157278b72 Safer SCOPE_EXIT 
It executes the code in the dtor, that should never throw.
 2021-04-06 21:10:37 +03:00
Nikolai Kochetov
9f39f5d52d Add more counters to MergeTreeDataSelectExecutor 2021-04-06 15:39:55 +03:00
Nikolai Kochetov
6102652c99 Merge branch 'master' into better-filter-push-down 2021-04-06 13:38:03 +03:00
alesapin
53a7836a9b Fix typos 2021-04-06 13:16:34 +03:00
alesapin
6456a1507c Support alter setting 2021-04-06 13:14:44 +03:00
Alexander Tokmakov
f596906f47 add transaction counters 2021-04-05 14:41:34 +03:00
alesapin
156ba8a7fc Disable setting by default and ban for old-syntax MergeTree 2021-04-05 12:16:38 +03:00
alesapin
8d5a787f6b Merge branch 'master' into merge_tree_deduplication 2021-04-05 10:40:03 +03:00
alexey-milovidov
b1e7624b3f
Merge pull request #22589 from ClickHouse/fix-comments 
Fix comments
 2021-04-05 00:22:27 +03:00
Alexey Milovidov
54caa61af5 Fix comments 2021-04-04 12:33:06 +03:00
Alexey Milovidov
8b5d0a598f Minor improvement in index deserialization 2021-04-04 12:17:54 +03:00
alesapin
548ec6d2cb Trying enabled setting 2021-04-04 11:02:09 +03:00
alesapin
1906c77884 Fix tidy one more time 2021-04-03 17:57:03 +03:00
alesapin
0204f5dd35 Merge branch 'master' into merge_tree_deduplication 2021-04-03 15:24:26 +03:00
alesapin
b127e31451 CI fixes 2021-04-03 14:56:56 +03:00
alexey-milovidov
c3a9cbe01c
Merge pull request #22421 from ClickHouse/fix-buffer-overflow-in-token-extractor 
Fix buffer overflow in token extractor
 2021-04-02 22:08:28 +03:00
alesapin
b884593186 Followup fix 2021-04-02 20:55:05 +03:00
alesapin
ca30c91868 More comments 2021-04-02 20:54:24 +03:00
alesapin
fc3afe1087 Fix typos 2021-04-02 20:51:57 +03:00
alesapin
b105153d2c Useful comment 2021-04-02 20:48:58 +03:00
alesapin
ab34873344 Add some comments 2021-04-02 20:46:01 +03:00
alesapin
a555d078a2 Add exception handling 2021-04-02 19:56:02 +03:00
alesapin
759dd79c76 Add tests 2021-04-02 19:45:18 +03:00
alesapin
6f36661575 Buildable code 2021-04-02 16:34:36 +03:00
tavplubix
c1ab9de720
Update MergeTreeDataPartChecksum.cpp 2021-04-02 15:57:30 +03:00
alesapin
2ab87d61eb Some build fixes 2021-04-02 15:37:42 +03:00
alesapin
14634689e5 Idea code 2021-04-02 14:46:42 +03:00
alesapin
4eaf585fe5 Almost nothing added 2021-04-01 22:49:51 +03:00
Mike Kot
c947280dfc Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-04-01 21:38:51 +03:00
alesapin
7c8f54e694 More changes 2021-04-01 11:07:56 +03:00
Alexey Milovidov
7db8fcdebd Fix buffer overflow in TokenExtractor #19233 2021-04-01 05:37:19 +03:00
alexey-milovidov
bbd41d64c2
Update DataPartsExchange.cpp 2021-04-01 02:59:47 +03:00
alexey-milovidov
538be4086b
Recursive removal is extremely dangerous. 2021-04-01 02:59:23 +03:00
alesapin
c15d7e009d Some initial code 2021-03-31 18:20:30 +03:00
Nicolae Vartolomei
5cd1dc485f Delete temporary fetch directory if it already exists 
It could be left there after a network timeout, a system crash, or
checksum issues.

The downside of this PR is that it would be harder to debug checksum
issues if replication will continuously delete and try to fetch the same
part.
 2021-03-31 14:14:58 +01:00
tavplubix
6c1e2a0d87
Merge pull request #21690 from ClickHouse/test_multiple_nodes 
Run three nodes with Replicated database and NuKeeper in CI
 2021-03-31 12:31:13 +03:00
Alexey Milovidov
e9bfc7456d Merge branch 'master' into virtualutil 2021-03-31 08:19:43 +03:00
alexey-milovidov
5c15b8a896
Merge branch 'master' into virtualutil 2021-03-30 23:59:44 +03:00
Anton Ivashkin
f87435213e Fix S3 Zero-Copy replication for hybrid storage 2021-03-30 20:38:04 +03:00
Alexander Tokmakov
4f896d2b07 fix tests 2021-03-30 19:46:23 +03:00
Nikolai Kochetov
8fad179800 Fix some tests. 2021-03-30 18:14:20 +03:00
Nikolai Kochetov
88bfa6c9ba Add ReadFromMergeTree step. 2021-03-30 13:25:26 +03:00
Nikolai Kochetov
78e27dc1b6
Merge pull request #22270 from liuneng1994/fix-query-failed 
fix constant condition conflict with column name and add stateless test
 2021-03-30 09:37:43 +03:00
Alexey Milovidov
230b2930b7 Merge branch 'master' into virtualutil 2021-03-30 00:43:00 +03:00
alexey-milovidov
1512ff832e
Merge branch 'master' into nv/replicated-fetches-timeouts 2021-03-29 19:25:13 +03:00
Anton Popov
ea82e7725f
Merge pull request #21562 from CurtizJ/serialization-refactoring-4 
Refactoring of data types serialization
 2021-03-29 16:36:44 +03:00
Neng Liu
0d78427fbf fix query failed and add stateless test 2021-03-29 16:12:02 +08:00
alexey-milovidov
7ab2890c4d
Merge pull request #22206 from ClickHouse/mmap-cache 
Add cache for mmap IO
 2021-03-29 05:17:55 +03:00
alexey-milovidov
c08289517a
Merge pull request #21766 from amosbird/indexfix 
Fix scalar subquery index analysis
 2021-03-29 02:03:27 +03:00
Alexey Milovidov
2a8ac01cdb Rename as suggested by Kita 2021-03-28 22:24:28 +03:00
Alexey Milovidov
6f9f7d118d Fixes 2021-03-28 04:27:26 +03:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
Alexander Kuzmenkov
a455762933
Merge pull request #22117 from ucasFL/useless-code 
remove useless code
 2021-03-26 17:18:14 +03:00
alexey-milovidov
f895bc895c
Merge pull request #22011 from ClickHouse/min_max_time_system_parts_datetime64 
Expose DateTime64 minmax part index in system.parts and system.parts_columns
 2021-03-25 16:02:33 +03:00
Anton Popov
6a15431be7 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-25 15:57:35 +03:00
Anton Popov
f58d470dd6
Merge pull request #21830 from foolchi/optimize_select_with_final 
optimize select final with prewhere primary key
 2021-03-25 15:32:13 +03:00
feng lv
b7622868fc remove useless code 2021-03-25 11:10:41 +00:00
Nikolai Kochetov
0e2f52518f Extract converting from UnionStep. 2021-03-25 12:57:14 +03:00
Mike Kot
285af08949 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-24 22:34:20 +03:00
Mike Kot
9f25ee3719 Replaced removing checksums.txt with version 
with the explicit data loading and verification.

If the function fails, the exception will re-throw upper,
cancelling the fetch in the handling of the replicated log entry,
but this event will also wake the PartCheckThread, which will
issue a re-fetch.
 2021-03-24 22:30:16 +03:00
alexey-milovidov
ffa6585170
Merge pull request #21837 from Jokser/disk-cache-cleanup-directory-before-move 
Cleanup destination directory during DiskCacheWrapper::moveFile()
 2021-03-24 21:19:48 +03:00
fuqi
487d6bdcd9 add test case 
fix order key check
 2021-03-24 16:36:45 +08:00
alexey-milovidov
612d4fb073
Update IMergeTreeDataPart.cpp 2021-03-24 02:03:14 +03:00
alesapin
36898bdc4a
Merge pull request #21142 from Jokser/unfreeze-partitions 
Add ALTER TABLE UNFREEZE command.
 2021-03-23 14:05:38 +03:00
Pavel Kovalenko
a92cf30b67 Code review fixes. 2021-03-23 13:33:07 +03:00
Alexey Milovidov
8d0210b510 Expose DateTime64 minmax part index in system.parts and system.parts_columns #18244 2021-03-23 01:16:41 +03:00
Alexey Milovidov
ce67afd9e3 Fix bad code #18244 2021-03-23 00:49:18 +03:00
Mike Kot
c55a73b752 Added the solution to handle the corruption case 
When the part data (e.g. data.bin) is corrupted, but the checksums.txt
is present -- explicitly deleting the checksums.txt.

Removed the extra logging, changes some exceptions message.
 2021-03-22 17:23:43 +03:00
Mike Kot
5789507e8b Investigating, why the checksums may match when they shouldn't. 2021-03-22 17:23:43 +03:00
Nikolai Kochetov
c3c393a7aa Merge branch 'master' into refactor-actions-dag 2021-03-18 14:33:07 +03:00
fuqi
cfa92f0045 fix code style 2021-03-18 00:25:43 +08:00
fuqi
b8edc12812 fix code style 2021-03-17 23:56:55 +08:00
Mike Kot
e309dd275a Trying to investigate the errors after attaching the part. 
+ A new entry is added to the part_log.
 2021-03-17 18:49:04 +03:00
fuqi
e116e8d5e8 fix bug 2021-03-17 22:19:10 +08:00
Mike Kot
831b90f272 Replaced the iteration algorithm in part finder 
Now, instead of iterating through the directories, we iterate though
directories of on of the table disks (which doesn't give us a
substantial boost but is a bit neater to read).

- Updated the system.replication_queue command types.
- Fixed the part ptr being empty (added the checksum loading and
initialization).
- Removed extra logging.
 2021-03-17 16:55:26 +03:00
Mike Kot
2ccdb7ef5c Multiple small code and test updates 
- Updated the docs to make everything clear.
- Multiple small logger fixes.
- Changed the attach_part command -- now it's after check for the
covering parts -- motivation is to do less work with the checksums
fetching.
- Better logging in the integration test.
 2021-03-17 16:52:35 +03:00
fuqi
9ab713c2e1 optimize select final with prewhere primary key 2021-03-17 18:57:58 +08:00
fuqi
5e20ea2c33 optimize select final with prewhere primary key 2021-03-17 18:49:24 +08:00
Pavel Kovalenko
5627ae9067 Make MergeTreeData::getPartitionMatcher const 2021-03-17 12:12:55 +03:00
alexey-milovidov
d02726bcac
Merge pull request #9404 from Enmk/DateTime64_extended_range 
Date time64 extended range
 2021-03-17 11:06:03 +03:00
Anton Popov
6247cd5503 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-16 16:50:15 +03:00
Pavel Kovalenko
a43c85b105 Add test for freeze/unfreeze partition with old MergeTree syntax 2021-03-16 16:36:05 +03:00
Alexey Milovidov
ea049b2cc0 Merge branch 'master' into DateTime64_extended_range 2021-03-16 14:08:02 +03:00
Pavel Kovalenko
39f3b17041 Merge remote-tracking branch 'origin/master' into unfreeze-partitions 2021-03-16 10:56:24 +03:00
alesapin
8fd3fbc5b3
Merge pull request #20387 from Avogar/optimize-on-insert 
Don't create empty parts on INSERT
 2021-03-16 10:20:34 +03:00
Amos Bird
f00e108410
Fix scalar subquery index analysis 2021-03-16 14:07:30 +08:00
Anton Popov
173d2ea1f4 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-16 02:50:14 +03:00
alexey-milovidov
378e10f929
Merge pull request #21351 from fuwhu/refine-load-data-parts 
Refine MergeTreeData::loadDataParts to not parse format version file and detached directory
 2021-03-15 22:38:30 +03:00
Alexey Milovidov
671395e8c8 Most likely improve performance 2021-03-15 22:23:27 +03:00
Mike Kot
406d037ebb Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-15 18:41:47 +03:00
alesapin
d502a217e5 Merge branch 'master' into Avogar-optimize-on-insert 2021-03-15 11:38:25 +03:00
alesapin
06eb2d8dfd
Merge branch 'master' into s3_zero_copy_replication 2021-03-13 22:32:54 +03:00
Anton Popov
81ac6382a3 slightly better performance 2021-03-13 21:05:18 +03:00
Anton Popov
1b07d28043 fix unwanted changes 2021-03-13 02:59:42 +03:00
Anton Popov
ed42437219 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-13 01:41:26 +03:00
alexey-milovidov
f6bedf55e9
Merge pull request #21642 from azat/fsync_part_directory-horizontal-merge-fix 
Fix fsync_part_directory for horizontal merge
 2021-03-12 21:22:55 +03:00
Anton Ivashkin
ace628d565 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-12 12:58:32 +03:00
alesapin
cca05da7fa
Merge pull request #16481 from amosbird/jbodbalance 
Auto-balance big parts in JBOD array
 2021-03-12 11:07:47 +03:00
Azat Khuzhin
69b2b2a159 Fix fsync_part_directory for horizontal merge 2021-03-11 21:41:27 +03:00
Nikolai Kochetov
4967227fe9 Remove commented code. 2021-03-11 20:11:57 +03:00
alesapin
5704405f8c
Disable new setting by default 2021-03-11 12:29:27 +03:00
Anton Ivashkin
cef0dda933 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-10 14:10:29 +03:00
Anton Ivashkin
deb0b83a12 Fix refreshState logic 2021-03-10 14:08:49 +03:00
Nikolai Kochetov
4f133ae036 Merge branch 'master' into refactor-actions-dag 2021-03-10 12:40:55 +03:00
Nikolai Kochetov
5ef9d87790 Fix some other tests. 2021-03-10 11:41:24 +03:00
fuwhu
69f516cd47 Refine MergeTreeData::loadDataParts to not parse format version file and detached directory 2021-03-10 10:10:25 +08:00
Amos Bird
e52b2fb8ae
update 2021-03-10 02:56:46 +08:00
Anton Ivashkin
265d293934 Use 'merge on single replica' option instead of zookeeper lock 2021-03-09 20:39:55 +03:00
Anton Popov
df6663dcb6 refactoring of serializations 2021-03-09 20:02:26 +03:00
Anton Popov
bc417cf54a refactoring of serializations 2021-03-09 17:46:52 +03:00
Amos Bird
2ec20c5d23
update and add tests 2021-03-08 17:38:07 +08:00
Amos Bird
2b11a2a855
Fix build 2021-03-08 11:10:35 +08:00
Amos Bird
e6522e1ebe
JBOD data balancer 2021-03-08 11:10:35 +08:00
Mikhail f. Shiryaev
efc04b29a6
Add check for growing age AND precision in rollup config 2021-03-06 18:35:53 +01:00
alesapin
5b3161e0b5 Get rid of const_cast 2021-03-05 20:24:06 +03:00
Anton Ivashkin
d08b481660 Fixes by review responces 2021-03-05 19:20:38 +03:00
Nikolai Kochetov
a669f7d641 Merge branch 'master' into refactor-actions-dag 2021-03-05 18:21:14 +03:00
Nikolai Kochetov
9a39459888 Refactor ActionsDAG 2021-03-04 20:38:12 +03:00
Amos Bird
909cb3c243
Fix again.... 2021-03-04 22:27:07 +08:00
Amos Bird
fac832227c
Fix again 2021-03-04 19:43:03 +08:00
Amos Bird
9205fad8c7
Better 2021-03-04 19:43:03 +08:00
Amos Bird
4c9b0f666c
Fix error 2021-03-04 19:43:03 +08:00
Amos Bird
93b661ad5a
partition id pruning 2021-03-04 19:43:03 +08:00
Anton Ivashkin
e69124a0a6 Merge master 2021-03-04 13:26:40 +03:00
Nikolai Kochetov
15921fbfcb Remove index by name from ActionsDAG 2021-03-03 23:01:07 +03:00
Mike Kot
6ea574525c Small fixes regarding the review 2021-03-03 16:51:41 +03:00
Pavel Kovalenko
c4523ec74c Add 'const auto' for variable. 2021-03-03 10:49:23 +03:00
Pavel Kovalenko
182ba7b5b4 Fixed argument name. 2021-03-03 10:35:31 +03:00
alesapin
6b1005aea5
Merge pull request #21334 from ClickHouse/fix_alter_partition_key 
Fix alter modify query for partition key and other metadata fields
 2021-03-03 10:15:11 +03:00
Anton Popov
a4c00ab5dc
Merge pull request #21303 from ucasFL/forbid 
Forbid to drop a column if it's referenced by materialized view
 2021-03-03 02:55:06 +03:00
Pavel Kovalenko
483e63ca45 ALTER TABLE UNFREEZE rework. 2021-03-03 00:10:09 +03:00
Pavel Kovalenko
a3c0c35dff ALTER TABLE UNFREEZE rework. 2021-03-02 23:28:42 +03:00
Nikolai Kochetov
4775ea305e Remove index by name from ActionsDAG 2021-03-02 20:51:54 +03:00
alesapin
c29d7c7f49 Shutup clang tidy 2021-03-02 19:13:36 +03:00
alesapin
4e33587043 Comments 2021-03-02 13:57:09 +03:00
Nikolai Kochetov
ffbea93b10
Merge pull request #19576 from traceon/row-policy-with-prewhere 
Allow row policies with PREWHERE
 2021-03-02 13:51:37 +03:00
alesapin
06678d650d Merge branch 'master' into fix_alter_partition_key 2021-03-02 13:43:41 +03:00
alesapin
9ebf1b4fad Get rid of separate minmax index fields 2021-03-02 13:33:54 +03:00
feng lv
a26c9e64a9 fix 
fix
 2021-03-02 03:20:03 +00:00
Mike Kot
4e1ac185b5 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-01 20:58:07 +03:00
Anton Ivashkin
98065ec56e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-01 13:33:40 +03:00
Anton Ivashkin
3c11d44494 Add description for getUniqueId method, fix typos 2021-03-01 13:31:36 +03:00
alesapin
9c8afbeb53 Fix alter modify query for partition key and other metadata fields 2021-03-01 12:59:19 +03:00
Nikolai Kochetov
976dbe8077
Merge pull request #20341 from ClickHouse/filter-push-down 
Filter push down
 2021-03-01 12:35:06 +03:00
alexey-milovidov
174f49a9f1
Merge pull request #21308 from fuwhu/refine-wal-restore-code 
remove unused code in MergeTreeWriteAheadLog::restore
 2021-03-01 01:44:23 +03:00
Denis Glazachev
eb80c189e4 Merge branch 'master' into row-policy-with-prewhere 2021-02-28 20:14:31 +04:00
fuwhu
2df33be7c2 Remove unused code in MergeTreeWriteAheadLog::restore 2021-02-28 23:17:31 +08:00
Azat Khuzhin
8b58dba094 Drop unused HexWriteBuffer 2021-02-28 10:52:09 +03:00
feng lv
51021c1164 forbid to drop a column if it's referenced by materialized view 2021-02-28 05:24:39 +00:00
Nikolai Kochetov
d91b8a3acb
Merge branch 'master' into filter-push-down 2021-02-26 19:33:27 +03:00
Nikolai Kochetov
d328bfa41f Review fixes. Add setting max_optimizations_to_apply. 2021-02-26 19:29:56 +03:00
tavplubix
09f62447af
Merge pull request #21060 from mxzlxy/fix-bug-for-replicated-engine-parameter 
fix default_replica_path and default_replica_name values are useless …
 2021-02-26 17:18:56 +03:00
Anton Ivashkin
5b267b7eec Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557 Fixes by review response 2021-02-26 12:48:57 +03:00
Pavel Kovalenko
ed44845f2c Fixed backup_path for unfreeze 2021-02-25 22:21:22 +03:00
Mike Kot
b2c898f58c Adding the part when found 2021-02-25 21:25:55 +03:00
alesapin
653d962cdd Fix style and add low-level check 2021-02-25 17:43:58 +03:00
alesapin
5c6c318737 Restrict mutations for engines which doesn't support them 2021-02-25 13:07:48 +03:00
Pavel Kovalenko
0df783e0ef Fix code style 2021-02-24 18:38:40 +03:00
Pavel Kovalenko
078dfce038 Add ALTER TABLE UNFREEZE command. 2021-02-24 17:28:30 +03:00
alesapin
427e5f295c
Merge pull request #21007 from CurtizJ/alter-bad-conversions 
Do not start mutation for alters with wrong type conversion
 2021-02-24 10:08:15 +03:00
tavplubix
4661073750
Update src/Storages/MergeTree/registerStorageMergeTree.cpp 2021-02-23 00:09:20 +03:00
jennyma
6433ab3da6 fix default_replica_path and default_replica_name values are useless on Replicated(*)MergeTree engine when the engine needs specify other parameters 2021-02-22 18:34:01 +08:00
Denis Glazachev
c1158b50f4 Merge branch 'master' into row-policy-with-prewhere 
* master: (155 commits)
  Update version_date.tsv after release 20.8.13.15
  Update version_date.tsv after release 20.12.7.3
  Update version_date.tsv after release 21.1.5.4
  Update version_date.tsv after release 21.2.4.6
  fix
  Add test to skip list
  Fix WriteBufferFromHTTPServerResponse usage in other places (add missing finalize())
  Fix WriteBufferFromHTTPServerResponse usage in odbc-bridge
  Update config.xml
  Suppress signed overflow in AggregateFunctionGroupArrayMoving 2
  Update BaseDaemon.cpp
  review suggestions
  Fix bash syntax in 01731_async_task_queue_wait
  Do not use view() in 01731_async_task_queue_wait to fix ANTLR parser
  Increase buffer for uncaught exception / std::terminate
  Even more better
  Fix uncaught exception when HTTP client goes away
  test for decimal ( p , s) in dictionaries
  Just little better
  Fixed style check
  ...
 2021-02-21 13:25:47 +04:00
Denis Glazachev
dfde9de0e9 Remove commented-out code 2021-02-20 23:30:21 +04:00
tavplubix
a697b578bc
Merge pull request #16193 from ClickHouse/database_replicated 
DatabaseReplicated: continuation of #10485
 2021-02-20 19:39:34 +03:00
Anton Popov
48e188681c do not start mutation for alters with wrong type conversion 2021-02-20 19:11:08 +03:00
Nikolai Kochetov
673e24d7ef Refactor 2021-02-20 14:00:16 +03:00
Denis Glazachev
85ea5cafc4 Always apply row_level_filter immediately 2021-02-20 05:23:48 +04:00
Denis Glazachev
8cc72b8fe0 Merge branch 'master' into row-policy-with-prewhere 
* master: (160 commits)
  Make Poco HTTP Server zero-copy again (#19516)
  Fixed documentation
  ccache 4.2+ does not requires any quirks for SOURCE_DATE_EPOCH
  Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string. (#19600)
  Reinterpret function added Decimal, DateTim64 support
  Add test
  Update InterpreterSelectQuery.cpp
  Improved serialization for data types combined of Arrays and Tuples. Improved matching enum data types to protobuf enum type. Fixed serialization of the Map data type. Omitted values are now set by default.
  Log stdout and stderr when failed to start docker in integration tests.
  Added comment
  Don't backport base commit of branch in the same branch (#20628)
  Fix fasttest retry for failed tests
  Dictionary create source with functions crash fix
  Added error reinterpretation tests
  Update run.sh
  Updated documentation
  fix subquery with limit
  Rename untyped function reinterpretAs into reinterpret
  ignore data store files
  Support vhost
  ...
 2021-02-19 19:49:37 +04:00
Denis Glazachev
5e5b3b80ce Remove debug printouts 2021-02-19 19:46:52 +04:00
Mike Kot
f1ef382cf9 Added part_checksum to Replicated...Entry serialization. 2021-02-19 16:04:12 +03:00
Ivan
414f470c79
Make Poco HTTP Server zero-copy again (#19516) 
* Refactoring: part 1

* Refactoring: part 2

* Handle request using ReadBuffer interface

* Struggles with ReadBuffer's

* Fix URI parsing

* Implement parsing of multipart/form-data

* Check HTTP_LENGTH_REQUIRED before eof() or will hang

* Fix HTTPChunkedReadBuffer

* Fix build and style

* Fix test

* Resist double-eof

* Fix arcadian build
 2021-02-19 15:51:26 +03:00
Mike Kot
ca83775711 Multiple small hotfixes 
Small fixes

Some fix for old bug

Another old code fix
 2021-02-16 16:39:18 +03:00
Mike Kot
a1fafc47c6 Correct checksums pre-calculation for ATTACH_PART in ...OutputStream 2021-02-16 15:40:00 +03:00
Alexander Tokmakov
bf6f64a3fb Merge branch 'master' into database_replicated 2021-02-16 01:28:19 +03:00
Nikolai Kochetov
85277d6a41 Comment debug output. 2021-02-15 22:54:47 +03:00
Nikolai Kochetov
ae73600fb0 Refactor row level security actions. 2021-02-15 22:48:06 +03:00
Mike Kot
0a02fe913a Add some code for the checksum pre-calculation in ...BlockOutputStream 
Added the comment explaining the double-get for the zookeeper header for
the part.
 2021-02-15 20:31:58 +03:00
Mike Kot
73f4740be5 Trying to re-write the solution by adding new command type 
ATTACH_PART into the replicated log.

The LogEntry now also has the pre-calculated part checksum for this
entry type, which is later used while searching in the detached/ folder
 2021-02-15 18:06:48 +03:00
Mike Kot
cd32803709 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-02-15 15:42:17 +03:00
Mike Kot
feff4c6a22 Started adding the new "ATTACH_PART" command into the replicated log 
The original ticket idea was to search for the possibly available data
into the /detached folders for the GET_PART command, but
@tavplubix pointed out this would be quite expensive for an every
fetch.

So a new command is going to be introduced, ATTACH_PART, which will
cover ALTER TABLE ATTACH PART and only for which the search will start.
 2021-02-15 01:59:13 +03:00
Denis Glazachev
45e90961f7 Store and process alias_actions in FilterInfo 2021-02-14 21:16:40 +04:00
Denis Glazachev
99a04b6c64 Merge branch 'master' into row-policy-with-prewhere 
* master: (759 commits)
  Suppress UBSan report in Decimal comparison
  Suppress UBSan report in Decimal comparison
  Fix UBSan report in arrayDifference
  Update README.md
  Non significant change in AggregationCommon
  Print stack trace on SIGTRAP
  Fix dependent test
  Fix tests for better parallel run
  Add test for already working code
  Revert "Fix access control manager destruction order"
  Update index.md
  Update index.md
  Update index.md
  Bit more complicated example for isIPv4String - ru
  Bit more complicated example for isIPv4String
  cleanup
  Replace database with ordinary
  Added comments
  Split tests to make them stable
  Fixes
  ...

# Conflicts:
#	src/Storages/MergeTree/MergeTreeRangeReader.cpp
 2021-02-14 02:24:00 +04:00
Denis Glazachev
79592b73f8 Store filter info in prewhere info instead of multiple prewheres 
Some cleanups
 2021-02-14 02:07:13 +04:00
Azat Khuzhin
68f23b7087 Improve logging during MergeTree reading 
- Remove "Not using primary index on part {}" message (too noisy)
- Add number of total marks before filtering by primary key into the
  common message
- Make "Index {} has dropped {} / {} granules." not per-part, but
  per-query
 2021-02-13 18:08:55 +03:00
Pavel Kruglov
f0163c2acf Don't create empty parts on INSERT 2021-02-12 17:02:04 +03:00
alesapin
a2bb586170
Merge pull request #20335 from ClickHouse/fix_scheduling_of_non_successful_background_tasks 
Fix backoff for failed background tasks in replicated merge tree
 2021-02-12 10:06:04 +03:00
alexey-milovidov
7b55eb8c58
Merge pull request #20178 from amosbird/delayinactive 
Delay or throw insertion when too many inactive parts
 2021-02-11 20:52:04 +03:00
alesapin
3253638969 Fix backoff for failed background tasks in replicated merge tree 2021-02-11 14:46:18 +03:00
alesapin
56453256b7
Merge pull request #20300 from ClickHouse/restrict_to_drop_version_column 
Don't allow to drop or rename version column
 2021-02-11 11:04:00 +03:00
alesapin
121c5c9605
Merge pull request #20279 from ClickHouse/relax_check_in_data_writer 
Relax check in data writer for non adaptive tables
 2021-02-11 10:57:07 +03:00
alesapin
c2bb2c2902
Merge pull request #20097 from ClickHouse/remove-adding-missed-step 
Build actions dag to evaluate missing defaults.
 2021-02-11 10:51:21 +03:00
alexey-milovidov
e33477a16e
Merge pull request #20282 from ClickHouse/dont_allow_ttl_for_old_syntax 
Don't allow TTL in old syntax
 2021-02-11 00:46:30 +03:00
Anton Popov
365e2fb746
Merge pull request #20231 from fuwhu/refine-loadDataParts-code 
refine code in MergeTreeData::loadDataParts to avoid parsing WAL file as data part
 2021-02-10 22:42:31 +03:00
alesapin
891fce3275 Don't allow to drop or rename version column 2021-02-10 22:23:48 +03:00
alesapin
a63d73db58 Don't allow TTL in old syntax 2021-02-10 15:09:12 +03:00
alesapin
420f2958e2
Merge pull request #19991 from ClickHouse/constant-propagation-fix 
Do not use inputs which values are known constants in ActionsDAG.
 2021-02-10 14:13:52 +03:00
alesapin
21faea1f68 Relax check in data writer for non adaptive tables 2021-02-10 13:43:31 +03:00
Amos Bird
afb5846a24
refactor 2021-02-10 11:22:24 +08:00
HuFuwang
4ac8860cea refine code in MergeTreeData::loadDataParts to avoid parsing WAL file as data part. 2021-02-10 10:13:43 +08:00
tavplubix
86530e603c
Merge pull request #20027 from bharatnc/ncb/event_time_micro_part_log 
add event_time_microseconds field to system.part_log table
 2021-02-09 14:46:40 +03:00
Nikolai Kochetov
229acbad5d
Merge pull request #20077 from ClickHouse/remove-adding-const-column-transform 
Remove AddingConstColumn step and transform.
 2021-02-09 11:28:44 +03:00
Alexander Tokmakov
5e0cbfe972 Merge branch 'master' into database_replicated 2021-02-08 23:01:12 +03:00
Nicolae Vartolomei
135f82ce94 Correctly document that settings apply only for fetch part requests 2021-02-08 19:44:03 +00:00
Nicolae Vartolomei
b153e8c190 Add support for custom fetchPart timeouts 2021-02-08 19:44:02 +00:00
Alexander Tokmakov
78c1d69b8c better code 2021-02-08 22:36:17 +03:00
Nicolae Vartolomei
c336cac306 Remove dead code 2021-02-08 17:36:58 +00:00
Nicolae Vartolomei
a8f37e7704 Remove dead code 2021-02-08 17:19:35 +00:00
bharatnc
60c8dfa22c PartLog - populate event_time_microseconds for MergeParts 2021-02-08 07:25:44 -08:00
Nikolai Kochetov
d7fa440356
Merge branch 'master' into remove-adding-const-column-transform 2021-02-08 15:49:23 +03:00
Anton Ivashkin
4d44d75bc7 Fix build after merge one more time 2021-02-08 14:45:10 +03:00
Amos Bird
109a392e0c
Fix ubsan 2021-02-08 19:41:16 +08:00
Anton Ivashkin
1b6ce056a5 Merge master 2021-02-08 12:34:20 +03:00
Amos Bird
ab5c7b75a4
Delay or throw insertion when too many inactive parts 2021-02-07 19:03:55 +08:00
Maxim Akhmedov
0cd3601928 Allow using MergeTreeWhereOptimizer not only with MergeTree-based storages 2021-02-06 18:08:42 +03:00
alesapin
011109c82a
Merge pull request #17348 from xjewer/alex/CLICKHOUSE-606_deduplication_UUID 
CLICKHOUSE-606: query deduplication based on parts' UUID
 2021-02-05 22:47:34 +03:00
Anton Ivashkin
e64c63c611 Merge master 2021-02-05 20:10:06 +03:00
Anton Ivashkin
df6c882aab Fix build after merge 2021-02-05 18:52:40 +03:00
Nikolai Kochetov
85c175883e Rename functions. 2021-02-05 18:11:26 +03:00
alesapin
7cbc135e72 More isolated code 2021-02-05 12:54:34 +03:00
Nikolai Kochetov
d9aa1096cf Build actions dag to evaluate missing defaults. 2021-02-04 23:36:50 +03:00
vdimir
30f98795bc
Merge remote-tracking branch 'upstream/master' into disk-s3-backup-restore-metadata 2021-02-04 18:38:28 +03:00
Nikolai Kochetov
e3d6ffd9dc Remove AddingConstColumn step and transform. 2021-02-04 17:25:11 +03:00
Alexey Milovidov
474f151707 More docs for MergeTreeSettings 2021-02-04 08:26:22 +03:00
Alexey Milovidov
dc2afc4795 Fix double whitespace #18383 2021-02-03 21:12:14 +03:00
Alexander Tokmakov
d010f97db0 Merge branch 'master' into database_replicated 2021-02-03 20:13:25 +03:00
Pavel Kovalenko
b7151aa754 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskDecorator.h
#	src/Disks/IDisk.h
#	src/Disks/S3/DiskS3.cpp
 2021-02-03 14:22:18 +03:00
Alexander Tokmakov
6456ccf0da better test 2021-02-02 22:39:04 +03:00
Alexey Milovidov
12485eee6b Fix some of the issues found by Coverity 2021-02-02 22:07:23 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID 
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
 2021-02-02 16:53:39 +00:00
Nikolai Kochetov
b2d7790794 Fix test. 2021-02-02 17:30:54 +03:00
Maksim Kita
d0151de4bb
Merge pull request #19608 from kreuzerkrieg/Add_IStoragePolicy_interface 
Add IStoragePolicy interface
 2021-02-02 11:03:20 +03:00
Pavel Kruglov
a3f1b825cc Fix build 2021-02-01 21:17:12 +03:00
alexey-milovidov
6b2f0435c8
Merge pull request #19375 from Avogar/select-final 
Improve do_not_merge_across_partitions_select_final optimization.
 2021-02-01 20:31:08 +03:00
alesapin
2aa8a6304b
Merge pull request #15450 from CurtizJ/fix-ttl-group-by 
Fix some cases of TTL expressions
 2021-02-01 16:48:07 +03:00
Nikolai Kochetov
19e4a33f9d
Merge pull request #19544 from amosbird/limitconcurrency 
Per MergeTree table query limit
 2021-02-01 16:09:12 +03:00
alesapin
d9598c47b4
Merge pull request #19702 from ClickHouse/fix_rare_bug_after_part_corruption 
Fix incorrect virtual_parts after part corruption
 2021-02-01 10:09:48 +03:00
alexey-milovidov
d19feb724b
Merge pull request #19799 from CurtizJ/fix-uint8-filtering 
Fix filtering by Uint8 greater than 127
 2021-01-29 21:34:42 +03:00
Denis Glazachev
31706e69bc Merge branch 'master' into row-policy-with-prewhere 
* master: (75 commits)
  Temporary retrun dicttoxml to integration tests runner
  Fix option
  Fix dependency on ODBC for Yandex internal build
  Remove useless headers
  Add test-connect tool
  Revert "Remove old non-automated test"
  Fix trivial bug in arrayEnumerateUniq
  Fix UBSan report in "round"
  Translate comment in test
  Added test to ANTLR skip_list.json
  Update Client.cpp
  Fixed race in tests
  Fixed test_kafka_flush_by_time test.
  LowCardinality UUID fix
  Update mergetree.md
  Clickhouse client query param CTE added test
  correct style zh doc typo
  Fixed data race in function DictGetNoType
  add empty line after error messages in client
  Fixed style issues
  ...

# Conflicts:
#	src/Storages/MergeTree/MergeTreeReadPool.h
 2021-01-29 22:26:11 +04:00
Pavel Kruglov
78371e15dc Update test, reduce num_threads_for_lonely_parts if data is small 2021-01-29 21:00:08 +03:00
alesapin
f4236fd765 Fix style 2021-01-29 20:12:53 +03:00
Nikolai Kochetov
47a7273fe3 Fix build 2021-01-29 19:21:52 +03:00
Nikolai Kochetov
afdc9635cb Update MergeTreeRangeReader 2021-01-29 19:12:53 +03:00
alesapin
c373d92a80 Less strict check 2021-01-29 18:50:08 +03:00
Nikolai Kochetov
4929fe2063 Update MergeTreeRangeReader 2021-01-29 18:13:09 +03:00
Pavel Kruglov
71f4acd48b Use one pool for lonely parts, update tests 2021-01-29 17:30:14 +03:00
Pavel Kruglov
a437ee4e31 Merge branch 'master' of github.com:ClickHouse/ClickHouse into select-final 2021-01-29 14:25:47 +03:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
Anton Popov
031132038b fix filtering by uint8 greater than 127 2021-01-29 10:39:18 +03:00
alexey-milovidov
fa7a795c3a
Merge pull request #19588 from ClickHouse/disable-checksums-on-read 
Allow to disable checksums on read
 2021-01-29 02:37:26 +03:00
alesapin
2881c830e3 Merge branch 'master' into fix_rare_bug_after_part_corruption 2021-01-28 23:16:52 +03:00
Denis Glazachev
009d1e3fc5 Merge branch 'master' into row-policy-with-prewhere 
* master: (247 commits)
  Update 01014_lazy_database_basic.sh
  Update 00459_group_array_insert_at.sql
  Same for ALTER
  Syntax and links
  Added test.
  Check where and prewhere identifiers exist.
  Remove redundant lsof
  Add lsof to fasttest
  Add new line to status
  Make integration odbc tests idempotent
  Minor code improvement in JOIN
  Arcadia does not have bitmaps
  Make Fuzzer more reliable
  Add missing lsof for fasttest docker image
  Update test
  The test most likely would not work in Arcadia
  Less parser depth
  Fix stack overflow in coroutine
  Fuzzer: better messages.
  Fixed build.
  ...
 2021-01-28 21:48:12 +04:00
alesapin
be1104d6c1 Fix typos 2021-01-28 19:07:47 +03:00
alesapin
c1d18fc8a2 Fix very rare case 2021-01-28 18:10:53 +03:00
alesapin
a7ca26fec4
Merge pull request #19708 from ClickHouse/fix_flaky_test_multiple_ttl 
Fix flaky test test_concurrent_ttl_merges
 2021-01-28 15:26:56 +03:00
alesapin
d14f4a525b Merge branch 'master' into fix_rare_bug_after_part_corruption 2021-01-28 11:09:41 +03:00
alesapin
402e031a22 Throw exception only in debug mode 2021-01-28 11:07:18 +03:00
Alexey Milovidov
065d22cf2b Merge branch 'master' into disable-checksums-on-read 2021-01-28 03:07:31 +03:00
Anton Popov
e5125b8c73 add comments and test for compatibility 2021-01-28 02:39:15 +03:00
alesapin
c74631c650 Fix test and add logical error 2021-01-27 21:54:05 +03:00
Anton Popov
a8f3078ce9 Merge remote-tracking branch 'upstream/master' into HEAD 2021-01-27 19:48:55 +03:00
alesapin
5622e6daa6 Fix rare max_number_of_merges_with_ttl_in_pool limit overrun for non-replicated MergeTree 2021-01-27 14:56:12 +03:00
alesapin
6fc39b10d3 Spelling 2021-01-27 13:11:48 +03:00
alesapin
01c8b9e1b1 Fix rare bug when some replicated operations (like mutation) cannot process some parts after data corruption 2021-01-27 13:07:18 +03:00
Alexey Milovidov
48b4d98b21 Amend 2021-01-27 04:48:41 +03:00
Alexey Milovidov
a6f5f40a65 Merge branch 'master' into disable-checksums-on-read 2021-01-27 04:09:00 +03:00
Alexey Milovidov
093108bf81 Normalize BigInt implementation 2021-01-27 03:54:57 +03:00
Anton Popov
c7070da85a better abstractions in disk interface 2021-01-26 17:49:35 +03:00
Denis Glazachev
0d1c9479f8 Fix compilation/linter 2021-01-26 18:39:12 +04:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
tavplubix
2d6a71fced
Merge pull request #19443 from ClickHouse/fix_virtual_parts_in_parts_to_do 
Addition to #15537
 2021-01-26 11:27:14 +03:00
Amos Bird
66fe97d8bd
Per MergeTree table query limit 2021-01-26 14:03:31 +08:00
Alexey Milovidov
8dfa933028 Amend 2021-01-25 23:48:10 +03:00
Alexey Milovidov
9ee5c1535e Allow to disable checksums on read 2021-01-25 23:29:04 +03:00
Denis Glazachev
11b53d3b9d Fix compilation/linter 2021-01-26 00:17:48 +04:00
Denis Glazachev
ff5ce1a5ae Fix compilation 2021-01-25 22:01:59 +04:00
Anton Popov
658f24dcff
Merge pull request #19358 from CurtizJ/fix-subcolumns 
Fix several cases, while reading subcolumns
 2021-01-25 20:26:07 +03:00
Denis Glazachev
4f6c880232 Pass and handle a chain of multiple prewhere infos 2021-01-25 18:31:59 +04:00
tavplubix
5f07bfb9f8
Update ReplicatedMergeTreeQueue.cpp 2021-01-25 16:15:47 +03:00
Alexander Tokmakov
3bd4d97353 Merge branch 'master' into database_replicated 2021-01-25 14:19:04 +03:00
tavplubix
a88a564aae
Update ReplicatedMergeTreeQueue.cpp 2021-01-25 12:51:06 +03:00
alexey-milovidov
73501102f3
Merge pull request #19528 from azat/merge_tree_min_for_concurrent_read-SIGSEGV 
Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX
 2021-01-25 00:18:42 +03:00
Azat Khuzhin
1c364b6ee3 Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX 
In case of 0 or too huge value it will try to read not existing marks
and got:

    Logical error: 'Trying to get non existing mark 11936128518282651045, while size is 2'.
 2021-01-24 14:39:57 +03:00