kssenii
3ba2d114d5
Revert "Merge pull request #26352 from ClickHouse/revert-25774-datatype-date32"
...
This reverts commit 94a210650f
, reversing
changes made to 0893b9ff8e
.
2021-07-15 11:41:52 +00:00
alexey-milovidov
9ba4064c86
Revert "Datatype Date32, support range 1925 to 2283"
2021-07-15 10:53:35 +03:00
Amos Bird
28b567a01b
Update
2021-07-14 23:09:17 +08:00
Amos Bird
0558ecdc3f
Aggressive IN index analysis for projections.
2021-07-14 22:56:52 +08:00
Anton Popov
e2249bf1e5
Merge pull request #26256 from zxc111/issue-822
...
Check hash function at table creation
2021-07-14 14:16:02 +03:00
Zhichang Yu
fe5d17680e
Merge remote-tracking branch 'origin/master' into zero_copy_hdfs
2021-07-14 02:12:28 +00:00
zxc111
7b2fed2951
use flag to check sample_condition
2021-07-13 21:17:22 +08:00
Ivan Lezhankin
ab26aed6f9
Merge remote-tracking branch 'upstream/master' into async-insert
2021-07-13 13:54:09 +03:00
Zhichang Yu
b4e6689bf9
fix test_hdfs_zero_copy_replication_move[tiered_copy-2]
2021-07-13 07:20:23 +00:00
Zhichang Yu
5047c758f4
fix per review
2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1
hdfs zero copy
2021-07-13 07:19:12 +00:00
alexey-milovidov
4b1e38eed9
Merge branch 'master' into projection-fix11
2021-07-13 04:52:15 +03:00
alexey-milovidov
5295f7ebb2
Merge pull request #26235 from ClickHouse/check-non-deterministic-functions-in-keys
...
Check for non-deterministic functions in keys, including constant expressions
2021-07-13 02:46:44 +03:00
Kseniia Sumarokova
4a6b4d6c26
Merge pull request #25774 from liuneng1994/datatype-date32
...
Datatype Date32, support range 1925 to 2283
2021-07-12 20:33:39 +03:00
zxc111
e0261866a4
add settings && add test case
2021-07-13 01:15:35 +08:00
mergify[bot]
b7ae4e4ced
Merge branch 'master' into async-insert
2021-07-12 09:21:37 +00:00
Alexey Milovidov
2a2aa69459
Handle non-existent "detached" directories
2021-07-12 10:01:26 +03:00
Alexey Milovidov
ff5c433f10
Check for non-deterministic functions in keys, including constant expressions
2021-07-12 08:58:19 +03:00
zxc111
cd97478c06
check sampling condition when create table
2021-07-12 00:38:20 +08:00
Alexey Milovidov
1ce40d6629
Fix bad code (default function argument)
2021-07-11 04:01:23 +03:00
Anton Popov
9f1ffa777f
remove unused code
2021-07-08 15:06:33 +03:00
mergify[bot]
446474932b
Merge branch 'master' into projection-fix11
2021-07-07 05:02:44 +00:00
Amos Bird
55981cb0ae
Fix normal projection
2021-07-07 13:01:30 +08:00
Anton Popov
9071ecd428
fix alter of settings in MergeTree
2021-07-05 15:44:58 +03:00
NengLiu
8b88988618
Merge branch 'master' into datatype-date32
...
# Conflicts:
# src/DataStreams/PostgreSQLBlockInputStream.cpp
2021-07-05 10:18:38 +08:00
NengLiu
dc0c65ef4f
Merge remote-tracking branch 'origin/master' into datatype-date32
2021-07-05 10:15:08 +08:00
alesapin
cf2fc94d9d
Merge pull request #25820 from ClickHouse/better_remove_empty_parts
...
Replace lost parts with empty parts instead of hacking replication queue
2021-07-04 18:55:49 +03:00
Aleksei Semiglazov
9a5365fc41
CLICKHOUSE-784: reset merge tree setting
...
Add an ability to reset custom setting to default and remove it from table's metadata.
This will allow to rollback the change without knowing the system/config's default.
Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>
2021-07-01 20:09:54 +01:00
mergify[bot]
98fa9f7951
Merge branch 'master' into better_remove_empty_parts
2021-07-01 11:32:43 +00:00
alesapin
94d4d32e4c
Merge pull request #25783 from ClickHouse/followup_drop_part_fix
...
Relax check during DROP PART
2021-06-30 12:25:42 +03:00
mergify[bot]
8182b6e890
Merge branch 'master' into better_remove_empty_parts
2021-06-30 08:42:26 +00:00
alesapin
0af53ace55
Fix comment
2021-06-29 22:49:41 +03:00
alesapin
ca1d21b663
Update src/Storages/MergeTree/MergeTreeData.cpp
...
Co-authored-by: tavplubix <avtokmakov@yandex-team.ru>
2021-06-29 22:48:46 +03:00
alesapin
a6834213a1
Add tests
2021-06-29 22:47:54 +03:00
Nikolai Kochetov
d5bd2b1fa9
Merge pull request #25719 from ClickHouse/remove-PrewhereDAGInfo
...
Remove PrewhereDAGInfo.
2021-06-29 18:58:03 +03:00
alesapin
7680eab0e4
one more fix
2021-06-29 13:03:00 +03:00
alesapin
941a6e539f
Fix
2021-06-29 10:36:20 +03:00
Ivan Lezhankin
1bb41593d1
Merge remote-tracking branch 'upstream/master' into async-insert
2021-06-28 22:55:56 +03:00
alesapin
d20feeb42c
Relax check during DROP PART
2021-06-28 15:05:22 +03:00
alesapin
4c213b639f
Merge pull request #25548 from nikitamikhaylov/background-processing
...
A little improvement in BackgroundJobsExecutor
2021-06-28 13:24:21 +03:00
Nikolai Kochetov
3544b059ce
Merge branch 'master' into remove-PrewhereDAGInfo
2021-06-25 17:52:03 +03:00
Nikolai Kochetov
6bc0a628cd
Remove PrewhereDAGInfo.
2021-06-25 17:49:28 +03:00
alesapin
e6da53dc46
better
2021-06-24 00:09:06 +03:00
alesapin
24876291d1
Followup fix
2021-06-24 00:01:07 +03:00
alesapin
4be4bc21e2
Fix for fix
2021-06-23 23:57:49 +03:00
alesapin
81c74435a3
Fix drop part bug
2021-06-23 22:25:30 +03:00
Nikita Mikhaylov
c66a3b22b5
done
2021-06-22 23:24:47 +00:00
Neng Liu
7ed1728a37
Merge branch 'master' into datatype-date32
...
# Conflicts:
# src/DataTypes/IDataType.h
# src/Functions/CustomWeekTransforms.h
2021-06-22 10:00:18 +08:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] ( #13652 )
...
* initial commit: add setting and stub
* typo
* added test stub
* fix
* wip merging new integration test and code proto
* adding steps interpreters
* adding firstly proposed solution (moving parts etc)
* added checking zookeeper path existence
* fixing the include
* fixing and sorting includes
* fixing outdated struct
* fix the name
* added ast ptr as level of indirection
* fix ref
* updating the changes
* working on test stub
* fix iterator -> reference
* revert rocksdb submodule update
* fixed show privileges test
* updated the test stub
* replaced rand() with thread_local_rng(), updated the tests
updated the test
fixed test config path
test fix
removed error messages
fixed the test
updated the test
fixed string literal
fixed literal
typo: =
* fixed the empty replica error message
* updated the test and the code with logs
* updated the possible test cases, updated
* added the code/test milestone comments
* updated the test (added more testcases)
* replaced native assert with CH one
* individual replicas recursive delete fix
* updated the AS db.name AST
* two small logging fixes
* manually generated AST fixes
* Updated the test, added the possible algo change
* Some thoughts about optimizing the solution:
ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH
* fix
* Removed the replica sync in test as it's invalid
* Some test tweaks
* tmp
* Rewrote the algo by using the executeQuery instead of
hand-crafting the ASTPtr.
Two questions still active.
* tr: logging active parts
* Extracted the parts moving algo into a separate helper function
* Fixed the test data and the queries slightly
* Replaced query to system.parts to direct invocation,
started building the test that breaks on various parts.
* Added the case for tables when at least one replica is alive
* Updated the test to test replicas restoration by detaching/attaching
* Altered the test to check restoration without replica restart
* Added the tables swap in the start if the server failed last time
* Hotfix when only /replicas/replica... path was deleted
* Restore ZK paths while creating a replicated MergeTree table
* Updated the docs, fixed the algo for individual replicas restoration case
* Initial parts table storage fix, tests sync fix
* Reverted individual replica restoration to general algo
* Slightly optimised getDataParts
* Trying another solution with parts detaching
* Rewrote algo without any steps, added ON CLUSTER support
* Attaching parts from other replica on restoration
* Getting part checksums from ZK
* Removed ON CLUSTER, finished working solution
* Multiple small changes after review
* Fixing parallel test
* Supporting rewritten form on cluster
* Test fix
* Moar logging
* Using source replica as checksum provider
* improve test, remove some code from parser
* Trying solution with move to detached + forget
* Moving all parts (not only Committed) to detached
* Edited docs for RESTORE REPLICA
* Re-merging
* minor fixes
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
2021-06-20 11:24:43 +03:00
Ivan Lezhankin
37365589ed
Merge branch 'master' into async-insert
2021-06-17 16:57:45 +03:00
Neng Liu
b6eab5fd2f
add datetype32 function support
2021-06-17 16:05:30 +08:00
Maksim Kita
67e9b85951
Merge ext into common
2021-06-16 23:28:41 +03:00
Kseniia Sumarokova
d6240f2bd4
Merge pull request #25309 from azat/fs-fix
...
Catch ErrnoException during parts cleaning
2021-06-16 22:03:54 +03:00
Nikolai Kochetov
8bf55ae2b5
Merge pull request #25166 from ClickHouse/always-detach-parts-with-wrong-partition-id
...
Always detach parts with wrong partition ID
2021-06-16 11:06:07 +03:00
Azat Khuzhin
de9689379f
Catch ErrnoException during parts cleaning
...
Avoids noisy messages like:
2021.06.15 09:27:35.546982 [ 58176 ] {} <Error> auto DB::IBackgroundJobExecutor::jobExecutingTask()::(anonymous class)::operator()() const: Code: 481, e.displayText() = DB::ErrnoException: Cannot check modification time for file: /var/lib/clickhouse/store/c52/c52619f8-99bd-435e-8526-19f899bdb35e/tmp_insert_202106_7010_7010_0/, errno: 2, strerror: No such file or directory, Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception() @ 0x903209a in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
1. DB::throwFromErrnoWithPath() @ 0x9033089 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
2. FS::getModificationTime() @ 0x907b166 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
3. DB::DiskLocal::getLastModified() @ 0xf9a6669 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
4. DB::MergeTreeData::clearOldTemporaryDirectories(long) @ 0x1069a6de in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
2021-06-16 06:56:55 +03:00
tavplubix
e2ecc51a1f
Merge pull request #25087 from ClickHouse/srmt_remove_copypaste
...
Remove copypaste from StorageReplicatedMergeTree
2021-06-15 12:43:55 +03:00
alexey-milovidov
30f0124e77
Merge branch 'master' into always-detach-parts-with-wrong-partition-id
2021-06-12 02:55:55 +03:00
Alexander Tokmakov
e5d8a97546
fix
2021-06-11 15:21:14 +03:00
Nikolai Kochetov
79c69ea316
Fix build.
2021-06-11 11:29:39 +03:00
Nikolai Kochetov
6222fcffe0
Never remove parts on start at all. Always move.
2021-06-10 15:47:55 +03:00
Nikolai Kochetov
aa90309abf
Never remove parts on start at all. Always move.
2021-06-10 15:46:03 +03:00
Nikolai Kochetov
847dfb9914
Merge remote-tracking branch 'origin' into always-detach-parts-with-wrong-partition-id
2021-06-10 12:37:46 +03:00
Nikolai Kochetov
d52b87d47d
Always detach proken parts with wrong partition id.
2021-06-10 12:37:33 +03:00
Amos Bird
4e358b28f1
Disable projection with final is used
2021-06-10 17:11:17 +08:00
Alexander Tokmakov
cef22688ff
make code less ugly
2021-06-09 15:36:47 +03:00
Alexander Tokmakov
6f5cf773ef
Merge branch 'master' into mvcc_prototype
2021-06-09 12:50:54 +03:00
Alexander Tokmakov
3ade38df82
remove copypaste
2021-06-08 22:17:45 +03:00
Alexander Tokmakov
9a88b9dacc
set maxtid on drop part
2021-06-08 21:17:18 +03:00
Ivan Lezhankin
973cea73b7
Merge branch 'master' into async-insert
2021-06-08 15:10:30 +03:00
Alexander Tokmakov
881889ef22
parts cleanup, fixes
2021-06-07 21:22:58 +03:00
tavplubix
1aff716f18
Merge pull request #24777 from ClickHouse/fix_intersection_with_lost_part
...
Do not crash on intersecting virtual parts
2021-06-03 15:42:46 +03:00
Alexander Tokmakov
9915bd0a8b
Merge branch 'master' into mvcc_prototype
2021-06-03 00:52:57 +03:00
Alexander Tokmakov
9a9e95172f
add test with mv
2021-06-02 23:03:44 +03:00
Ivan Lezhankin
791cbcf0b1
Merge remote-tracking branch 'upstream/master' into async-insert
2021-06-02 18:50:37 +03:00
Ivan Lezhankin
84c23dc060
Merge branch 'master' into async-insert
2021-06-02 18:06:21 +03:00
Alexander Tokmakov
7d7c5638a5
Merge branch 'master' into fix_intersection_with_lost_part
2021-06-01 16:32:59 +03:00
Alexander Tokmakov
5969891611
do not crash on intersecting parts
2021-06-01 16:25:23 +03:00
Nikolai Kochetov
e615858b10
Merge branch 'master' into refactor-MergeTreeDataSelectExecutor
2021-06-01 16:25:04 +03:00
alexey-milovidov
c7bd3b8674
Merge pull request #23315 from CurtizJ/empty-parts-fix
...
Fix waiting while dropping empty parts
2021-06-01 13:52:49 +03:00
Nikolai Kochetov
b54ff696e5
Merge branch 'master' into refactor-MergeTreeDataSelectExecutor
2021-06-01 12:08:37 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs
...
Poco::File to std::filesystem
2021-05-31 23:17:02 +03:00
Nikolai Kochetov
b9aa404508
Review fixes.
2021-05-31 21:25:43 +03:00
Alexander Tokmakov
5692db736c
Merge branch 'master' into mvcc_prototype
2021-05-31 21:11:22 +03:00
Nikolai Kochetov
afc1fe7f3d
Make ContextPtr const by default.
2021-05-31 17:49:02 +03:00
Alexander Tokmakov
16647fe8ce
some unrelated fixes
2021-05-31 00:29:37 +03:00
kssenii
2a631aaf08
Final fixes
2021-05-29 00:34:44 +03:00
Anton Popov
336abfa475
Merge remote-tracking branch 'upstream/master' into HEAD
2021-05-28 01:58:30 +03:00
Nikolai Kochetov
936bd8a011
Merge branch 'master' into refactor-MergeTreeDataSelectExecutor
2021-05-27 20:47:39 +03:00
Nikolai Kochetov
1aeb705b20
Fix some tests.
2021-05-27 19:53:58 +03:00
Nikolai Kochetov
cbdf3752ef
Part 3.
2021-05-27 16:40:33 +03:00
kssenii
73f16ee9ee
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-26 23:08:08 +03:00
alesapin
069d45790f
Merge pull request #24157 from kssenii/modulo-partition-key
...
Adjust modulo function when used in partition key
2021-05-26 10:30:54 +03:00
kssenii
31107816bb
fs::absolute to fs::canonical
2021-05-24 20:37:54 +03:00
Anton Ivashkin
a98286b4c4
Merge master
2021-05-24 17:46:29 +03:00
kssenii
659e57e60f
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into modulo-partition-key
2021-05-23 22:46:00 +00:00
kssenii
bbf56e35ac
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-23 10:54:48 +03:00
kssenii
94b5763dbd
Revert "Different approach to fix modulo"
...
This reverts commit 665f1aa0f6
.
2021-05-21 16:14:01 +00:00
kssenii
30845a383f
Fix comparisons with modulo key
2021-05-21 15:01:41 +00:00
kssenii
665f1aa0f6
Different approach to fix modulo
2021-05-21 14:30:08 +00:00
Anton Ivashkin
6b1b80f5cc
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication
2021-05-21 16:01:48 +03:00
Anton Ivashkin
29336a4a34
Use keep_s3_on_delet flag instead of DeleteOnDestroyKeepS3 state
2021-05-21 15:29:10 +03:00
kssenii
3dee003f9b
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-20 19:20:09 +03:00
kssenii
477214a57f
Fix partitionPrunner with modulo
2021-05-19 17:36:32 +00:00
mergify[bot]
230bec70fb
Merge branch 'master' into modulo-partition-key
2021-05-19 10:08:41 +00:00
alesapin
4d6c71801c
Merge branch 'master' into nv/parts-uuid-move-shard
2021-05-19 13:03:14 +03:00
tavplubix
8b13e73ac2
Merge pull request #24208 from ClickHouse/fix_intersecting_parts_again
...
Addition to #23997
2021-05-18 21:13:25 +03:00
Alexander Tokmakov
529d1aeb19
fix merges of uncommitted parts
2021-05-18 20:07:29 +03:00
Kseniia Sumarokova
0ab15d0bc4
Merge branch 'master' into modulo-partition-key
2021-05-18 17:45:32 +03:00
kssenii
5f083b779d
A little better, fix checks
2021-05-18 14:25:19 +03:00
Alexander Tokmakov
f2ab5a05c6
Merge branch 'master' into mvcc_prototype
2021-05-18 13:36:38 +03:00
Alexander Tokmakov
1fd6142c1a
try enable merges
2021-05-18 13:35:48 +03:00
Alexander Tokmakov
20b9af7b26
do not allow concurrent alter partition
2021-05-18 00:20:58 +03:00
kssenii
9b8df78fdd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-17 17:42:05 +03:00
Anton Popov
e27acc26be
better dropPart and dropPartition
2021-05-17 17:26:36 +03:00
Anton Popov
e043cc45bf
Merge remote-tracking branch 'upstream/master' into HEAD
2021-05-17 16:11:12 +03:00
Ilya Yatsishin
8ede5c736a
Merge pull request #23992 from MaxWk/enable-datatime64-version-column
...
enable DateTime64 to be version column
2021-05-17 14:34:15 +03:00
alesapin
17f229857c
Merge branch 'master' into nvartolomei-parts-move
2021-05-17 13:52:48 +03:00
Anton Ivashkin
39a30b77fe
Merge master
2021-05-17 11:47:48 +03:00
alesapin
eae2df6478
Merge pull request #23997 from ClickHouse/fix_intersecting_parts
...
Try fix intersecting parts
2021-05-17 10:46:58 +03:00
kssenii
3b1bf2bae6
Poco::Path substitution
2021-05-16 23:38:23 +03:00
kssenii
c7f51fe60e
Modulo adjustment for partition key
2021-05-15 19:51:02 +00:00
Amos Bird
8de016d33a
Fix empty key projection query analysis
2021-05-14 22:26:26 +08:00
Alexander Tokmakov
41776ba5d9
fix
2021-05-14 15:55:30 +03:00
Alexander Tokmakov
970ad94851
Merge branch 'master' into fix_intersecting_parts
2021-05-14 13:17:02 +03:00
Alexander Tokmakov
df2849be90
Merge branch 'master' into mvcc_prototype
2021-05-13 21:48:36 +03:00
Alexander Tokmakov
2da12f0b59
fix intersection of drop range and merge
2021-05-13 17:04:36 +03:00
Alexander Tokmakov
e114c7eb8b
fix virtual parts in REPLACE_RANGE
2021-05-13 14:29:59 +03:00
kssenii
0527f0ea33
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-12 16:54:18 +03:00
kssenii
393b026f2e
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-11 16:49:23 +03:00
alesapin
46e136b5c4
Merge branch 'master' into nv/parts-uuid-move-shard
2021-05-11 15:36:40 +03:00
Amos Bird
074c2ead9c
Fix
2021-05-11 18:22:23 +08:00
Amos Bird
5e9fab6d10
Comment debug info out
2021-05-11 18:12:29 +08:00
Amos Bird
5836e2e277
Fix issues in normal projection
2021-05-11 18:12:29 +08:00
Amos Bird
ddd4256a15
More fixes
2021-05-11 18:12:28 +08:00
Amos Bird
718c284437
Fix more tests
2021-05-11 18:12:28 +08:00
Amos Bird
ba17acbd63
Fix tests
2021-05-11 18:12:28 +08:00
Amos Bird
a3fd0b6f2e
Fix tests
2021-05-11 18:12:28 +08:00
Amos Bird
42f161b1dc
Fix more
2021-05-11 18:12:28 +08:00
Amos Bird
9c069ebdbf
support prewhere, row_filter, read_in_order and decent projection selection
...
TODO set index analysis in projection
2021-05-11 18:12:27 +08:00
Amos Bird
35961c0c5d
Use ActionsDAG to rewrite projection queries
2021-05-11 18:12:27 +08:00
Nikolai Kochetov
29d7038e30
Add ProjectionDescription::Type enum. Remove some checks.
2021-05-11 18:12:27 +08:00
Amos Bird
ebaf42a448
Reformat and fix some tests
2021-05-11 18:12:27 +08:00
Nikolai Kochetov
672cfedd13
Disable normal projection by the number of granules.
2021-05-11 18:12:26 +08:00
Amos Bird
483bd165e7
Check if pipeline is simple and add more comments
2021-05-11 18:12:26 +08:00
Nikolai Kochetov
3296c9292f
Try to merge projectons faster.
2021-05-11 18:12:26 +08:00
Amos Bird
264cff6415
Projections
...
TODO (suggested by Nikolai)
1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies
3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)
Also need to figure out how prewhere works for projections, and
row_filter_policies.
wip
2021-05-11 18:12:23 +08:00
Kruglov Pavel
49e7ed6e60
Merge pull request #23673 from amosbird/partitionvalue
...
Add _partition_value virtual column
2021-05-11 11:23:00 +03:00
Alexander Tokmakov
eef899ae63
better code, add test
2021-05-10 21:03:37 +03:00
万康
308b964ca4
enable DateTime64 to be version column
2021-05-10 20:30:52 +08:00
kssenii
ab1a05a1f4
Poco::Path to fs::path, less concatination
2021-05-09 14:59:49 +03:00