Commit Graph

3953 Commits

Author SHA1 Message Date
Azat Khuzhin
b24d1251de Introduce helper for removing one part from fs for ReplicatedMergeTree 2021-04-11 14:38:46 +03:00
alexey-milovidov
0f6fc5bd72
Merge pull request #22962 from kitaisreal/storage-external-distributed-arcadia-fix
StorageExternalDistributed arcadia fix
2021-04-11 13:34:03 +03:00
Azat Khuzhin
940ff6cb76 Do not use c-style cast for std::min(max_part_removal_threads) 2021-04-11 12:22:39 +03:00
Azat Khuzhin
6e71757814 Fix thread accounting for max_part_removal_threads 2021-04-11 11:43:30 +03:00
Azat Khuzhin
dfe3699e77 Respect max_part_removal_threads for ReplicatedMergeTree 2021-04-11 11:33:45 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297)
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Maksim Kita
685da4e234 StorageExternalDistributed arcadia fix 2021-04-10 21:52:17 +03:00
Azat Khuzhin
a6703c5ac2 Fix inactive_parts_to_throw_insert=0 with inactive_parts_to_delay_insert>0 2021-04-10 09:57:08 +03:00
Alexander Kuzmenkov
e44b3822e3
Merge pull request #21850 from fastio/handle_errors_for_kafka_engine
Handle errors for Kafka engine
2021-04-09 22:59:40 +03:00
Alexander Kuzmenkov
0264124146
Merge pull request #21942 from ucasFL/distributed_depth
Add settings max_distributed_depth
2021-04-09 15:52:58 +03:00
Nikolai Kochetov
28ca191102
Merge pull request #22087 from ClickHouse/better-filter-push-down
Better filter push down
2021-04-09 10:22:17 +03:00
alesapin
4b0f973f93
Merge pull request #18215 from filimonov/quorum-dedup-issue
Fix issue with quorum retries behaviour
2021-04-09 10:19:50 +03:00
alexey-milovidov
42b865c9c1
Merge pull request #22829 from azat/dist-send-log
Slightly improve logging messages for Distributed async sends
2021-04-09 09:51:53 +03:00
alexey-milovidov
b02a82250b
Merge pull request #22858 from vitlibar/fix-hanging-in-temporary-live-view-cleaner
Fix hanging in TemporaryLiveViewCleaner
2021-04-09 09:25:38 +03:00
Anton Popov
1c28878f5d
Merge pull request #22726 from azat/mt-reverse-total-rows-approx-fix
Fix approx total rows accounting for reverse reading from MergeTree
2021-04-09 00:16:49 +03:00
Nikolai Kochetov
794d7c89b6 Remove ReverseRowsStep 2021-04-08 16:18:17 +03:00
Vitaly Baranov
5bcb7a0fed Fix hanging in TemporaryLiveViewCleaner 2021-04-08 16:14:49 +03:00
Nikolai Kochetov
febb187da0 Added test, 2021-04-08 15:54:27 +03:00
Nikolai Kochetov
7ffbeac9df Add info about indexes to ReadFromMergeTree step. 2021-04-08 14:48:54 +03:00
alesapin
e762d02e37 More const 2021-04-08 13:38:40 +03:00
alesapin
e17444757b Fix rare case when quorum insert is not really quorum because of deduplication 2021-04-08 13:35:38 +03:00
Nikolai Kochetov
7c5a9133df Add index info to ReadFromStorageStep. 2021-04-08 11:19:04 +03:00
alesapin
1533f9b9aa
Merge pull request #14113 from johnskopis/dynamic-interserver-creds-v20
Support interserver credential rotation
2021-04-08 11:01:40 +03:00
Azat Khuzhin
c27b931f6a Slightly improve logging messages for Distributed async sends
- add took time (in ms)
- add rows/bytes
2021-04-08 08:10:39 +03:00
alesapin
b2a370ad49 Fix PVS 2021-04-07 19:06:20 +03:00
alesapin
2987bbc948 Small improvements 2021-04-07 16:52:11 +03:00
Nikita Mikhaylov
48af7a898c
Merge pull request #22299 from nikitamikhaylov/keen-wolf-storagefile-column-oriented
Merging #21302
2021-04-07 16:01:12 +03:00
Kruglov Pavel
5b52657428
Merge pull request #22725 from azat/buffer-lock-metrics
Add metric to track how much time is spend during waiting for Buffer layer lock
2021-04-07 14:35:57 +03:00
Vladimir
864ab4a7ab
Merge pull request #21370 from vdimir/fix-merge-engine-join-group-by 2021-04-07 11:52:27 +03:00
Vladimir
db0550013c
Merge pull request #22325 from azat/dist-async-insert-header-check
Check only column name and type for async distributed blocks
2021-04-07 11:51:12 +03:00
alesapin
9fd251eaaf
Merge pull request #22514 from ClickHouse/merge_tree_deduplication
Non replicated merge tree deduplication
2021-04-07 10:19:43 +03:00
alexey-milovidov
a7e216969a
Merge pull request #22592 from azat/uncaught-exceptions
Fix some uncaught exceptions (in SCOPE_EXIT) under memory pressure
2021-04-07 07:17:26 +03:00
Nikita Mikhailov
71a2c85d9a better 2021-04-06 22:23:16 +03:00
Nikita Mikhailov
af85ce2fd1 better 2021-04-06 22:23:16 +03:00
Nikita Mikhailov
37f48d13b4 add test 2021-04-06 22:23:16 +03:00
Nikita Mikhailov
282775b10c save 2021-04-06 22:23:16 +03:00
keenwolf
1a4961499b the Native format is not supported after test 2021-04-06 22:23:16 +03:00
keenwolf
94c8e7af1a remove const for value-copy-return 2021-04-06 22:23:16 +03:00
keenwolf
3d3923b518 update comments 2021-04-06 22:23:16 +03:00
keenwolf
b95f54edce Small fix 2021-04-06 22:23:16 +03:00
keenwolf
139bb678a9 Branch with Format isColumnOriented() or not 2021-04-06 22:23:15 +03:00
keenwolf
4eaf718fd2 fix the getColumsForNames() to bring the whole column info from metadata 2021-04-06 22:23:15 +03:00
keenwolf
9d6e2b4bad update comments 2021-04-06 22:23:15 +03:00
keenwolf
de365a5852 fix 2021-04-06 22:23:15 +03:00
keenwolf
6f9bad682f Only read needed columns for formats as parquet etc 2021-04-06 22:23:15 +03:00
Maksim Kita
e38206ee02
Merge pull request #22721 from kitaisreal/memory-storage-sync-comments-and-code
MemoryStorage sync comments and code
2021-04-06 21:56:41 +03:00
Azat Khuzhin
60b000ed51 Fix approx total rows accounting for reverse reading from MergeTree 2021-04-06 21:13:51 +03:00
Azat Khuzhin
c4a7e81287 Add metric to track how much time is spend during waiting for Buffer layer lock
It uses very fast CLOCK_MONOTONIC_COARSE, so this should not be a
problem.
Also note that there is no sense in using microseconds/nanoseconds since
accuracy of CLOCK_MONOTONIC_COARSE usually milliseconds.
2021-04-06 21:13:24 +03:00
Azat Khuzhin
f157278b72 Safer SCOPE_EXIT
It executes the code in the dtor, that should never throw.
2021-04-06 21:10:37 +03:00
Nikolai Kochetov
9f39f5d52d Add more counters to MergeTreeDataSelectExecutor 2021-04-06 15:39:55 +03:00
Maksim Kita
af19f89273 MemoryStorage sync comments and code 2021-04-06 13:41:48 +03:00
Nikolai Kochetov
6102652c99 Merge branch 'master' into better-filter-push-down 2021-04-06 13:38:03 +03:00
alesapin
53a7836a9b Fix typos 2021-04-06 13:16:34 +03:00
alesapin
6456a1507c Support alter setting 2021-04-06 13:14:44 +03:00
vdimir
735154c81a
Handle not plain where tree in StorageMerge modifySelect 2021-04-06 12:29:29 +03:00
alesapin
86a843bb51
Merge pull request #21509 from kssenii/library-bridge
clickhouse-library-bridge for library dictionary source
2021-04-06 12:26:08 +03:00
tavplubix
8b1d320633
Merge pull request #22217 from kssenii/replicas-shards-for-mysql-and-postgres
Replicas and shards for mysql and postgres storages
2021-04-06 11:05:24 +03:00
alesapin
3cd571f92b
Merge pull request #22669 from ClickHouse/fix_two_bugs_with_mutation_wait
Fix two stupid bugs in mutation wait
2021-04-06 11:04:16 +03:00
Azat Khuzhin
27d4fbd13b Compare Block itself for distributed async INSERT batches
INSERT into Distributed with insert_distributed_sync=1 stores the
distributed batches on the disk for sending in background.

But types may be a little bit different for the Distributed and it's
underlying table, so the initiator need to know whether conversion is
required or not.

Before this patch those on disk distributed batches contains header,
which includes dumpStructure() for the block in that batch, however it
checks not only names and types and plus dumpStructure() is a debug
method.

So instead of storing string representation for the block header we
should store empty block in the file header (note, that we cannot store
the empty block not in header, since this will require reading all
blocks from file, due to some trickery of the readers interface).

Note, that this patch also contains tiny refactoring:
- s/header/distributed_header/

v1: dumpNamesAndTypes()
v2: dump empty block into the batch itself
v3: move empty block into the header
2021-04-06 10:05:21 +03:00
Maksim Kita
de71553c9d
Merge pull request #22593 from zlobober/introduce_distributed_write
Introduce IStorage::distributedWrite method for distributed INSERT SELECTS
2021-04-05 22:59:57 +03:00
kssenii
dc42d5189d Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-04-05 15:36:35 +00:00
alesapin
c830397317 Fix two stupid bugs in mutation wait 2021-04-05 18:08:43 +03:00
feng lv
f6e4e13f89 fix conflict 2021-04-05 14:08:30 +00:00
feng lv
56073db22d max distributed depth
Add settings max_distributed_depth

fix style

fix

fix

fix

fix test

fix

fix
2021-04-05 14:00:54 +00:00
kssenii
9dba529494 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-04-05 13:14:51 +00:00
alesapin
156ba8a7fc Disable setting by default and ban for old-syntax MergeTree 2021-04-05 12:16:38 +03:00
alesapin
8d5a787f6b Merge branch 'master' into merge_tree_deduplication 2021-04-05 10:40:03 +03:00
alexey-milovidov
6cefb8e620
Merge pull request #22583 from azat/log-rwlock-ub-fix
Avoid UB in *Log engines for rwlock unlock due to unlock from another thread
2021-04-05 05:20:16 +03:00
Maxim Akhmedov
725fa17961 Introduce IStorage::distributedWrite method for distributed INSERT SELECT. 2021-04-05 02:14:27 +03:00
alexey-milovidov
b1e7624b3f
Merge pull request #22589 from ClickHouse/fix-comments
Fix comments
2021-04-05 00:22:27 +03:00
alesapin
0ceb95aa55 Merge branch 'master' into merge_tree_deduplication 2021-04-04 18:32:27 +03:00
Azat Khuzhin
44abb7588a StorageStripeLog: avoid UB for rwlock unlock due to unlock from another thread 2021-04-04 13:25:44 +03:00
Azat Khuzhin
89a96cf14d StorageLog: avoid UB for rwlock unlock due to unlock from another thread 2021-04-04 13:25:44 +03:00
Azat Khuzhin
5d795661ed StorageTinyLog: add regression test for UB due to unlock from different thread
Refs: #22560
2021-04-04 13:25:44 +03:00
Azat Khuzhin
5ced6e2e9a StorageStripeLog: drop mutable qualifier for rwlock 2021-04-04 13:25:43 +03:00
Azat Khuzhin
5ef845af40 StorageLog: drop mutable qualifier for rwlock 2021-04-04 13:25:43 +03:00
Alexey Milovidov
54caa61af5 Fix comments 2021-04-04 12:33:06 +03:00
Alexey Milovidov
8b5d0a598f Minor improvement in index deserialization 2021-04-04 12:17:54 +03:00
alesapin
548ec6d2cb Trying enabled setting 2021-04-04 11:02:09 +03:00
alexey-milovidov
67fdfd4b89
Merge pull request #22560 from azat/tiny-log-rwlock-ub-fix
Fix UB by unlocking the rwlock of the TinyLog from the same thread
2021-04-03 21:59:01 +03:00
alesapin
1906c77884 Fix tidy one more time 2021-04-03 17:57:03 +03:00
Azat Khuzhin
6cebd3ee1f Fix UB by unlocking the rwlock of the TinyLog from the same thread
Before this patch the build with libstdc++ hangs for
00967_insert_into_distributed_different_types test (and I guess some
others), this is because the rwlock is acquired from different thread as
it was unlocked which causes UB [1], fix this by moving unlock into
writeSuffix().

  [1]: The pthread_rwlock_unlock() function shall release a lock held on
       the read-write lock object referenced by rwlock. Results are undefined
       if the read-write lock rwlock is not held by the **calling thread**.
2021-04-03 16:04:39 +03:00
Azat Khuzhin
10d35867e3 Drop mutable qualifier for StorageTinyLog::rwlock 2021-04-03 16:03:17 +03:00
alesapin
0204f5dd35 Merge branch 'master' into merge_tree_deduplication 2021-04-03 15:24:26 +03:00
alesapin
b127e31451 CI fixes 2021-04-03 14:56:56 +03:00
kssenii
13d1f17d3e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-04-03 08:00:09 +00:00
alexey-milovidov
c3a9cbe01c
Merge pull request #22421 from ClickHouse/fix-buffer-overflow-in-token-extractor
Fix buffer overflow in token extractor
2021-04-02 22:08:28 +03:00
alesapin
b884593186 Followup fix 2021-04-02 20:55:05 +03:00
alesapin
ca30c91868 More comments 2021-04-02 20:54:24 +03:00
alesapin
fc3afe1087 Fix typos 2021-04-02 20:51:57 +03:00
alesapin
b105153d2c Useful comment 2021-04-02 20:48:58 +03:00
alesapin
ab34873344 Add some comments 2021-04-02 20:46:01 +03:00
alesapin
a555d078a2 Add exception handling 2021-04-02 19:56:02 +03:00
alesapin
759dd79c76 Add tests 2021-04-02 19:45:18 +03:00
kssenii
2de6b99f84 Better names 2021-04-02 16:12:14 +00:00
robot-clickhouse
39b5674bef Auto version update to [21.5.1.1] [54450] 2021-04-02 18:34:29 +03:00
Alexander Kuzmenkov
e7df8893df
Merge pull request #22300 from ClickHouse/trying_parallel_func_tests
Trying parallel func tests
2021-04-02 18:09:58 +03:00
tavplubix
93bd2c3b67
Merge pull request #18978 from myrrc/feature/attach-partition-local
Lookup parts/partitions in replica's own detached/ folder before downloading them from other replicas
2021-04-02 17:46:03 +03:00
alesapin
6f36661575 Buildable code 2021-04-02 16:34:36 +03:00
tavplubix
c1ab9de720
Update MergeTreeDataPartChecksum.cpp 2021-04-02 15:57:30 +03:00
alesapin
2ab87d61eb Some build fixes 2021-04-02 15:37:42 +03:00
alesapin
14634689e5 Idea code 2021-04-02 14:46:42 +03:00
Maksim Kita
d06fb1197a
Merge pull request #22096 from kitaisreal/hierarchy-dictionaries-updated
Refactored hierarchy dictionaries interface
2021-04-02 11:38:01 +03:00
alesapin
4eaf585fe5 Almost nothing added 2021-04-01 22:49:51 +03:00
Mike Kot
c947280dfc Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-04-01 21:38:51 +03:00
Mike Kot
60e5946449 Removed needless logging, fixed data part initialization while checking
in the detached folder
2021-04-01 20:11:14 +03:00
kssenii
99f0b08ac3 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-04-01 13:48:33 +00:00
Alexey Milovidov
39c3efdb00 Fix build 2021-04-01 16:38:04 +03:00
Alexander Kuzmenkov
c7bf01fa33 typo 2021-04-01 16:08:18 +03:00
Alexander Kuzmenkov
ba64bc8557 quick fix for inaccessible DatabaseReplicated 2021-04-01 15:21:29 +03:00
kssenii
8300a74f5a Fix 2021-04-01 12:09:44 +00:00
vdimir
3f464595eb
Accurate removing 'join' part for queries to 'Merge' engine 2021-04-01 14:31:57 +03:00
vdimir
fff8043e55
Revert "Fix joined table access with Merge engine and aggregation"
This reverts commit 5476e68d6c.
2021-04-01 14:31:46 +03:00
kssenii
522d170588 Add default port 2021-04-01 11:30:20 +00:00
vdimir
5476e68d6c
Fix joined table access with Merge engine and aggregation 2021-04-01 14:26:28 +03:00
kssenii
747967a9bb Better 2021-04-01 10:27:24 +00:00
alesapin
7c8f54e694 More changes 2021-04-01 11:07:56 +03:00
Alexey Milovidov
7db8fcdebd Fix buffer overflow in TokenExtractor #19233 2021-04-01 05:37:19 +03:00
alexey-milovidov
bbd41d64c2
Update DataPartsExchange.cpp 2021-04-01 02:59:47 +03:00
alexey-milovidov
538be4086b
Recursive removal is extremely dangerous. 2021-04-01 02:59:23 +03:00
alesapin
c15d7e009d Some initial code 2021-03-31 18:20:30 +03:00
Peng Jian
26b5482b4d remove the flag in the parser 2021-03-31 22:25:51 +08:00
kssenii
4e849f3b77 Less logging 2021-03-31 14:06:24 +00:00
kssenii
ba5ddd5c32 Do not parse replicas in PoolWithFailover 2021-03-31 14:03:52 +00:00
Nicolae Vartolomei
5cd1dc485f Delete temporary fetch directory if it already exists
It could be left there after a network timeout, a system crash, or
checksum issues.

The downside of this PR is that it would be harder to debug checksum
issues if replication will continuously delete and try to fetch the same
part.
2021-03-31 14:14:58 +01:00
Maksim Kita
55984e849c
Merge branch 'master' into hierarchy-dictionaries-updated 2021-03-31 15:55:07 +03:00
tavplubix
6c1e2a0d87
Merge pull request #21690 from ClickHouse/test_multiple_nodes
Run three nodes with Replicated database and NuKeeper in CI
2021-03-31 12:31:13 +03:00
Peng Jian
ec54f2a361 fix compilation error 2021-03-31 17:16:51 +08:00
Peng Jian
278d2d7df0 fix typo 2021-03-31 17:16:51 +08:00
Peng Jian
cad54d4e2c reset the parser if exception catched. 2021-03-31 17:16:51 +08:00
Peng Jian
85cfb927e6 move HandleKafkaErrorMode to KafkaSettings 2021-03-31 17:16:51 +08:00
Peng Jian
909d5ad2b5 Handle errors for Kafka engine 2021-03-31 17:15:57 +08:00
Alexey Milovidov
e9bfc7456d Merge branch 'master' into virtualutil 2021-03-31 08:19:43 +03:00
alexey-milovidov
a21fe0b492
Merge pull request #22378 from ianton-ru/s3_zero_copy_replication
Fix S3 Zero-Copy replication for hybrid storage
2021-03-31 04:15:47 +03:00
alexey-milovidov
59c158300a
Merge pull request #22365 from ClickHouse/aku/processes-database
add current database to system.processes
2021-03-31 04:14:17 +03:00
alexey-milovidov
5c15b8a896
Merge branch 'master' into virtualutil 2021-03-30 23:59:44 +03:00
Anton Ivashkin
f87435213e Fix S3 Zero-Copy replication for hybrid storage 2021-03-30 20:38:04 +03:00
kssenii
ce05087b1b Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-03-30 17:27:10 +00:00
Alexander Tokmakov
4f896d2b07 fix tests 2021-03-30 19:46:23 +03:00
Nikolai Kochetov
8fad179800 Fix some tests. 2021-03-30 18:14:20 +03:00
Alexander Kuzmenkov
0252f60c23 add current database to system.processes 2021-03-30 16:36:08 +03:00
Vladimir
5ae9f2c5bb
Merge pull request #21329 from depressed-pho/isIPAddressContainedIn 2021-03-30 13:26:49 +03:00
alesapin
2f891521db
Merge pull request #22291 from ClickHouse/no_logical_error_on_mutation
Relax covering part check on mutation
2021-03-30 13:25:27 +03:00
Nikolai Kochetov
88bfa6c9ba Add ReadFromMergeTree step. 2021-03-30 13:25:26 +03:00
Nikolai Kochetov
78e27dc1b6
Merge pull request #22270 from liuneng1994/fix-query-failed
fix constant condition conflict with column name and add stateless test
2021-03-30 09:37:43 +03:00
alexey-milovidov
b7b55ce61d
Merge pull request #20088 from nvartolomei/nv/replicated-fetches-timeouts
Replicated fetches timeouts
2021-03-30 03:46:09 +03:00
Alexey Milovidov
230b2930b7 Merge branch 'master' into virtualutil 2021-03-30 00:43:00 +03:00
tavplubix
3c0f5a57ec
Merge pull request #21535 from ClickHouse/distributed_ddl_improvements
Distributed DDL improvements
2021-03-29 22:40:11 +03:00
alexey-milovidov
9d84f3113a
Merge pull request #21512 from azat/optimize_skip_unused_shards_limit
Add optimize_skip_unused_shards_limit
2021-03-29 21:13:18 +03:00
alexey-milovidov
1512ff832e
Merge branch 'master' into nv/replicated-fetches-timeouts 2021-03-29 19:25:13 +03:00
Anton Popov
ea82e7725f
Merge pull request #21562 from CurtizJ/serialization-refactoring-4
Refactoring of data types serialization
2021-03-29 16:36:44 +03:00
Alexander Kuzmenkov
0171ab95fa
Merge pull request #21640 from vdimir/fix-join-where-database-engine
Remove unknown columns from joined table in where for queries to external database engines
2021-03-29 16:14:10 +03:00
alesapin
f439e17ece Fix build 2021-03-29 15:41:26 +03:00
alesapin
2c06eb8382 Relax covering part check on mutation 2021-03-29 14:59:28 +03:00
Neng Liu
0d78427fbf fix query failed and add stateless test 2021-03-29 16:12:02 +08:00
Vladimir
0e9926e6c6
Update src/Storages/transformQueryForExternalDatabase.cpp
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
2021-03-29 09:34:43 +03:00
alexey-milovidov
7ab2890c4d
Merge pull request #22206 from ClickHouse/mmap-cache
Add cache for mmap IO
2021-03-29 05:17:55 +03:00
alexey-milovidov
c08289517a
Merge pull request #21766 from amosbird/indexfix
Fix scalar subquery index analysis
2021-03-29 02:03:27 +03:00
Alexey Milovidov
a0a3380d91 Remove useless headers 2021-03-28 22:46:45 +03:00
Alexey Milovidov
2a8ac01cdb Rename as suggested by Kita 2021-03-28 22:24:28 +03:00
kssenii
1bd1a705ce Fix clang tidy 2021-03-28 12:45:49 +00:00
kssenii
0a66f11f69 Fix style, fix build 2021-03-28 10:27:37 +00:00
Alexey Milovidov
6f9f7d118d Fixes 2021-03-28 04:27:26 +03:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
kssenii
c17748a0ba Add proper settings 2021-03-27 21:57:35 +00:00
kssenii
ef537b802f Better comments 2021-03-27 21:10:44 +00:00
kssenii
22b515fbc9 Add namespace, simplify names 2021-03-27 20:14:02 +00:00
kssenii
f40c582e7a Replicas/shards for postgres storage 2021-03-27 20:11:48 +00:00
kssenii
f141f027f4 Make common class for external storages 2021-03-27 17:12:47 +00:00
kssenii
95e8a8b9f0 Support shards 2021-03-27 14:40:07 +00:00
kssenii
ae868208c2 Use pool with failover in mysql storage 2021-03-27 14:39:45 +00:00
alesapin
3b119355d2
Merge pull request #22118 from ClickHouse/try_to_fix992
Trying to fix my favorite test
2021-03-27 10:17:14 +03:00
Maksim Kita
9f2f0d1095 Refactored hierarchy dictionaries interface 2021-03-26 18:42:32 +03:00
Alexander Kuzmenkov
a455762933
Merge pull request #22117 from ucasFL/useless-code
remove useless code
2021-03-26 17:18:14 +03:00
alesapin
35d1443a9c Don't wait when table shutdown called 2021-03-26 15:54:08 +03:00
tavplubix
ae62bee939
Merge pull request #21913 from ClickHouse/fix_context_ref
Fix possibly dangling reference to Context
2021-03-26 10:39:13 +03:00
PHO
1868ca34b8 Fix build on GCC where std::isinf() cannot be called without the namespace prefix 2021-03-26 12:11:22 +09:00
Alexey Milovidov
177a017c5c Minor modification #22115 2021-03-25 20:51:32 +03:00
alexey-milovidov
f895bc895c
Merge pull request #22011 from ClickHouse/min_max_time_system_parts_datetime64
Expose DateTime64 minmax part index in system.parts and system.parts_columns
2021-03-25 16:02:33 +03:00
Anton Popov
6a15431be7 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-25 15:57:35 +03:00
Anton Popov
f58d470dd6
Merge pull request #21830 from foolchi/optimize_select_with_final
optimize select final with prewhere primary key
2021-03-25 15:32:13 +03:00
feng lv
b7622868fc remove useless code 2021-03-25 11:10:41 +00:00
Nikolai Kochetov
0e2f52518f Extract converting from UnionStep. 2021-03-25 12:57:14 +03:00
kssenii
bc36baef26 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-25 07:48:07 +00:00
Nikolai Kochetov
977ebe8b44
Merge pull request #22058 from azat/system.errors-trace
Convert system.errors.stack_trace from String into Array(UInt64)
2021-03-25 10:23:05 +03:00
Mike Kot
285af08949 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-24 22:34:20 +03:00
Mike Kot
9f25ee3719 Replaced removing checksums.txt with version
with the explicit data loading and verification.

If the function fails, the exception will re-throw upper,
cancelling the fetch in the handling of the replicated log entry,
but this event will also wake the PartCheckThread, which will
issue a re-fetch.
2021-03-24 22:30:16 +03:00
kssenii
d9326cc229 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-24 18:28:55 +00:00
alexey-milovidov
ffa6585170
Merge pull request #21837 from Jokser/disk-cache-cleanup-directory-before-move
Cleanup destination directory during DiskCacheWrapper::moveFile()
2021-03-24 21:19:48 +03:00
fuqi
487d6bdcd9 add test case
fix order key check
2021-03-24 16:36:45 +08:00
Azat Khuzhin
7c07b43597 Convert system.errors.stack_trace from String into Array(UInt64)
This should decrease overhead for the errors collecting.
2021-03-24 06:52:48 +03:00
alexey-milovidov
612d4fb073
Update IMergeTreeDataPart.cpp 2021-03-24 02:03:14 +03:00
Anton Popov
8a56d04714 Merge branch 'merge_tree_part_block_number' of git://github.com/ahrefs/ClickHouse into merge_21760 2021-03-23 16:47:14 +03:00
Alexander Tokmakov
c938f4f2fe fix 2021-03-23 14:29:29 +03:00
alesapin
36898bdc4a
Merge pull request #21142 from Jokser/unfreeze-partitions
Add ALTER TABLE UNFREEZE command.
2021-03-23 14:05:38 +03:00
Pavel Kovalenko
a92cf30b67 Code review fixes. 2021-03-23 13:33:07 +03:00
Alexey Milovidov
8d0210b510 Expose DateTime64 minmax part index in system.parts and system.parts_columns #18244 2021-03-23 01:16:41 +03:00
Alexey Milovidov
ce67afd9e3 Fix bad code #18244 2021-03-23 00:49:18 +03:00
Alexander Kuzmenkov
2f5dbf57b6
Merge pull request #21839 from kssenii/add-postgres-connection-pool
Add connection pool for postgres engine
2021-03-22 19:49:51 +03:00
tavplubix
5dfd0045a6
Merge branch 'master' into distributed_ddl_improvements 2021-03-22 18:13:19 +03:00
Mike Kot
da67e06aa0 Added another test case to handle missing part data 2021-03-22 17:52:21 +03:00
Mike Kot
c55a73b752 Added the solution to handle the corruption case
When the part data (e.g. data.bin) is corrupted, but the checksums.txt
is present -- explicitly deleting the checksums.txt.

Removed the extra logging, changes some exceptions message.
2021-03-22 17:23:43 +03:00
Mike Kot
5789507e8b Investigating, why the checksums may match when they shouldn't. 2021-03-22 17:23:43 +03:00
kssenii
8a650c4408 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-22 13:38:25 +00:00
Maksim Kita
8be263062d
Merge pull request #21910 from kitaisreal/table-function-dictionary
Added table function dictionary
2021-03-22 16:08:56 +03:00
tavplubix
0e5362a02c
Merge pull request #21903 from ClickHouse/tavplubix-patch-3
Show path to data directory of EmbeddedRocksDB
2021-03-22 12:46:53 +03:00
Nikolai Kochetov
730785f4c2
Merge pull request #21459 from ClickHouse/refactor-actions-dag
Refactor actions dag
2021-03-22 11:58:16 +03:00
Maksim Kita
849a980644 Updated ExternalDictionariesLoader interface with context parameter 2021-03-20 18:02:09 +03:00
kssenii
698b1b043c Fix typos check 2021-03-19 21:54:59 +00:00
kssenii
3b32c8fe9a Block on empty connection pool 2021-03-19 21:00:45 +00:00
kssenii
e066feb81c Slightly better 2021-03-19 18:50:41 +00:00
kssenii
50743b19c4 Add one more test 2021-03-19 16:12:33 +00:00
kssenii
f374a71165 Fix 2021-03-19 16:07:53 +00:00
Alexander Tokmakov
1845df25f3 fix possibly dangling reference to Context 2021-03-19 17:02:48 +03:00
Maksim Kita
32831e37ba Added table function dictionary 2021-03-19 15:47:27 +03:00
Maksim Kita
a53c90e509 Upated system.dictionaries table 2021-03-19 12:43:57 +03:00
kssenii
a0f717619c Remove initial filling 2021-03-19 08:33:04 +00:00
kssenii
9057aad798 Better version 2021-03-19 08:11:36 +00:00
tavplubix
fc0b87c5c4
Update StorageEmbeddedRocksDB.h 2021-03-19 10:24:27 +03:00
kssenii
f1ef87d966 Fix 2021-03-18 20:04:54 +00:00
Nikolai Kochetov
c3c393a7aa Merge branch 'master' into refactor-actions-dag 2021-03-18 14:33:07 +03:00
kssenii
7847b78af9 Fix style, build 2021-03-17 20:49:56 +00:00
fuqi
cfa92f0045 fix code style 2021-03-18 00:25:43 +08:00
fuqi
b8edc12812 fix code style 2021-03-17 23:56:55 +08:00
Mike Kot
e309dd275a Trying to investigate the errors after attaching the part.
+ A new entry is added to the part_log.
2021-03-17 18:49:04 +03:00
kssenii
3903d59d30 Better 2021-03-17 14:34:04 +00:00
fuqi
e116e8d5e8 fix bug 2021-03-17 22:19:10 +08:00
kssenii
ae64a24844 Add connection pool 2021-03-17 13:55:47 +00:00
Mike Kot
831b90f272 Replaced the iteration algorithm in part finder
Now, instead of iterating through the directories, we iterate though
directories of on of the table disks (which doesn't give us a
substantial boost but is a bit neater to read).

- Updated the system.replication_queue command types.
- Fixed the part ptr being empty (added the checksum loading and
initialization).
- Removed extra logging.
2021-03-17 16:55:26 +03:00
Mike Kot
2ccdb7ef5c Multiple small code and test updates
- Updated the docs to make everything clear.
- Multiple small logger fixes.
- Changed the attach_part command -- now it's after check for the
covering parts -- motivation is to do less work with the checksums
fetching.
- Better logging in the integration test.
2021-03-17 16:52:35 +03:00
kssenii
34dea63248 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-17 13:32:43 +00:00
Nikita Mikhaylov
f51b41b7a4
Merge pull request #21710 from kssenii/replica-priorities
Support replicas priority for postgres dictionary source
2021-03-17 16:02:00 +03:00
Kseniia Sumarokova
f8fc4281f2
Update PostgreSQLReplicaConnection.h 2021-03-17 15:56:47 +03:00
fuqi
9ab713c2e1 optimize select final with prewhere primary key 2021-03-17 18:57:58 +08:00
fuqi
5e20ea2c33 optimize select final with prewhere primary key 2021-03-17 18:49:24 +08:00
Pavel Kovalenko
5627ae9067 Make MergeTreeData::getPartitionMatcher const 2021-03-17 12:12:55 +03:00
Nikolai Kochetov
3a3f800f6e
Merge pull request #21579 from azat/dist-query-zero-shards-fix
Fix optimize_skip_unused_shards for zero shards case
2021-03-17 11:54:08 +03:00
Nikolai Kochetov
4f1f344d4d
Merge pull request #21529 from azat/system.errors-improvements
Add last_error_time/last_error_message/last_error_stacktrace/remote columns for system.errors
2021-03-17 11:51:29 +03:00
alexey-milovidov
d02726bcac
Merge pull request #9404 from Enmk/DateTime64_extended_range
Date time64 extended range
2021-03-17 11:06:03 +03:00
tavplubix
a093980767
Merge pull request #21711 from kssenii/non-default-schema
Support non-default table schema for postgres storage/table-function
2021-03-17 02:10:32 +03:00
Azat Khuzhin
9dee842b60 Distinguish remote and local error info 2021-03-16 22:31:49 +03:00
Azat Khuzhin
a337691b06 Fix modernize-use-bool-literals clang-tidy warning in StorageSystemErrors 2021-03-16 22:31:49 +03:00
Azat Khuzhin
efdd04c958 Drop last_ prefix for ErrorCodes::ValuePair::error_time_ms 2021-03-16 22:31:49 +03:00
Azat Khuzhin
44c9dc753d Add system.errors.last_error_stacktrace column 2021-03-16 22:31:49 +03:00
Azat Khuzhin
775f8f7682 Add system.errors.last_error_message column 2021-03-16 22:31:49 +03:00
Azat Khuzhin
c8852331a2 Add system.errors.last_error_time column 2021-03-16 22:31:49 +03:00
Azat Khuzhin
0d01eaf94f Guard ErrorCodes with mutex over atomic 2021-03-16 22:31:49 +03:00
Azat Khuzhin
259e5ba88e Separate accounting of remote exceptions in system.errors 2021-03-16 22:31:46 +03:00
tavplubix
848bb59175
Merge pull request #21728 from ClickHouse/fix_alter_decimal
Fix alter modify for decimal columns when type size doesn't change
2021-03-16 20:01:35 +03:00
Anton Popov
6247cd5503 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-16 16:50:15 +03:00
Pavel Kovalenko
a43c85b105 Add test for freeze/unfreeze partition with old MergeTree syntax 2021-03-16 16:36:05 +03:00
Nikita Mikhaylov
7d7a2bc724
Merge pull request #21630 from nikitamikhaylov/pf-local
Parallel formatting for clickhouse-local
2021-03-16 14:37:40 +03:00
kssenii
fb5c871140 Merge branch 'non-default-schema' of https://github.com/kssenii/ClickHouse into non-default-schema 2021-03-16 11:17:45 +00:00
Alexey Milovidov
ea049b2cc0 Merge branch 'master' into DateTime64_extended_range 2021-03-16 14:08:02 +03:00
Alexey Milovidov
eadf0248d0 Build fixes 2021-03-16 14:07:53 +03:00
alexey-milovidov
f52e29cfec
Merge pull request #21624 from vdimir/storage-s3-fail
Try to fix test_storage_s3: crash in WriteBufferFromS3
2021-03-16 13:38:48 +03:00
kssenii
4bdfc799f5 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into non-default-schema 2021-03-16 10:05:10 +00:00
Nikolai Kochetov
e8d7349c79
Merge branch 'master' into dist-query-zero-shards-fix 2021-03-16 12:00:08 +03:00
Pavel Kovalenko
39f3b17041 Merge remote-tracking branch 'origin/master' into unfreeze-partitions 2021-03-16 10:56:24 +03:00
alesapin
23a7c00ba4
Merge branch 'master' into fix_alter_decimal 2021-03-16 10:24:46 +03:00
alesapin
8fd3fbc5b3
Merge pull request #20387 from Avogar/optimize-on-insert
Don't create empty parts on INSERT
2021-03-16 10:20:34 +03:00
Amos Bird
f00e108410
Fix scalar subquery index analysis 2021-03-16 14:07:30 +08:00
Anton Popov
173d2ea1f4 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-16 02:50:14 +03:00
ip
1a356af579
StorageMergeTree: allocate temp part blocknumber in dest table 2021-03-15 17:03:34 -04:00
alexey-milovidov
378e10f929
Merge pull request #21351 from fuwhu/refine-load-data-parts
Refine MergeTreeData::loadDataParts to not parse format version file and detached directory
2021-03-15 22:38:30 +03:00
Alexey Milovidov
671395e8c8 Most likely improve performance 2021-03-15 22:23:27 +03:00
Kseniia Sumarokova
4e35cf5509
Update src/Storages/StoragePostgreSQL.cpp
Co-authored-by: tavplubix <tavplubix@gmail.com>
2021-03-15 19:26:44 +03:00
Mike Kot
406d037ebb Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-15 18:41:47 +03:00
Nikita Mikhaylov
23a1c63460
Update StorageURL.cpp 2021-03-15 17:40:51 +03:00
Alexey Milovidov
eaeb4a3bf0 Merge branch 'master' into DateTime64_extended_range 2021-03-15 16:59:32 +03:00
Nikita Mikhaylov
1f92c8ce58 parallel formatting everywhere 2021-03-15 14:51:24 +03:00
alexey-milovidov
8d1bf002e9
Merge pull request #21716 from azat/concurrent-OPTIMIZE-DROP-fix
[RFC] Fix concurrent OPTIMIZE and DROP for ReplicatedMergeTree
2021-03-15 14:12:27 +03:00
Vitaly Baranov
6153f52811
Merge pull request #21641 from vitlibar/allowed-grantees
Add new clause GRANTEES for CREATE/ALTER USER
2021-03-15 12:47:22 +03:00
alesapin
d502a217e5 Merge branch 'master' into Avogar-optimize-on-insert 2021-03-15 11:38:25 +03:00
tavplubix
6a265f468e
Merge branch 'master' into distributed_ddl_improvements 2021-03-15 11:24:14 +03:00
alesapin
69b8ded578 Fix alter modify for decimal columns when type size doesn't change 2021-03-15 11:22:15 +03:00
Kseniia Sumarokova
5b7ef512f5
Update PostgreSQLReplicaConnection.h 2021-03-15 09:14:30 +03:00
Azat Khuzhin
402bf77783 Fix concurrent OPTIMIZE and DROP for ReplicatedMergeTree
Found with fuzzer [1] for 00992_system_parts_race_condition_zookeeper:

    2021.03.13 11:12:30.385188 [ 42042 ] {2d3a8e17-26be-47c1-974f-bd2c9fc7c3af} <Debug> executeQuery: (from [::1]:58192, using production parser) (comment: '/usr/share/clickhouse-test/queries/1_stateful/00153_aggregate_arena_race.sql') CREATE TABLE alter_tabl
    e (a UInt8, b Int16, c Float32, d String, e Array(UInt8), f Nullable(UUID), g Tuple(UInt8, UInt16)) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_3.alter_table', 'r1') ORDER BY a PARTITION BY b % 10 SETTINGS old_parts_lifetime = 1, cleanup_delay_p
    eriod = 1, cleanup_delay_period_random_add = 0;
    ...
    2021.03.13 11:12:30.678387 [ 42042 ] {528cafc5-a02b-4df8-a531-a9a98e37b478} <Debug> executeQuery: (from [::1]:58192, using production parser) (comment: '/usr/share/clickhouse-test/queries/1_stateful/00153_aggregate_arena_race.sql') CREATE TABLE alter_table2 (a UInt8, b Int16, c Float32, d String, e Array(UInt8), f Nullable(UUID), g Tuple(UInt8, UInt16)) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_3.alter_table', 'r2') ORDER BY a PARTITION BY b % 10 SETTINGS old_parts_lifetime = 1, cleanup_delay_period = 1, cleanup_delay_period_random_add = 0;
    ...
    2021.03.13 11:12:40.671994 [ 4193 ] {d96ee93c-69b0-4e89-b411-16c382ae27a8} <Debug> executeQuery: (from [::1]:59714, using production parser) (comment: '/usr/share/clickhouse-test/queries/1_stateful/00153_aggregate_arena_race.sql') OPTIMIZE TABLE alter_table FINAL
    ...
    2021.03.13 11:12:40.990174 [ 2298 ] {a80f9306-3a73-4778-a921-db53249247e3} <Debug> executeQuery: (from [::1]:59768, using production parser) (comment: '/usr/share/clickhouse-test/queries/1_stateful/00153_aggregate_arena_race.sql') DROP TABLE alter_table;
    ...
    2021.03.13 11:12:41.333054 [ 2298 ] {a80f9306-3a73-4778-a921-db53249247e3} <Debug> test_3.alter_table (d4fedaca-e0f6-4c22-9a4f-9f4d11b6b705): Removing part from filesystem 7_0_0_0
    ...
    2021.03.13 11:12:41.335380 [ 2298 ] {a80f9306-3a73-4778-a921-db53249247e3} <Debug> DatabaseCatalog: Waiting for table d4fedaca-e0f6-4c22-9a4f-9f4d11b6b705 to be finally dropped
    ...
    2021.03.13 11:12:41.781032 [ 4193 ] {d96ee93c-69b0-4e89-b411-16c382ae27a8} <Debug> test_3.alter_table (d4fedaca-e0f6-4c22-9a4f-9f4d11b6b705): Waiting for queue-0000000085 to disappear from r2 queue
    ...
    2021.03.13 11:12:41.900039 [ 371 ] {} <Trace> test_3.alter_table2 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000085 of type MERGE_PARTS for part 7_0_0_1 because part 7_0_0_0 is not ready yet (log entry for that part is being processed).
    2021.03.13 11:12:41.900213 [ 365 ] {} <Trace> test_3.alter_table2 (ReplicatedMergeTreeQueue): Cannot execute alter metadata queue-0000000056 with version 22 because another alter 21 must be executed before
    2021.03.13 11:12:41.900231 [ 13762 ] {} <Trace> test_3.alter_table2 (ae877c49-0d30-416d-9afe-27fd457d8fc4): Executing log entry to merge parts -7_0_0_0 to -7_0_0_1
    2021.03.13 11:12:41.900330 [ 13762 ] {} <Debug> test_3.alter_table2 (ae877c49-0d30-416d-9afe-27fd457d8fc4): Don't have all parts for merge -7_0_0_1; will try to fetch it instead
    ...

  [1]: https://clickhouse-test-reports.s3.yandex.net/21691/eb3710c164b991b8d4f86b1435a65f9eceb8f1f5/stress_test_(address).html#fail1
2021-03-14 22:35:50 +03:00
kssenii
8688ae9346 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-14 19:32:21 +00:00
Vitaly Baranov
37ce6e26d3 Add a new clause GRANTEES to commands CREATE USER and ALTER USER. 2021-03-14 19:12:42 +03:00
Vitaly Baranov
d6e0342c30 Improvements in implementations of the classes AccessRights and GrantedRoles. 2021-03-14 19:12:35 +03:00
kssenii
6e69299191 Fix style 2021-03-14 10:43:23 +00:00
kssenii
797063ae0f Support non-default table schema for postgres storage/table-function 2021-03-14 10:35:10 +00:00
alexey-milovidov
53e21443eb
Merge branch 'master' into DateTime64_extended_range 2021-03-14 12:27:17 +03:00
kssenii
8eea6a87e9 Support replicas priority for postgres dictionary source 2021-03-14 08:51:24 +00:00
alesapin
06eb2d8dfd
Merge branch 'master' into s3_zero_copy_replication 2021-03-13 22:32:54 +03:00
Anton Popov
81ac6382a3 slightly better performance 2021-03-13 21:05:18 +03:00
Anton Popov
1b07d28043 fix unwanted changes 2021-03-13 02:59:42 +03:00
Anton Popov
ed42437219 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-13 01:41:26 +03:00
kssenii
323fb54a8e Fix split build finally 2021-03-12 21:12:34 +00:00
alexey-milovidov
f6bedf55e9
Merge pull request #21642 from azat/fsync_part_directory-horizontal-merge-fix
Fix fsync_part_directory for horizontal merge
2021-03-12 21:22:55 +03:00
vdimir
7b494c63b8
Fix unused variable in transformQueryForExternalDatabase 2021-03-12 17:03:33 +03:00
vdimir
91180fa06f
Fix test gtest_transform_query_for_external_database 2021-03-12 17:03:14 +03:00
Anton Ivashkin
ace628d565 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-12 12:58:32 +03:00
alesapin
cca05da7fa
Merge pull request #16481 from amosbird/jbodbalance
Auto-balance big parts in JBOD array
2021-03-12 11:07:47 +03:00
alesapin
b98e27ba82
Merge branch 'master' into s3_zero_copy_replication 2021-03-12 11:07:20 +03:00
kssenii
70f8f90511 Fix split build 2021-03-12 08:04:20 +00:00
kssenii
9e0101e5fc Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-12 07:47:22 +00:00
Denis Glazachev
290a6d273e
Add Kerberos support for authenticating existing users when accessing over HTTP 2021-03-11 23:41:10 +03:00
Azat Khuzhin
69b2b2a159 Fix fsync_part_directory for horizontal merge 2021-03-11 21:41:27 +03:00
vdimir
2156401506
Remove unknown columns from joined table in where for queries to external database engines 2021-03-11 20:53:29 +03:00
Nikolai Kochetov
4967227fe9 Remove commented code. 2021-03-11 20:11:57 +03:00
Alexander Kuzmenkov
3303371b64
Merge pull request #21526 from azat/optimize_skip_unused_shards-once
Avoid processing optimize_skip_unused_shards twice
2021-03-11 17:31:37 +03:00
Alexander Kuzmenkov
3e5447ccec
Merge pull request #21480 from Avogar/slowdown_count
Add slowdown_count and show actual errors_count in system.clusters
2021-03-11 17:18:05 +03:00
vdimir
1194d50e48
Try to fix test_storage_s3: crash in WriteBufferFromS3 2021-03-11 15:21:13 +03:00
alesapin
5704405f8c
Disable new setting by default 2021-03-11 12:29:27 +03:00
Anton Ivashkin
109cb634e1 Remove debug lines 2021-03-11 12:08:19 +03:00
Anton Ivashkin
6f7800ecb4 Fix PVS detected errors 2021-03-10 16:27:08 +03:00
Anton Ivashkin
cef0dda933 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-10 14:10:29 +03:00
Anton Ivashkin
deb0b83a12 Fix refreshState logic 2021-03-10 14:08:49 +03:00
Pervakov Grigorii
eb87afc4d4 recreate S3 client if credentials changed 2021-03-10 13:36:57 +03:00
Nikolai Kochetov
4f133ae036 Merge branch 'master' into refactor-actions-dag 2021-03-10 12:40:55 +03:00
Nikolai Kochetov
5ef9d87790 Fix some other tests. 2021-03-10 11:41:24 +03:00
Azat Khuzhin
61d40c3600 Fix optimize_skip_unused_shards for zero shards case
v2: move check to the beginning of the StorageDistributed::read()
2021-03-10 09:05:14 +03:00
fuwhu
69f516cd47 Refine MergeTreeData::loadDataParts to not parse format version file and detached directory 2021-03-10 10:10:25 +08:00
Amos Bird
e52b2fb8ae
update 2021-03-10 02:56:46 +08:00
Amos Bird
02604185f2
Correctly check constant expr 2021-03-10 02:48:46 +08:00
Anton Ivashkin
aff13c0c52 Make method StorageReplicatedMergeTree::fetchExistsPart 2021-03-09 20:49:50 +03:00
Anton Ivashkin
265d293934 Use 'merge on single replica' option instead of zookeeper lock 2021-03-09 20:39:55 +03:00
Alexander Tokmakov
4d19c84d90 Merge branch 'master' into distributed_ddl_improvements 2021-03-09 20:06:41 +03:00
tavplubix
5a6cc876e4
Merge pull request #20626 from ClickHouse/fix_force_drop_materialized_view
Make force_drop_table work with MaterializedView
2021-03-09 20:05:51 +03:00
Anton Popov
df6663dcb6 refactoring of serializations 2021-03-09 20:02:26 +03:00
Anton Popov
bc417cf54a refactoring of serializations 2021-03-09 17:46:52 +03:00
alesapin
d80c2cef06 Slightly better 2021-03-09 11:45:41 +03:00
alesapin
c00c8c36b7 Merge branch 'master' into ianton-ru-s3_zero_copy_replication 2021-03-09 10:59:11 +03:00
Azat Khuzhin
3474ea044e Avoid processing optimize_skip_unused_shards twice 2021-03-09 10:05:56 +03:00
alexey-milovidov
0a3275e169
Merge pull request #21496 from Felixoid/rollup_grow
Add check for growing age AND precision in rollup config
2021-03-09 07:31:07 +03:00
kssenii
5ee2925123 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into library-bridge 2021-03-08 19:53:48 +00:00
Alexander Tokmakov
2484781070 fix MV recovery in Replicated db 2021-03-08 20:26:38 +03:00
Amos Bird
2ec20c5d23
update and add tests 2021-03-08 17:38:07 +08:00
Azat Khuzhin
ed09897eb1 Pass optimize_skip_unused_shards_limit to the bottom layer
And now optimize_skip_unused_shards_limit=0 is not a special case
anymore.
2021-03-08 10:05:56 +03:00
Azat Khuzhin
16f4c02d42 Add optimize_skip_unused_shards_limit
Limit for number of sharding key values, turns off
optimize_skip_unused_shards if the limit is reached
2021-03-26 06:09:00 +03:00
Amos Bird
2b11a2a855
Fix build 2021-03-08 11:10:35 +08:00
Amos Bird
e6522e1ebe
JBOD data balancer 2021-03-08 11:10:35 +08:00
Amos Bird
091894f8ca
Add more comments 2021-03-08 11:09:06 +08:00
Alexey Milovidov
460658aeab Update tests 2021-03-08 03:18:55 +03:00
kssenii
61d8e27ea7 Common base for bridges part 2 2021-03-07 11:21:49 +00:00
Alexey Milovidov
db63668d03 Merge branch 'master' into DateTime64_extended_range 2021-03-06 23:51:27 +03:00
Mikhail f. Shiryaev
efc04b29a6
Add check for growing age AND precision in rollup config 2021-03-06 18:35:53 +01:00
robot-clickhouse
4e61f45dbb Auto version update to [21.4.1.1] [54449] 2021-03-06 14:43:30 +03:00
Pavel Kruglov
dbae58f30b Add slowdown_count and show actual information in system.clusters 2021-03-06 02:45:17 +03:00
alesapin
5b3161e0b5 Get rid of const_cast 2021-03-05 20:24:06 +03:00
Anton Ivashkin
d08b481660 Fixes by review responces 2021-03-05 19:20:38 +03:00
Nikolai Kochetov
a669f7d641 Merge branch 'master' into refactor-actions-dag 2021-03-05 18:21:14 +03:00
Alexander Tokmakov
e8987f799e Merge branch 'fix_force_drop_materialized_view' into just_another_fix_for_ddl_worker 2021-03-05 15:59:27 +03:00
Alexey Milovidov
3e77c9b1c5 Merge branch 'master' into DateTime64_extended_range 2021-03-05 03:07:19 +03:00
Nikolai Kochetov
9a39459888 Refactor ActionsDAG 2021-03-04 20:38:12 +03:00
Amos Bird
909cb3c243
Fix again.... 2021-03-04 22:27:07 +08:00
Amos Bird
fac832227c
Fix again 2021-03-04 19:43:03 +08:00
Amos Bird
9205fad8c7
Better 2021-03-04 19:43:03 +08:00
Amos Bird
4c9b0f666c
Fix error 2021-03-04 19:43:03 +08:00
Amos Bird
2f8f4e9697
Fix tests 2021-03-04 19:43:03 +08:00
Amos Bird
93b661ad5a
partition id pruning 2021-03-04 19:43:03 +08:00
Anton Ivashkin
bec9de7161 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-04 13:29:45 +03:00
Anton Ivashkin
e69124a0a6 Merge master 2021-03-04 13:26:40 +03:00
Alexey Milovidov
4e8239e098 Merge branch 'master' into DateTime64_extended_range 2021-03-03 23:43:20 +03:00
Azat Khuzhin
79ed35876e DirectoryMonitor: Remove const qualifier and lots of mutable qualifiers 2021-03-03 23:30:24 +03:00
Azat Khuzhin
45ee650e26 Distributed: check for bytes_to_throw/delay_insert only before INSERT
Before it was checked for each block.
2021-03-03 23:30:24 +03:00
Azat Khuzhin
6965ac26c3 Distributed: Add ability to delay/throttle INSERT until pending data will be reduced
Add two new settings for the Distributed engine:
- bytes_to_delay_insert
- max_delay_to_insert

If at the beginning of INSERT there will be too much pending data, more
then bytes_to_delay_insert, then the INSERT will wait until it will be
shrinked, and not more then max_delay_to_insert seconds.

If after this there will be still too much pending, it will throw an
exception.

Also new profile events were added (by analogy to the MergeTree):
- DistributedDelayedInserts (although you can use system.errors instead
  of this, but still)
- DistributedRejectedInserts
- DistributedDelayedInsertsMilliseconds
2021-03-03 23:30:23 +03:00
Azat Khuzhin
15f7459cae Distributed/DirectoryMonitor: protect metric_pending_files with metrics_lock
Since there is local value, that is not atomic, anyway we already have
lock for metrics, so it is fine.
2021-03-03 23:30:03 +03:00
Azat Khuzhin
70049db143 CurrentMetrics/Increment: Introduce add() 2021-03-03 23:30:03 +03:00
Azat Khuzhin
017c054a35 Distributed/DirectoryMonitor: Use std::lock_guard over std::unique_lock
It is more natural, since we do not need lazy locking.
2021-03-03 23:30:03 +03:00
Azat Khuzhin
fcf49a4914 Distributed: Calculate counters for async INSERT at INSERT time
Previous patch fixes the inaccuracy, but it's done using iterating over
directory on each request (to system.distribution_queue or to check
bytes_to_throw_insert), and like previous patch alredy stated, it may
have pretty huge overhead (especially when you have lots of distributed
files pending).

This patch remove that recalculation (but it will still be done, and
if there is different, there will be a log message), and replace it with
proper account at INSERT time (and after file has been sent, or marked
as broken).
2021-03-03 23:30:03 +03:00
Azat Khuzhin
b43046ba06 Distributed: More accurate distribution_queue counters
So now system.distribution_queue will show accurate statistics, so tests
does not requires sleep anymore.

But note that with too much distributed pending this will iterate over
all directories.
2021-03-03 23:30:03 +03:00
Azat Khuzhin
b5a5778589 Distributed: Add ability to limit amount of pending bytes for async INSERT
Right now with distributed_directory_monitor_batch_inserts=1 and
insert_distributed_sync=0 INSERT into Distributed table will store
blocks that should be sent to remote (and in case of
prefer_localhost_replica=0 to the localhost too) on the local
filesystem, and sent it in background.

However there is no limit for this storage, and if the remote is
unavailable (or some other error), these pending blocks may take
significant space, and this is not always desired behaviour.

Add new Distributed setting - bytes_to_throw_insert, that will set the
limit for how much pending bytes is allowed, if the limit will be
reached an exception will be throw.

By default was set to 0, to avoid surprises.
2021-03-03 23:30:00 +03:00
Azat Khuzhin
ce09b7ff89 Distributed: Implement totalBytes() (system.tables.total_bytes) 2021-03-03 23:29:11 +03:00
Azat Khuzhin
456cbaf747 Distributed: Hide private part of the interface 2021-03-03 23:29:11 +03:00
Nikolai Kochetov
15921fbfcb Remove index by name from ActionsDAG 2021-03-03 23:01:07 +03:00
Mike Kot
6ea574525c Small fixes regarding the review 2021-03-03 16:51:41 +03:00
Pavel Kovalenko
c4523ec74c Add 'const auto' for variable. 2021-03-03 10:49:23 +03:00
Pavel Kovalenko
182ba7b5b4 Fixed argument name. 2021-03-03 10:35:31 +03:00
alesapin
6b1005aea5
Merge pull request #21334 from ClickHouse/fix_alter_partition_key
Fix alter modify query for partition key and other metadata fields
2021-03-03 10:15:11 +03:00
Anton Popov
a4c00ab5dc
Merge pull request #21303 from ucasFL/forbid
Forbid to drop a column if it's referenced by materialized view
2021-03-03 02:55:06 +03:00
Kruglov Pavel
fbe0a14a2f
Merge pull request #21035 from anmolarora/system-parts-requested-columns-fill
Fill only requested columns when querying system.parts & system.parts…
2021-03-03 00:39:45 +03:00
Pavel Kovalenko
483e63ca45 ALTER TABLE UNFREEZE rework. 2021-03-03 00:10:09 +03:00
Pavel Kovalenko
a3c0c35dff ALTER TABLE UNFREEZE rework. 2021-03-02 23:28:42 +03:00
Nikolai Kochetov
4775ea305e Remove index by name from ActionsDAG 2021-03-02 20:51:54 +03:00
alesapin
c29d7c7f49 Shutup clang tidy 2021-03-02 19:13:36 +03:00
alesapin
4e33587043 Comments 2021-03-02 13:57:09 +03:00
Nikolai Kochetov
ffbea93b10
Merge pull request #19576 from traceon/row-policy-with-prewhere
Allow row policies with PREWHERE
2021-03-02 13:51:37 +03:00
alesapin
06678d650d Merge branch 'master' into fix_alter_partition_key 2021-03-02 13:43:41 +03:00
alesapin
9ebf1b4fad Get rid of separate minmax index fields 2021-03-02 13:33:54 +03:00
Pavel Kruglov
780a65e3a0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into system-parts-requested-columns-fill 2021-03-02 10:29:26 +03:00
feng lv
a26c9e64a9 fix
fix
2021-03-02 03:20:03 +00:00
Mike Kot
4e1ac185b5 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-01 20:58:07 +03:00
Anton Ivashkin
98065ec56e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-01 13:33:40 +03:00
Anton Ivashkin
3c11d44494 Add description for getUniqueId method, fix typos 2021-03-01 13:31:36 +03:00
Kruglov Pavel
caaa0c5e86
Restart tests. 2021-03-01 13:25:20 +03:00
alesapin
9c8afbeb53 Fix alter modify query for partition key and other metadata fields 2021-03-01 12:59:19 +03:00
Nikolai Kochetov
976dbe8077
Merge pull request #20341 from ClickHouse/filter-push-down
Filter push down
2021-03-01 12:35:06 +03:00
alexey-milovidov
174f49a9f1
Merge pull request #21308 from fuwhu/refine-wal-restore-code
remove unused code in MergeTreeWriteAheadLog::restore
2021-03-01 01:44:23 +03:00
Denis Glazachev
eb80c189e4 Merge branch 'master' into row-policy-with-prewhere 2021-02-28 20:14:31 +04:00
fuwhu
2df33be7c2 Remove unused code in MergeTreeWriteAheadLog::restore 2021-02-28 23:17:31 +08:00
Azat Khuzhin
8b58dba094 Drop unused HexWriteBuffer 2021-02-28 10:52:09 +03:00
feng lv
51021c1164 forbid to drop a column if it's referenced by materialized view 2021-02-28 05:24:39 +00:00
Anmol Arora
2d9b524bdd Fill only requested columns when querying system.parts & system.parts_columns 2021-02-28 01:58:38 +00:00
alexey-milovidov
b8fba768e5
Merge pull request #21264 from ClickHouse/fix_zookeeper_update
Fix several bugs with ZooKeeper client
2021-02-28 01:57:04 +03:00
alesapin
9e93d7f507 Fix tidy and add comments 2021-02-27 11:07:14 +03:00
Nikolai Kochetov
d91b8a3acb
Merge branch 'master' into filter-push-down 2021-02-26 19:33:27 +03:00
Nikolai Kochetov
d328bfa41f Review fixes. Add setting max_optimizations_to_apply. 2021-02-26 19:29:56 +03:00
tavplubix
09f62447af
Merge pull request #21060 from mxzlxy/fix-bug-for-replicated-engine-parameter
fix default_replica_path and default_replica_name values are useless …
2021-02-26 17:18:56 +03:00
Vladimir
78024ae2d6
Merge pull request #21009 from vdimir/fix-race-storage-join 2021-02-26 14:26:10 +03:00
Anton Ivashkin
5b267b7eec Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557 Fixes by review response 2021-02-26 12:48:57 +03:00
Maksim Kita
749dd8d555
Merge pull request #21193 from kitaisreal/storage-rabbit-mq-added-uvloop
StorageRabbitMQ added UVLoop
2021-02-26 11:04:02 +03:00
alesapin
5b72cd231f
Merge pull request #21183 from ClickHouse/fix_mutation_for_storages
Restrict mutations for engines which doesn't support them
2021-02-26 09:54:09 +03:00
Pavel Kovalenko
ed44845f2c Fixed backup_path for unfreeze 2021-02-25 22:21:22 +03:00
Mike Kot
2b3b335eda Exit fixes 2021-02-25 21:41:09 +03:00
Mike Kot
b2c898f58c Adding the part when found 2021-02-25 21:25:55 +03:00
kssenii
1247c0f36c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into rabbit-fix 2021-02-25 16:49:29 +00:00
vdimir
9c1ba71416
Merge remote-tracking branch 'upstream/master' into fix-race-storage-join 2021-02-25 19:13:49 +03:00
alesapin
653d962cdd Fix style and add low-level check 2021-02-25 17:43:58 +03:00
Maksim Kita
29e783999c Fixed typo 2021-02-25 17:43:32 +03:00
Maksim Kita
f077a25a7a StorageRabbitMQ added UVLoop 2021-02-25 17:02:34 +03:00
vdimir
6f864ddb42
Minor changes in Hash/StorageJoin, add TableLockHolder to ExecutableFunctionJoinGet 2021-02-25 14:21:06 +03:00
alesapin
5c6c318737 Restrict mutations for engines which doesn't support them 2021-02-25 13:07:48 +03:00
vdimir
2f70e895fc
Update StorageJoin locking
Move joinGet into StorageJoin

Protect JoinSource with lock, add test

Add comments about locking logic
2021-02-25 12:31:22 +03:00
fastio
fea2836673 get zookeeper from global context 2021-02-25 14:19:58 +08:00
fastio
3ddb729e4a fix metadata leak when drop Replicated*MergeTree 2021-02-25 14:19:58 +08:00
kssenii
6eaf9281b5 Review fix 2021-02-24 16:52:14 +00:00
vdimir
6aa4791be2
Remove unused *Locked methods from HashJoin, add comments 2021-02-24 19:19:04 +03:00
Pavel Kovalenko
0df783e0ef Fix code style 2021-02-24 18:38:40 +03:00
Vasily Nemkov
4fcc23ec9a Fixed build for GCC-10 2021-02-24 17:08:43 +02:00
Vasily Nemkov
2d03d330bc Extended range of DateTime64 to years 1925 - 2238
The Year 1925 is a starting point because most of the timezones
switched to saner (mostly 15-minutes based) offsets somewhere
during 1924 or before. And that significantly simplifies implementation.

2238 is to simplify arithmetics for sanitizing LUT index access;
there are less than 0x1ffff days from 1925.

* Extended DateLUTImpl internal LUT to 0x1ffff items, some of which
  represent negative (pre-1970) time values.
  As a collateral benefit, Date now correctly supports dates up to 2149
  (instead of 2106).
* Added a new strong typedef ExtendedDayNum, which represents dates
  pre-1970 and post 2149.
* Functions that used to return DayNum now return ExtendedDayNum.
* Refactored DateLUTImpl to untie DayNum from the dual role of being
  a value and an index (due to negative time). Index is now a different
  type LUTIndex with explicit conversion functions from DatNum, time_t,
  and ExtendedDayNum.
* Updated DateLUTImpl to properly support values close to epoch start
  (1970-01-01 00:00), including negative ones.
* Reduced resolution of DateLUTImpl::Values::time_at_offset_change
  to multiple of 15-minutes to allow storing 64-bits of time_t in
  DateLUTImpl::Value while keeping same size.
* Minor performance updates to DateLUTImpl when building month LUT
  by skipping non-start-of-month days.
* Fixed extractTimeZoneFromFunctionArguments to work correctly
  with DateTime64.
* New unit-tests and stateless integration tests for both DateTime
  and DateTime64.
2021-02-24 17:08:35 +02:00
Pavel Kovalenko
078dfce038 Add ALTER TABLE UNFREEZE command. 2021-02-24 17:28:30 +03:00
kssenii
3728480f04 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into rabbit-fix 2021-02-24 14:18:25 +00:00
kssenii
96f22b4290 Fix rabbit bug 2021-02-24 10:41:08 +00:00
alesapin
427e5f295c
Merge pull request #21007 from CurtizJ/alter-bad-conversions
Do not start mutation for alters with wrong type conversion
2021-02-24 10:08:15 +03:00
vdimir
c006b41afc
Lock before reuseJoinedData 2021-02-24 09:48:12 +03:00
tavplubix
4661073750
Update src/Storages/MergeTree/registerStorageMergeTree.cpp 2021-02-23 00:09:20 +03:00
jennyma
6433ab3da6 fix default_replica_path and default_replica_name values are useless on Replicated(*)MergeTree engine when the engine needs specify other parameters 2021-02-22 18:34:01 +08:00
Alexey Milovidov
46ac365629 Merge branch 'master' into topvisor/file-engine-settings 2021-02-21 23:10:26 +03:00
alexey-milovidov
2ac5c4b71b
Merge pull request #20996 from azat/system-stack-trace-diagnostics
Add log message when stacktrace cannot be obtained for thread
2021-02-21 22:18:13 +03:00
Denis Glazachev
c1158b50f4 Merge branch 'master' into row-policy-with-prewhere
* master: (155 commits)
  Update version_date.tsv after release 20.8.13.15
  Update version_date.tsv after release 20.12.7.3
  Update version_date.tsv after release 21.1.5.4
  Update version_date.tsv after release 21.2.4.6
  fix
  Add test to skip list
  Fix WriteBufferFromHTTPServerResponse usage in other places (add missing finalize())
  Fix WriteBufferFromHTTPServerResponse usage in odbc-bridge
  Update config.xml
  Suppress signed overflow in AggregateFunctionGroupArrayMoving 2
  Update BaseDaemon.cpp
  review suggestions
  Fix bash syntax in 01731_async_task_queue_wait
  Do not use view() in 01731_async_task_queue_wait to fix ANTLR parser
  Increase buffer for uncaught exception / std::terminate
  Even more better
  Fix uncaught exception when HTTP client goes away
  test for decimal ( p , s) in dictionaries
  Just little better
  Fixed style check
  ...
2021-02-21 13:25:47 +04:00
Denis Glazachev
dfde9de0e9 Remove commented-out code 2021-02-20 23:30:21 +04:00
tavplubix
103d569885
Merge branch 'master' into fix_force_drop_materialized_view 2021-02-20 20:41:03 +03:00
tavplubix
a697b578bc
Merge pull request #16193 from ClickHouse/database_replicated
DatabaseReplicated: continuation of #10485
2021-02-20 19:39:34 +03:00
Anton Popov
48e188681c do not start mutation for alters with wrong type conversion 2021-02-20 19:11:08 +03:00
vdimir
6cc2fb5e9f
Try to fix race in storage join: block parralel inserts 2021-02-20 18:00:59 +03:00
Nikolai Kochetov
673e24d7ef Refactor 2021-02-20 14:00:16 +03:00
alexey-milovidov
b4196c8e68
Merge pull request #20168 from ClickHouse/in-memory-compression
Compression for Memory tables
2021-02-20 09:49:37 +03:00
Azat Khuzhin
7c04f15c80 Add log message when stacktrace cannot be obtained for thread
This is to provide better diagnostics for 01051_system_stack_trace
failure [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/20881/866dfaec793f764dc9ba167d3ac9f6521b9b3381/functional_stateless_tests_(release,_wide_parts_enabled).html#fail1
2021-02-20 09:13:00 +03:00
Denis Glazachev
85ea5cafc4 Always apply row_level_filter immediately 2021-02-20 05:23:48 +04:00
Alexander Tokmakov
2a36d6cb55 review suggestions 2021-02-20 02:41:58 +03:00
M0r64n
2f7d0ba926 Replace direct truncate with O_TRUNC flag 2021-02-20 03:27:23 +04:00
alexey-milovidov
2299930c31
Merge pull request #20545 from kssenii/rabbit-format-settings
rabbitmq: add missing format factory settings
2021-02-19 20:24:17 +03:00
Denis Glazachev
8cc72b8fe0 Merge branch 'master' into row-policy-with-prewhere
* master: (160 commits)
  Make Poco HTTP Server zero-copy again (#19516)
  Fixed documentation
  ccache 4.2+ does not requires any quirks for SOURCE_DATE_EPOCH
  Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string. (#19600)
  Reinterpret function added Decimal, DateTim64 support
  Add test
  Update InterpreterSelectQuery.cpp
  Improved serialization for data types combined of Arrays and Tuples. Improved matching enum data types to protobuf enum type. Fixed serialization of the Map data type. Omitted values are now set by default.
  Log stdout and stderr when failed to start docker in integration tests.
  Added comment
  Don't backport base commit of branch in the same branch (#20628)
  Fix fasttest retry for failed tests
  Dictionary create source with functions crash fix
  Added error reinterpretation tests
  Update run.sh
  Updated documentation
  fix subquery with limit
  Rename untyped function reinterpretAs into reinterpret
  ignore data store files
  Support vhost
  ...
2021-02-19 19:49:37 +04:00
Denis Glazachev
5e5b3b80ce Remove debug printouts 2021-02-19 19:46:52 +04:00
Mike Kot
eb1826a5e3 Trying to figure out why iteration doesn't work 2021-02-19 17:28:29 +03:00
Mike Kot
f1ef382cf9 Added part_checksum to Replicated...Entry serialization. 2021-02-19 16:04:12 +03:00