Kruglov Pavel
f6f71ac5f1
Merge branch 'master' into fix_bug_in_copier
2022-12-01 12:29:50 +01:00
Alexey Milovidov
cf27a0cdeb
Fix log messages in clickhouse-copier
2022-11-25 22:29:02 +01:00
lzydmxy
0462132dd6
Ensure consistency when copier update status and current_partition_attach_is_done after partition attach is done
2022-11-24 11:15:44 +08:00
Azat Khuzhin
44f23c2568
Make disks checks only for clickhouse-server
...
This will fix clickhouse-disks
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-20 16:28:35 +01:00
Vitaly Baranov
8d72f75556
Make the password wiping an option of IAST::format().
2022-11-16 01:35:06 +01:00
taiyang-li
64d7087084
merge master and fix conflict
2022-11-15 10:33:10 +08:00
Robert Schulze
e996e2ff15
Fix clang-tidy
2022-10-26 07:48:37 +00:00
Robert Schulze
c119cd2f00
Merge branch 'master' into update-libcxx-to-15
2022-10-24 08:29:37 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Robert Schulze
f435e4fbe2
Un-clusterf$ck incomplete type errors
2022-10-20 14:48:46 +00:00
Robert Schulze
50c932e566
Separate TaskShard and TaskTable
2022-10-20 14:00:20 +00:00
Robert Schulze
df318d8719
Un-inline TaskTableAndShard
2022-10-20 13:43:15 +00:00
Robert Schulze
f6de964eaa
Fix static_assert in random generator
2022-10-20 11:20:18 +00:00
Robert Schulze
cdae7082a3
Up-inline ShardPartition, ShardPartitionPiece and TaskCluster
2022-10-20 11:16:05 +00:00
Alexey Milovidov
35cce03125
Remove dlopen
2022-09-17 03:02:34 +02:00
Yakov Olkhovskiy
2e34b384c1
update tcp protocol, add quota_key
2022-08-03 15:44:08 -04:00
Robert Schulze
62149111d3
Fix build
2022-07-14 14:36:31 +00:00
Nikolai Kochetov
6e924cdc77
Fix some more tests.
2022-06-01 15:21:47 +00:00
Nikolai Kochetov
3d84aae0ab
Better.
2022-05-24 20:06:08 +00:00
Nikolai Kochetov
333fd09dbf
Fixing build.
2022-05-24 19:29:00 +00:00
Amos Bird
4a5e4274f0
base should not depend on Common
2022-04-29 10:26:35 +08:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains"
...
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
2022-04-18 23:53:11 +02:00
liyang830
0a46532005
add check copier drop partition success
2022-03-14 18:25:28 +08:00
taiyang-li
d78493eeb2
commit again
2022-03-08 16:41:16 +08:00
taiyang-li
e21ab223c5
finish dev
2022-03-08 16:11:21 +08:00
vdimir
e7d3dbeebd
cluster discovery init only with zookeeper
2021-12-27 15:26:08 +03:00
Raúl Marín
b2cfa70541
Reduce dependencies on ASTFunction.h
...
481 -> 230
2021-11-26 18:21:54 +01:00
Raúl Marín
051dddd8df
Reduce dependencies on ASTIdentifier.h
...
Goes from rebuilding 483 objects to 165 when it's modified
2021-11-26 16:49:40 +01:00
Nikolay Degterinsky
ba50e8387d
Merge branch 'master' into query_parameters
2021-11-15 19:55:54 +00:00
Azat Khuzhin
48e9456e75
Disable log_queries in clickhouse-copier
...
This will avoid calling Context::addQueryFactoriesInfo()
v2: disable it in ClusterCopier::reloadTaskDescription() too
v3: just create local context and pass it to the ClusterCopier instead
of global context
2021-11-15 10:32:30 +03:00
Azat Khuzhin
6274c7bfc2
Do not use global context for query execution in clickhouse-copier
2021-11-15 10:32:19 +03:00
Nikolay Degterinsky
d76976e272
Merge branch 'master' into query_parameters
2021-10-18 17:34:20 +03:00
Nikolai Kochetov
a08c98d760
Move some files.
2021-10-16 17:03:50 +03:00
Nikolai Kochetov
067eaadadd
Merge branch 'master' into removing-data-streams-folder
2021-10-16 09:46:05 +03:00
Nikolai Kochetov
c668696047
Merge pull request #30171 from ClickHouse/remove-stream-interfaces
...
Remove stream interfaces
2021-10-16 09:34:01 +03:00
Nikolai Kochetov
fd14faeae2
Remove DataStreams folder.
2021-10-15 23:18:20 +03:00
Nikolai Kochetov
700504e7d6
Update Internals.cpp
2021-10-14 22:09:44 +03:00
Nikolai Kochetov
ab28c6c855
Remove BlockInputStream interfaces.
2021-10-14 13:25:43 +03:00
Nikolai Kochetov
3d3e143c29
Merge pull request #30001 from ClickHouse/remove-streams-from-formats
...
Remove streams from formats.
2021-10-14 12:39:07 +03:00
Nikolay Degterinsky
49c1a52f34
Merge branch 'master' into query_parameters
2021-10-13 03:04:40 +03:00
Nikolai Kochetov
ec18340351
Remove streams from formats.
2021-10-11 19:11:50 +03:00
Nikita Mikhaylov
32d77aafe2
Merge pull request #29921 from azat/copier-fixes
...
copier: add ability to configure retries and delays between them
2021-10-11 12:11:38 +03:00
Azat Khuzhin
452b3b443d
copier: add ability to configure retries and delays between them
...
The following options had been introduced:
- max-table-tries
- max-shard-partition-tries
- max-shard-partition-piece-tries-for-alter
- retry-delay-ms
2021-10-10 08:58:35 +03:00
Nikolai Kochetov
340b53ef85
Remove some more streams.
2021-10-08 17:03:54 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams ( #29491 )
...
* Remove more streams.
* Fixing build.
* Fixing build.
* Rename files.
* Fix fast test.
* Fix StorageKafka.
* Try fix kafka test.
* Move createBuffer to KafkaSource ctor.
* Revert "Move createBuffer to KafkaSource ctor."
This reverts commit 81fa94d27e2107e54969
2021-10-07 11:26:08 +03:00
Nikolay Degterinsky
e03bf0c526
Merge branch 'master' into query_parameters
2021-10-04 17:23:17 +03:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
Nikolay Degterinsky
7f3d395bf3
Merge branch 'master' into query_parameters
2021-09-30 02:01:33 +03:00
Nikolai Kochetov
998d29ebc7
Merge branch 'master' into rewrite-pushing-to-views
2021-09-23 13:10:27 +03:00
Nikolay Degterinsky
1b1d90deb8
Merge branch 'master' into query_parameters
2021-09-22 13:53:56 +03:00
Dmitry Novik
fdd2e5d530
Update programs/copier/ClusterCopier.cpp
2021-09-21 01:19:40 +03:00
Eugene Konkov
130552330f
Fix error message
...
Fix typo in error message. Better wording.
2021-09-21 00:58:22 +03:00
Nikolai Kochetov
a790d391c0
Merge branch 'master' into rewrite-pushing-to-views
2021-09-20 19:43:15 +03:00
Nikolai Kochetov
341553febd
Fix build.
2021-09-16 20:40:42 +03:00
Nikolai Kochetov
b997214620
Rename QueryPipeline to QueryPipelineBuilder.
2021-09-14 20:48:18 +03:00
Azat Khuzhin
c99bd06195
copier: fix partition_key alias using WITH
...
After those changes it is not possible to add an alias for the tuple
element anymore, i.e.:
$ clickhouse-local -q 'select ((1,2) = (1,2) as foo), foo'
1 1
$ clickhouse-upstream local -q 'select ((1,2) = ((1,2) as foo)), foo'
1 (1,2)
But these can be done using WITH statement.
2021-09-12 00:15:31 +03:00
Nikolay Degterinsky
22f59e1714
Add support for table and database query parameters
2021-09-07 01:19:02 +03:00
Nikolai Kochetov
66a76ab70f
Rewrite PushingToViewsBlockOutputStream part 6
2021-09-03 20:29:36 +03:00
Raúl Marín
34b0bb6b7f
Move formatBlock to its own file
2021-07-29 16:39:42 +02:00
Amos Bird
dbfb699690
Asynchronously drain connections.
2021-07-19 21:53:29 +08:00
Nikita Mikhaylov
0d0ea91a0f
fix
2021-06-17 15:58:05 +00:00
Maksim Kita
67e9b85951
Merge ext into common
2021-06-16 23:28:41 +03:00
Nikita Mikhaylov
f6932026d5
delete partition from original table in case of errors while moving
2021-06-04 14:21:47 +00:00
Nikita Mikhaylov
68c92cd93e
style
2021-06-03 23:58:47 +00:00
Nikita Mikhaylov
b978405f4a
fix
2021-06-03 15:50:44 +00:00
Nikita Mikhaylov
4382aac25d
support for ReplicatedTable creation without arguments
2021-06-02 15:46:37 +03:00
Nikita Mikhaylov
312bb96eeb
Merge branch 'master' of github.com:ClickHouse/ClickHouse into copier-stuck
2021-06-02 01:04:47 +03:00
Nikita Mikhaylov
f57863b974
fix tests
2021-06-02 01:03:08 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs
...
Poco::File to std::filesystem
2021-05-31 23:17:02 +03:00
Nikolai Kochetov
afc1fe7f3d
Make ContextPtr const by default.
2021-05-31 17:49:02 +03:00
Nikita Mikhaylov
33dcebbb24
style
2021-05-31 17:41:35 +03:00
Nikita Mikhaylov
01511d0cbe
better
2021-05-31 17:40:07 +03:00
Nikita Mikhaylov
1c548058e6
add status json
2021-05-31 17:38:55 +03:00
Nikita Mikhaylov
ac072243d6
style + enable trivial test
2021-05-31 17:38:55 +03:00
Nikita Mikhaylov
6d19dea761
better
2021-05-31 17:38:20 +03:00
Nikita Mikhaylov
e9f66a0045
delete alter table optimize
2021-05-31 17:38:20 +03:00
Nikita Mikhaylov
fa22b41d5a
added allow_to_drop_target_partitions with test
2021-05-31 17:38:20 +03:00
Nikita Mikhaylov
9022646633
support copying of tables with data skipping indices
2021-05-31 17:38:20 +03:00
Nikita Mikhaylov
24af47063e
allow to copy materialized and alias
2021-05-31 17:37:10 +03:00
Nikita Mikhaylov
c41cc36046
support TTL on columns
2021-05-31 17:37:10 +03:00
Nikita Mikhaylov
90ab394769
better
2021-05-31 17:37:10 +03:00
Nikita Mikhaylov
2017d2f918
save
2021-05-31 17:37:10 +03:00
kssenii
2a631aaf08
Final fixes
2021-05-29 00:34:44 +03:00
kssenii
31107816bb
fs::absolute to fs::canonical
2021-05-24 20:37:54 +03:00
kssenii
2124113aa2
Update programs/*
2021-05-17 01:06:09 +03:00
Amos Bird
264cff6415
Projections
...
TODO (suggested by Nikolai)
1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies
3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)
Also need to figure out how prewhere works for projections, and
row_filter_policies.
wip
2021-05-11 18:12:23 +08:00
Alexey Milovidov
4ff812db7f
Maybe better support for paths with whitespaces
2021-04-24 22:47:52 +03:00
Egor O'Sten
c77dd36ca9
Readability of error message for ORDER BY
2021-04-11 20:40:00 -04:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Azat Khuzhin
f157278b72
Safer SCOPE_EXIT
...
It executes the code in the dtor, that should never throw.
2021-04-06 21:10:37 +03:00
Nikita Mikhaylov
18dc213cee
Merge pull request #21912 from songenjie/clickhouse-copier-create-destination-once
...
[ClickHouse][Copier] Improve copier work
2021-03-30 14:42:00 +03:00
turbo jason
22c9cb1aa3
fix code style
2021-03-23 15:35:15 +08:00
turbo jason
f4be05ca25
fix code style
2021-03-23 15:04:25 +08:00
turbo jason
6aa89d0ee3
remove duplicated code
2021-03-23 11:00:06 +08:00
turbo jason
1b1d425e42
remove duplicated code
2021-03-22 10:27:32 +08:00
turbo jason
a4d3732091
[ClickHouse][Copier] Improve copier work
2021-03-19 20:51:29 +08:00
Anton Popov
bc417cf54a
refactoring of serializations
2021-03-09 17:46:52 +03:00
madianjun
198a8d3b27
Drop helping tables when the whole table is done in clickhouse-copier
2021-02-08 10:54:03 +08:00
alexey-milovidov
8c0ec5105b
Add a patch from @FishermanZzhang, #19952
2021-02-02 06:53:11 +03:00
Azat Khuzhin
5b3ab48861
More forward declaration for generic headers
...
The following headers are pretty generic, so use forward declaration as
much as possible:
- Context.h
- Settings.h
- ConnectionTimeouts.h
(Also this shows that some missing some includes -- this has been fixed)
And split ConnectionTimeouts.h into ConnectionTimeoutsContext.h (since
module part cannot be added for it, due to recursive build dependencies
that will be introduced)
Also remove Settings from the RemoteBlockInputStream/RemoteQueryExecutor
and just pass the context, since settings was passed only in speicifc
places, that can allow making a copy of Context (i.e. Copier).
Approx results (How much units will be recompiled after changing file X?):
- ConnectionTimeouts.h
- mainline: 100
- Context.h:
- mainline: ~800
- patched: 415
- Settings.h:
- mainline: 900-1K
- patched: 440 (most of them because of the Context.h)
2020-12-12 17:43:10 +03:00
