thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
102,531 Commits 9,328 Branches 1,783 Tags 2.1 GiB
6979dc9f2f
Commit Graph

1034 Commits

Author SHA1 Message Date
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
Alexander Tokmakov
0035997e02 fix race on enqueue query 2021-05-31 16:31:03 +03:00
kssenii
0a3589524e File movement 2021-05-29 00:57:53 +03:00
kssenii
2a631aaf08 Final fixes 2021-05-29 00:34:44 +03:00
Arslan G
fbc5734302 Clean up code and fix tests 2021-05-28 20:00:26 +03:00
kssenii
a78c9e63e7 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-23 12:07:26 +00:00
kssenii
866b29fb5a Return list fds with Poco, more canonical 2021-05-23 10:56:13 +03:00
kssenii
1c43b333ea Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-22 21:26:48 +03:00
kssenii
3e6d3d4ecb Postgres schema for insert 2021-05-22 08:50:23 +00:00
alexey-milovidov
a25f1910cc
Merge pull request #24320 from ClickHouse/remove-auto-array 
Remove AutoArray
 2021-05-22 02:19:59 +03:00
kssenii
096d117f68 Review fixes 2021-05-22 00:12:46 +03:00
Alexey Milovidov
ea30bb7abf Improve implementation 2021-05-21 04:17:18 +03:00
Azat Khuzhin
4f41ebcae3 Add settings (for connections) for MySQL storage engine 
Default settings are not very efficient, since they do not even reuse
connections.
And when each query requires connection you can have only ~80 QPS, while
by simply enabling connection reuse (connection_auto_close=false) you
can have ~500 QPS (and by increasing connection_pool_size you can have
better QPS throughput).

So this patch allows to pass through some connection related settings
for the StorageMySQL engine, like:
- connection_pool_size=16
- connection_max_tries=3
- connection_auto_close=true

v2: remove connection_pool_default_size
v3: remove num_tries_on_connection_loss
 2021-05-20 22:44:52 +03:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
kssenii
5f083b779d A little better, fix checks 2021-05-18 14:25:19 +03:00
kssenii
9b8df78fdd Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-17 17:42:05 +03:00
Arslan G
dde9e30043 Merge branch 'master' into add-sqlite-support 2021-05-17 14:38:24 +03:00
Arslan G
ea51840f85 Init SQLite integration 2021-05-17 14:02:35 +03:00
kssenii
1ee77eae89 Better way to distinguish between user query and replication thread query 2021-05-16 21:35:37 +00:00
kssenii
8a4711b716 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-16 20:55:18 +00:00
Azat Khuzhin
23981ffd0e Introduce IStorage::flushAndShutdown() 2021-05-16 13:23:53 +03:00
Azat Khuzhin
f20799f683 Destroy Buffer() tables before others (within one database) 
This will reduce amount of data that may be lost at shutdown.

v2: replace dynamic_cast with IStorage::isBuffer (@kitaisreal)
v3: replace IStorage::isBuffer with IStorage::flush (@alexey-milovidov)
v4: flush() for StorageProxy/StorageTableFunction
 2021-05-16 13:23:53 +03:00
feng lv
c6f8ab9826 fix 2021-05-13 02:05:53 +00:00
kssenii
7c81103ea3 Add replication identifier, allow proper access to _version and _sign columns from nested 2021-05-12 22:48:42 +00:00
kssenii
407db17e2e Remove setting, use native transaction 2021-05-12 22:33:15 +00:00
kssenii
0955e5f0be Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-12 15:11:15 +00:00
kssenii
0527f0ea33 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-12 16:54:18 +03:00
Nikolai Kochetov
a1ba67e029
Merge pull request #20202 from amosbird/projection 
Projections
 2021-05-12 09:38:07 +03:00
kssenii
81797393f1 fs::last_write_time, fs::space 2021-05-12 09:19:11 +03:00
Maksim Kita
72d46beca0
Merge pull request #23979 from azat/dict-preallocate 
Reimplement preallocate for hashed/sparse_hashed dictionaries
 2021-05-11 20:15:46 +03:00
feng lv
9035e3a643 fix test and conflict 2021-05-11 16:22:24 +00:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
Amos Bird
264cff6415
Projections 
TODO (suggested by Nikolai)

1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies

3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)

Also need to figure out how prewhere works for projections, and
row_filter_policies.

wip
 2021-05-11 18:12:23 +08:00
kssenii
626e87bae5 Some fixes 2021-05-10 13:51:05 +00:00
kssenii
ae1191d0c0 Get rid of metadata file, rely only on lsn 2021-05-10 09:10:02 +00:00
kssenii
4ac023e511 Small fixes 2021-05-10 08:45:52 +00:00
kssenii
993823bcc4 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-09 21:26:41 +00:00
kssenii
ab1a05a1f4 Poco::Path to fs::path, less concatination 2021-05-09 14:59:49 +03:00
Azat Khuzhin
80f4059deb Drop unused getDictionaryConfigurationFromAST.h 2021-05-09 14:25:45 +03:00
kssenii
e26cb4bee0 Fix style check, fix yandex check 2021-05-09 13:45:50 +03:00
Alexey Milovidov
3025f9e141 More fixes for PVS-Studio 2021-05-08 19:12:49 +03:00
Alexey Milovidov
8d62c42eb9 A bunch of changes for PVS-Studio 2021-05-08 19:11:50 +03:00
kssenii
5c24f0943c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-08 14:54:44 +00:00
kssenii
266567e04c Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-08 17:49:05 +03:00
alexey-milovidov
680860aae7
Merge pull request #23912 from azat/CU 
Cleanup IDatabase.h from extra headers
 2021-05-08 16:42:56 +03:00
kssenii
02288359c5 Less manual concatenation of paths 2021-05-08 13:59:55 +03:00
kssenii
35f999bf04 Poco::createFile to fs::createFile 2021-05-08 02:41:47 +03:00
kssenii
c3eab60b8b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-08 02:40:17 +03:00
kssenii
140bf7e2ea Less Poco 2021-05-07 15:29:26 +03:00
kssenii
912878ad7f Review fixes 2021-05-07 12:05:34 +00:00
Azat Khuzhin
3672af8bbe Make DatabaseConnectionMySQL self compilable (missing Core/NamesAndTypes.h) 2021-05-06 21:03:03 +03:00
kssenii
b96f047db7 Fix clang tidy 2021-05-06 07:02:13 +00:00
Azat Khuzhin
6348497d1b Cleanup IDatabase.h from extra headers 
From IDatabase.h depends ~300 CU.

- drop unused
- use forward declaration whenever is possible
 2021-05-06 09:07:59 +03:00
kssenii
8900e93bd8 New impl 2021-05-06 00:52:59 +00:00
kssenii
ed5c2321cc Some fixes 2021-05-04 10:57:05 +00:00
kssenii
eff26f9d54 Add comments, small improvements 2021-05-03 21:42:06 +00:00
kssenii
3e3396bd9c Refactor code, add comments 2021-05-03 09:52:32 +00:00
feng lv
39f68bf5ff fix conflict 2021-05-02 16:33:45 +00:00
kssenii
14d355a6ac Fix background update 2021-05-02 11:53:03 +00:00
kssenii
32b2917141 Update s3/* 2021-05-01 18:10:04 +03:00
kssenii
ee06936596 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-01 17:24:31 +03:00
kssenii
14be3a0783 Better reload from snapshot 2021-05-01 11:49:45 +00:00
kssenii
7643d84095 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-30 22:02:51 +00:00
kssenii
e717b4c5c7 Update databases/* 2021-04-30 22:38:07 +03:00
kssenii
6e0cfdaeb5 Update Databases/* part 2 2021-04-30 20:24:36 +03:00
kssenii
1b75f12fcb Update Databases/* 2021-04-30 19:48:42 +03:00
Maksim Kita
318c4bb80d Add examples folder filter to ya.make.in 2021-04-30 11:25:52 +03:00
Maksim Kita
2e5f97c6ff Fixed code review issues 2021-04-27 22:09:03 +03:00
Maksim Kita
8e508a37b7 Fixed dictionaries_lazy_load during StorageDictionary CREATE query 2021-04-27 12:15:35 +03:00
Maksim Kita
f37599112a Added EXCHANGE DICTIONARIES query support 2021-04-26 23:36:15 +03:00
Maksim Kita
c93666e0de Fixed code review issues 2021-04-26 13:52:41 +03:00
Maksim Kita
43680d8ced Fixed tests 2021-04-26 13:52:41 +03:00
Maksim Kita
561b045991 Fixed tests 2021-04-26 13:52:41 +03:00
Maksim Kita
64f49a981b Fixed tests 2021-04-26 13:52:41 +03:00
Maksim Kita
2e3e90b25b Fixed tests 2021-04-26 13:52:41 +03:00
Maksim Kita
e5079c87db Added test 2021-04-26 13:52:41 +03:00
Maksim Kita
3cb4d1bdcb Fixed style check 2021-04-26 13:52:41 +03:00
Maksim Kita
a9917a6540 Refactored StorageDictionary 2021-04-26 13:52:41 +03:00
tavplubix
fc2e2848bf
Merge pull request #23413 from TCeason/modify_MaterializeMySQL_var_check 
modify MaterializeMySQL check MySQL var logical
 2021-04-26 09:52:59 +03:00
feng lv
0f5c05854c fix conflict 2021-04-23 12:22:51 +00:00
feng lv
4ffe199d39 Implement table comments 2021-04-23 12:18:23 +00:00
TCeason
51fe9fc771 modify MaterializeMySQL check MySQL var logical 
Supprot for MySQL 5.7.9: https://github.com/ClickHouse/ClickHouse/issues/4006#issuecomment-817622685
 2021-04-21 10:16:04 +08:00
tavplubix
70c89eeff6
Merge pull request #23200 from ClickHouse/fix_prewhere_in_materialize_mysql 
Fix prewhere in MaterializeMySQL
 2021-04-20 20:05:07 +03:00
tavplubix
738d8a757b
Merge pull request #23189 from ClickHouse/minor_fixes_in_attach_query 
Minor fixes in ATTACH query
 2021-04-20 20:04:26 +03:00
alexey-milovidov
cbbff1bc4b
Merge pull request #23319 from ClickHouse/mysql-insert-without-database 
Allow to INSERT into remote MySQL storage with empty database #19281
 2021-04-20 19:30:44 +03:00
tavplubix
bef0ab182e
Merge branch 'master' into fix_prewhere_in_materialize_mysql 2021-04-20 15:38:48 +03:00
Alexander Tokmakov
4108a235f5 make separate functions 2021-04-20 14:22:02 +03:00
Kseniia Sumarokova
b43656bf89
Merge pull request #22697 from TCeason/feature/add_MaterializeMySQL_sync_bytes_judgment 
Add MySQL read history data bytes judgment
 2021-04-20 10:26:42 +03:00
Alexey Milovidov
ac68dd8e97 Fix bad code; allow to INSERT into MySQL without database name 2021-04-20 00:29:37 +03:00
TCeason
472c131420 Add MySQL read history data bytes judgment 2021-04-19 10:51:50 +08:00
kssenii
88b50621d4 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-18 14:40:37 +00:00
Alexander Tokmakov
0354312294 minor fixes in attach query 2021-04-17 22:43:02 +03:00
Alexander Tokmakov
3b4bd18809 fix prewhere in materialize mysql 2021-04-16 18:51:27 +03:00
Alexander Tokmakov
ebf0a3119e fix data_type_default_nullable on attach 2021-04-16 13:29:33 +03:00
Christian
969b63d1ac
Excludes views from syncing in MaterializeMySQL (#22760) 
* Excludes views from syncing in MaterializeMySQL

* Adds integration test for ignoring MySQL views on initial replication

* Fixes bug in initial integration test for excluding views from replicating to ClickHouse on initial replication

* Fixes bug in initial integration test for excluding views from replicating to ClickHouse on initial replication

* Replace assert with check_query

Co-authored-by: Christian Frøystad <christian@securepractice.no>
Co-authored-by: Ivan <5627721+abyss7@users.noreply.github.com>
 2021-04-14 14:19:07 +03:00
kssenii
01075677cf Better 2021-04-11 20:22:33 +00:00
kssenii
027f67affb Fix checks 2021-04-11 09:54:10 +00:00
kssenii
beae1c5fa4 Use ContextPtr 2021-04-11 08:08:32 +00:00
kssenii
c1adece94f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-11 06:00:18 +00:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
kssenii
bc228f4010 Better way to drop nested table for single storage 2021-04-10 19:26:53 +00:00
kssenii
1c501e7d97 Support replica identity index 2021-04-10 16:10:28 +00:00
kssenii
12f98e8b11 Much better 2021-04-09 10:22:06 +00:00
kssenii
f0be5c6938 Fix style, build, ya check 2021-04-09 08:07:56 +00:00
kssenii
3e41c82df1 Add mutex to tryGetTable only for materialized table (not nested) 2021-04-08 23:38:27 +00:00
kssenii
bd207daa74 Use only atomic database 2021-04-08 22:38:46 +00:00
kssenii
4482a35a3a Check table in a suggested way 2021-04-08 20:39:16 +00:00
kssenii
8586565ce9 Move connection-related filed to Core/PostgreSQL 2021-04-08 12:43:02 +00:00
kssenii
9fa21a6272 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-08 12:18:54 +00:00
Azat Khuzhin
f157278b72 Safer SCOPE_EXIT 
It executes the code in the dtor, that should never throw.
 2021-04-06 21:10:37 +03:00
Kseniia Sumarokova
ad85467128
Merge pull request #22668 from kssenii/postgres-add-bool 
Add bool type in postgres engine
 2021-04-06 13:29:33 +03:00
kssenii
cc2fdc6c62 Add bool type 2021-04-05 14:49:52 +00:00
kssenii
13d1f17d3e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-04-03 08:00:09 +00:00
kssenii
2de6b99f84 Better names 2021-04-02 16:12:14 +00:00
kssenii
99f0b08ac3 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-04-01 13:48:33 +00:00
kssenii
522d170588 Add default port 2021-04-01 11:30:20 +00:00
kssenii
747967a9bb Better 2021-04-01 10:27:24 +00:00
kssenii
ba5ddd5c32 Do not parse replicas in PoolWithFailover 2021-03-31 14:03:52 +00:00
Maksim Kita
55984e849c
Merge branch 'master' into hierarchy-dictionaries-updated 2021-03-31 15:55:07 +03:00
kssenii
ce05087b1b Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-03-30 17:27:10 +00:00
Alexander Tokmakov
a5b07f2f03 Merge branch 'master' into test_multiple_nodes 2021-03-29 22:43:25 +03:00
tavplubix
3c0f5a57ec
Merge pull request #21535 from ClickHouse/distributed_ddl_improvements 
Distributed DDL improvements
 2021-03-29 22:40:11 +03:00
tavplubix
e3cf2d99b7
Update DatabaseReplicated.cpp 2021-03-29 16:44:41 +03:00
kssenii
22b515fbc9 Add namespace, simplify names 2021-03-27 20:14:02 +00:00
kssenii
f40c582e7a Replicas/shards for postgres storage 2021-03-27 20:11:48 +00:00
kssenii
ae868208c2 Use pool with failover in mysql storage 2021-03-27 14:39:45 +00:00
Maksim Kita
9f2f0d1095 Refactored hierarchy dictionaries interface 2021-03-26 18:42:32 +03:00
tavplubix
5dfd0045a6
Merge branch 'master' into distributed_ddl_improvements 2021-03-22 18:13:19 +03:00
kssenii
ed158e2b77 Proper setting 2021-03-20 14:42:03 +00:00
kssenii
4c37ff3fbc Less lambdas 2021-03-20 13:17:30 +00:00
kssenii
91b79f95bc Better templates 2021-03-20 12:24:35 +00:00
kssenii
caffccd73e Fixes 2021-03-20 11:53:41 +00:00
kssenii
6bb81630e5 Small fixes 2021-03-20 10:32:11 +00:00
kssenii
619f0f51fd Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-03-20 09:47:05 +00:00
kssenii
3b32c8fe9a Block on empty connection pool 2021-03-19 21:00:45 +00:00
kssenii
9057aad798 Better version 2021-03-19 08:11:36 +00:00
kssenii
f1ef87d966 Fix 2021-03-18 20:04:54 +00:00
Alexander Tokmakov
ce31d4608b fix tests 2021-03-18 22:13:21 +03:00
Alexander Tokmakov
343b8c906d Merge branch 'master' into test_multiple_nodes 2021-03-18 16:01:04 +03:00
Alexander Tokmakov
fdae70df6c fix tests 2021-03-18 15:50:03 +03:00
kssenii
3903d59d30 Better 2021-03-17 14:34:04 +00:00
kssenii
ae64a24844 Add connection pool 2021-03-17 13:55:47 +00:00
Nikita Mikhaylov
f51b41b7a4
Merge pull request #21710 from kssenii/replica-priorities 
Support replicas priority for postgres dictionary source
 2021-03-17 16:02:00 +03:00
kssenii
87c740730b Rename to MaterializePostgreSQL 2021-03-17 10:52:12 +00:00
Alexander Tokmakov
ff79be8af9 fix 2021-03-16 23:01:20 +03:00
kssenii
f8a99804c1 Add double quotes 2021-03-16 11:15:43 +00:00
kssenii
a03e849a8f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-03-16 10:06:10 +00:00
tavplubix
6a265f468e
Merge branch 'master' into distributed_ddl_improvements 2021-03-15 11:24:14 +03:00
tavplubix
6ff8f468f6
Update DatabaseReplicated.cpp 2021-03-15 11:22:42 +03:00
kssenii
8eea6a87e9 Support replicas priority for postgres dictionary source 2021-03-14 08:51:24 +00:00
tavplubix
26ebd54eaf
Merge pull request #20961 from havardk/reconnect 
MaterializeMySQL: Attempt to reconnect to MySQL if the connection is lost
 2021-03-10 20:07:11 +03:00
Alexander Tokmakov
ba3d51383c fix 2021-03-10 00:41:04 +03:00
tavplubix
5a6cc876e4
Merge pull request #20626 from ClickHouse/fix_force_drop_materialized_view 
Make force_drop_table work with MaterializedView
 2021-03-09 20:05:51 +03:00
Alexander Tokmakov
83b3e4e0f5 allow to use Cluster from Replicated db 2021-03-09 20:05:24 +03:00
Alexander Tokmakov
00b939e5a3 fix 2021-03-09 13:24:08 +03:00
Alexander Tokmakov
5c5d11cfce enable some tests 2021-03-09 03:48:37 +03:00
Alexander Tokmakov
163e27b299 support query and session settings for distributed DDL 2021-03-09 01:57:53 +03:00
Alexander Tokmakov
5070b5b85b detach permanently setting for Replicated db 2021-03-09 00:34:52 +03:00
Alexander Tokmakov
2022b90919 configurable output mode for distributed DDL 2021-03-08 23:35:09 +03:00
Alexander Tokmakov
2484781070 fix MV recovery in Replicated db 2021-03-08 20:26:38 +03:00
Alexander Tokmakov
e8987f799e Merge branch 'fix_force_drop_materialized_view' into just_another_fix_for_ddl_worker 2021-03-05 15:59:27 +03:00
Haavard Kvaalen
641b1b249e Attempt to reconnect to MySQL 
For MaterializeMySQL databases, attempt to reconnect if the connection
to MySQL is lost.  The existing setting
`max_wait_time_when_mysql_unavailable` is used to control how often we
attempt to reconnect.  This setting can now be set to a negative value
to disable reconnects.
 2021-03-04 08:49:06 +01:00
Haavard Kvaalen
8e95d6e174 Add method to metadata class to start replication 
Previously this was done by the constructor.  This makes it
possible to restart replication with an existing metadata object.
 2021-03-04 08:49:06 +01:00
Haavard Kvaalen
e6711675a1 Make handling of unavailable MySQL consistent 
If MySQL was unavailable when loading an existing database on startup,
we would previously load the database, fail to start synchronization,
but allow queries to the database.  Change this to only allow queries
if the `allows_queries_when_mysql_lost` setting is on, so that the
behavior is consistent with what happens if the connection to MySQL is
lost while ClickHouse is running.

Also retry connection to MySQL if MySQL is unavailable when ClickHouse
is started (we would previously reconnect only if the connection was
lost during the initial dump of existing data).
 2021-03-04 08:49:06 +01:00
kssenii
a3c7402200 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-03-02 07:07:56 +00:00
tavplubix
10aa26576c
Update DatabaseReplicatedWorker.cpp 2021-03-01 17:11:45 +03:00
Azat Khuzhin
955974a8b5 DDLWorker: avoid NULL dereference on termination and failed zookeeper initialization 
Log snipped shows the problem:

    2021.02.24 04:40:29.349181 [ 39 ] {} <Warning> DDLWorker: DDLWorker is configured to use multiple threads. It's not recommended because queries can be reordered. Also it may cause some unknown issues to appear.
    2021.02.24 04:40:29.349516 [ 39 ] {} <Information> Application: Ready for connections.
    2021.02.24 04:40:29.349602 [ 74 ] {} <Debug> DDLWorker: Started DDLWorker cleanup thread
    2021.02.24 04:40:29.349639 [ 73 ] {} <Debug> DDLWorker: Starting DDLWorker thread
    2021.02.24 04:40:29.349698 [ 73 ] {} <Debug> DDLWorker: Started DDLWorker thread
    2021.02.24 04:40:29.352548 [ 73 ] {} <Error> virtual void DB::DDLWorker::initializeMainThread(): Code: 999, e.displayText() = Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: 192.168.112.3:2181
    Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 21.3.1.1), 192.168.112.3:2181
    Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 21.3.1.1), 192.168.112.3:2181
    Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 21.3.1.1), 192.168.112.3:2181
     (Connection loss), Stack trace (when copying this message, always include the lines below):

    0. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Coordination::Error, int) @ 0xfe93923 in /usr/bin/clickhouse
    1. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Coordination::Error) @ 0xfe93ba2 in /usr/bin/clickhouse
    2. Coordination::ZooKeeper::connect(std::__1::vector<Coordination::ZooKeeper::Node, std::__1::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0xfed3a01 in /usr/bin/clickhouse
    3. Coordination::ZooKeeper::ZooKeeper(std::__1::vector<Coordination::ZooKeeper::Node, std::__1::allocator<Coordination::ZooKeeper::Node> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Timespan, Poco::Timespan, Poco::Timespan) @ 0xfed2222 in /usr/bin/clickhouse
    4. zkutil::ZooKeeper::init(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfe961cd in /usr/bin/clickhouse
    5. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfe97a96 in /usr/bin/clickhouse
    6. void std::__1::allocator_traits<std::__1::allocator<zkutil::ZooKeeper> >::__construct<zkutil::ZooKeeper, Poco::Util::AbstractConfiguration const&, char const (&) [10]>(std::__1::integral_constant<bool, true>, std::__1::allocator<zkutil::ZooKeeper>&, zkutil::ZooKeeper*, Poco::Util::AbstractConfiguration const&, char const (&) [10]) @ 0xed98387 in /usr/bin/clickhouse
    7. DB::Context::getZooKeeper() const @ 0xed75190 in /usr/bin/clickhouse
    8. DB::DDLWorker::getAndSetZooKeeper() @ 0xedb81c9 in /usr/bin/clickhouse
    9. DB::DDLWorker::initializeMainThread() @ 0xedc9eb0 in /usr/bin/clickhouse
    10. DB::DDLWorker::runMainThread() @ 0xedb5d01 in /usr/bin/clickhouse
    11. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() @ 0xedcafa1 in /usr/bin/clickhouse
    12. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x892651f in /usr/bin/clickhouse
    13. ? @ 0x8929fb3 in /usr/bin/clickhouse
    14. start_thread @ 0x8ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so
    15. __clone @ 0xfddef in /lib/x86_64-linux-gnu/libc-2.31.so
     (version 21.3.1.1)
    ...
    2021.02.24 04:40:30.025278 [ 41 ] {} <Trace> BaseDaemon: Received signal 15
    2021.02.24 04:40:30.025336 [ 41 ] {} <Information> Application: Received termination signal (Terminated)
    ...
    2021.02.24 04:40:30.582078 [ 39 ] {} <Information> Application: Closed all listening sockets.
    2021.02.24 04:40:30.582124 [ 39 ] {} <Information> Application: Closed connections.
    2021.02.24 04:40:30.583770 [ 39 ] {} <Information> Application: Shutting down storages.
    2021.02.24 04:40:30.583932 [ 39 ] {} <Information> Context: Shutdown disk data
    2021.02.24 04:40:30.583951 [ 39 ] {} <Information> Context: Shutdown disk default
    2021.02.24 04:40:30.584163 [ 46 ] {} <Trace> SystemLog (system.query_log): Terminating
    2021.02.24 04:40:30.586025 [ 39 ] {} <Trace> BackgroundSchedulePool/BgSchPool: Waiting for threads to finish.
    2021.02.24 04:40:34.352701 [ 73 ] {} <Debug> DDLWorker: Initialized DDLWorker thread
    2021.02.24 04:40:34.352758 [ 73 ] {} <Debug> DDLWorker: Scheduling tasks
 2021-02-28 05:45:12 +03:00
kssenii
6554965711 Fix build 2021-02-23 10:46:41 +00:00
Kseniia Sumarokova
bf88d693d5
Update fetchPostgreSQLTableStructure.cpp 2021-02-23 11:59:38 +03:00
Kseniia Sumarokova
6c5781cf34
Update fetchPostgreSQLTableStructure.cpp 2021-02-23 11:48:51 +03:00
Kseniia Sumarokova
b2850ec94f
Merge branch 'master' into fix-postgres-numeric 2021-02-23 11:40:28 +03:00
kssenii
4cdb55babe Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-02-22 12:39:35 +00:00
kssenii
ace76bb7fa Better, remove debug logs 2021-02-22 12:35:53 +00:00
kssenii
8a48bb24ce Fix typos, comments 2021-02-21 23:13:58 +00:00
kssenii
427aad80a1 Avoid table does not exist errors if nested is unavailable 2021-02-21 11:48:13 +00:00
tavplubix
aa48d967b7
Update DatabaseAtomic.cpp 2021-02-21 01:13:14 +03:00
kssenii
d590c32708 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-02-20 18:31:26 +00:00
Alexey Milovidov
64e76a4a8d Minor changes in Decimal 2021-02-20 21:13:36 +03:00
tavplubix
103d569885
Merge branch 'master' into fix_force_drop_materialized_view 2021-02-20 20:41:03 +03:00
kssenii
0424770f68 Handle ddl part 1 2021-02-20 16:21:53 +00:00
Alexander Tokmakov
863c099254 fix 2021-02-20 15:57:07 +03:00
kssenii
a38a31c954 Fix typos check 2021-02-20 09:58:24 +00:00
kssenii
f820047cc8 Fix 2021-02-20 06:52:10 +00:00
Alexander Tokmakov
2a36d6cb55 review suggestions 2021-02-20 02:41:58 +03:00
kssenii
3107f82a3b Fix clang tidy 2021-02-18 18:25:49 +00:00
kssenii
2aad067e7c Support conversion for postgres numeric without precision and scale 2021-02-18 12:16:58 +00:00
kssenii
1480e95179 Fix build, fix arcadia 2021-02-18 06:06:37 +00:00
Alexander Tokmakov
13ae988eff make force_drop_table work with materialized view 2021-02-17 17:23:07 +03:00
Alexander Tokmakov
7b54b892b5 fix 2021-02-16 17:05:58 +03:00
kssenii
272431bcae Fix types check, better tests, try fix build 2021-02-15 23:18:53 +00:00
Alexander Tokmakov
bf6f64a3fb Merge branch 'master' into database_replicated 2021-02-16 01:28:19 +03:00
Alexander Tokmakov
cf57c3b4a2 update comments 2021-02-15 23:01:21 +03:00
Alexander Tokmakov
9c7cf9e92e remove some debug code 2021-02-15 13:26:34 +03:00
Alexander Tokmakov
320fd6b264 startup without zk 2021-02-15 03:04:46 +03:00
kssenii
95c93aca41 Better table sync 2021-02-13 21:03:25 +00:00
kssenii
44f4f1a412 Allow to replicate a subset of database tables 2021-02-13 20:09:01 +00:00
kssenii
219dece1d0 Slightly better 2021-02-12 21:15:00 +00:00
kssenii
5bc0010f94 Better 2021-02-12 21:14:36 +00:00
kssenii
ed6d19b8a6 Add PostgreSQLReplica database engine 2021-02-12 21:14:24 +00:00
Alexander Tokmakov
f64f9b672b fix 2021-02-12 19:22:01 +03:00
Alexander Tokmakov
c9cf63e958 fix 2021-02-12 01:23:40 +03:00
Alexander Tokmakov
9afb16759e fix 2021-02-11 18:14:38 +03:00
Alexander Tokmakov
15256d86e5 better replica recovery and queue cleanup 2021-02-10 23:30:40 +03:00
Alexander Tokmakov
2c6a0e74fb better replica creation 2021-02-09 18:14:20 +03:00
Alexander Tokmakov
78c1d69b8c better code 2021-02-08 22:36:17 +03:00
Alexander Tokmakov
91d0924665 write dictionaries metadata to zk 2021-02-08 12:46:30 +03:00
Alexander Tokmakov
7ce0ef2561 show clusters for replicated db 2021-02-08 12:14:17 +03:00
Alexander Tokmakov
18f6b5bbad add timeouts 2021-02-04 22:41:44 +03:00
Alexander Tokmakov
066fb4c82b fix 2021-02-03 23:02:37 +03:00
Alexander Tokmakov
d010f97db0 Merge branch 'master' into database_replicated 2021-02-03 20:13:25 +03:00
Alexander Tokmakov
6456ccf0da better test 2021-02-02 22:39:04 +03:00
Alexander Tokmakov
0073c87d5d fix 2021-02-02 13:32:42 +03:00
Alexander Tokmakov
9da445e740 execute initial query in the same thread 2021-02-01 22:29:47 +03:00
Vitaly Baranov
7c8deae0e1 Fix starting the server with tables having columns' default expressions containing dictGet(). 
Allow getting return type of dictGet() without loading dictionary.
 2021-01-30 19:06:48 +03:00
Alexander Tokmakov
f20d5e3b41 fix 2021-01-26 20:51:25 +03:00
Alexander Tokmakov
3bd4d97353 Merge branch 'master' into database_replicated 2021-01-25 14:19:04 +03:00
tavplubix
4be33e81ee
Merge pull request #19344 from zhang2014/patch-2 
ISSUES-18684 fix MaterializeMySQL integration test failure
 2021-01-25 12:46:53 +03:00
zhang2014
4593ca80ff ISSUES-18684 fix MaterializeMySQL integration failure 2021-01-24 15:50:37 +08:00
kssenii
a98c6d333c Make sure not to log non-existent objects 2021-01-23 12:05:37 +00:00
kssenii
daab2c91bb Better 2021-01-21 21:15:11 +00:00
kssenii
c1702f34ee Add factories info into system.query_log 2021-01-21 15:46:37 +00:00
Alexander Tokmakov
7f97a11c84 Merge branch 'master' into database_replicated 2021-01-18 17:09:39 +03:00
tavplubix
071a6aeb4d
trigger CI 2021-01-18 16:53:33 +03:00
TCeason
a3cc6707b4 add MySQL Var check 
add MySQL var check: log_bin_use_v1_row_events=OFF
 2021-01-17 17:50:35 +08:00
kssenii
2222741e33 Fix style check 2021-01-16 09:17:04 +00:00
Alexey Milovidov
45380d45c8 Fix potential (safe) SQL-injection 2021-01-16 11:15:43 +03:00
kssenii
11ffa6293d Fix integration tests 2021-01-16 08:04:24 +00:00
Alexey Milovidov
593b6f5c83 Add assert 2021-01-16 00:25:58 +03:00
Alexey Milovidov
67f207d8ca More safe connection string 2021-01-16 00:18:51 +03:00
Alexey Milovidov
ef319bfa75 Slightly better 2021-01-15 23:33:04 +03:00
Alexey Milovidov
c16ecb0bdd It feels more correct 2021-01-15 23:18:55 +03:00
Alexey Milovidov
b1b09ed5ae Fix SQL injection 2021-01-15 23:14:51 +03:00
Alexey Milovidov
52cfc1d110 Minor changes 2021-01-15 23:09:31 +03:00
Alexey Milovidov
ba1921f380 Minor changes 2021-01-15 22:59:49 +03:00
Alexey Milovidov
a19e7edd14 Merge branch 'master' into kssenii-pg2ch 2021-01-15 17:33:19 +03:00
kssenii
38a9cba850 Fix 2021-01-11 10:55:38 +00:00
kssenii
c6c6b2d23c Fix style and types check 2021-01-10 13:01:15 +00:00
kssenii
d952b0897e Minor adjustments 2021-01-10 12:06:18 +00:00
kssenii
fc9de76f7d Fix incorrect getCreateTableQueryImpl with multidim arrays 2021-01-10 09:00:35 +00:00
Alexey Milovidov
76149947ef Remove useless headers 2021-01-10 05:48:57 +03:00
kssenii
0088d66d59 Hopefully better array conversions for insert query 2021-01-09 22:42:42 +00:00
Alexey Milovidov
8af19c3251 Fix Arcadia 2021-01-07 15:29:02 +03:00
Alexey Milovidov
60d4db421c Fix Arcadia 2021-01-07 06:45:12 +03:00
alexey-milovidov
72b142a00a
Merge branch 'master' into pg2ch 2021-01-06 23:18:59 +03:00
kssenii
e100aec086 Fix typos check 2020-12-31 11:15:05 +00:00
kssenii
62b06de97e Try fix build 2020-12-30 16:00:05 +00:00
spongedc
494c9ac809 Merge remote-tracking branch 'origin/master' into support_show_create_view 2020-12-29 10:20:06 +08:00
kssenii
fa3297bf72 Fix build, update libpq 2020-12-28 19:16:10 +00:00
kssenii
9b25890674 Generate ya.make, fix fast test 2020-12-28 12:54:52 +00:00
kssenii
8f8920a7ee Add table cache, better drop table 2020-12-27 15:52:15 +00:00
kssenii
00a37404ca Add postgresql database engine 2020-12-27 12:26:29 +00:00
Alexander Tokmakov
42911e2438 implement CREATE OR REPLACE TABLE 2020-12-25 19:32:09 +03:00
Yatsishin Ilya
ce1e2b18a3 Fix codespell warnings. Split style checks. Update style checks docker 2020-12-24 13:17:52 +03:00
spongedc
a277308ac6 Merge remote-tracking branch 'origin/master' into support_show_create_view 2020-12-23 11:56:46 +08:00
Alexander Tokmakov
52ef4289fe enable optimize_on_insert for MaterializeMySQL 2020-12-17 16:05:38 +03:00
tavplubix
7e73b6acd8
Merge pull request #17642 from filimonov/detach-permanently 
DETACH TABLE PERMANENTLY
 2020-12-17 14:26:51 +03:00
Nikolai Kochetov
44d2194326 Try fix integration tests. 2020-12-16 11:52:04 +03:00
spongedc
38c0827881 Merge remote-tracking branch 'origin/master' into support_show_create_view 2020-12-16 12:26:56 +08:00
spongedc
a0083e23d6 Fix compile error 2020-12-16 12:07:50 +08:00
Mikhail Filimonov
9df7ecb8e6
fixes 2020-12-15 18:06:23 +01:00
Mikhail Filimonov
27ff3a5214
Rework using a separate flag file instead of renames 2020-12-15 15:46:13 +01:00
spongedc
dc27e0a4fd Fix error code 2020-12-15 20:29:26 +08:00
spongedc
8b2feaeca2 Support SHOW CREATE VIEW Syntax 2020-12-14 22:37:25 +08:00
Mikhail Filimonov
24abcc970b
Tests and fixes 2020-12-14 15:31:13 +01:00
Mikhail Filimonov
87ea1528cb
support for Lazy database 2020-12-14 15:31:13 +01:00
Mikhail Filimonov
4ba80996b0
Detach permanently for DatabaseConnectionMySQL 2020-12-14 15:31:12 +01:00
Mikhail Filimonov
799141362d
Prevent attempts to recreate the table which was detached permanently 2020-12-14 15:31:11 +01:00
Mikhail Filimonov
f6097cf818
Adding DETACH TABLE PERMANENTLY 2020-12-14 15:28:59 +01:00
Alexander Tokmakov
58a31e4882 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-12-11 16:14:03 +03:00
tavplubix
5f78280aec
Merge pull request #17915 from zhang2014/better_exception_message 
Better exception message for MaterializeMySQL
 2020-12-10 19:26:20 +03:00
tavplubix
f13de96afc
Merge pull request #17366 from zhang2014/fix/ISSUES-16835 
ISSUES-16835 try fix miss match header with MySQL SHOW statement
 2020-12-09 23:12:47 +03:00
zhang2014
bf30aa5ff1 Code style 2020-12-09 16:50:40 +08:00
zhang2014
ef15b1f1fb Better exception message for MaterializeMySQL 2020-12-09 16:46:54 +08:00
Ivan
0ba712a2d3
Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-12-08 16:45:51 +03:00
Winter Zhang
972cb41fae
Merge branch 'master' into fix/ISSUES-16835 2020-12-08 15:32:58 +08:00
Alexander Tokmakov
39532f7d9e slightly better DDLWorker initialization and restarting 2020-12-03 21:14:27 +03:00
Alexander Tokmakov
a0ef3a113a Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-12-02 18:29:02 +03:00
Alexander Tokmakov
1a4bd67736 fixes 2020-12-01 20:20:42 +03:00
Alexander Tokmakov
19c8399eb0 Merge branch 'master' into database_replicated 2020-12-01 17:33:07 +03:00
Vasily Nemkov
87945fb68b Merge branch 'master' into HEAD 2020-11-30 23:25:57 +03:00
Alexander Tokmakov
ab197a49c8 better code, fixes 2020-11-30 23:12:43 +03:00
Alexander Tokmakov
f1a52a609b separate DatabaseReplicatedDDLWorker 2020-11-27 17:04:03 +03:00
zhang2014
a74235397e ISSUES-16835 try fix miss match header with MySQL SHOW statement 2020-11-24 23:31:43 +08:00
Alexander Tokmakov
e82c63e05c fixes 2020-11-24 15:28:54 +03:00
Alexander Tokmakov
dce0bf8efc Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-11-24 15:28:31 +03:00
Alexander Tokmakov
5ccf975a4a Merge branch 'master' into database_replicated 2020-11-24 14:38:02 +03:00
Alexander Tokmakov
dad21ee684 maintain metadata in zk 2020-11-24 13:24:39 +03:00
taichong
a6c2a90756 modify integration test materialize mysql database 
1. fix asan test err

2. reset pymysql conn if ping err

3. finish mysql & ck killed test

4. modify check_query
 2020-11-23 17:55:07 +08:00
Alexey Milovidov
fbbf439338 Merge branch 'master' into Enmk-DateTime64_fixes_comparison 2020-11-23 05:46:17 +03:00
tavplubix
5cc9cb01cd
Merge pull request #16751 from amosbird/globalcontext 
Make global_context consistent.
 2020-11-22 18:46:17 +03:00
Nikolai Kochetov
bac1def5f9
Merge pull request #17134 from abyss7/tcp-port 
Implement tcpPort() function for tests
 2020-11-20 20:32:55 +03:00
Alexander Tokmakov
7ab4445e99 try another approach 2020-11-20 19:06:27 +03:00
tavplubix
3e335c1e95
Merge pull request #17091 from zhang2014/fix/binlog_checksum 
Fix MaterializeMySQL SYNC failure when modify binlog_checksum
 2020-11-20 13:25:14 +03:00
Amos Bird
1d9d586e20
Make global_context consistent. 2020-11-20 18:23:14 +08:00
Alexander Tokmakov
2283906a11 try support replica recovery 2020-11-19 13:34:45 +03:00
Ivan Lezhankin
fdd1c182ab Merge remote-tracking branch 'upstream/master' into tcp-port 2020-11-18 16:22:00 +03:00
Ivan Lezhankin
f897f7c93f Refactor IFunction to execute with const arguments 2020-11-17 16:24:45 +03:00
alexey-milovidov
b494b18748
Merge pull request #17108 from ClickHouse/ya-make-owners 
Add "OWNER" to ya.make files.
 2020-11-17 05:19:46 +03:00
Alexey Milovidov
24f4fa6edf Follow Arcadia ya.make rules 2020-11-17 00:16:50 +03:00
Alexey Milovidov
3df04ce0c2 Follow Arcadia ya.make rules 2020-11-16 21:24:58 +03:00
Winter Zhang
f6a205d64b
Merge branch 'master' into fix/binlog_checksum 2020-11-16 21:53:22 +08:00
zhang2014
6ef93ac73c Try fix MaterializeMySQL SYNC with modify binlog_checksum 2020-11-16 21:46:36 +08:00
Vasily Nemkov
63d168d34e Fixed MySQL and DateTime64 integration. 
Also updated other places, like type helpers/dispatchers where DateTime64
was implicitly supported before as Decimal64.
 2020-11-16 09:02:41 +03:00
Alexander Tokmakov
b0262b3d06 better replica creation 2020-11-13 21:35:45 +03:00
Alexander Tokmakov
0731bfb7be Merge branch 'master' into database_replicated 2020-11-12 21:04:18 +03:00
Alexander Tokmakov
2a6c0b9180 try reuse DDLWorker in DatabaseReplicated 2020-11-12 20:35:29 +03:00
taichong
010e12eb4b Replace std::*stringstreams 2020-11-12 13:49:56 +08:00
TCeason
3912ed4965 Merge branch 'master' into add_sync_user_privs_exception 2020-11-12 09:34:08 +08:00
taichong
e2a9fe44b0 modify test error 2020-11-12 08:17:13 +08:00
taichong
5c1a01897a Revise the code according to the review 2020-11-11 19:27:37 +08:00
taichong
d907afa77e throw exception when mysql sync user privs error. 2020-11-11 19:27:37 +08:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
Alexander Tokmakov
62ff00ee8b use WriteBuffer in formatAST(...) 2020-11-09 19:05:40 +03:00
tavplubix
6fb077c2f3
Merge pull request #16704 from TCeason/make_sync_ddl_execute_atomic 
sync MySQL DDL atomicly
 2020-11-09 13:47:02 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
taichong
9ce6c4dcb8 according to review modify code 2020-11-06 15:04:05 +08:00
Alexey Milovidov
b56486510f Improve Arcadia 2020-11-05 12:55:01 +03:00
taichong
fb5e03ebaf sync MySQL DDL atomicly 
Now ClickHouse will write metadata and then execute DDL from MySQL.

But If ClickHouse is crashed and it just begin to execute DDL, we
restart the server and then the sync will may be stopped.

e.g.:

MySQL : create table foo.a(id int key); uuid1:1
ClickHouse : write uuid1:1 to .metadata -> server crashed;

Because of ClickHouse's metadata records: uuid1:1, if MySQL run a
query:`insert into foo.a values(1);` ClickHouse will throw a exception:
the table foo.a is not exist.
 2020-11-05 17:01:14 +08:00
Alexander Tokmakov
d8ae9fcdb4 fixes, add shard name 2020-11-03 14:30:53 +03:00
Alexander Tokmakov
ac32809b6a fix #16482 2020-11-02 19:40:39 +03:00
Maxim Akhmedov
3627fabfb9 Remove -g0 form Arcadia build settings. 2020-10-29 17:37:23 +03:00
alexey-milovidov
bf3acecf6f
Merge pull request #16032 from Avogar/mysql_bug 
Don't touch MySQL database if it's unnecessary
 2020-10-29 04:14:26 +03:00
tavplubix
22847613d1
Merge pull request #16437 from ClickHouse/addition_to_16127 
Addition to #16127
 2020-10-28 13:03:13 +03:00
alesapin
8a6be1602b
Merge pull request #16364 from ClickHouse/fix_uuid_mapping_exists 
Minor improvements in DatabaseCatalog
 2020-10-28 10:54:39 +03:00
Alexander Tokmakov
b5ccb5ed5b review suggestions 2020-10-27 23:52:49 +03:00
Alexander Tokmakov
dba8c85ae4 do not wait while holding lock 2020-10-27 22:56:54 +03:00
Alexander Kuzmenkov
6509f1eb39
Merge branch 'master' into minumum 2020-10-27 21:38:23 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell 2020-10-27 12:04:03 +01:00
Alexander Tokmakov
cd14f095ab fix tests 2020-10-26 18:12:16 +03:00
Alexander Tokmakov
636da1318b Merge branch 'master' into database_replicated 2020-10-26 15:00:43 +03:00
Alexey Milovidov
d5e277f080 Better exceptions rethrow 2020-10-26 06:35:57 +03:00
Alexander Tokmakov
f8f2d62d70 minor improvements in DatabaseCatalog 2020-10-25 20:27:21 +03:00
Alexander Tokmakov
478eb0b8a5 fix 2020-10-22 23:32:47 +03:00
Alexander Tokmakov
72dbb43235 Merge branch 'master' into fork-valbaturin-replication-db-engine 2020-10-20 19:14:54 +03:00
Azat Khuzhin
32c449c34f Add IDatabaseTablesIterator::reset() interface 2020-10-19 00:16:10 +03:00
alesapin
0bb4480fee
Merge pull request #16031 from ClickHouse/database_atomic_patch 
Minor fix for DatabaseAtomic
 2020-10-16 10:08:04 +03:00
Alexander Tokmakov
2f9611f1ae fix logical error with explicit UUIDs 2020-10-15 19:01:23 +03:00
Pavel Kruglov
ee8a9d2cf1 Don't touch MySQL database if it's unnecessary 2020-10-15 18:57:17 +03:00
Alexander Tokmakov
72b1339656 Revert "Revert "Write structure of table functions to metadata"" 
This reverts commit c65d1e5c70.
 2020-10-14 15:19:29 +03:00
tavplubix
c65d1e5c70
Revert "Write structure of table functions to metadata" 2020-10-14 13:59:29 +03:00
alexey-milovidov
f60ccb4edf
Merge pull request #14295 from ClickHouse/write_structure_of_table_functions 
Write structure of table functions to metadata
 2020-10-13 23:56:09 +03:00
Azat Khuzhin
d16adddb41 Fix drop of materialized view with inner table in Atomic database 
Materialized view drop the inner table (if any) drop the drop() method,
and this will lead to essure recursive drop() from the drop worker
thread, which will hang:

    3  std::__1::condition_variable::wait<> ()
    4  DB::DatabaseCatalog::waitTableFinallyDropped (this=0x7ffff7831500, uuid=...) at ../src/Interpreters/DatabaseCatalog.cpp:828
    5  DB::InterpreterDropQuery::executeToTable (this=this@entry=0x7fff16ff88a0, table_id_=..., query=...) at ../src/Interpreters/InterpreterDropQuery.cpp:135
    6  DB::InterpreterDropQuery::execute (this=this@entry=0x7fff16ff88a0) at ../contrib/libcxx/include/string:1474
    7  DB::executeDropQuery (kind=kind@entry=DB::ASTDropQuery::Drop, global_context=..., target_table_id=...) at ../src/Storages/StorageMaterializedView.cpp:156
    8  DB::StorageMaterializedView::drop (this=0x7ffefc348f40) at ../src/Storages/StorageMaterializedView.cpp:169
    9  DB::DatabaseCatalog::dropTableFinally (this=this@entry=0x7ffff7831500, table=...) at ../src/Interpreters/DatabaseCatalog.cpp:775
    10 DB::DatabaseCatalog::dropTableDataTask (this=0x7ffff7831500) at ../src/Interpreters/DatabaseCatalog.cpp:745

Fix this by dropping the inner table just before scheduling job the the
drop worker thread.
 2020-10-12 21:46:07 +03:00
Alexey Milovidov
269b6383f5 Check for #pragma once in headers 2020-10-10 21:37:02 +03:00
Alexander Tokmakov
6b2da861dc Merge branch 'master' into write_structure_of_table_functions 2020-10-02 20:46:58 +03:00
Alexander Tokmakov
32776da69a Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-10-02 20:09:37 +03:00
tavplubix
2f95609f98
Merge pull request #15448 from ClickHouse/allow_synchronous_drop_for_atomic_db 
Allow synchronous DROP/DETACH TABLE for Atomic
 2020-10-01 01:14:31 +03:00
tavplubix
c9eeb149fd
Merge pull request #15384 from zhang2014/fix/ISSUES-15365 
ISSUES-15365 fix attach mysql database database
 2020-09-30 11:27:15 +03:00
Alexander Tokmakov
e156e6115e add setting 2020-09-30 01:30:34 +03:00
Alexander Tokmakov
5d19ed04f6 allow synchronous drop/detach table for Atomic 2020-09-29 16:47:08 +03:00
zhang2014
9e94c077e7 IESSUES-15365 try fix integration test 2020-09-29 10:39:30 +08:00
zhang2014
e5a6d06820 ISSUES-15365 try fix integration test failure 2020-09-28 21:15:02 +08:00
zhang2014
e40d23cfe5 ISSUES-15365 add ya make 2020-09-28 19:11:40 +08:00
zhang2014
d3047d0783 ISSUES-15365 fix attach mysql database failure 2020-09-28 18:02:30 +08:00
Alexander Tokmakov
b0d99217fb Merge branch 'master' into write_structure_of_table_functions 2020-09-27 14:26:47 +03:00
Alexander Tokmakov
aa5e5013f2 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-09-27 14:11:53 +03:00
Alexander Tokmakov
34addcf61f add review suggestions 2020-09-26 22:18:28 +03:00
alexey-milovidov
e29c4c3cc4
Merge pull request #15024 from ClickHouse/fix_renameat2_einval_on_macos 
Fix EINVAL in renameat2 on MacOS
 2020-09-21 21:08:17 +03:00
tavplubix
7f629c8185
Update DatabaseMemory.cpp 2020-09-21 14:12:45 +03:00
alesapin
5ffee8808a
Alter remove column properties and TTLs (#14742) 2020-09-20 16:27:33 +03:00
Alexander Tokmakov
46aa307d0a fix EINVAL in renameat2 on MacOS 2020-09-19 17:19:06 +03:00
tavplubix
6e0bdaf46d
Merge pull request #14535 from zhang2014/fix/datetime 
ISSUES-4006 support decimal data type for MaterializedMySQL
 2020-09-19 14:05:32 +03:00
Alexander Tokmakov
1ca9a92b21 Merge branch 'master' into write_structure_of_table_functions 2020-09-18 21:09:23 +03:00
tavplubix
0597ad3150
fix typo 2020-09-17 21:47:21 +03:00
Alexander Tokmakov
1c37ac7c9d fix 2020-09-17 21:19:02 +03:00
Alexander Tokmakov
41e99cf261 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-09-17 20:28:50 +03:00
zhang2014
64032e22a2 Merge branch 'master' into fix/datetime 2020-09-18 00:28:09 +08:00
zhang2014
c5b56c24f8 ISSUES-14894 fix MaterializeMySQL temp metadata file exists 2020-09-17 14:44:05 +08:00
Alexander Tokmakov
0ecf8e595f cleanup code, add test 2020-09-15 16:41:53 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build (#14809) 2020-09-15 12:55:57 +03:00
Alexander Tokmakov
3671c91c37 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-09-15 12:43:01 +03:00
Alexander Tokmakov
a812f7eb75 allow using Atomic database inside MaterializeMySQL 2020-09-14 22:25:02 +03:00
BohuTANG
00b8d91eb8 Fix mysql replication TableMapEvent optional metadata ignore 2020-09-10 19:46:59 +08:00
zhang2014
094897fe69 Merge branch 'master' into fix/datetime 2020-09-09 23:37:54 +08:00
Alexander Tokmakov
d278d849e4 Merge branch 'master' into write_structure_of_table_functions 2020-09-09 17:12:35 +03:00
Vasily Nemkov
3973a17530
MySql datatypes dateTime64 and decimal (#11512) 2020-09-09 15:18:02 +03:00
Winter Zhang
18b4c5997d
Merge branch 'master' into fix/datetime 2020-09-08 11:32:48 +08:00
Alexey Milovidov
e3924b8057 Fix "Arcadia" 2020-09-08 01:14:13 +03:00
zhang2014
b7e9d5e72d ISSUES-4006 support decimal for MaterializedMySQL engine 2020-09-07 17:43:25 +08:00
Alexander Tokmakov
b840d741d0 Merge branch 'master' into write_structure_of_table_functions 2020-09-04 13:00:07 +03:00
zhang2014
dd0feeeaa1 ISSUES-4006 support datetime with precision 2020-09-03 11:03:28 +08:00
Alexey Milovidov
96038e9d54 Fix for Lazy database 2020-09-03 04:06:12 +03:00
Alexander Tokmakov
41dd393824 fixes 2020-09-01 20:55:35 +03:00
Alexander Tokmakov
56695727b2 Merge branch 'master' into write_structure_of_table_functions 2020-09-01 20:15:13 +03:00
Alexander Tokmakov
3a9779adb4 slightly better code 2020-09-01 17:41:49 +03:00
alesapin
a0b4cc78d6 Throw exception on alter for storages created from table functions 2020-09-01 10:26:31 +03:00
zhang2014
4182596961 ISSUES-14231 try fix wrong lexer 2020-08-29 12:28:04 +08:00
alesapin
c1ca09653e Update custom parts of storage AST only if it has extended definition 2020-08-28 14:30:53 +03:00
alesapin
0896b49533 Small fixes 2020-08-28 13:18:56 +03:00
Amos Bird
078b14610d
ALTER MODIFY SAMPLE BY 2020-08-27 22:31:30 +08:00
Alexander Tokmakov
969940b4c9 write table tructure for table function remote(...) 2020-08-26 23:55:40 +03:00
tavplubix
cff92c8ad3
Merge pull request #13820 from BohuTANG/mysql_replica_gtid_issue_4006 
ISSUES-4006 support MySQL GTID based replication #4006
 2020-08-26 15:02:53 +03:00
tavplubix
3b8ac01ed0
Merge pull request #13756 from ClickHouse/test-drop-database-race 
Fix race condition in DROP DATABASE
 2020-08-19 13:27:50 +03:00
Alexander Tokmakov
e686616756 add exclusive DDLGuard for database 2020-08-18 18:15:27 +03:00
BohuTANG
8c9236fefd ISSUES-4006 adapted GTID replication to Materialize MySQL 2020-08-17 22:12:41 +08:00
BohuTANG
adc6cce96e ISSUES-4006 fetch executed_gtid_set from replication position #4006 2020-08-17 15:38:09 +08:00
tavplubix
18ef911fe8
Merge pull request #13672 from zhang2014/refactor/materialize_mysql_database 
ISSUES-4006 split mysql protocol to multiple packets
 2020-08-16 13:55:11 +03:00
zhang2014
fe49bd3414 Merge branch 'master' of github.com:yandex/ClickHouse into refactor/materialize_mysql_database 2020-08-15 16:12:46 +08:00
Alexey Milovidov
52c3857225 Fix server restart after stress test 2020-08-15 06:10:57 +03:00
Alexander Tokmakov
0f1c4bc9c4 attach all system tables 2020-08-13 20:09:40 +03:00
zhang2014
96bd3ac34b ISSUES-4006 split mysql protocol 2020-08-13 20:41:36 +08:00
zhang2014
34f4c8972e ISSUES-4006 split replication packets 2020-08-13 20:07:02 +08:00
zhang2014
a8d3b5f65d ISSUES-4006 better exception message 2020-08-13 10:14:59 +08:00
tavplubix
4aff48dc3f
Merge pull request #10851 from zhang2014/feature/materialize_mysql_database 
ISSUES-4006 support MaterializeMySQL database engine
 2020-08-12 15:59:28 +03:00
zhang2014
2311cda334 ISSUES-4006 convert _sign & _version to materialized column 2020-08-10 22:32:05 +08:00
Alexey Milovidov
0c15f3b6c8 Allow server to startup if there are leftovers from unsuccessfull database creations 2020-08-10 05:58:08 +03:00
zhang2014
933f287db4 ISSUES-4006 resolve review comment & support alter renmae command 2020-08-09 20:10:30 +08:00
zhang2014
6fca8ad7df ISSUES-4006 fix segfault on ClickHouse shutdown when MySQL is not available 2020-08-07 16:49:27 +08:00
zhang2014
d6bba8cbbb ISSUES-4006 add settings for allows_query_when_mysql_lost & move dump event to logger 2020-08-07 14:12:27 +08:00
zhang2014
bd21f8e6b9 ISSUES-4006 fix conflicts commit when merge 2020-08-07 12:55:45 +08:00
zhang2014
e496225641 ISSUES-4006 try fix build failure & review comment 2020-08-07 12:53:33 +08:00
zhang2014
a3bc18bed2 ISSUES-4006 try resolve review comment 2020-08-06 12:03:27 +08:00
Alexey Milovidov
d91471ea00 Merge branch 'master' into stack-frams-size-limit 2020-08-04 15:28:47 +03:00
zhang2014
34f7fac76c ISSUES-4006 try fix test failure 2020-08-04 12:22:12 +08:00
Alexander Tokmakov
cf78a1f8fc try fix build 2020-08-04 10:31:27 +08:00
zhang2014
7ea2eee98a ISSUES-4006 add some check & add comment 2020-08-04 10:31:27 +08:00
zhang2014
c173bd391f ISSUES-4006 try fix build & test failure 2020-08-04 10:31:27 +08:00
zhang2014
6b452cf83d ISSUES-4006 try fix build failure & fix float 32 parser 2020-08-04 10:31:27 +08:00
zhang2014
c0db408c38 ISSUES-4006 try fix build failure 2020-08-04 10:31:27 +08:00
zhang2014
b76f218d90 ISSUES-4006 add mysql dml test 2020-08-04 10:31:27 +08:00
zhang2014
9996166860 ISSUES-4006 try fix check style 2020-08-04 10:31:27 +08:00
zhang2014
bd65a8f213 ISSUES-4006 try fix build failure 2020-08-04 10:31:27 +08:00
zhang2014
9d8234afd7 ISSUES-4006 try fix integration test 2020-08-04 10:31:27 +08:00
zhang2014
c5946f7697 ISSUES-4006 add mysql 8.0 integration test 2020-08-04 10:31:27 +08:00
zhang2014
74f354d840 ISSUES-4006 add some check before create materialize mysql database 2020-08-04 10:31:27 +08:00
zhang2014
95b8afb8eb ISSUES-4006 try fix unit test failure 2020-08-04 10:31:26 +08:00
zhang2014
e471f805b5 ISSUES-4006 try fix build failure & update integration test 2020-08-04 10:31:26 +08:00
zhang2014
1c17878f31 ISSUES-4006 try fix build failure 2020-08-04 10:31:26 +08:00
zhang2014
1930b8f584 ISSUES-4006 support drop materialize mysql database engine 2020-08-04 10:31:26 +08:00
zhang2014
eace0153bd ISSUES-4006 fix test failure 2020-08-04 10:31:26 +08:00
zhang2014
8f28df3514 ISSUES-4006 rename locality_data to MaterializeMySQL 2020-08-04 10:31:26 +08:00
zhang2014
cee16292a6 ISSUES-4006 fix restart failure 2020-08-04 10:31:26 +08:00
zhang2014
094ce895da ISSUES-4006 fix test failure 2020-08-04 10:31:26 +08:00
zhang2014
035760df9d ISSUES-4006 refactor for external ddl 2020-08-04 10:31:26 +08:00
zhang2014
6562597064 ISSUES-4006 refactor interpreter 2020-08-04 10:31:26 +08:00
zhang2014
1583f67681 ISSUES-4006 remove unused code 2020-08-04 10:31:26 +08:00
zhang2014
54c3db0567 ISSUES-4006 some refactor for query convert 2020-08-04 10:31:26 +08:00
zhang2014
5fb92359d2 ISSUES-4006 some refactor for query convert 2020-08-04 10:31:26 +08:00
zhang2014
3afffaf303 ISSUES-4006 rename sign and version column 2020-08-04 10:31:26 +08:00
zhang2014
101a13b5c3 ISSUES-4006 remove unused code 2020-08-04 10:31:26 +08:00
zhang2014
a9e1794bd3 ISSUES-4006 filter sign = -1 rows 2020-08-04 10:31:26 +08:00
zhang2014
9324c0ee49 ISSUES-4006 fix crash when dump data 2020-08-04 10:31:26 +08:00
zhang2014
1df8bfe4fb ISSUES-4006 fix build failure after refactor 2020-08-04 10:31:25 +08:00
zhang2014
4a99ca0d80 ISSUES-4006 refactor 2020-08-04 10:31:25 +08:00
zhang2014
0f75b2d0f0 ISSUES-4006 fix buidl failure 2020-08-04 10:31:25 +08:00
zhang2014
0336a4ad58 ISSUES-4006 suport synchronous MySQL incremental data[part 2] 2020-08-04 10:31:23 +08:00
zhang2014
0c52d425ba ISSUES-4006 suport synchronous MySQL incremental data 2020-08-04 10:29:54 +08:00
zhang2014
7203c6e186 ISSUES-4006 unlock table on exception 2020-08-04 10:29:54 +08:00
zhang2014
86c6be4cd9 ISSUES-4006 fix create query convert 2020-08-04 10:29:52 +08:00
zhang2014
bd18c1cbf8 ISSUES-4006 support dump all mysql data when create database 2020-08-04 10:27:57 +08:00
zhang2014
5d569f56a7 ISSUES-4006 support convert mysql create to clickhouse create 2020-08-04 10:26:23 +08:00
zhang2014
c53a48fe01 ISSUES-4006 support parse MySQL create query 2020-08-04 10:26:23 +08:00
Alexander Tokmakov
5b92f43f12 Merge branch 'master' into database_atomic_improvements 2020-08-03 14:35:40 +03:00
Alexey Milovidov
0fd7af8a63 Merge branch 'master' into stack-frams-size-limit 2020-08-03 00:06:00 +03:00
Alexey Milovidov
41111d1e57 Merge branch 'master' into stack-frams-size-limit 2020-08-02 19:18:33 +03:00
Alexey Milovidov
6f690b7c0d Normalize ya.make files, fix "Arcadia" build 2020-08-02 16:57:38 +03:00
Vitaly Baranov
668653600c Use SettingMaxThreads only in Settings, call getNumberOfPhysicalCPUCores() instead of SettingMaxThreads::getAuto(). 2020-07-31 19:11:27 +03:00
Vitaly Baranov
30b34e6a15 Use SettingSeconds only in Settings, use std::chrono::seconds in other places. 2020-07-31 19:11:27 +03:00
Alexey Milovidov
50bbfc848d Merge branch 'master' into stack-frams-size-limit 2020-07-21 14:45:15 +03:00
Alexander Tokmakov
cc25355989 fix 2020-07-18 19:33:39 +03:00
Alexander Tokmakov
c27de9b424 fixes 2020-07-17 16:11:44 +03:00
Alexander Tokmakov
ebf98ba006 implement reneme dictionary query 2020-07-17 00:41:26 +03:00
Alexander Tokmakov
d10b4c504d rename database with dictionaries 2020-07-16 17:25:39 +03:00
Alexander Tokmakov
f6949b2f47 use dictionary ids 2020-07-15 22:25:31 +03:00
Alexander Tokmakov
31e62e713e write uuid for ddl dictionaries 2020-07-14 00:21:01 +03:00
Alexander Tokmakov
15b395da51 fixes 2020-07-13 17:24:00 +03:00
Alexander Tokmakov
445acd0294 Merge branch 'master' into database_atomic_improvements 2020-07-09 16:02:54 +03:00
Alexander Tokmakov
9c4efa1f36 fix, add test 2020-07-09 01:50:15 +03:00
Alexander Tokmakov
7a5d532c7b implement rename database for atomic 2020-07-07 15:11:58 +03:00
Alexander Tokmakov
d1be5ec641 fix symlinks, add short syntax of attach database 2020-07-06 16:41:43 +03:00
Vitaly Baranov
99e9b15e00 Make code clearer: use enum instead of bool internal. 2020-07-05 17:14:29 +03:00
Val
e591fe5014 database replicated feedback mechanism prototype 2020-07-04 19:32:23 +03:00
Vitaly Baranov
ab343132de Fix handling dependency of table with ENGINE=Dictionary on dictionary. 2020-07-03 22:05:53 +03:00
Alexander Tokmakov
383c583bdd add uuid to atomic databases 2020-07-02 23:39:31 +03:00
alexey-milovidov
948af0bfc5
Merge pull request #12045 from ClickHouse/fix_loading_of_tables 
Parse metadata in parallel when loading tables
 2020-06-30 21:16:40 +03:00
Alexander Tokmakov
987e64acfe parse metadata in parallel when loading tables 2020-06-30 03:12:16 +03:00
Val
8273248c4e add log_name_to_exec to dbreplicated 2020-06-28 18:43:30 +03:00
Alexey Milovidov
1462a66d1e Fix typos 2020-06-27 22:05:00 +03:00
Val
e23c7a313e address pr comments 2020-06-27 17:04:12 +03:00
Val
9635ea64be Add desc of propose idatabase method 2020-06-27 17:04:11 +03:00
Val
d293e002a7 address pr comments 2020-06-27 17:04:11 +03:00
Val
16e50e33d7 fix typo 2020-06-27 17:04:11 +03:00
Val
67588edcf5 clean up db replicated files and add more tests 2020-06-27 17:04:11 +03:00
Val
82f5281cfe remove redundant includes 2020-06-27 17:04:11 +03:00
Val
4fc4b1d195 db replicated minor enhancements 2020-06-27 17:02:06 +03:00
Val
f928c897cf change replication algorithm, remove zk lock 
In this version of the databaseReplicated sequential persistent zk nodes
are used to order DDL queries. Db replicated ddl queries are executed
in the backgrould pool no matter whether it's proposed by the same
replica or not.
 2020-06-27 17:02:03 +03:00
Val
469f9738df refactor save state in db replicated 2020-06-27 17:01:20 +03:00
Val
a0af67b636 Add one more test for db replicated and fix related bug 2020-06-27 17:01:20 +03:00
Val
0e9f516738 add comment for replicated db class 2020-06-27 17:01:20 +03:00
Val
31910e9bf1 Use ClientInf::QueryKind to distinguish replicated db log queries 2020-06-27 17:01:18 +03:00
Val
cbcd1bea0e provide better comments and information 2020-06-27 17:00:44 +03:00
Val
4921dc6dab db replicated refactoring 2020-06-27 16:59:11 +03:00
Val
5e076b464e add replicated db snapshot, integration test, repl alter queries, etc 
add an option to create replicated tables within replicated db without specifying zk path and replica id
add replicated sch pool
disable replication of alter queries for replicated tables in replicated dbs
snapshot prototype. amend of replicated db workflow
add prototype of integration tests for replicated db
 2020-06-27 16:58:15 +03:00
Val
f103e24a09 make db replicated inherited from atomic 2020-06-27 16:52:35 +03:00
Val
d7a354b24d create query fix for replicated dbs 2020-06-27 16:51:47 +03:00
Val
d61259cd7b ddl replication works 2020-06-27 16:51:47 +03:00
Val
5eea58039c fix not initialized last entry in zk 2020-06-27 16:51:47 +03:00
Val
0a860c0c2b log based replicated 2020-06-27 16:51:36 +03:00
Val
319256ef4f an attempt to replicated create query from create query 2020-06-27 16:49:40 +03:00
Val
0a4c1783a1 Make drop work by fixing namespace bug 
data dir wasn't set right. now it's fixed.
add non-replicated table to test sql
 2020-06-27 16:49:40 +03:00
Val
8b0366ff4f an attempt to make something meaningful 2020-06-27 16:49:40 +03:00