thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-11 08:52:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
127,702 Commits 9,280 Branches 1,782 Tags 2.1 GiB
4e263956e0
Commit Graph

1271 Commits

Author SHA1 Message Date
Robert Schulze
45c0ec44b0
Fix build, pt. II 2022-05-04 10:15:25 +02:00
Vitaly Baranov
409edfd3fa Rework RestoreCoordination: make restore deterministic. 2022-05-03 11:01:44 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Nikita Mikhaylov
93a65463d0
Added SYSTEM SYNC DATABASE query (#35944) 2022-05-01 15:40:18 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Vitaly Baranov
641a5f5e35
Merge pull request #36198 from vitlibar/backup-improvements-3 
Backup for replicated tables and other improvements
 2022-04-28 09:23:53 +02:00
alesapin
92296484e7
Merge pull request #36348 from rschu1ze/erase_if3 
Replace remove-erase idiom by C++20 erase()/erase_if()
 2022-04-25 23:34:18 +02:00
Vitaly Baranov
129eca7307 CREATE queries generated by RESTORE command is now considered as internal. 2022-04-25 16:34:34 +02:00
Vitaly Baranov
68a020ecea Implement BACKUP/RESTORE ON CLUSTER. 2022-04-25 16:34:33 +02:00
tavplubix
2c54381520
Merge pull request #36200 from ClickHouse/fix_create_as_replicated_default_args 
Fix CREATE AS for ReplicatedMergeTree with default arguments
 2022-04-20 12:01:13 +03:00
Robert Schulze
b6d7367538
Merge remote-tracking branch 'origin/master' into erase_if3 
Conflicts:
- Interpreters/ActionsDAG.cpp
 2022-04-20 10:02:59 +02:00
Antonio Andelic
bbb0be6a44
Merge pull request #36402 from rschu1ze/clang-tidy-contains 
Activate clang-tidy warning "readability-container-contains"
 2022-04-20 08:15:46 +02:00
xiedeyantu
89d0c30f38
Fix the repeated call of func to get the table when drop table (#36248) 2022-04-19 13:51:51 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
liyang830
b2e2b4cce7 fix attach table dictionaries function name normalizer 2022-04-18 19:58:57 +08:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
Robert Schulze
1e1df8e101
Replace remove-erase idiom by C++20 erase()/erase_if() 
- makes the code less verbose while being as efficient
 2022-04-17 12:04:47 +02:00
Alexander Tokmakov
819eac6297 fix 2022-04-14 14:48:24 +02:00
Alexander Tokmakov
40fea2b833 do not unfold {uuid} macro 2022-04-13 16:51:59 +02:00
Alexander Tokmakov
f2c4f88295 avoid moving replicated tables to Ordinary db 2022-04-13 14:38:20 +02:00
Raúl Marín
6e346a7fc3 Support atomic replace under OSX 2022-04-11 14:32:55 +02:00
tavplubix
2894672ecc
Update DatabaseReplicatedWorker.cpp 2022-04-08 22:12:57 +03:00
Alexander Tokmakov
7f54e7b422 Merge branch 'master' into mvcc_prototype 2022-04-07 15:14:06 +02:00
Alexander Tokmakov
37a06eec1a fixes 2022-04-05 17:36:53 +02:00
Alexander Tokmakov
4e9ec5dc2f make some replicated DDL faster 2022-04-05 00:51:48 +02:00
Alexander Tokmakov
5a50ad9de3 Merge branch 'master' into mvcc_prototype 2022-03-31 11:35:04 +02:00
Kruglov Pavel
771f34d793
Merge pull request #35691 from Avogar/fix-matview 
Fix creating materialized view with subquery after server restart
 2022-03-30 14:57:14 +02:00
Antonio Andelic
9990abb76a Use compile-time check for Exception messages, fix wrong messages 2022-03-29 13:16:11 +00:00
avogar
36ea9ef11f Fix creating materialized view with subquery after server restart 2022-03-28 19:52:10 +00:00
Alexander Tokmakov
208b242188 Merge branch 'master' into mvcc_prototype 2022-03-28 19:58:06 +02:00
Nikita Mikhaylov
64f79f0c69
Added an ability to specify cluster secret in replicated database (#35333) 2022-03-25 00:14:26 +01:00
Alexander Tokmakov
3c762f566d Merge branch 'master' into mvcc_prototype 2022-03-21 20:16:29 +01:00
Alexander Tokmakov
9e05b12d2c Merge branch 'master' into mvcc_prototype 2022-03-20 22:42:26 +01:00
Vitaly Baranov
ce25afb2e9 Storages and databases are hollow by default now. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
5a883c6b5a Fixed checking access rights while producing backup, add IStorage::hasHollowBackup(). 2022-03-20 20:01:54 +01:00
kssenii
1b06a2033e Support uuid 2022-03-18 12:24:50 +01:00
Alexander Tokmakov
4b3e13a4fe Merge branch 'master' into mvcc_prototype 2022-03-16 21:06:19 +01:00
Kseniia Sumarokova
b4e03eda8b
Update DatabasePostgreSQL.cpp 2022-03-16 01:04:40 +01:00
kssenii
3a784fe611 Fix 2022-03-16 00:52:51 +01:00
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
Maksim Kita
1d674123a9 Fix clang-tidy warnings in Databases, DataTypes, Dictionaries folders 2022-03-14 18:17:35 +00:00
Maksim Kita
ad6b3693e1
Merge pull request #35123 from zhanghuajieHIT/fix_build_fail_with_gcc 
fix build fail with gcc
 2022-03-14 10:36:15 +01:00
zhanghuajie
53a8987b3b fix build fail with gcc --fix warnings without disabling some parameters 2022-03-11 21:59:19 +08:00
kssenii
3cd1da1e11 Fix 2022-03-10 11:11:59 +01:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
tavplubix
b7d4c78f13
Merge pull request #34487 from zzsmdfj/issue/32977_MaterializedMySQL_add_table_list_settings 
MaterializedMySQL add materialized_mysql_tables_list settings
 2022-03-03 12:44:51 +01:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
zzsmdfj
4d9bdfa58c to 32977_MaterializedMySQL_add_table_list_settings 2022-03-01 10:21:58 +08:00
mergify[bot]
b4574a777f
Merge branch 'master' into mvcc_prototype 2022-02-28 11:24:39 +00:00
Kseniia Sumarokova
ba6032a353
Update src/Databases/SQLite/SQLiteUtils.cpp 
Co-authored-by: Vladimir C <vdimir@clickhouse.com>
 2022-02-26 01:31:04 +01:00
kssenii
40075d4c6b Fix 2022-02-25 15:04:14 +01:00
Alexander Tokmakov
5a26f856d9 remove trash that shouldn't have been merged 2022-02-22 23:41:33 +03:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
Alexander Tokmakov
1e4e569151 Merge branch 'master' into mvcc_prototype 2022-02-15 02:26:47 +03:00
alesapin
89373155fc Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-13 21:07:54 +03:00
tavplubix
6d1fae834a
Merge pull request #34187 from qoega/default-table-engine 
Default table engine
 2022-02-11 18:14:29 +03:00
Alexander Tokmakov
07e66e690d Merge branch 'master' into mvcc_prototype 2022-02-11 15:53:32 +03:00
Alexander Tokmakov
9e893dfda7 fixes 2022-02-10 23:35:03 +03:00
zzsmdfj
a4f3dbf11c MaterializedMySQL add table_list_settings-fix-build 2022-02-10 23:27:42 +08:00
Anton Popov
298838f891 avoid unnecessary copying of Settings 2022-02-10 12:13:51 +03:00
zzsmdfj
ac264e33d5 MaterializedMySQL_add_table_list_settings 2022-02-10 13:52:13 +08:00
alesapin
02a93cb852 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-08 19:42:27 +03:00
Alexey Milovidov
9f51fe1bef Fix wrong engine in SHOW CREATE DATABASE with engine Memory #34225 2022-02-06 03:47:40 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
alesapin
8e33427e19 Fix check for existing table 2022-02-03 15:17:55 +03:00
Azat Khuzhin
cf93732de9 DatabaseSQLite: fix fmt error in DatabaseSQLite::checkSQLiteTable() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
Alexander Tokmakov
5fad3fdffc throw exception on non-transactional queries 2022-02-01 01:27:55 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
tavplubix
9f1da9b8b8
Update MaterializedMySQLSyncThread.cpp 2022-01-25 17:25:34 +03:00
zzsmdfj
e380e59b16 MaterializeMySQL support SET YEAR TIME GEOMETRY dataType 2022-01-21 12:24:11 +08:00
Kseniia Sumarokova
6587cd0ec6
Merge pull request #33231 from kssenii/settings-changes-with-named-conf 
Pass settings as key value or config for storage with settings
 2022-01-11 12:30:41 +03:00
kssenii
1984a80750 Merge remote-tracking branch 'upstream/master' into materialized-postgresql-better-startup 2022-01-09 21:23:49 +03:00
kssenii
bfaab1356f Merge remote-tracking branch 'upstream/master' into settings-changes-with-named-conf 2022-01-09 17:56:44 +03:00
kssenii
61f85e9afc More fixes 2022-01-08 20:44:17 +03:00
kssenii
64538cf20f Merge remote-tracking branch 'upstream/master' into materialized-postgresql-better-startup 2022-01-08 00:49:08 +03:00
kssenii
90972e3752 Fix race 2022-01-08 00:27:29 +03:00
Kruglov Pavel
dd7f61b337
Merge branch 'master' into schema-inference 2021-12-29 12:59:05 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
Maksim Kita
9ef359ce2c Dictionaries fix comment 2021-12-28 23:50:48 +03:00
kssenii
55dfaef4de Settings changes as key value 2021-12-27 17:45:00 +03:00
kssenii
ef22ebad85 Better startup 2021-12-25 21:30:36 +03:00
kssenii
dd902b6875 Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-17 22:03:58 +03:00
Kseniia Sumarokova
77b3ebe44e
Merge pull request #32734 from kssenii/mysql-fix-timeouts 
Allow to control connection timeouts for mysql
 2021-12-17 10:13:33 +03:00
alexey-milovidov
6d82644094
Update DatabaseMySQL.cpp 2021-12-16 02:02:36 +03:00
Alexander Tokmakov
07bfbc26d6 fix restarting with unavailable mysql 2021-12-15 15:08:08 +03:00
kssenii
47f549166d Table Override for PostgreSQL 2021-12-14 20:36:43 +03:00
kssenii
f0d0714e47 Pass timeouts for mysql 2021-12-13 22:12:33 +00:00
Kseniia Sumarokova
a9ca84ff7d
Merge pull request #32325 from stigsb/table-overrides 
Add TABLE OVERRIDE feature for MaterializedMySQL databases
 2021-12-13 20:39:42 +03:00
Azat Khuzhin
5fad4f6204 clickhouse-local: fix CREATE DATABASE with Atomic engine 
Before it fails to create due to "metadata" directory had not been
created, since metadata_path is different for Atomic database, see
InterpreterCreateQuery.cpp.
 2021-12-08 21:23:00 +03:00
Stig Bakken
bb521cd363 Add TABLE OVERRIDE feature for MaterializedMySQL databases 2021-12-07 09:45:02 +01:00
tavplubix
657db07795
Merge pull request #32288 from ClickHouse/fix_xml_dict_cyclic_dependency 
Fix cyclic dependency with XML dictionary and StorageDictionary
 2021-12-06 20:02:38 +03:00
Alexander Tokmakov
7797a72907 fix false-positive ciclyc dependency with XML dict 2021-12-06 16:35:43 +03:00
Alexander Tokmakov
90eba0c0f6 fix uncaught exception in DatabaseLazy 2021-12-03 17:45:53 +03:00
alexey-milovidov
6020188cc0
Merge pull request #31781 from havardk/tabletable 
MaterializedMySQL: Fix issue with table named 'table'
 2021-12-01 10:34:59 +03:00
kssenii
0e24eebc80 Merge branch 'master' of github.com:ClickHouse/ClickHouse into materialized-postgresql-fix 2021-11-30 12:26:10 +03:00
alesapin
9c4002658c
Merge pull request #31960 from ClickHouse/fix_segfault_in_materialized_mysql 
Fix segfault in MaterializedMySQL
 2021-11-30 10:16:27 +03:00
Alexander Tokmakov
b96fcd90e4 fix segfault in MaterializedMySQL 2021-11-29 22:02:32 +03:00
Raúl Marín
7781fc12ed Reduce dependencies on ASTSelectWithUnionQuery.h 
521 -> 77 files requiring changes
 2021-11-26 19:27:16 +01:00
Raúl Marín
b2cfa70541 Reduce dependencies on ASTFunction.h 
481 -> 230
 2021-11-26 18:21:54 +01:00
Raúl Marín
051dddd8df Reduce dependencies on ASTIdentifier.h 
Goes from rebuilding 483 objects to 165 when it's modified
 2021-11-26 16:49:40 +01:00
Haavard Kvaalen
fa7b547212 Always quote table name 
This fixes an issue if the database contained a table named 'table'.
We would previously generate a query that ClickHouse could not parse.
 2021-11-25 11:49:23 +01:00
tavplubix
bebee01b34
Merge pull request #31638 from ClickHouse/fix_rename_dictionary 
Fix rename dictionary
 2021-11-24 20:25:35 +03:00
tavplubix
5eae7710ea
Update DatabaseOnDisk.cpp 2021-11-24 12:06:19 +03:00
Alexander Tokmakov
9237b251d5 allow rename dict in Ordrinary, fix another issue 2021-11-23 20:53:45 +03:00
tavplubix
7a43a87f5b
Use shard and replica name from Replicated database arguments (#31488) 
* fix another issue

* use shard and replica name from Replicated database

* fix

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-11-23 12:41:54 +03:00
tavplubix
d2aa56fc1f
Update DatabaseOnDisk.cpp 2021-11-22 20:44:06 +03:00
tavplubix
667dbef4a0
Merge branch 'master' into fix_rename_dictionary 2021-11-22 18:48:22 +03:00
Alexander Tokmakov
d56950a207 fix rename dictionary 2021-11-22 18:41:35 +03:00
tavplubix
dc6d48ff83
Merge pull request #31292 from stigsb/materialized-mysql-atomic-only 
Drop support for DatabaseOrdinary in MaterializedMySQL
 2021-11-22 14:03:50 +03:00
SuperDJY
dd2972b8c3
return fake create query when executing show create table on system's tables. (#31391) 
* clarify that cannot show create table of system's tables

* clarify that cannot show create table of system's tables in the document

* bypass test style-check for required `database=currentDatabase()` because `show create table` do not have condition

* rename supportsShowCreateTable to isSystemStorage

* build fake create query for show create table of system's tables

* fix tests for show create table on system's table:

1. only build fake create query when cannot get real query
2. only test rocksdb when it's enabled.

* fix test for show create table system.tables

* fix test for ASTCreateQuery set table and database

* change to LOGICAL_ERROR when cannot get in_memory metadata

Co-authored-by: tavplubix <tavplubix@gmail.com>

* implement common getCreateQueryFromStorage

Co-authored-by: tavplubix <tavplubix@gmail.com>
 2021-11-19 13:25:55 +03:00
Stig Bakken
b91f21d658 Don't use setInternalQuery() when forwarding drop calls to database 2021-11-18 11:46:51 +01:00
Stig Bakken
9190a883d3 Print CREATE, not ATTACH 2021-11-18 11:46:51 +01:00
Stig Bakken
eb5115adb8 Fail when loading MaterializeMySQL database without uuid 
and print instructions on how to remove and recreate.
 2021-11-18 11:46:51 +01:00
Stig Bakken
18ce50fd7f Add hasReplicationThread() and stopReplication() methods to IDatabase 2021-11-18 11:46:51 +01:00
Stig Bakken
ff46e8bb51 Drop support for DatabaseOrdinary in MaterializedMySQL 
1. Dropped support for DatabaseOrdinary for MaterializeMySQL. It
   is marked as experimental, and dropping support makes the code
   more maintaible, and speeds up integration tests by 50%.

2. Get rid of thread name logic for StorageMaterializeMySQL wrapping,
   use setInternalQuery instead (similar to MaterializedPostgreSQL).
 2021-11-18 11:46:51 +01:00
alexey-milovidov
2a5c70c5f9
Merge pull request #31400 from ClickHouse/remove-thread-local-std-string 
Remove thread_local std::string
 2021-11-17 10:29:34 +03:00
Nikolay Degterinsky
2f6a565b06 Merge branch 'master' into query_parameters 2021-11-16 10:37:58 +00:00
Nikolay Degterinsky
ba50e8387d Merge branch 'master' into query_parameters 2021-11-15 19:55:54 +00:00
Alexander Tokmakov
416a82b99a Merge branch 'master' into check_dependencies_on_drop 2021-11-15 13:35:45 +03:00
Alexey Milovidov
fe3ee85d80 Fix MaterializedMySQL 2021-11-14 10:04:45 +03:00
Azat Khuzhin
f2c74a81f7 Fix QueryScope in MaterializedMySQLSyncThread 
QueryScope should exist for the whole query execution time.
 2021-11-13 20:06:08 +03:00
Nikolay Degterinsky
0d58c5231f Better 2021-11-11 13:28:18 +00:00
Nikolay Degterinsky
7b5954b736 Merge branch 'master' into query_parameters 2021-11-11 11:43:00 +00:00
kssenii
26f6a697b5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into materialized-postgresql-fix 2021-11-04 13:25:48 +03:00
Alexander Tokmakov
c4108ab672 support alter 2021-11-02 17:30:22 +03:00
Alexander Tokmakov
598dbe04db fix tests, support rename 2021-11-02 15:58:45 +03:00
Alexander Tokmakov
2b566c5bdf basic check for dependencies on drop table 2021-11-01 22:20:18 +03:00
Alexey Milovidov
993f3faefd Miscellaneous 2021-10-31 18:11:46 +03:00
tavplubix
a5901f2901
Update DatabasesCommon.cpp 2021-10-29 19:58:13 +03:00
Alexander Tokmakov
3933cd8259 support alter table for Memory database 2021-10-29 19:47:57 +03:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
tavplubix
d07d53f1b1
Merge pull request #29328 from aiven/kmichel-recover-replica-race 
Fix hanging DDL queries on Replicated database
 2021-10-27 16:43:21 +03:00
Kseniia Sumarokova
7383bdd5fa
Merge pull request #25969 from ucasfl/filelog-engine 
Implementation of a table engine to consume application log files in ClickHouse
 2021-10-24 10:14:05 +03:00
Kseniia Sumarokova
00d0665bf2
Merge pull request #30477 from kssenii/postgres-fix-query 
More reliable fetch query in postgres database
 2021-10-23 19:07:34 +03:00
Kseniia Sumarokova
f362d420b8
Update fetchPostgreSQLTableStructure.cpp 2021-10-23 11:38:18 +03:00
kssenii
1fa123ee5c Properly done 2021-10-23 01:33:17 +03:00
kssenii
c95a29edac Fix quoting 2021-10-22 10:07:23 +00:00
Anton Popov
b1a74af44d
Merge pull request #30180 from CurtizJ/remove-sample-by 
Allow to remove `SAMPLE BY` expression
 2021-10-21 17:09:53 +03:00
kssenii
a223526f45 Fix 2021-10-20 23:29:35 +00:00
Nikolay Degterinsky
d76976e272 Merge branch 'master' into query_parameters 2021-10-18 17:34:20 +03:00
feng lv
1e3df5780c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-18 07:33:11 +00:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
Nikolai Kochetov
067eaadadd Merge branch 'master' into removing-data-streams-folder 2021-10-16 09:46:05 +03:00
Nikolai Kochetov
c668696047
Merge pull request #30171 from ClickHouse/remove-stream-interfaces 
Remove stream interfaces
 2021-10-16 09:34:01 +03:00
feng lv
8b7721ae6f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-16 05:14:11 +00:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
Anton Popov
476d7a411f allow to remove SAMPLE BY expression 2021-10-14 17:05:20 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Maksim Kita
f589084a94
Merge pull request #29899 from Enmk/Governance/dictionary_comment 
Governance/dictionary comment
 2021-10-14 12:35:10 +03:00
feng lv
21f7ecbd9f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-14 02:09:51 +00:00
Nikolay Degterinsky
63faf0212d Merge branch 'master' into query_parameters 2021-10-13 23:55:09 +03:00
Nikolai Kochetov
2957971ee3 Remove some last streams. 2021-10-13 21:22:02 +03:00
tavplubix
7742b96497
Remove metadata leftovers on drop database (#30054) 
* remove metadata leftovers on drop database

* Update InterpreterDropQuery.cpp

* Update DatabaseCatalog.cpp
 2021-10-13 13:34:18 +03:00
Nikolay Degterinsky
49c1a52f34 Merge branch 'master' into query_parameters 2021-10-13 03:04:40 +03:00
feng lv
b2a17cb320 fix build and conflict 2021-10-12 16:50:50 +00:00
feng lv
a57e97d2e0 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-11 06:09:43 +00:00
Azat Khuzhin
f37dac03f4 Fix lock-order-inversion between DROP TABLE for DatabaseMemory and LiveView 
CI stress founds [1], TSan report:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=509)
  Cycle in lock order graph: M71436201 (0x7b5800a91308) => M237700395169415472 (0x000000000000) => M71436201

  Mutex M237700395169415472 acquired here while holding mutex M71436201 in thread T723:
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x154b0db9)
    4 DB::DatabaseWithOwnTablesBase::tryGetTable() const obj-x86_64-linux-gnu/../src/Databases/DatabasesCommon.cpp:37:21 (clickhouse+0x154b0db9)
    5 DB::DatabaseMemory::tryGetTableUUID() const obj-x86_64-linux-gnu/../src/Databases/DatabaseMemory.cpp:95:22 (clickhouse+0x15466bb5)
    6 DB::Context::resolveStorageID() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2672:90 (clickhouse+0x155e6aa1)
    7 DB::JoinedTables::getLeftTableStorage() obj-x86_64-linux-gnu/../src/Interpreters/JoinedTables.cpp:200:29 (clickhouse+0x15eee962)
    8 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:321:33 (clickhouse+0x15b792be)
    9 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:160:7 (clickhouse+0x15b78160)
    10 DB::StorageLiveView::collectMergeableBlocks() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:113:28 (clickhouse+0x16aae192)
    11 DB::StorageLiveView::getNewBlocks() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:384:33 (clickhouse+0x16ab393a)
    12 DB::StorageLiveView::refresh(bool) obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:528:13 (clickhouse+0x16ab6395)
    13 DB::StorageLiveView::read() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:545:9 (clickhouse+0x16ab6395)
    21 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:950:30 (clickhouse+0x16101441)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M71436201 acquired here while holding mutex M237700395169415472 in thread T723:
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x16ab5c29)
    4 DB::StorageLiveView::drop() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:477:21 (clickhouse+0x16ab5c29)
    5 DB::DatabaseMemory::dropTable() obj-x86_64-linux-gnu/../src/Databases/DatabaseMemory.cpp:44:16 (clickhouse+0x15465cd6)
    8 DB::InterpreterDropQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:62:16 (clickhouse+0x15afa679)

  [1]: https://clickhouse-test-reports.s3.yandex.net/29856/42ca2b4bb241827edf69bbd6938d6b19c31935f1/stress_test_(thread).html#fail1

v2: fix attachTableUnlocked() (as pointed by @tavplubix)
v3: use SCOPE_EXIT() and so avoid holding a lock during removing from fs
 2021-10-10 23:51:36 +03:00
Vasily Nemkov
67ff0f5dba Refactoring: moved comment out of ASTStorage to ASTCreateQuery 2021-10-09 12:25:27 +03:00
tavplubix
f0e3122507
Merge pull request #29804 from ClickHouse/improvement_for_28373 
Improvement for #28373
 2021-10-08 14:30:00 +03:00
feng lv
4841b31b56 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-07 13:20:08 +00:00
Shoh Jahon
298f613557
Update fetchPostgreSQLTableStructure.cpp 
added partitioned table prefix 'p' for the query for fetching replica identity index @kssenii !
 2021-10-07 14:59:23 +05:00
Nikolai Kochetov
78e1db209f
Remove more data streams (#29491) 
* Remove more streams.

* Fixing build.

* Fixing build.

* Rename files.

* Fix fast test.

* Fix StorageKafka.

* Try fix kafka test.

* Move createBuffer to KafkaSource ctor.

* Revert "Move createBuffer to KafkaSource ctor."

This reverts commit 81fa94d27e.

* Revert "Try fix kafka test."

This reverts commit 2107e54969.

* Comment some rows in test.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-10-07 11:26:08 +03:00
Alexander Tokmakov
c0ee6d46f7 resolve dependency of StorageDictionary 2021-10-06 15:39:14 +03:00
feng lv
c801fa65df fix and add new test 
fix

fix
 2021-10-06 10:56:42 +00:00
Maksim Kita
4fde987ad1 Parse query from metadata exception throw fix 2021-10-05 19:34:25 +03:00
Nikolay Degterinsky
e03bf0c526 Merge branch 'master' into query_parameters 2021-10-04 17:23:17 +03:00
Anton Popov
362e84a336
Merge pull request #29429 from Enmk/Governance/database_comment 
Implemented creating databases with comments
 2021-10-04 15:12:33 +03:00
kssenii
7ea9037939 Fix 2021-10-03 16:41:12 +00:00
kssenii
7bb2649744 Fix checks 2021-10-03 14:36:32 +00:00
Kseniia Sumarokova
ca7395e374
Merge branch 'master' into materialized-postgresql-fix 2021-10-03 11:56:57 +03:00
kssenii
e4d1d43a34 Fixes 2021-10-03 08:55:50 +00:00
Mike Kot
57e2744264 Fixing other imports 2021-10-02 21:47:35 +02:00
Mike Kot
65e6e211b4 Merge remote-tracking branch 'upstream/master' into improvement/fn-traits 2021-10-02 21:37:50 +02:00
kssenii
28aeb30d4a Support full schema list 2021-10-02 11:15:24 +00:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Mike Kot
38b02b121a Merge remote-tracking branch 'upstream/master' into improvement/fn-traits 2021-10-01 16:55:28 +02:00
Vasily Nemkov
2f16806cd0 Implemented creating databases with comments 2021-10-01 11:51:26 +03:00
Nikolay Degterinsky
7f3d395bf3 Merge branch 'master' into query_parameters 2021-09-30 02:01:33 +03:00
Mike Kot
7670ea50c1 Replacing std::function to Fn in some places 
to avoid dynamic allocating while keeping desired function signature
clear.
Simplifying SimpleCache (CachedFn) using C++20
 2021-09-29 13:09:39 +02:00
Alexander Tokmakov
462088f500 fix table name escaping 2021-09-28 13:14:05 +03:00
Nikolai Kochetov
236d71ea94
Merge pull request #28582 from ClickHouse/rewrite-pushing-to-views 
Rewrite PushingToViews
 2021-09-27 21:19:11 +03:00
tavplubix
e9749b0027
Merge pull request #29323 from ClickHouse/fix_assertion_ddl_dependency 
Fix assertion in DDLDependencyVisitor
 2021-09-27 18:06:32 +03:00
alesapin
ddd80c8e72
Merge pull request #29264 from Enmk/governance/table_comment_update 
Implemented modifying table comments with `ALTER TABLE t MODIFY COMMENT 'value'`
 2021-09-27 18:02:14 +03:00
Nikolai Kochetov
78a7665f43 Merge branch 'master' into rewrite-pushing-to-views 2021-09-27 10:56:50 +03:00
Nikolai Kochetov
efbd8e4911 Remove ExceptionHandlingSink. 2021-09-26 17:54:59 +03:00
Alexey Milovidov
cd7f9d981c Remove ya.make 2021-09-25 04:22:54 +03:00
Kevin Michel
aa3f4003c8
Fix hanging DDL queries on Replicated database 
There was a race condition when issuing a DDL query on a replica just
after a new replica was added.

If the DDL query is issued after the new replica adds itself to the
list of replicas, but before the new replica has finished its
recovery, then the first replica adds the new replica to the list of
replicas to wait to confirm the query was replicated.

Meanwhile, the new replica is still in recovery and applies queries
from the /metadata snapshot. When it's done, it bumps its log_ptr
without marking the corresponding log entries (if any) as finished.

The first replica then waits until distributed_ddl_task_timeout
expires and wrongly assumes the query was not replicated.

The issue is fixed by remembering the max_log_ptr at the exact point
where the replica adds itself to the list of replicas, then mark as
finished all queries that happened between that max_log_ptr and the
max_log_ptr of the metadata snapshot used in recovery.

The bug was randomly observed during a downstream test. It can be
reproduced more easily by inserting a sleep of a few seconds at the
end of createReplicaNodesInZooKeeper, enough to have time to issue a
DDL query on the first replica.
 2021-09-24 12:51:52 +02:00
Alexander Tokmakov
165765659d fix assertion in DDLDependencyVisitor 2021-09-24 13:22:22 +03:00
Nikolai Kochetov
998d29ebc7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-23 13:10:27 +03:00
Vasily Nemkov
cb1ca9b33e Implemented modifying table comments with ALTER TABLE t MODIFY COMMENT 'value' 2021-09-23 01:14:00 +03:00
Nikolai Kochetov
4f802d1cea Fix test_executable_table_function 2021-09-22 22:31:12 +03:00
kssenii
a84fc77b52 Fixes 2021-09-22 18:10:25 +03:00
Nikolay Degterinsky
bdc0d65c6b Fixes after merge 2021-09-22 16:57:17 +03:00
Nikolay Degterinsky
1b1d90deb8 Merge branch 'master' into query_parameters 2021-09-22 13:53:56 +03:00
kssenii
b51840a205 Merge branch 'master' of github.com:ClickHouse/ClickHouse into named-collections 2021-09-22 11:45:22 +03:00
tavplubix
e54bd40102
Merge pull request #29216 from ClickHouse/fix_29122 
Fix terminate on uncaught exception
 2021-09-21 20:12:51 +03:00
Alexander Tokmakov
7e346ec6ec fix terminate on uncaught exception 2021-09-21 11:04:22 +03:00
bharatnc
51e45050c1 Update outdated ya.make files 
This runs the `utils/generate-ya-make/generate-ya-make.sh` script to
update the ya.make files that are not updated.

I wonder why they were not updated and how it was missed.
 2021-09-20 18:28:07 -07:00
Nikolai Kochetov
a790d391c0 Merge branch 'master' into rewrite-pushing-to-views 2021-09-20 19:43:15 +03:00
tavplubix
922cf7ee20
Merge pull request #28373 from ClickHouse/tables_topsort 
Resolve table dependencies on metadata loading
 2021-09-20 14:46:47 +03:00
Nikolai Kochetov
341553febd Fix build. 2021-09-16 20:40:42 +03:00
kssenii
1650c92407 Better 2021-09-16 11:38:26 +03:00
kssenii
221c09589c Review fixes 2021-09-15 21:11:49 +03:00
Nikolay Degterinsky
6adfdd2f79 Fix style 2021-09-15 19:36:55 +03:00
kssenii
489a92c067 Review fixes 2021-09-15 18:36:49 +03:00
kssenii
84c8757049 Merge branch 'master' of github.com:ClickHouse/ClickHouse into named-collections 2021-09-15 12:28:31 +03:00
kssenii
710c8d12dd Merge branch 'master' of github.com:ClickHouse/ClickHouse into materialized-postgresql 2021-09-15 12:28:09 +03:00
Nikolai Kochetov
b997214620 Rename QueryPipeline to QueryPipelineBuilder. 2021-09-14 20:48:18 +03:00
Nikolai Kochetov
0e267c50b4 Merge branch 'master' into rewrite-pushing-to-views 2021-09-14 16:13:54 +03:00
Alexander Tokmakov
df56e99b87 fixes after review 2021-09-13 22:11:16 +03:00
Nikolay Degterinsky
9199b100ee Fix build 2021-09-13 17:44:02 +03:00
Alexander Tokmakov
8ac19caca9 Merge branch 'master' into tables_topsort 2021-09-13 16:58:39 +03:00
Nikolay Degterinsky
84aea776c2 Merge branch 'master' into query_parameters 2021-09-13 12:54:48 +03:00
kssenii
ff602c1fdd Fix build check 2021-09-11 19:29:23 +03:00
Azat Khuzhin
ff17de531c Join MaterializedMySQLSyncThread only if it is joinable 
It is possible to trigger
MaterializedMySQLSyncThread::stopSynchronization() from the same thread
in case of some exception at startup, when some interpreter holds the
storage refcnt, and later MaterializedMySQLSyncThread will try to join
itself from the main thread.

Here is a stack trace for example:

<details>

    4  0x000000000f7ae45c in Poco::Event::wait (this=0x7f1b90c38170) at ../contrib/poco/Foundation/include/Poco/Event.h:97
    5  ThreadFromGlobalPool::join (this=0x7f1b90c23120) at ../src/Common/ThreadPool.h:210
    6  DB::MaterializeMySQLSyncThread::stopSynchronization (this=0x7f1b8024ca68) at ../src/Databases/MySQL/MaterializeMySQLSyncThread.cpp:229
    7  DB::MaterializeMySQLSyncThread::~MaterializeMySQLSyncThread (this=0x7f1b8024ca68) at ../src/Databases/MySQL/MaterializeMySQLSyncThread.cpp:85
    8  0x000000000f816dc8 in DB::DatabaseMaterializeMySQL<DB::DatabaseAtomic>::~DatabaseMaterializeMySQL (this=0x7f1b8024c918) at ../src/Databases/MySQL/DatabaseMaterializeMySQL.h:21
    11 std::__1::shared_ptr<DB::IDatabase>::~shared_ptr (this=<optimized out>) at ../contrib/libcxx/include/memory:3212
    12 0x000000000f8726a6 in DB::InterpreterCreateQuery::createTable (this=<optimized out>, create=...) at ../src/Interpreters/InterpreterCreateQuery.cpp:952
    13 0x000000000f87735c in DB::InterpreterCreateQuery::execute (this=0x7f1aeef59860) at ../src/Interpreters/InterpreterCreateQuery.cpp:1225
    14 0x000000000fe22253 in DB::executeQueryImpl (begin=<optimized out>, end=<optimized out>, context=..., internal=true, stage=DB::QueryProcessingStage::Complete, has_query_tail=<optimized out>, istr=<optimized out>) at ../src/Interpreters/executeQuery.cpp:574
    15 0x000000000fe208e3 in DB::executeQuery (query=..., context=..., internal=<optimized out>, stage=DB::QueryProcessingStage::FetchColumns, may_have_embedded_data=<optimized out>) at ../src/Interpreters/executeQuery.cpp:933
    16 0x000000000faafcde in DB::MySQLInterpreter::InterpreterMySQLDDLQuery<DB::MySQLInterpreter::InterpreterCreateImpl>::execute (this=<optimized out>) at ../src/Interpreters/MySQL/InterpretersMySQLDDLQuery.h:75
    17 0x000000000faade78 in DB::InterpreterExternalDDLQuery::execute (this=<optimized out>) at ../src/Interpreters/InterpreterExternalDDLQuery.cpp:64
    18 0x000000000fe22253 in DB::executeQueryImpl (begin=<optimized out>, end=<optimized out>, context=..., internal=true, stage=DB::QueryProcessingStage::Complete, has_query_tail=<optimized out>, istr=<optimized out>) at ../src/Interpreters/executeQuery.cpp:574
    19 0x000000000fe208e3 in DB::executeQuery (query=..., context=..., internal=<optimized out>, stage=DB::QueryProcessingStage::FetchColumns, may_have_embedded_data=<optimized out>) at ../src/Interpreters/executeQuery.cpp:933
    20 0x000000000f7ba406 in DB::tryToExecuteQuery (query_to_execute=..., query_context=..., database=..., comment=...) at ../src/Databases/MySQL/MaterializeMySQLSyncThread.cpp:69
    21 0x000000000f7d4b88 in DB::dumpDataForTables() (connection=..., query_prefix=..., database_name=..., mysql_database_name=..., context=..., is_cancelled=..., need_dumping_tables=...) at ../src/Databases/MySQL/MaterializeMySQLSyncThread.cpp:326
    22 DB::MaterializeMySQLSyncThread::prepareSynchronized(DB::MaterializeMetadata&)::$_1::operator()() const (this=<optimized out>) at ../src/Databases/MySQL/MaterializeMySQLSyncThread.cpp:391
    29 DB::commitMetadata() (function=..., persistent_tmp_path=..., persistent_path=...) at ../src/Databases/MySQL/MaterializeMetadata.cpp:197
    30 0x000000000f80a000 in DB::MaterializeMetadata::transaction(DB::MySQLReplication::Position const&, std::__1::function<void ()> const&) (this=0x7f1b6375c3d8, position=..., fun=...) at ../src/Databases/MySQL/MaterializeMetadata.cpp:230
    31 0x000000000f7b2231 in DB::MaterializeMySQLSyncThread::prepareSynchronized (this=0x7f1b8024ca68, metadata=...) at ../src/Databases/MySQL/MaterializeMySQLSyncThread.cpp:388
    32 0x000000000f7b169c in DB::MaterializeMySQLSyncThread::synchronization (this=0x7f1b8024ca68) at ../src/Databases/MySQL/MaterializeMySQLSyncThread.cpp:180
    33 0x000000000f7d4074 in DB::MaterializeMySQLSyncThread::startSynchronization()::$_0::operator()() const (this=<optimized out>) at ../src/Databases/MySQL/MaterializeMySQLSyncThread.cpp:236

</details>
 2021-09-11 09:27:12 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
kssenii
f26da04cdf Fix checks 2021-09-09 09:53:58 +03:00
kssenii
bed2688dad Review fixes 2021-09-09 01:25:08 +03:00
kssenii
797f858bc6 Merge branch 'master' of github.com:ClickHouse/ClickHouse into materialized-postgresql 2021-09-08 22:52:48 +03:00
kssenii
ab6937ee45 Merge branch 'master' of github.com:ClickHouse/ClickHouse into named-collections 2021-09-08 20:43:37 +03:00
mergify[bot]
9f0c50335f
Merge branch 'master' into tables_topsort 2021-09-08 10:55:11 +00:00
Maksim Kita
7a4a0b0ede
Merge pull request #28474 from azat/mysql-connection_no_block 
Introduce connection_wait_timeout for MySQL engine.
 2021-09-07 12:28:02 +03:00
Nikolay Degterinsky
22f59e1714 Add support for table and database query parameters 2021-09-07 01:19:02 +03:00
kssenii
76e49334fa Minor fixes 2021-09-05 01:59:44 +03:00
kssenii
a1e4d2e230 Fix style check and pvs check 2021-09-05 01:03:15 +03:00
kssenii
ac2d9a73a8 User managed slots 2021-09-04 14:49:40 +03:00
Nikolai Kochetov
66a76ab70f Rewrite PushingToViewsBlockOutputStream part 6 2021-09-03 20:29:36 +03:00
kssenii
a5f56bb588 Some other 2021-09-03 16:25:40 +03:00
kssenii
f17d1bc7a5 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into materialized-postgresql 2021-09-03 09:03:48 +00:00
kssenii
6efbee320b Done MySQL 2021-09-03 11:30:28 +03:00
Azat Khuzhin
ed70ed6f71 Introduce connection_no_wait setting for MySQL engine. 
This will allow to avoid superfluous sleep during query execution, since
this not only not desired behavoiur, but also may hang the server, since
if you will execute enough queries that will use MySQL database but will
not allow enough connections (or your MySQL server is too slow) then you
may run out of threads in the global thread pool.

Also note that right now it is possible to get deadlock when the mysql
pool is full, consider the following scenario:

- you have m1 and m2 mysql tables
- you have q1 and q2 queries, bot queries join m1 and m2
- q1 allocated connection for m1 but cannot allocate connection for m2
- q2 allocated connection for m2 but cannot allocate connection for m1
- but to resolve the lock one should give up on the locking while it is not possible right now...

And then you got no free threads and this:

    # grep -h ^202 /proc/$(pgrep clickhouse-serv)/task/*/syscall  | cut -d' ' -f2 | sort | uniq -c | sort -nr | head
       1554 0x7ffb60b92fe8 # mutex in mysqlxx::PoolWithFailover::get
       1375 0x7ffb9f1c4748 # mutex in ::PoolEntryHelper::~PoolEntryHelper from DB::MultiplexedConnections::invalidateReplica
       1160 0x7ffb612918b8 # mutex in mysqlxx::PoolWithFailover::get
         42 0x7ffb9f057984 # mutex in ThreadPoolImpl<std::__1::thread>::worker

    *NOTE: 202 is a `futex` with WAIT*

(Went with `syscall` because debugging 10k+ threads is not easy, and
eventually it may TRAP)
 2021-09-02 22:23:37 +03:00
Alexander Tokmakov
1e2844f999 support function IN 2021-09-02 16:48:41 +03:00
Alexander Tokmakov
666a3aee99 add another test 2021-09-02 16:34:46 +03:00
kssenii
7a45775f4f Complete postgres 2021-09-02 13:46:19 +03:00
Alexander Tokmakov
024a24aaf7 better code, moar logging 2021-09-02 12:19:37 +03:00
Alexander Tokmakov
c8d8f0a38c fix 2021-08-31 15:07:00 +03:00
Alexander Tokmakov
b387f05d9c resolve table dependencies on metadata loading 2021-08-30 22:29:22 +03:00
tavplubix
b47f34aa17
Merge pull request #28313 from amosbird/fastload 
Fast data parts loading by delaying table startup process
 2021-08-30 13:53:55 +03:00
Amos Bird
de9e591619
Fast load by delaying table startup 2021-08-30 10:15:32 +08:00
kssenii
9d0444774a Fix tests 2021-08-29 11:50:03 +03:00
kssenii
378e1f74aa Fix tests 2021-08-28 19:51:04 +03:00
kssenii
cc90d787f8 Slightly better 2021-08-28 17:45:15 +03:00
kssenii
fd621381c7 Allow modify some other settings 2021-08-28 17:36:27 +03:00
kssenii
a2b0996ac3 Dynamically remove tables 2021-08-28 17:35:59 +03:00
kssenii
174340074c Dynamically add tables complete 2021-08-28 17:35:44 +03:00
kssenii
4cd62227cf Almost done 2021-08-28 17:34:23 +03:00
tavplubix
c36fe210b8
Revert "Add test for #13398" 2021-08-27 21:07:10 +03:00
kssenii
0d5000f917 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into mysql 2021-08-24 07:30:42 +00:00
kssenii
962f415740 Add test and settings 2021-08-23 22:18:04 +00:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Vitaly Baranov
2ac97c7fc3 Make IDatabase::getTablesIterator() const. 2021-08-23 23:41:45 +03:00
kssenii
200d75646a Fix 2021-08-13 20:58:03 +00:00
Maksim Kita
124a87684f Removed some data streams 2021-08-11 23:39:01 +03:00
Mikhail Filimonov
af6249c8ec
Less include <Common/Stopwatch.h> 2021-08-11 16:20:59 +02:00
Maksim Kita
b8b24684b4
Merge pull request #27273 from ClickHouse/remove-streams-from-dicts 
Remove streams from dicts
 2021-08-09 15:58:37 +03:00
Maksim Kita
1db4eddafc
Merge pull request #27225 from kitaisreal/database-replicated-worker-logs-to-keep-race-fix 
DatabaseReplicatedWorker logs_to_keep race fix
 2021-08-06 14:16:36 +03:00
tavplubix
14ef66e49b
Merge pull request #25895 from ClickHouse/improve_create_or_replace 
Improve CREATE OR REPLACE query
 2021-08-06 11:58:46 +03:00
Nikolai Kochetov
39ff5ab478 Merge branch 'master' into remove-streams-from-dicts 2021-08-06 11:46:27 +03:00
Nikolai Kochetov
13f95f3fdf Streams -> Processors for dicts, part 3. 2021-08-06 11:41:45 +03:00
Kseniia Sumarokova
5edd9e0513
Merge pull request #27198 from kssenii/postgres-db-schema 
Support schema for postgres database engine
 2021-08-05 18:30:47 +03:00
Maksim Kita
1ae59b6cce DatabaseReplicatedWorker logs_to_keep race fix 2021-08-05 17:05:03 +03:00
kssenii
e8dd257468 Update DatabaseFactory.cpp 2021-08-05 09:04:00 +00:00
kssenii
1e38d48364 Fixes 2021-08-05 07:25:47 +00:00
kssenii
f06703a7c9 Support schema for postgres database engine 2021-08-04 20:52:45 +00:00
Alexander Tokmakov
ff2572c243 fix 2021-08-03 16:28:03 +03:00
Alexander Tokmakov
42a8bb6872 fix assertions in Replicated database 2021-08-02 16:19:11 +03:00
Alexander Tokmakov
fc9ab2cda7 Merge branch 'master' into rename_materialize_mysql 2021-07-27 22:38:40 +03:00
Alexander Kuzmenkov
ece7d000dd
Merge pull request #26736 from Algunenano/ch_local_remote_localhost 
CH local: Treat localhost:port as a remote database
 2021-07-27 15:30:54 +03:00
Alexander Tokmakov
a7e644f774 support old name 2021-07-26 21:24:32 +03:00
Alexander Tokmakov
63ab38ee09 rename MaterializeMySQL to MaterializedMySQL 2021-07-26 21:17:28 +03:00
kssenii
f3129b80f9 Fix sqlite attach 2021-07-26 07:20:41 +00:00
Raúl Marín
383c982715 CH local: Treat localhost:port as a remote database 2021-07-23 13:16:35 +02:00
mergify[bot]
39f71f4a08
Merge branch 'master' into fix_no_column_materialized_mysql 2021-07-19 12:27:38 +00:00
kssenii
dd519f9434 Fix 2021-07-15 19:02:31 +03:00
kssenii
f20ea87bfb Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into add-sqlite-database-path-check 2021-07-14 20:49:05 +00:00
Kseniia Sumarokova
e42a26a585
Merge pull request #26234 from jasine/master 
convert timestamp and  timestamptz data types to DateTime64 in postgres engine
 2021-07-14 23:42:51 +03:00
kssenii
4fb0762bbb Merge branch 'add-sqlite-database-path-check' of github.com:kssenii/ClickHouse into add-sqlite-database-path-check 2021-07-14 20:08:11 +00:00
kssenii
e875f1e289 Fix yandex check 2021-07-14 20:07:55 +00:00
Kseniia Sumarokova
ef0ebaf55c
Update SQLiteUtils.cpp 2021-07-14 20:20:41 +03:00
kssenii
d62cbbb8a5 Unification 2021-07-14 13:16:30 +00:00
kssenii
b393603b57 Fix build 2021-07-13 22:25:26 +00:00
Nikita Mikhaylov
601d04f55d ignore 2021-07-13 22:16:27 +00:00
kssenii
da09ce3f20 Remove unused 2021-07-13 19:51:02 +00:00
kssenii
a90df59b92 Better 2021-07-13 19:46:06 +00:00
mergify[bot]
d5898ecba6
Merge branch 'master' into fix_no_column_materialized_mysql 2021-07-13 14:03:05 +00:00
kssenii
9688efde96 Merge branch 'add-sqlite-database-path-check' of github.com:kssenii/ClickHouse into add-sqlite-database-path-check 2021-07-13 12:40:43 +00:00
kssenii
f86817d57c Check inclusion inside user_files 2021-07-13 12:38:37 +00:00
Kseniia Sumarokova
c615084ba7
Update DatabaseSQLite.cpp 2021-07-13 14:40:07 +03:00
kssenii
48ffd3b81d Add check for sqlite database path 2021-07-13 09:40:46 +00:00
jasine
4ec8d23fce
Merge branch 'ClickHouse:master' into master 2021-07-13 15:12:40 +08:00
kssenii
bcc77fca62 Merge branch 'add-sqlite-support' of github.com:g-arslan/ClickHouse into g-arslan-add-sqlite-support 2021-07-12 22:39:34 +00:00
kssenii
976f4dffda Add table function sqlite, fix type conversions 2021-07-12 21:09:23 +00:00
Kseniia Sumarokova
08ab42483c
Update DatabaseFactory.cpp 2021-07-12 20:40:30 +03:00
jasine
192ee2df98
Update fetchPostgreSQLTableStructure.cpp 2021-07-12 12:39:17 +08:00
kssenii
10611ef92a Normal exceptions 2021-07-10 11:00:59 +00:00
kssenii
18c2abaaf9 Fix database engine attach, more tests 2021-07-10 10:31:21 +00:00
kssenii
d250d79e33 Generate ya make 2021-07-10 08:07:55 +00:00
kssenii
78094a82ab Better 2021-07-10 08:05:23 +00:00
kssenii
1bf9a9f29f Add .cmake 2021-07-09 10:27:53 +00:00
kssenii
ff56f1787c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into g-arslan-add-sqlite-support 2021-07-07 15:19:12 +00:00
Alexander Tokmakov
0f377fc4d8 Merge branch 'master' into fix_no_column_materialized_mysql 2021-07-05 22:07:56 +03:00
kssenii
32b7d7b750 More correct startup on create query 2021-07-04 19:53:05 +00:00
kssenii
b39888f199 Do not allow inconsistency in publication 2021-07-04 08:54:41 +00:00
kssenii
0ce4c565d1 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-07-03 11:48:15 +00:00
kssenii
f01c8edbff Fix access to tables after restart 2021-07-01 07:33:58 +00:00
Kostiantyn Storozhuk
a9510f25f9 Style fix 2021-06-29 15:22:08 +08:00
Kostiantyn Storozhuk
296593716d Reverted constructor change 2021-06-29 14:37:53 +08:00
Kostiantyn Storozhuk
c2c78929cb Implemented MySQL column comments support 2021-06-28 22:44:38 +08:00
Kseniia Sumarokova
ff8c44179c
Update DatabaseAtomic.cpp 2021-06-28 17:37:22 +03:00
Kseniia Sumarokova
a96c7a3a70
Update DatabaseAtomic.cpp 2021-06-28 17:32:03 +03:00
Kseniia Sumarokova
b6b7924413
Update DatabaseAtomic.cpp 2021-06-28 10:20:33 +03:00
kssenii
aa7133c239 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-27 20:08:00 +00:00
kssenii
c3a2fc0584 Review fixes 2021-06-27 19:17:17 +00:00
alexey-milovidov
fb89505656
Merge pull request #25538 from kssenii/pg-dix 
Fix postrges array conversion
 2021-06-27 19:35:24 +03:00
alexey-milovidov
6e25d11e1f
Merge pull request #25582 from azat/atomic-unlink-symlink 
Remove only symlinks during force_restore_data of Atomic engine
 2021-06-27 18:19:59 +03:00
kssenii
3d90f0710c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-26 22:04:47 +00:00
Maksim Kita
3c9ae7b5ba Added more tests 2021-06-26 14:28:13 +03:00
Azat Khuzhin
50d9d3efd5 Remove only symlinks during force_restore_data of Atomic engine 2021-06-23 10:17:31 +03:00
Kseniia Sumarokova
77a3a1416d
Update fetchPostgreSQLTableStructure.cpp 2021-06-21 17:26:14 +03:00
kssenii
0f9fc33a4e Fix postgres arrays 2021-06-21 11:27:10 +00:00
Maksim Kita
2c4c2680f7
Merge pull request #25430 from kitaisreal/database-mysql-rename 
DatabaseMySQL rename
 2021-06-18 01:30:52 +03:00
Maksim Kita
cf77e9f2c6 DatabaseMySQL rename 2021-06-17 22:26:34 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
kssenii
0521851afa Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into postgres-fixes-schema 2021-06-08 17:20:23 +00:00
kssenii
e23bc14582 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-03 19:44:55 +00:00
mergify[bot]
438b45e6c0
Merge branch 'master' into add-sqlite-support 2021-06-03 12:26:12 +00:00
Arslan G
4fe9a24472 Fix review a little, try add simple test 2021-06-02 03:53:30 +03:00
Arslan G
dc0a1001d8 Fix style 2021-06-02 01:21:01 +03:00
Arslan G
6f7dc47b2b Merge branch 'master' into add-sqlite-support 2021-06-02 00:24:29 +03:00
Arslan G
bf4171e0ea Some review fixes 2021-06-01 23:52:12 +03:00
kssenii
12ea45e956 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-01 17:43:34 +00:00
tavplubix
e2941a8006
Update DatabaseReplicated.cpp 2021-06-01 15:15:13 +03:00
Alexander Tokmakov
94d2aed336 fix tests 2021-06-01 13:24:06 +03:00
Alexander Tokmakov
fe482109da Merge branch 'master' into fix_database_replicated_enqueue_race 2021-06-01 11:49:29 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs 
Poco::File to std::filesystem
 2021-05-31 23:17:02 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
Alexander Tokmakov
0035997e02 fix race on enqueue query 2021-05-31 16:31:03 +03:00
kssenii
0a3589524e File movement 2021-05-29 00:57:53 +03:00
kssenii
2a631aaf08 Final fixes 2021-05-29 00:34:44 +03:00
Arslan G
fbc5734302 Clean up code and fix tests 2021-05-28 20:00:26 +03:00
kssenii
a78c9e63e7 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-23 12:07:26 +00:00
kssenii
866b29fb5a Return list fds with Poco, more canonical 2021-05-23 10:56:13 +03:00
kssenii
1c43b333ea Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-22 21:26:48 +03:00
kssenii
3e6d3d4ecb Postgres schema for insert 2021-05-22 08:50:23 +00:00
alexey-milovidov
a25f1910cc
Merge pull request #24320 from ClickHouse/remove-auto-array 
Remove AutoArray
 2021-05-22 02:19:59 +03:00
kssenii
096d117f68 Review fixes 2021-05-22 00:12:46 +03:00
Alexey Milovidov
ea30bb7abf Improve implementation 2021-05-21 04:17:18 +03:00
Azat Khuzhin
4f41ebcae3 Add settings (for connections) for MySQL storage engine 
Default settings are not very efficient, since they do not even reuse
connections.
And when each query requires connection you can have only ~80 QPS, while
by simply enabling connection reuse (connection_auto_close=false) you
can have ~500 QPS (and by increasing connection_pool_size you can have
better QPS throughput).

So this patch allows to pass through some connection related settings
for the StorageMySQL engine, like:
- connection_pool_size=16
- connection_max_tries=3
- connection_auto_close=true

v2: remove connection_pool_default_size
v3: remove num_tries_on_connection_loss
 2021-05-20 22:44:52 +03:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
kssenii
5f083b779d A little better, fix checks 2021-05-18 14:25:19 +03:00
kssenii
9b8df78fdd Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-17 17:42:05 +03:00
Arslan G
dde9e30043 Merge branch 'master' into add-sqlite-support 2021-05-17 14:38:24 +03:00
Arslan G
ea51840f85 Init SQLite integration 2021-05-17 14:02:35 +03:00
kssenii
1ee77eae89 Better way to distinguish between user query and replication thread query 2021-05-16 21:35:37 +00:00
kssenii
8a4711b716 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-16 20:55:18 +00:00
Azat Khuzhin
23981ffd0e Introduce IStorage::flushAndShutdown() 2021-05-16 13:23:53 +03:00
Azat Khuzhin
f20799f683 Destroy Buffer() tables before others (within one database) 
This will reduce amount of data that may be lost at shutdown.

v2: replace dynamic_cast with IStorage::isBuffer (@kitaisreal)
v3: replace IStorage::isBuffer with IStorage::flush (@alexey-milovidov)
v4: flush() for StorageProxy/StorageTableFunction
 2021-05-16 13:23:53 +03:00
feng lv
c6f8ab9826 fix 2021-05-13 02:05:53 +00:00
kssenii
7c81103ea3 Add replication identifier, allow proper access to _version and _sign columns from nested 2021-05-12 22:48:42 +00:00
kssenii
407db17e2e Remove setting, use native transaction 2021-05-12 22:33:15 +00:00
kssenii
0955e5f0be Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-12 15:11:15 +00:00
kssenii
0527f0ea33 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-12 16:54:18 +03:00
Nikolai Kochetov
a1ba67e029
Merge pull request #20202 from amosbird/projection 
Projections
 2021-05-12 09:38:07 +03:00
kssenii
81797393f1 fs::last_write_time, fs::space 2021-05-12 09:19:11 +03:00
Maksim Kita
72d46beca0
Merge pull request #23979 from azat/dict-preallocate 
Reimplement preallocate for hashed/sparse_hashed dictionaries
 2021-05-11 20:15:46 +03:00
feng lv
9035e3a643 fix test and conflict 2021-05-11 16:22:24 +00:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
Amos Bird
264cff6415
Projections 
TODO (suggested by Nikolai)

1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies

3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)

Also need to figure out how prewhere works for projections, and
row_filter_policies.

wip
 2021-05-11 18:12:23 +08:00
kssenii
626e87bae5 Some fixes 2021-05-10 13:51:05 +00:00
kssenii
ae1191d0c0 Get rid of metadata file, rely only on lsn 2021-05-10 09:10:02 +00:00
kssenii
4ac023e511 Small fixes 2021-05-10 08:45:52 +00:00
kssenii
993823bcc4 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-09 21:26:41 +00:00
kssenii
ab1a05a1f4 Poco::Path to fs::path, less concatination 2021-05-09 14:59:49 +03:00
Azat Khuzhin
80f4059deb Drop unused getDictionaryConfigurationFromAST.h 2021-05-09 14:25:45 +03:00
kssenii
e26cb4bee0 Fix style check, fix yandex check 2021-05-09 13:45:50 +03:00
Alexey Milovidov
3025f9e141 More fixes for PVS-Studio 2021-05-08 19:12:49 +03:00
Alexey Milovidov
8d62c42eb9 A bunch of changes for PVS-Studio 2021-05-08 19:11:50 +03:00
kssenii
5c24f0943c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-08 14:54:44 +00:00
kssenii
266567e04c Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-08 17:49:05 +03:00
alexey-milovidov
680860aae7
Merge pull request #23912 from azat/CU 
Cleanup IDatabase.h from extra headers
 2021-05-08 16:42:56 +03:00
kssenii
02288359c5 Less manual concatenation of paths 2021-05-08 13:59:55 +03:00
kssenii
35f999bf04 Poco::createFile to fs::createFile 2021-05-08 02:41:47 +03:00
kssenii
c3eab60b8b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-08 02:40:17 +03:00
kssenii
140bf7e2ea Less Poco 2021-05-07 15:29:26 +03:00
kssenii
912878ad7f Review fixes 2021-05-07 12:05:34 +00:00
Azat Khuzhin
3672af8bbe Make DatabaseConnectionMySQL self compilable (missing Core/NamesAndTypes.h) 2021-05-06 21:03:03 +03:00
kssenii
b96f047db7 Fix clang tidy 2021-05-06 07:02:13 +00:00
Azat Khuzhin
6348497d1b Cleanup IDatabase.h from extra headers 
From IDatabase.h depends ~300 CU.

- drop unused
- use forward declaration whenever is possible
 2021-05-06 09:07:59 +03:00
kssenii
8900e93bd8 New impl 2021-05-06 00:52:59 +00:00
kssenii
ed5c2321cc Some fixes 2021-05-04 10:57:05 +00:00
kssenii
eff26f9d54 Add comments, small improvements 2021-05-03 21:42:06 +00:00
kssenii
3e3396bd9c Refactor code, add comments 2021-05-03 09:52:32 +00:00
feng lv
39f68bf5ff fix conflict 2021-05-02 16:33:45 +00:00
kssenii
14d355a6ac Fix background update 2021-05-02 11:53:03 +00:00
kssenii
32b2917141 Update s3/* 2021-05-01 18:10:04 +03:00
kssenii
ee06936596 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-01 17:24:31 +03:00
kssenii
14be3a0783 Better reload from snapshot 2021-05-01 11:49:45 +00:00
kssenii
7643d84095 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-30 22:02:51 +00:00