thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-08 15:31:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
115,051 Commits 9,205 Branches 1,782 Tags 2 GiB
9dbe9507e7
Commit Graph

348 Commits

Author SHA1 Message Date
Vitaly Baranov
ce233761d7 Fix making a query scope for async backups. 2022-07-15 13:35:04 +02:00
Vitaly Baranov
2f47be5da7 Check that the destination for a backup is not in use. 2022-07-15 13:34:58 +02:00
Vitaly Baranov
847cda87f9 BACKUP/RESTORE ON CLUSTER use async mode on replicas now. 2022-07-08 22:26:01 +02:00
Vitaly Baranov
5dcc271856 More careful destructors. 2022-07-07 11:16:44 +02:00
Vitaly Baranov
7f84cf3968 Fix style. 2022-07-06 16:36:59 +02:00
Vitaly Baranov
5d7ad46f6a Move files and write comments. 2022-07-06 11:09:31 +02:00
Vitaly Baranov
1ac46c5e48 Fix making backups containing multiple ACL tables. 2022-07-05 20:57:01 +02:00
Vitaly Baranov
f9204315b5 Store columns.txt in backups for the Memory table engine too. 2022-07-05 19:03:20 +02:00
Vitaly Baranov
43d35eec1b Write unfinished mutations to backup. 2022-07-05 14:51:09 +02:00
Vitaly Baranov
92e0ee0b6f More detailed error messages. 2022-07-03 14:20:19 +02:00
Vitaly Baranov
e367d96964 Fix style. 2022-06-30 15:10:33 +02:00
Vitaly Baranov
5456bde4a2 Improve gathering metadata for storing ACL in backups. 2022-06-30 09:46:37 +02:00
Vitaly Baranov
031ca28fdc Add test for partition clause. More checks for data compatibility on restore. 2022-06-30 08:37:18 +02:00
Vitaly Baranov
11b51d2878 Implement storing UDF in backups. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
aa97bf5125 Improve handling predefined databases and tables. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
7689e0c36f Improve gathering metadata for backup - part 6. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
6ca400fd89 Improve gathering metadata for backup - part 5. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
aaf7f66549 Improve gathering metadata for backup - part 4. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
44db346fea Improve gathering metadata for backup - part 3. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
461a31f237 Improve gathering metadata for backup - part 2. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
64b51a3772 Improve gathering metadata for backup. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
18b4413df8
Merge pull request #38299 from vitlibar/backup-improvements-7 
Backup improvements 7
 2022-06-23 11:37:03 +02:00
Vitaly Baranov
5ae8fce1ef Attach threads to thread groups better. 2022-06-22 18:51:41 +02:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
Vitaly Baranov
a6fc0dea4e Fix clang-tidy more. 2022-06-20 11:04:37 +02:00
Vitaly Baranov
638ea23399 Fix build. 2022-06-20 03:44:59 +02:00
Vitaly Baranov
8a7c970ce0 Fix style. 2022-06-19 15:58:26 +02:00
Vitaly Baranov
01aaaf7395 More accurate access checking for RESTORE. 2022-06-19 11:26:41 +02:00
Vitaly Baranov
de9a07d18d Fix RESTORE ALL for tables without database in backup. 2022-06-18 14:07:01 +02:00
Vitaly Baranov
36475c5b98 Fix handling empty files in backups. 2022-06-18 12:28:32 +02:00
Vitaly Baranov
a0c558a17e Implement backup/restore for ACL system tables (system.users, system.roles, etc.) 2022-06-17 18:14:31 +02:00
Vitaly Baranov
c2c35fad82 Refactoring of the code getting create table queries for backup. 2022-06-15 20:32:35 +02:00
Vitaly Baranov
c0f06c5e16 Require new privilige 'BACKUP' to make a backup. 2022-06-15 20:32:35 +02:00
Vitaly Baranov
0102626532 Disable the 'BACKUP ALL' command (it's not quite clear what to do with predefined databases). 2022-06-15 20:32:35 +02:00
Vitaly Baranov
cb9bf62e77 Change syntax RESTORE ALL DATABASES => RESTORE ALL 2022-06-15 20:32:35 +02:00
Vitaly Baranov
1198e86295 Fix storing temporary tables and skipping system tables while making a backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
6877b8f864 Fix renaming visitor. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
d78a2cda72 Restore tables regarding their dependencies. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
cf34883000 Use QualifiedTableName instead of DatabaseAndTableName. Remove mode 'ALL TEMPORARY TABLES' 2022-06-15 20:32:34 +02:00
Vitaly Baranov
21f3bed435 Simplify path calculations in backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
592f568f83 Move backup/restore code to storages and databases - part 2. 2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57 Move backup/restore code to storages and databases - part 1. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
ce1836f0d2 Lock tables for share before backup and restore. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21 Rework collecting replicated parts. 2022-06-15 20:28:42 +02:00
Vitaly Baranov
d00b4a7fdb Remove obsolete function IRestoreCoordination::getReplicatedTableDataPath() 2022-06-15 20:28:42 +02:00
Vitaly Baranov
e891eba80e Finalize write buffers used in backups. 2022-06-15 20:26:27 +02:00
Maksim Kita
98a89b50ff Use pdqsort instead of standard sort 2022-06-13 15:31:08 +02:00
Alexander Gololobov
8cc41521ad tidy build fix 2022-05-17 14:35:12 +02:00
Vitaly Baranov
c1baad0763 Fix style. 2022-05-15 14:09:42 +02:00
Vitaly Baranov
ecbbfca698 Fix handling timeouts. 2022-05-14 12:38:19 +02:00
Vitaly Baranov
feb2de8542 Fix access checking for BACKUP and RESTORE. 2022-05-14 10:48:35 +02:00
Vitaly Baranov
dfa1053b9f Use query scopes for async backup/restore. 2022-05-13 10:35:02 +02:00
Vitaly Baranov
23322b0bf6 Add async tests. 2022-05-12 19:42:05 +02:00
Vitaly Baranov
2c92fe21a9 Implement restoring to a bigger or smaller cluster. 
Remove backup settings allow_storing_multiple_replicas: now it's always allowed.
 2022-05-12 14:55:06 +02:00
Vitaly Baranov
30005a1eff BACKUP ON CLUSTER correctly collects data of a replicated table from all replicas now, 
and if some part doesn't exist on either replica it's no problem anymore.
 2022-05-12 13:33:42 +02:00
Vitaly Baranov
41a41d3a31 Set max_free_threads=0 for BackupsWorker. 2022-05-08 10:43:12 +02:00
Vitaly Baranov
1b2eb4fe27 Use more clear syntax for BACKUP/RESTORE. 2022-05-08 10:37:02 +02:00
Vitaly Baranov
160bc288d3 Fix implementation of totalBytes() & totalRows() for Log family. 2022-05-04 00:15:21 +02:00
Vitaly Baranov
202dd864ed Fix compilation. 2022-05-03 18:34:29 +02:00
Vitaly Baranov
484c2c9c4a Use SYSTEM SYNC DATABASE REPLICA to make code better. 2022-05-03 16:59:41 +02:00
Vitaly Baranov
828f45f078 Add new restore setting 'allow_non_empty_tables'. 2022-05-03 16:18:45 +02:00
Vitaly Baranov
cb9d867f5f Fix restore coordination for creating tables in replicated databases. 2022-05-03 11:03:16 +02:00
Vitaly Baranov
5257ce31f8 Improved using ThreadPool for making backup or restoring, changed columns in system.backups. 2022-05-03 11:03:13 +02:00
Vitaly Baranov
409edfd3fa Rework RestoreCoordination: make restore deterministic. 2022-05-03 11:01:44 +02:00
Vitaly Baranov
2c754f44fc Make calculation of shard_num & replica_num not dependant on match of the cluster's definitions on nodes. 2022-05-03 11:01:44 +02:00
Vitaly Baranov
b1295311c9 Fix crash when BACKUP & RESTORE are called without ON CLUSTER for replicated DB. 2022-05-03 11:01:44 +02:00
Vitaly Baranov
2a645bb187 Fix sending 'create_table' and 'create_database' restore settings to cluster. 2022-05-03 11:01:44 +02:00
Vitaly Baranov
bddec55d35 Added ASTBackupQuery::setDatabase(). 2022-05-03 11:01:44 +02:00
Amos Bird
9d30be2c59
Fix build again 2022-04-29 14:45:27 +08:00
Vitaly Baranov
a8e924caf6 Make BACKUP & RESTORE synchronous by default. 2022-04-26 18:45:39 +02:00
Vitaly Baranov
eb1917f9de Use sequential nodes for counters. Add comments. 2022-04-26 18:45:35 +02:00
Vitaly Baranov
a89ef54c69 Fix tests and compilation. 2022-04-26 13:32:23 +02:00
Vitaly Baranov
16f8c71eb4 Disable usage of archives with backups on clusters. 2022-04-26 10:13:49 +02:00
Vitaly Baranov
1c0b731ea6 Fix compilation. 2022-04-26 10:13:08 +02:00
Vitaly Baranov
78bcb96098 Rename backup & restore setting 'async' -> 'sync', and make backup & restore async by default. 2022-04-26 09:51:19 +02:00
Vitaly Baranov
000b184691 Fix style & compilation. 2022-04-25 23:05:35 +02:00
Vitaly Baranov
5127eeee7b Remove debug sleep. 2022-04-25 17:11:19 +02:00
Vitaly Baranov
ed0af9d4a0 Improve shutdown with async backup/restore. 2022-04-25 16:34:34 +02:00
Vitaly Baranov
97cd68239f Change represenation of shards inside backup. 2022-04-25 16:34:34 +02:00
Vitaly Baranov
042dc4abb2 Rename settings shard -> shard_num, replica -> replica_num. 2022-04-25 16:34:34 +02:00
Vitaly Baranov
129eca7307 CREATE queries generated by RESTORE command is now considered as internal. 2022-04-25 16:34:34 +02:00
Vitaly Baranov
68a020ecea Implement BACKUP/RESTORE ON CLUSTER. 2022-04-25 16:34:33 +02:00
Vitaly Baranov
144d3aefeb Add system table system.backups; implement async mode for commands BACKUP & RESTORE 2022-04-25 16:34:33 +02:00
Vitaly Baranov
ec5b89a6ac Revert backup version 2. Add parameter data_file_path to handle duplicate files more easily without introducing a new backup format. 2022-04-25 16:34:33 +02:00
Vitaly Baranov
c92a8925e3 Refactoring of backup implementation to support archives more easily. 2022-04-25 16:34:33 +02:00
Vitaly Baranov
3966ee1e30 Use SeekableReadBuffer instead of ReadBuffer in IBackupEntry. 2022-04-25 16:34:33 +02:00
Vitaly Baranov
f14613f433 Implemented backup version 2: now files in backups are named by their checksums. 
It will allow to store duplicate files only one time.
 2022-04-25 16:34:33 +02:00
Vitaly Baranov
6c3333b50b RESTORE TABLE doesn't create containing database anymore 2022-04-25 16:34:33 +02:00
Vitaly Baranov
d20b3d78c5 Rename some restore settings. 2022-04-25 16:34:17 +02:00
Vitaly Baranov
6743eb6bef Correct error messages shown by File & Disk backup engines. 2022-04-25 16:33:38 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Vitaly Baranov
0ae63dfdbd
Fix code style in registerBackupEnginesFileAndDisk.cpp 2022-04-11 12:45:16 +02:00
Vitaly Baranov
bcd727d535 Allow creating directories containing a new archive. 2022-04-09 01:32:17 +02:00
Vitaly Baranov
fcd28fc3f4 Make backup metadata clearer with new setting 'use_base'. 2022-04-09 01:07:14 +02:00
Vitaly Baranov
71d54db215 Better handling arguments of backup engines. 2022-04-08 19:52:52 +02:00
Vitaly Baranov
46457c4d0c Improve removing of backup files after error. 2022-04-08 14:25:40 +02:00
Vitaly Baranov
51fc556fcd Implement BACKUP and RESTORE for MatViews. 2022-03-21 11:45:21 +01:00
Vitaly Baranov
ce25afb2e9 Storages and databases are hollow by default now. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
18173d71fb A corresponding file in base backup can be searched by checksum too, not only by name. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
c884cf901f Use BackupEntryFromAppendOnlyFile for Log engine, 
and use a BackupEntriesBatch for Memort engine to improve performance.
A lot of minor corrections.
 2022-03-20 20:02:15 +01:00
Vitaly Baranov
5f195d6ead Implement writing backups as zip archive. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
ef57a87394 Add settings "throw_if_table_has_different_definition" and "throw_if_database_has_different_definition". 2022-03-20 20:02:15 +01:00
Vitaly Baranov
fd1d24f624 Add settings "throw_if_database_exists" and "throw_if_table_exists". 2022-03-20 20:02:15 +01:00
Vitaly Baranov
6849f51c3b Add setting "structure_only." 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7f89b98308 Rework BackupSettings and RestoreSettings a little, pass StorageRestoreSettings to storages. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
258a472001 Shorter names: rename IRestoreFromBackupTask -> IRestoreTask. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
e8fc9cf476 Move renameInCreateQuery() to a better place. 2022-03-20 20:02:12 +01:00
Vitaly Baranov
54d554f55d Simplify elements of ASTBackupQuery. 2022-03-20 20:01:54 +01:00
Vitaly Baranov
5a883c6b5a Fixed checking access rights while producing backup, add IStorage::hasHollowBackup(). 2022-03-20 20:01:54 +01:00
Vitaly Baranov
c160494f43 Make BACKUP more consistent with RESTORE. 2022-03-20 20:01:52 +01:00
Vitaly Baranov
7a63feb3f7 Make restore tasks explicit. 2022-03-20 20:01:31 +01:00
Maksim Kita
07b0845110 Fix clang-tidy warnings in Backups, Client, Columns folders 2022-03-14 18:17:35 +00:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0 Fix split build 2022-03-08 18:05:55 +01:00
Maksim Kita
1f5837359e clang-tidy check performance-noexcept-move-constructor fix 2022-03-02 18:15:27 +00:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Raúl Marín
051dddd8df Reduce dependencies on ASTIdentifier.h 
Goes from rebuilding 483 objects to 165 when it's modified
 2021-11-26 16:49:40 +01:00
Nikolay Degterinsky
ba50e8387d Merge branch 'master' into query_parameters 2021-11-15 19:55:54 +00:00
Nikolay Degterinsky
5c5897c1cd Apply suggestions from code review 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2021-11-15 17:57:26 +00:00
Nikolay Degterinsky
0d58c5231f Better 2021-11-11 13:28:18 +00:00
Nikolay Degterinsky
7b5954b736 Merge branch 'master' into query_parameters 2021-11-11 11:43:00 +00:00
Vitaly Baranov
71359433dd Add parameters "backups.allowed_path" and "backups.allowed_disk" to config. 2021-11-10 11:03:09 +03:00
Vitaly Baranov
606e94f4cc Fix backup of database. 2021-11-10 11:03:09 +03:00
Vitaly Baranov
e0f475c724 Split backup implementation into two parts to help implementing other backup engines. 2021-11-10 11:03:03 +03:00
Vitaly Baranov
bf8606a931 Add UUID and timestamp to backup metadata. 2021-11-09 23:16:45 +03:00
Vitaly Baranov
8a93001245 Switch to XML format for backup's metadata. 2021-11-09 23:16:45 +03:00
Vitaly Baranov
cf77c0b3fc New syntax for BACKUP/RESTORE: set backup engine explicitly. 2021-11-09 23:16:40 +03:00
Azat Khuzhin
93c34824c0 Fix explicit 0 size for createReadBufferFromFileBase() 2021-10-24 18:58:22 +03:00
Nikolay Degterinsky
677eb5bbda Fix tests 2021-10-21 15:53:02 +03:00
Nikolay Degterinsky
e03bf0c526 Merge branch 'master' into query_parameters 2021-10-04 17:23:17 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Nikolay Degterinsky
7f3d395bf3 Merge branch 'master' into query_parameters 2021-09-30 02:01:33 +03:00
Alexey Milovidov
cd7f9d981c Remove ya.make 2021-09-25 04:22:54 +03:00
Nikolay Degterinsky
1b1d90deb8 Merge branch 'master' into query_parameters 2021-09-22 13:53:56 +03:00
tavplubix
922cf7ee20
Merge pull request #28373 from ClickHouse/tables_topsort 
Resolve table dependencies on metadata loading
 2021-09-20 14:46:47 +03:00
Anton Popov
fc17936c12 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-14 23:02:30 +03:00
Alexander Tokmakov
f5c38fe027 fix 2021-09-14 00:39:50 +03:00
Alexander Tokmakov
df56e99b87 fixes after review 2021-09-13 22:11:16 +03:00
Nikolay Degterinsky
84aea776c2 Merge branch 'master' into query_parameters 2021-09-13 12:54:48 +03:00
Sergei Semin
f74350c148 fix special build 2021-09-12 02:49:31 +03:00
Nikolay Degterinsky
22f59e1714 Add support for table and database query parameters 2021-09-07 01:19:02 +03:00
Anton Popov
5e421ab272 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-02 02:42:50 +03:00
Alexey Milovidov
ca6f6306f3 Merge branch 'master' into async-reads 2021-08-31 02:52:58 +03:00
Dmitrii Kovalkov
2c8e2c6995 Fix arcadia build 2021-08-30 12:19:13 +03:00
Anton Popov
6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Alexey Milovidov
8f57216180 Progress on development 2021-08-25 00:45:58 +03:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00