thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 03:12:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
97,000 Commits 9,326 Branches 1,783 Tags 2.1 GiB
842c636354
Commit Graph

9334 Commits

Author SHA1 Message Date
alesapin
f608e62c27 Fix call 2022-08-03 14:17:31 +02:00
alesapin
56a4d26e87 Better total part size calculation on mutation 2022-08-03 14:15:45 +02:00
Maksim Kita
3e6b663020 ASTSelectQuery added hasJoin method 2022-08-03 11:25:45 +02:00
Anton Popov
a333cc4146 Merge remote-tracking branch 'upstream/master' into HEAD 2022-08-02 12:57:43 +00:00
Frank Chen
1c9b462c80 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 20:18:22 +08:00
Igor Nikonov
56697125d3 Merge remote-tracking branch 'origin/master' into distinct_sorted_simplify 2022-08-02 11:18:46 +00:00
Igor Nikonov
4bf42ce867 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-08-02 10:50:25 +00:00
Wangyang Guo
91e3e2f18b KeyCondition: optimize applyFunction in multi-thread scenario 
Construct and deconstruct args (ColumnsWithTypeAndName) will inc/dec
ref_count (actually this is a atomic lock inc/dec operation) to share_ptr,
which may share the same DataTypePtr among different threads. This will
have a lock contention issue in large parallel situation.

The patch try to minimize `args` scope and reduce unnecessary
construct/destory of instances. It will improve the performance in
multi-thread cases.
 2022-08-02 16:16:16 +08:00
Frank Chen
7d8a8f26cd Fix build check 2022-08-02 13:31:36 +08:00
Frank Chen
8eb254c0c8 Fix merge problem 2022-08-02 10:23:51 +08:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
Alexander Tokmakov
1567b65473 Merge branch 'master' into replicated_database_improvements 2022-08-01 19:12:37 +02:00
Alexey Milovidov
3de747a6e2
Merge pull request #39746 from ClickHouse/cleanup-projection-setting 
Cleanup usages of `allow_experimental_projection_optimization` setting, part 1
 2022-08-01 19:21:30 +03:00
Robert Schulze
bf574b9154
Merge pull request #39760 from ClickHouse/bit-fiddling 
Use std::popcount, ::countl_zero, ::countr_zero functions
 2022-08-01 17:04:51 +02:00
Antonio Andelic
9ec27c0ab4
Merge pull request #39757 from ClickHouse/fix-rocksdb-filter-with-params 
Use params correctly for filtering by key in EmbeddedRocksDB
 2022-08-01 12:17:10 +02:00
Dmitry Novik
ab5a147065
Merge pull request #38725 from azat/fix-order-by-projection 
Fix ORDER BY that matches projections ORDER BY
 2022-08-01 00:32:59 +02:00
Nikolai Kochetov
22fbfe19a4 Merge branch 'master' into use-dag-in-key-condition 2022-07-31 21:54:12 +02:00
kssenii
e5f4a619ed Merge master 2022-07-31 20:24:40 +03:00
Robert Schulze
a7734672b9
Use std::popcount, ::countl_zero, ::countr_zero functions 
- Introduced with the C++20 <bit> header

- The problem with __builtin_c(l|t)z() is that 0 as input has an
  undefined result (*) and the code did not always check. The std::
  versions do not have this issue.

- In some cases, we continue to use buildin_c(l|t)z(), (e.g. in
  src/Common/BitHelpers.h) because the std:: versions only accept
  unsigned inputs (and they also check that) and the casting would be
  ugly.

(*) https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
 2022-07-31 15:16:51 +00:00
Robert Schulze
52d08d9db4
Merge pull request #39520 from ClickHouse/no-split-binary 
Remove SPLIT_BINARY
 2022-07-31 14:23:31 +02:00
Antonio Andelic
eea1aaef2c Use params for filtering by key in EmbeddedRocksDB 2022-07-31 10:44:01 +00:00
Igor Nikonov
7f0adb5eb0 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-07-31 07:07:36 +00:00
Anton Popov
74eb8119ef
Merge branch 'master' into fix-custom-serializations 2022-07-31 01:51:52 +02:00
Alexey Milovidov
8fb70abe3e
Merge pull request #39178 from azat/dist-insert-log 
Add connection info for Distributed sends log message
 2022-07-31 02:22:22 +03:00
Alexey Milovidov
441f2feb49 Cleanup usages of allow_experimental_projection_optimization setting, part 1 2022-07-30 23:40:21 +02:00
Igor Nikonov
a7cfad105e
Merge branch 'master' into distinct_sorted_simplify 2022-07-30 21:57:53 +02:00
Igor Nikonov
7b0b38e997 DistinctSortedTransform works only if columns contains sort prefix of 
sort description
 2022-07-29 20:01:07 +00:00
Kruglov Pavel
ccd1e1bdb8
Merge branch 'master' into fix-cancel-insert-into-function 2022-07-29 20:27:32 +02:00
Maksim Kita
8fc6bad4f4 Join enums refactoring 2022-07-29 18:35:05 +02:00
Alexander Tokmakov
6f88065cc1 check metadata consistency 2022-07-29 18:33:16 +02:00
Nikolai Kochetov
59a11b32ad
Merge branch 'master' into use-dag-in-key-condition 2022-07-29 17:01:33 +02:00
Igor Nikonov
fb5799fe7f More generic getInputOrderInfo() in query info 2022-07-29 14:36:36 +00:00
Igor Nikonov
ae82bfb7f3 Merge remote-tracking branch 'origin/master' into distinct_sorted_simplify 2022-07-29 13:12:03 +00:00
Robert Schulze
3d1797f75f
Merge remote-tracking branch 'origin/master' into no-split-binary 2022-07-29 12:17:43 +00:00
Anton Popov
57e4fb2e30 Merge remote-tracking branch 'upstream/master' into HEAD 2022-07-29 11:42:11 +00:00
Anton Popov
bb551566e7 fix build 2022-07-29 11:41:53 +00:00
Azat Khuzhin
498c8b3c52 Fix clang-tidy in utils/examples 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-29 11:36:51 +03:00
Robert Schulze
199e254777
Merge remote-tracking branch 'origin/master' into no-split-binary 2022-07-28 15:54:22 +00:00
Anton Popov
04fd72cdef fix alter column rename 2022-07-28 13:19:23 +00:00
Vitaly Baranov
f8ebe76826
Merge pull request #39503 from vitlibar/improve-system-backups-table 
Improve system.backups table
 2022-07-27 20:25:39 +02:00
Vladimir C
115506356c
Merge branch 'master' into refactor-prepared-sets 2022-07-27 19:57:23 +02:00
Nikolai Kochetov
6206e468a4 Ignor materialize function for ActionsDAG in PK analysis. 2022-07-27 17:02:48 +00:00
Nikolai Kochetov
51c4c9c957 Fixing build. 2022-07-27 15:45:23 +00:00
Anton Popov
56039c8780 fix alter of LowCardinality 2022-07-27 15:35:09 +00:00
Anton Popov
4cf0353892 try to fix Nested 2022-07-27 14:05:16 +00:00
Filatenkov Artur
a218597546
Merge branch 'master' into annoy-2 2022-07-27 15:37:58 +03:00
FArthur-cmd
6b5e92e68a change index policy 2022-07-27 15:36:07 +03:00
Nikolai Kochetov
f9a8a4cd32 Fixing read-in-order for Merge 2022-07-27 12:00:55 +00:00
vdimir
8eecb9ef82
upd PreparedSets: rename/change signature of methods, add comments 2022-07-27 11:22:16 +00:00
Alexander Tokmakov
8fc075a527
Merge pull request #39619 from ClickHouse/try-do-enable-global-allocations-for-attach 
Replace MemoryTrackerBlockerInThread to LockMemoryExceptionInThread
 2022-07-27 13:13:04 +03:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache 2022-07-27 11:35:36 +02:00
Vitaly Baranov
794eeb5d51 Split "total_size" to "uncompressed_size" and "compressed_size". 2022-07-27 10:36:56 +02:00
Vitaly Baranov
a79fc17235 Add test for system.backups 2022-07-27 09:04:10 +02:00
Vitaly Baranov
1cfe0b10f7 Add columns "total_size" and "num_files" to system.backups 2022-07-27 09:04:10 +02:00
Vitaly Baranov
35c267b3b1 Replace column "status_changed_time" with columns "start_time" and "end_time". 2022-07-27 09:04:10 +02:00
Vitaly Baranov
fc16a15ecf Rename column "uuid" -> "id" in system.backups and allow user to set it in a query. 2022-07-27 09:04:10 +02:00
Vitaly Baranov
131019ba49 Rename column "backup_name" -> "name" in system.backups. 2022-07-27 09:04:10 +02:00
Vitaly Baranov
afd0982187 Remove column "internal" from system.backups 2022-07-27 09:04:10 +02:00
Alexander Gololobov
a64aa00869
Merge pull request #37893 from zhangjmruc/feature/sql-standard-delete 
Support SQL standard "delete from ... where ..." syntax and lightweight implementation on merge tree tables
 2022-07-26 23:39:07 +02:00
Nikolai Kochetov
1adb021df6 Fixing tests. 2022-07-26 18:42:27 +00:00
vdimir
5ce2960f03
Get rid of SelectQueryInfoBase -> SelectQueryInfo 2022-07-26 18:39:09 +00:00
vdimir
d9928ac93d
Add methods to SubqueryForSet, do not use refernce to SetPtr 2022-07-26 18:39:09 +00:00
vdimir
1e3fa2e01f
Refactor PreparedSets/SubqueryForSet 2022-07-26 18:39:02 +00:00
Anton Popov
9321ca34cf try to fix Nested 2022-07-26 17:32:37 +00:00
Nikolai Kochetov
33ce662d3e Replace MemoryTrackerBlockerInThread to LockMemoryExceptionInThread in some places. Reduced MemoryTrackerBlockerInThread level to User. 2022-07-26 15:22:00 +00:00
Kruglov Pavel
c683cb252f
Merge pull request #39227 from amosbird/rename-log1 
Rename log when rename merge tree tables
 2022-07-26 17:12:44 +02:00
Nikolai Kochetov
d7cc3831ea Push predicate over StorageMerge. 2022-07-26 15:01:39 +00:00
Nikolai Kochetov
c5110bb164 Add a query plan step for StorageMerge 2022-07-26 14:43:05 +00:00
jianmei zhang
c1834d183b ReplicatedMergeTree support lightweight delete 2022-07-26 11:50:09 +08:00
Alexander Gololobov
25deba2c1b
Merge branch 'master' into feature/sql-standard-delete 2022-07-25 22:13:20 +02:00
Nikolai Kochetov
be9c7ed52c Add ReadFromMerge step. 2022-07-25 19:41:43 +00:00
Roman Vasin
b462366415 Merge branch 'master' of github.com:ClickHouse/ClickHouse into ADQM-485 2022-07-25 17:55:47 +00:00
Roman Vasin
b412ea5f6d Improve generateRandom() for Date32; fix tests 01087_table_function_generate, 01277_fromUnixTimestamp64, 01691_DateTime64_clamp and 01702_toDateTime_from_string_clamping 2022-07-25 17:06:11 +00:00
Alexander Tokmakov
778dcf6994
Merge pull request #39230 from Algunenano/improve_replicated_merge_logging 
Improve logging around replicated merges
 2022-07-25 18:42:12 +03:00
Alexander Gololobov
48de02a7b8 Capitalized const name 2022-07-25 16:32:16 +02:00
Alexander Gololobov
7b44950d1d More detailed comment 2022-07-25 16:14:33 +02:00
Nikolai Kochetov
b70be40804
Merge branch 'master' into use-dag-in-key-condition 2022-07-25 14:30:22 +02:00
Nikolai Kochetov
1ce8e714d3 Fixing build. 2022-07-25 12:30:01 +00:00
Alexey Milovidov
388d06fda1
Merge pull request #39535 from ClickHouse/stringref 
Less usage of StringRef
 2022-07-25 04:06:11 +03:00
Alexander Gololobov
460950ecdc
Merge branch 'master' into feature/sql-standard-delete 2022-07-24 21:27:22 +02:00
Robert Schulze
4333750985
Less usage of StringRef 
... replaced by std::string_view, see #39262
 2022-07-24 18:33:52 +00:00
Alexey Milovidov
0209012e5e
Merge pull request #39517 from pkit/pkit/fix_examples 
fix `-DENABLE_EXAMPLES=1` in master
 2022-07-24 17:41:23 +03:00
Igor Nikonov
eacc57b6a4 Simplify DistinctSorted 
+ use ordinary Distinct for deduplication in MergeTree when no sorting provided
 2022-07-24 14:26:25 +00:00
Azat Khuzhin
22d8e532ed Fix ORDER BY that matches projections ORDER BY 
In case of projection is seleted and it is comlete (all parts has such
projection) and ORDER BY matches projection ORDER BY, and
optimize_read_in_order=1 (default), then the sorting is simply not done,
because projections has separate plan, and so
InterpreterSelectQuery::executeOrder() -> SortingStep has
pipeline.getNumStreams() == 0 and it cannot do sorting.

Fix this, by adding sorting when creating plan for reading from
projections.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-24 15:23:13 +03:00
Kseniia Sumarokova
04944ff6f5
Merge pull request #39509 from monadbobo/remove_duplicate_code 
Removed some duplicate code.
 2022-07-24 14:24:40 +03:00
Alexander Gololobov
6692770113 Allow to update "system" virtual columns such as _row_exists 2022-07-24 12:21:47 +02:00
Alexander Gololobov
c8b3c574a4 Disable lightweight delete if table has projections 2022-07-24 12:21:47 +02:00
Alexander Gololobov
be64b45583 Properly handle empty parts list 2022-07-24 12:21:47 +02:00
Alexander Gololobov
594195451e Cleanups 2022-07-24 12:21:18 +02:00
Alexey Milovidov
071374b152 Remove SPLIT_BINARY 2022-07-24 01:15:54 +02:00
Kseniia Sumarokova
f146183400
Merge pull request #39491 from ClickHouse/remove_unused_constructor 
Remove unused constructor
 2022-07-23 21:16:39 +02:00
mergify[bot]
c01ff2d38a
Merge branch 'master' into improve_replicated_merge_logging 2022-07-23 19:10:10 +00:00
Constantine Peresypkin
d645d03e89 fix -DENABLE_EXAMPLES=1 in master 2022-07-23 19:00:52 +02:00
Kseniia Sumarokova
77ddb8eda0
Merge pull request #39506 from CurtizJ/fix-direct-io 
Fix reading from async read buffer with direct io
 2022-07-23 06:08:48 +02:00
Anton Popov
3d03b2714b try to fix Nested 2022-07-22 23:05:11 +00:00
Nikolai Kochetov
dd65a36c59 Fix another one test. 2022-07-22 19:11:25 +00:00
Nikolai Kochetov
53175e43f3 Better support for projections. 2022-07-22 19:11:25 +00:00
Simon Liu
2305f282dc Removed some duplicate code. 2022-07-23 02:27:17 +08:00
Anton Popov
49627aa554 Merge remote-tracking branch 'upstream/master' into HEAD 2022-07-22 17:16:06 +00:00
Anton Popov
c1a5514e46 fix reading from async read buffer with direct io 2022-07-22 16:58:30 +00:00
Alexander Tokmakov
bed2206ae9
Merge pull request #39460 from ClickHouse/remove_some_dead_and_commented_code 
Remove some dead and commented code
 2022-07-22 13:24:34 +03:00
alesapin
69984b28e0 Remove unused constructor 2022-07-22 12:21:10 +02:00
Robert Schulze
99579ab440
Merge pull request #39364 from ClickHouse/stringref-to-string_view 
Even less usage of StringRef
 2022-07-22 09:30:49 +02:00
Alexander Tokmakov
0441d3629e
Merge pull request #39440 from liangliangpan/optimize-cluster-table-func 
optimize select query from cluster table function
 2022-07-21 20:40:31 +03:00
Mikhail f. Shiryaev
cf0b01b189
Update version to 22.8.1.1 2022-07-21 19:21:28 +02:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache 2022-07-21 18:59:14 +02:00
Robert Schulze
ea0a3bf600
Merge branch 'master' into stringref-to-string_view 2022-07-21 18:33:06 +02:00
zhangxiao871
5549b64c06 Fix issues 39469 2022-07-21 23:52:33 +08:00
Anton Popov
10719fd61d fix style check 2022-07-21 15:06:55 +00:00
Anton Popov
e0d2c8fb37 fix json type with sparse columns 2022-07-21 14:47:19 +00:00
Alexander Tokmakov
8bd674186a
Update getStructureOfRemoteTable.cpp 2022-07-21 16:15:42 +03:00
Alexander Tokmakov
a8da5d96fc remove some dead and commented code 2022-07-21 15:05:48 +02:00
avogar
52b3a87ed1 Fix typo 2022-07-21 12:38:18 +00:00
avogar
6b541aa98f Fix WriteBuffer finalize when cancel insert into function 2022-07-21 12:18:37 +00:00
Alexander Tokmakov
f79f12408d fixes 2022-07-21 13:57:24 +02:00
Alexander Gololobov
1ea9f143ff Leave only _row_exists-based implementation of lightweight delete 2022-07-21 11:26:13 +02:00
Nikolai Kochetov
e15967e9db
Merge pull request #38475 from ClickHouse/additional-filters 
Additional filters for a table (from setting)
 2022-07-21 07:52:04 +02:00
Mingliang Pan
e019b6ba69
Update getStructureOfRemoteTable.cpp 2022-07-21 12:11:33 +08:00
Alexey Milovidov
844042fc18
Merge pull request #39433 from ClickHouse/revert-39396-try-fix-write-buffer-terminate 
Revert "Fix WriteBuffer finalize in destructor when cacnel query"
 2022-07-21 07:04:07 +03:00
Mingliang Pan
3f76c8d7fd
optimize select query from cluster table function 
Use local node as first priority to get Structure Of Remote Table.
we have many distributed queries( like   select xx from cluster('xx',view  (xxxx)   ) on a clickhouse cluster.  we found that the first node (shard_num=1)  have 2 times of query number compared to other shards.
The reason is that  the getStructureOfRemoteTableInShard func always take the first shard to execute  "DESC TABLE xx" query.
The better way is to  use local node as first priority which save the network rpc and reduce the pressure of first shard .
 2022-07-21 11:56:35 +08:00
Alexey Milovidov
fd691000b7
Merge pull request #39342 from azat/bg-executor-fix 
Fix possible UB in MergeTreeBackgroundExecutor (leads to SIGSEGV on race with DROP/DETACH)
 2022-07-21 05:22:51 +03:00
Kruglov Pavel
92995a832b
Revert "Fix WriteBuffer finalize in destructor when cacnel query" 2022-07-21 01:45:16 +02:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Nikolai Kochetov
91aa2385ca
Merge branch 'master' into additional-filters 2022-07-20 18:56:24 +02:00
Kruglov Pavel
0efed4644d
Merge pull request #38957 from Avogar/compatibility-setting 
Add compatibility setting and system.settings_changes system table
 2022-07-20 17:46:06 +02:00
Nikolai Kochetov
4e8cd70b1d
Merge branch 'master' into use-dag-in-key-condition 2022-07-20 17:38:33 +02:00
Dmitry Novik
a09422de7b
Merge pull request #38642 from ClickHouse/group-by-use-nulls 
Add group_by_use_nulls setting
 2022-07-20 14:42:42 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache 2022-07-20 13:30:42 +02:00
Vladimir C
7694c72154
Merge pull request #38887 from bigo-sg/hive_file_crash 
fixed: crash caused by IHiveFile be shared among threads
 2022-07-20 12:15:57 +02:00
Kruglov Pavel
7722b647b7
Merge pull request #39396 from Avogar/try-fix-write-buffer-terminate 
Fix WriteBuffer finalize in destructor when cacnel query
 2022-07-20 12:06:20 +02:00
lgbo-ustc
4b46d80169 update codes 2022-07-20 18:03:34 +08:00
Alexander Tokmakov
988569bdb1
Merge pull request #39366 from ClickHouse/fix_race_in_replication_queue 
Do not enqueue uneeded parts for check
 2022-07-20 00:35:02 +03:00
Nikolai Kochetov
f570cde815 Fixing build. 2022-07-19 20:19:57 +00:00
avogar
5c16d6b553 Fix WriteBuffer finalize in destructor when cacnel query 2022-07-19 19:21:30 +00:00
Nikolai Kochetov
a00bcd389f Fixing build 2022-07-19 19:33:14 +02:00
Nikolai Kochetov
eaeb30a71a Merge branch 'master' into use-dag-in-key-condition 2022-07-19 18:39:52 +02:00
Dmitry Novik
50989bdb68
Merge branch 'master' into group-by-use-nulls 2022-07-19 14:58:01 +02:00
Alexander Tokmakov
a761da9f1b add comment 2022-07-19 13:58:59 +02:00
kssenii
21ebf8874e Merge remote-tracking branch 'upstream/master' into make-cache-composable 2022-07-19 13:36:42 +02:00
Alexander Tokmakov
3e7414b356 add comment 2022-07-19 12:47:20 +02:00
Nikolai Kochetov
d0db7135cb
Merge pull request #38946 from amosbird/nullable-partition-fix1 
Fix incorrect partition pruning when there is a nullable partition
 2022-07-19 12:14:36 +02:00
Kseniia Sumarokova
f168e0c5d0
Merge pull request #39272 from kssenii/fixes-for-materialized-pg 
Fix segfault in materialised postgresql
 2022-07-19 11:56:10 +02:00
Robert Schulze
81ef1099cc
Even less usage of StringRef 
--> see #39300
 2022-07-19 07:01:06 +00:00
FArthur-cmd
3b30cd9259 improvements 2022-07-19 00:59:07 +03:00
Alexander Tokmakov
d82f378a9d do not enqueue uneeded parts for check 2022-07-18 23:37:07 +02:00
Alexander Gololobov
614cb9a87f Disable trivial count() optimization if _row_exists column is present 2022-07-18 20:08:01 +02:00
Alexander Gololobov
ae0d00083c Renamed __row_exists to _row_exists 2022-07-18 20:07:36 +02:00
Alexander Gololobov
f324ca9921 Cleanups 2022-07-18 20:07:22 +02:00
Alexander Gololobov
9de72d995a POC lightweight delete using __row_exists virtual column and prewhere-like filtering 2022-07-18 20:06:42 +02:00
Robert Schulze
6df3c9d799
Merge pull request #39300 from ClickHouse/stringref-to-stringview 
First try at reducing the use of StringRef
 2022-07-18 19:20:16 +02:00
Azat Khuzhin
c6f7698f9a Fix possible UB in MergeTreeBackgroundExecutor 
Lambda erase_from_active() captures the item (TaskRuntimeDataPtr), most
of the code path is OK, since it explicitly reset the item->task.

However one is not, when it moves the item to pending list, which will
be cleaned up when the table will be DROP/DETACH'ed, from
MergeTreeBackgroundExecutor::removeTasksCorrespondingToStorage(),
and in this case if IStorage will be already removed, then it will lead
to use-after-free on destroying the lambda, since it captures the item
by value.

And I belive that CI founds this issue here [1]:

<details>

<summary>stack trace</summary>

    4 0x268d1354 in DB::ReplicatedMergeTreeQueue::CurrentlyExecuting::~CurrentlyExecuting() build_docker/../src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:1510:33
    5 0x268ead1a in std::__1::default_delete<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting>::operator()(DB::ReplicatedMergeTreeQueue::CurrentlyExecuting*) const build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:54:5
    6 0x268ead1a in std::__1::unique_ptr<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting, std::__1::default_delete<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting> >::reset(DB::ReplicatedMergeTreeQueue::CurrentlyExecuting*) build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:315:7
    7 0x268ead1a in std::__1::unique_ptr<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting, std::__1::default_delete<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting> >::~unique_ptr() build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:269:19
    8 0x268ead1a in DB::ReplicatedMergeTreeQueue::SelectedEntry::~SelectedEntry() build_docker/../src/Storages/MergeTree/ReplicatedMergeTreeQueue.h:351:12
    9 0x268ead1a in void std::__1::__destroy_at<DB::ReplicatedMergeTreeQueue::SelectedEntry, 0>(DB::ReplicatedMergeTreeQueue::SelectedEntry*) build_docker/../contrib/libcxx/include/__memory/construct_at.h:56:13
    ...
    16 0x265e9abb in DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>)::'lambda'()::~() build_docker/../src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:127:30
    17 0x265e9abb in DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) build_docker/../src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:235:1
    18 0x265ea730 in DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() build_docker/../src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:261:13

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/533c7888038453c047df816f3f65292cca05a54f/stateless_tests__ubsan__actions_.html

And I also I think that the following issue will also be fixed [2]:

<details>

<summary>stack trace</summary>

    [ 680 ] {} <Fatal> : Logical error: 'Memory tracker: allocations not allowed.'.
    ...
    [ 23430 ] {} <Fatal> BaseDaemon: 23. /build/build_docker/../src/Common/formatReadable.cpp:46: formatReadableSizeWithBinarySuffix(double, int) @ 0x1713bb98 in /usr/bin/clickhouse
    ...
    [ 23430 ] {} <Fatal> BaseDaemon: 31. ../src/Common/MemoryTracker.cpp:112: MemoryTracker::logPeakMemoryUsage() @ 0x170e0ab9 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 32. /build/build_docker/../src/Common/MemoryTracker.cpp:98: MemoryTracker::~MemoryTracker() @ 0x170e063a in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 33. /build/build_docker/../src/Storages/MergeTree/MergeList.cpp:144: DB::MergeListElement::~MergeListElement() @ 0x279fb290 in /usr/bin/clickhouse
    ...
    [ 23430 ] {} <Fatal> BaseDaemon: 38. /build/build_docker/../src/Storages/MergeTree/BackgroundProcessList.h:41: DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>::~BackgroundProcessListEntry() @ 0x276ce6c7 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 39. /build/build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:54: std::__1::default_delete<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo> >::operator()(DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>*) const @ 0x276ce60b in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 40. /build/build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:316: std::__1::unique_ptr<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>, std::__1::default_delete<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo> > >::reset(DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>*) @ 0x276ce57c in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 41. /build/build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:269: std::__1::unique_ptr<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>, std::__1::default_delete<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo> > >::~unique_ptr() @ 0x276ce399 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 42. /build/build_docker/../src/Storages/MergeTree/MutatePlainMergeTreeTask.h:22: DB::MutatePlainMergeTreeTask::~MutatePlainMergeTreeTask() @ 0x27defceb in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 43. /build/build_docker/../contrib/libcxx/include/__memory/construct_at.h:57: void std::__1::__destroy_at<DB::MutatePlainMergeTreeTask, 0>(DB::MutatePlainMergeTreeTask*) @ 0x27dd69c1 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 44. /build/build_docker/../contrib/libcxx/include/__memory/construct_at.h:82: void std::__1::destroy_at<DB::MutatePlainMergeTreeTask, 0>(DB::MutatePlainMergeTreeTask*) @ 0x27dd6955 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: Integrity check of the executable skipped because the reference checksum could not be read. (calculated checksum: 91F5937571C11255DFE73230B52CE9C0)
    [ 602 ] {} <Fatal> Application: Child process was terminated by signal 6.

</details>

  [2]: https://s3.amazonaws.com/clickhouse-test-reports/39222/a068c397dfd7943359a8b554566c3c70b78baf8d/stateless_tests__debug__actions__%5B1/3%5D.html

Refs: https://github.com/ClickHouse/ClickHouse/pull/29614#discussion_r720455032 (cc @nikitamikhaylov)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-18 19:01:41 +03:00
kssenii
0aed5a602a Merge master 2022-07-18 16:16:28 +02:00
Kseniia Sumarokova
b4eae94889
Merge pull request #39259 from rnbondarenko/patch-1 
removed default queue arguments
 2022-07-18 15:32:31 +02:00
avogar
80b5bc5236 Add system table settings_changes 2022-07-18 13:19:41 +00:00
Anton Popov
5605b60c97
Merge pull request #39039 from CurtizJ/randomize-more-settings 
Add more settings for randomization
 2022-07-18 15:05:41 +02:00
kssenii
a6655bea1c Some fixes 2022-07-18 14:23:08 +02:00
Robert Schulze
1ced1d11fd
Merge pull request #39285 from azat/bg-exe 
[RFC] Remove superior atomic from MergeTreeBackgroundExecutor and annotations for TSA
 2022-07-18 12:46:54 +02:00
Kseniia Sumarokova
eb23491957
Merge branch 'master' into fixes-for-materialized-pg 2022-07-18 12:00:58 +02:00
Alexey Milovidov
13a8f85ed0
Merge branch 'master' into nullable-partition-fix1 2022-07-18 00:34:58 +03:00
Robert Schulze
13482af4ee
First try at reducing the use of StringRef 
- to be replaced by std::string_view
- suggested in #39262
 2022-07-17 17:26:02 +00:00
kssenii
8ad26b3111 Merge master 2022-07-16 22:50:03 +02:00
Vitaly Baranov
e0fb03d1b6
Merge pull request #38970 from vitlibar/select-from-system-db-requires-grant 
SELECT from the system database requires grant now
 2022-07-16 22:37:47 +02:00
Kseniia Sumarokova
3f43641b18
Update MergedColumnOnlyOutputStream.cpp 2022-07-16 16:32:33 +02:00
Azat Khuzhin
accb0e38da Remove superior atomic from MergeTreeBackgroundExecutor and annotations for TSA 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-16 14:00:39 +03:00
kssenii
9f1d6814fc Fix 2022-07-15 18:01:54 +02:00
alesapin
5cf4bd50de
Merge pull request #39214 from azat/s3-fix-fetching-memory-part 
Fix fetch of in-memory part with allow_remote_fs_zero_copy_replication
 2022-07-15 17:52:05 +02:00
Alexander Tokmakov
5ef7f5f494
Merge pull request #39229 from azat/fix-noexcept-scope 
[RFC] Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception)
 2022-07-15 18:14:56 +03:00
Raúl Marín
d68b3dfd43 Fix destructor order 2022-07-15 15:48:35 +02:00
Raúl Marín
aea045f297 Improve logging around replicated merges 2022-07-15 15:48:35 +02:00
Vitaly Baranov
c7cef91d4d Two ways to provide access to system.users: "GRANT SHOW USERS ON *.*" and "GRANT SELECT ON system.users" 2022-07-15 15:44:29 +02:00
Vitaly Baranov
fbb2e14d54 Add new table function viewIfPermitted(). 2022-07-15 15:44:29 +02:00
Anton Popov
76eec62469
Merge branch 'master' into randomize-more-settings 2022-07-15 14:12:49 +02:00
Kruglov Pavel
a944a92d4c
Merge pull request #39224 from Avogar/string-view-by-value 
Pass const std::string_view by value, not by reference
 2022-07-15 13:05:56 +02:00
rnbondarenko
a7fe1cf5f0
removed default queue arguments 
declare queue without default arguments x-max-length and x-overflow.
 2022-07-15 13:46:52 +03:00
Alexander Tokmakov
11ea455632
Merge pull request #39245 from ClickHouse/tavplubix-patch-1 
Remove incorrect assertion
 2022-07-15 13:26:14 +03:00
jianmei zhang
197d3796ec Killed mutation has empty commands may cause SegV 2022-07-15 15:37:36 +08:00
jianmei zhang
ca42f649da Rewrite logic for loading deleted mask related to getDeletedMask() 2022-07-15 15:31:10 +08:00
jianmei zhang
0793add9b5 Fix clang tidy compile error, remove std::move for const getDeletedRowsPtr() 2022-07-15 12:32:42 +08:00
jianmei zhang
8b1114fa60 fix clang tidy compile error 2022-07-15 12:32:42 +08:00
jianmei zhang
d37152a5d6 Remove loadDeletedMask() and get deleted mask when needed 2022-07-15 12:32:42 +08:00
jianmei zhang
4268ac8b0e Add supportLightweightDeleteMutate() in IMergeTreeDataPart to disable LWD for part with projections 2022-07-15 12:32:42 +08:00
jianmei zhang
9d1dff515c code changes for strange name and adding exception for mutation entry 2022-07-15 12:32:42 +08:00
jianmei zhang
198674e994 Consider lightweight delete as ordinary for compact part in selectPartsToMutate() 2022-07-15 12:32:42 +08:00
jianmei zhang
780cdfb8f0 Code changes based on review opinions. Make ordinary single alter delete lightwight by default 2022-07-15 12:32:42 +08:00
jianmei zhang
6e6f77ef8a Fix compile error in clang tidy build 2022-07-15 12:32:41 +08:00
jianmei zhang
7e433859ea Change deleted rows mask from String to Native UInt8 format 2022-07-15 12:32:41 +08:00
jianmei zhang
dcc7367ac4 Fix code style error 2022-07-15 12:32:41 +08:00
jianmei zhang
6248486790 Fix lightweight delete bugs: add skip files and use source part's columns to avoid metadata updated cases 2022-07-15 12:32:41 +08:00
jianmei zhang
8ad2bb7c33 Code changes due to master new fixes, and update reference for mutations table 2022-07-15 12:32:41 +08:00
jianmei zhang
11fdea6e4b Add missing code for deleted_mask_filter_holder 2022-07-15 12:32:41 +08:00
jianmei zhang
9d27af7ee2 For some columns mutations, skip to apply deleted mask when read some columns. Also add unit test case 2022-07-15 12:32:41 +08:00
jianmei zhang
2c74e9b866 Merge support to apply lightweight delete mask 2022-07-15 12:32:41 +08:00
jianmei zhang
b4a37e1e22 Disable optimizations for count() when lightweight delete exists, add hasLightweightDelete() function in IMergeTreeDataPart 2022-07-15 12:32:41 +08:00
jianmei zhang
8696319d62 Support lightweight delete execution using string as deleted rows mask,also part of select can handle LWD 2022-07-15 12:32:41 +08:00
jianmei zhang
8df7b7a030 Support new mutation type for lightweight 2022-07-15 12:32:41 +08:00
Sergei Trifonov
62f975da60
Merge pull request #39095 from ClickHouse/remote-io-bandwidth-limits 
add max_remote_{read,write}_network_bandwidth_for_server settings
 2022-07-14 23:14:53 +02:00
Alexander Tokmakov
435d98738d
Update ReplicatedMergeTreeQueue.cpp 2022-07-14 22:41:01 +03:00
avogar
9291d33080 Pass const std::string_view & by value, not by reference 2022-07-14 16:11:57 +00:00
Yakov Olkhovskiy
1f09303d54
Merge pull request #39181 from azat/part_log-merge_algorithm 
Add merge_algorithm to system.part_log
 2022-07-14 11:43:04 -04:00
Sergei Trifonov
b2317ff710 renaming to be consistent 2022-07-14 17:33:22 +02:00
Anton Popov
d4cb2184c2
Merge branch 'master' into randomize-more-settings 2022-07-14 17:01:07 +02:00
Nikolai Kochetov
b24eeedf53
Merge branch 'master' into nullable-partition-fix1 2022-07-14 16:41:38 +02:00
Yakov Olkhovskiy
9ecf19c883
Merge pull request #39212 from azat/stack_trace-filter-v2 
Optimize accesses to system.stack_trace (filter by name before sending signal)
 2022-07-14 10:33:26 -04:00
Azat Khuzhin
b20407fab9 Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception) 
Current implementation of NOEXCEPT_SCOPE will not work, you cannot
rethrow exception outside the catch block, this will simply terminate
(via std::terminate) the program.
In other words NOEXCEPT_SCOPE macro will simply call std::terminate on
exception and will lost original exception.

But if NOEXCEPT_SCOPE will accept the code that should be runned w/o
exceptions, then it can catch exception and log it, rewrite it in this
way.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 17:16:18 +03:00
Amos Bird
96bb6e0cd2
Rename log when rename merge tree tables 2022-07-14 21:22:46 +08:00
Anton Popov
51533adee8 fix build 2022-07-14 11:33:23 +00:00
FArthur-cmd
ad459bc6e4 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy-2 2022-07-14 14:10:44 +03:00
Sergei Trifonov
8a693469f0 Merge branch 'master' into remote-io-bandwidth-limits 2022-07-14 12:38:35 +02:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache 2022-07-14 12:29:54 +02:00
Kseniia Sumarokova
530dac6487
Merge pull request #37213 from aaapetrenko/master 
add TableFunctionMongoDB and tests
 2022-07-14 12:09:07 +02:00
Azat Khuzhin
531e3b13ce Optimize accesses to system.stack_trace (filter by name before sending signal) 
Because of failed rebase/push this part of the patch was lost, so
resending separatelly (sigh).

Follow-up for: #39177 (cc @yakov-olkhovskiy)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 11:49:34 +03:00
Azat Khuzhin
3e77b80b88 Fix fetch of in-memory part with allow_remote_fs_zero_copy_replication 
CI founds the following error during trying to fetch in-memory part [1]:

    2022.07.13 08:06:54.231033 [ 141093 ] {} <Error> InterserverIOHTTPHandler: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/disks/s3/store/886/88635b40-e4e3-4fe6-a0a0-1b6755463358/all_0_0_0/data.bin, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

<details>

<summary>stacktrace</summary>

    2022.07.13 08:06:54.101825 [ 189342 ] {8dbd11b3-f38a-4d5d-9ded-148987adb71d} <Debug> executeQuery: (from [::1]:54570) (comment: 01643_replicated_merge_tree_fsync_smoke.sql) select 'memory in_memory_parts_insert_sync'; (stage: Complete)
    2022.07.13 08:06:54.131309 [ 691 ] {} <Debug> test_26u6kx.rep_fsync_r2 (39c3823c-22e5-4c05-9dec-cdffd8872c40): Fetching part all_0_0_0 from /clickhouse/tables/test_26u6kx/rep_fsync/replicas/r1
    2022.07.13 08:06:54.231033 [ 141093 ] {} <Error> InterserverIOHTTPHandler: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/disks/s3/store/886/88635b40-e4e3-4fe6-a0a0-1b6755463358/all_0_0_0/data.bin, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):
    0. DB::Exception::Exception() @ 0xba0191a in /usr/bin/clickhouse
    1. DB::throwFromErrnoWithPath() @ 0xba029ca in /usr/bin/clickhouse
    2. DB::OpenedFile::open() const @ 0x156e7fb0 in /usr/bin/clickhouse
    3. DB::OpenedFile::getFD() const @ 0x156e8003 in /usr/bin/clickhouse
    4. DB::ReadBufferFromFilePReadWithDescriptorsCache::ReadBufferFromFilePReadWithDescriptorsCache() @ 0x156e5f23 in /usr/bin/clickhouse
    5. ? @ 0x156e53f0 in /usr/bin/clickhouse
    6. DB::createReadBufferFromFileBase() @ 0x156e52b5 in /usr/bin/clickhouse
    7. DB::DiskLocal::readFile() const @ 0x15e45ea8 in /usr/bin/clickhouse
    8. DB::MetadataStorageFromDisk::readFileToString() const @ 0x15e6ab8b in /usr/bin/clickhouse
    9. DB::MetadataStorageFromDisk::readMetadataUnlocked() const @ 0x15e6cdeb in /usr/bin/clickhouse
    10. DB::MetadataStorageFromDisk::getSerializedMetadata() const @ 0x15e6cfc4 in /usr/bin/clickhouse
    11. DB::DiskObjectStorage::getSerializedMetadata() const @ 0x15e19e2e in /usr/bin/clickhouse
    12. DB::DiskDecorator::getSerializedMetadata() const @ 0x15e54ed1 in /usr/bin/clickhouse
    13. DB::DiskDecorator::getSerializedMetadata() const @ 0x15e54ed1 in /usr/bin/clickhouse
    14. DB::DataPartsExchange::Service::sendPartFromDiskRemoteMeta() @ 0x1700bb9e in /usr/bin/clickhouse
    15. DB::DataPartsExchange::Service::processQuery(DB::HTMLForm const&, DB::ReadBuffer&, DB::WriteBuffer&, DB::HTTPServerResponse&) @ 0x1700a649 in /usr/bin/clickhouse
    16. DB::InterserverIOHTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::InterserverIOHTTPHandler::Output&) @ 0x17433c53 in /usr/bin/clickhouse
    17. DB::InterserverIOHTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x174344f1 in /usr/bin/clickhouse
    18. DB::HTTPServerConnection::run() @ 0x1768714d in /usr/bin/clickhouse
    19. Poco::Net::TCPServerConnection::start() @ 0x1a398093 in /usr/bin/clickhouse
    20. Poco::Net::TCPServerDispatcher::run() @ 0x1a399411 in /usr/bin/clickhouse
    21. Poco::PooledThread::run() @ 0x1a54b7bb in /usr/bin/clickhouse
    22. Poco::ThreadImpl::runnableEntry(void*) @ 0x1a548ec0 in /usr/bin/clickhouse
    23. ? @ 0x7fdf1c204609 in ?
    24. clone @ 0x7fdf1c129133 in ?
     (version 22.7.1.1781 (official build))

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/8b6e31cc615ca52c80724b6e5097777cb9514f07/stateless_tests__release__s3_storage__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 11:43:35 +03:00
Yakov Olkhovskiy
d6a09acb38
Merge pull request #39177 from azat/stack_trace-filter 
Optimize accesses to system.stack_trace
 2022-07-14 01:47:14 -04:00
Anton Popov
089009c996 fix reading from StorageLog with mmap 2022-07-13 20:36:18 +00:00
Sergei Trifonov
15ab3bc99f use context->getWriteSettings() 2022-07-13 19:48:57 +02:00
Sergei Trifonov
af4a5a5f3a fix conflicts 2022-07-13 19:44:29 +02:00
kssenii
615221f73b Minor changes 2022-07-13 19:22:47 +02:00
Sergei Trifonov
f854507729 merge master and resolve conflicts 2022-07-13 19:07:28 +02:00
Kseniia Sumarokova
71aba5b4c6
Merge pull request #38860 from kssenii/object-storages-refactor 
Refactoring of code around object storages, added LocalObjectStorage (extracted this diff from PR #36171)
 2022-07-13 16:51:56 +02:00
kssenii
0944183eec Fix merge 2022-07-13 16:50:31 +02:00
Azat Khuzhin
3720a3843c Add merge_algorithm to system.part_log 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 17:16:25 +03:00
Azat Khuzhin
4f25a08b7c Add connection info for Distributed sends log message 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 16:04:12 +03:00
Anton Popov
54a0985c42 randomize one more setting 2022-07-13 12:50:12 +00:00
Azat Khuzhin
8122c1b07f Optimize accesses to system.stack_trace 
- filter by thread_id
- filter by thread_name
- filter by requested columns (query_id/trace, thread_name)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 15:17:09 +03:00
FArthur-cmd
8ddc2578a2 refacroring 2022-07-13 14:48:02 +03:00
kssenii
0d2d3ac2b4 Merge master 2022-07-13 13:26:15 +02:00
kssenii
83a09fd961 Merge remote-tracking branch 'petrenko/master' into petrenko-master 2022-07-13 11:28:52 +02:00
kssenii
ae324622d3 Add one more test 2022-07-13 11:27:43 +02:00
Amos Bird
0795cdd36a
Slightly better interface of waitForMutation 2022-07-13 12:34:12 +08:00
Dmitry Novik
5f65b45269
Merge branch 'master' into group-by-use-nulls 2022-07-12 22:36:04 +02:00
kssenii
6f29eada80 Merge master 2022-07-12 19:28:18 +02:00
Kseniia Sumarokova
0fb35b773f
Update StorageMongoDB.cpp 2022-07-12 18:37:19 +02:00
Kseniia Sumarokova
5468a2fde9
Update StorageMongoDB.cpp 2022-07-12 17:20:20 +02:00
Kseniia Sumarokova
869da0d923
Update StorageMongoDB.cpp 2022-07-12 17:19:49 +02:00
kssenii
f46d641440 Fix style check and black check 
t:wq# Date:      Tue Jul 12 14:02:09 2022 +0200
 2022-07-12 16:29:11 +02:00
Nikolai Kochetov
8efbe6d44d
Merge pull request #39119 from ClickHouse/removed-projection-and-old-part 
Properly remove projection from part in case it was removed from table metadata.
 2022-07-12 16:24:30 +02:00
Robert Schulze
69de9ee0e8
Merge pull request #39108 from ClickHouse/prefix_cmake_commands 
Prefix overridden add_executable() command with "clickhouse_"
 2022-07-12 15:05:48 +02:00
FArthur-cmd
6e2812490b refactoring 2022-07-12 14:29:11 +03:00
Alexander Tokmakov
c99fe6e6c8
Merge branch 'master' into removed-projection-and-old-part 2022-07-12 14:00:55 +03:00
FArthur-cmd
0e89fbff53 refactoring 2022-07-12 13:57:16 +03:00
Alexander Tokmakov
b35ad0dbdc
Merge pull request #39091 from ClickHouse/tavplubix-patch-1 
Follow-up to #39036
 2022-07-12 13:47:37 +03:00
Nikolai Kochetov
945b5d4b3f
Merge pull request #38993 from azat/fix-fsync_part_directory-on-fetch 
Fix fsync_part_directory for fetches
 2022-07-12 11:48:42 +02:00
Nikolai Kochetov
cd4211298e Fix typo. 2022-07-12 09:37:00 +00:00
Nikolai Kochetov
03d9301425 Fix log. 2022-07-12 08:31:09 +00:00
Nikolai Kochetov
631be9219b Properly remove projection from part in case it was removed from table metadata. 2022-07-12 08:28:22 +00:00
FArthur-cmd
783e6e398c merge with other working branch 2022-07-11 23:02:58 +03:00
aaapetrenko
aabf0ed67f refactor Storage 2022-07-11 22:00:57 +02:00
FArthur-cmd
6ae227a6bb add test 2022-07-11 22:50:10 +03:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_" 
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
 2022-07-11 19:36:18 +02:00
Sergei Trifonov
6ed777d2a8 add azure support 2022-07-11 19:19:57 +02:00
Sergei Trifonov
add0d1698e respect new limits in HDFS 2022-07-11 17:30:23 +02:00
FArthur-cmd
868628d3fd make stable prototype with arrays support 2022-07-11 18:02:50 +03:00
Sergei Trifonov
43779ec280 add max_remote_{read,write}_network_bandwidth_for_server settings 2022-07-11 14:59:39 +02:00
Alexander Tokmakov
e9ddbc247f
Merge pull request #39016 from Algunenano/clearOldLogs_zkerror 
clearOldLogs: Don't report KEEPER_EXCEPTION on concurrent deletes
 2022-07-11 15:16:18 +03:00
Alexander Tokmakov
048d27189e
Update MergeTreeData.cpp 2022-07-11 15:07:31 +03:00
FArthur-cmd
7b7a9521c7 change submodule path 2022-07-10 18:10:47 +03:00
FArthur-cmd
2f9f5f172a Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy_3 2022-07-10 17:25:21 +03:00
kssenii
4c88527c8e Review fixes 2022-07-09 19:35:07 +02:00
kssenii
bf1c99daab Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
Frank Chen
da57a993e4 Fix CI 2022-07-09 13:43:10 +08:00
Alexander Tokmakov
51130d723c
Update MergeTreeData.cpp 2022-07-08 20:17:29 +03:00
Raúl Marín
a965438eea clearOldLogs: Don't report errors on concurrent deletes 2022-07-08 13:57:53 +02:00
Azat Khuzhin
b2f7f9c0f0 Fix fsync_part_directory for fetches 
Before it produces the following error:

    2022.07.07 23:40:03.131044 [ 44869 ] {} <Error> default.rep_data: auto DB::StorageReplicatedMergeTree::processQueueEntry(ReplicatedMergeTreeQueue::SelectedEntryPtr)::(anonymous class)::operator()(DB::StorageReplicatedMergeTree::LogEntryPtr &) const: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/tmp-fetch_36_25458_25458_0, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

    0. Poco::Exception::Exception() @ 0x1764634c in /usr/bin/clickhouse
    1. DB::Exception::Exception() @ 0xb42caba in /usr/bin/clickhouse
    2. DB::throwFromErrnoWithPath() @ 0xb42d79b in /usr/bin/clickhouse
    3. DB::LocalDirectorySyncGuard::LocalDirectorySyncGuard() @ 0x1404bbd4 in /usr/bin/clickhouse
    4. DB::DiskLocal::getDirectorySyncGuard() const @ 0x140466a4 in /usr/bin/clickhouse
    5. DB::DataPartsExchange::Fetcher::downloadPartToDisk() @ 0x15122af1 in /usr/bin/clickhouse
    6. DB::DataPartsExchange::Fetcher::fetchPart() @ 0x1511de39 in /usr/bin/clickhouse

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-08 13:48:51 +03:00
lgbo-ustc
e01b6cf8a5 update codes 2022-07-08 09:25:45 +08:00
Alexander Tokmakov
b0f8970d07
Merge pull request #38961 from Algunenano/reduce_zk_eexists 
Stop reporting Zookeeper "Node exists" exceptions in system.errors when they are expected
 2022-07-07 23:16:46 +03:00
Dmitry Novik
d1df66687b
Merge branch 'master' into group-by-use-nulls 2022-07-07 20:54:38 +02:00
Raúl Marín
9b0c7714e5 Style 2022-07-07 17:15:23 +02:00
Kseniia Sumarokova
48220f64d6
Merge pull request #38214 from bigo-sg/storagehive_alter 
support alter on `StorageHive` table
 2022-07-07 17:06:43 +02:00
Raúl Marín
c3935000a8 Avoid reporting ZK EEXIST errors on checkPartChecksumsAndCommit 2022-07-07 16:10:05 +02:00
Raúl Marín
33ffb71453 Avoid 'Node exists' errors when creating Ephemeral Locks 2022-07-07 16:10:04 +02:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
alesapin
1bd97fa6fa
Merge pull request #38851 from Algunenano/rmarin_read_only_on_drop 
Deactivate mutations_finalizing_task during shutdown
 2022-07-07 12:55:50 +02:00
Frank Chen
93dc109e36 Fix code 2022-07-07 17:44:19 +08:00
Frank Chen
d3d89f59ca Add tracing support to distributed insert 2022-07-07 17:43:09 +08:00
Frank Chen
57a7e4a7c9 Remove old API reference 2022-07-07 17:42:35 +08:00
FArthur-cmd
8968ff8995 small inmprovements 2022-07-07 11:33:06 +03:00
Amos Bird
19bd3d8d9f
Nullable partition pruning fix 2022-07-07 16:02:38 +08:00
lgbo-ustc
337f379bc3 update codes 2022-07-07 09:45:49 +08:00
lgbo-ustc
cda081dc57 update codes 2022-07-07 09:41:52 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9 
Backup Improvements 9
 2022-07-07 02:24:47 +02:00
FArthur-cmd
a3407ba6db Merge branch 'master' of github.com:FArthur-cmd/ClickHouse into annoy-2 2022-07-06 19:42:55 +03:00
Vitaly Baranov
7f84cf3968 Fix style. 2022-07-06 16:36:59 +02:00
Maksim Kita
b94489d52c
Merge pull request #38859 from kitaisreal/merge-tree-merge-disable-batch-optimization 
MergeTree merge disable batch optimization
 2022-07-06 15:59:40 +02:00
Vladimir C
3c7fdb604d
Merge pull request #35363 from vdimir/kv-join 2022-07-06 15:57:30 +02:00
FArthur-cmd
c94d6a57ec add arrays 2022-07-06 14:39:49 +03:00
Raúl Marín
86d1c319f4 Remove unnecessary log 2022-07-06 12:23:45 +02:00
lgbo-ustc
aeb4f9c6be fixed: crash caused by IHiveFile be shared among threads 2022-07-06 16:56:17 +08:00
lgbo-ustc
946a9b8c63 update codes 2022-07-06 09:26:02 +08:00
Kseniia Sumarokova
a26643a7a6
Merge pull request #38363 from kssenii/pg-add-auto-close 
Support `auto_close` option for postgres engine connection
 2022-07-06 01:10:05 +02:00
kssenii
e55169ebad Fix 2022-07-05 20:15:35 +02:00
Vitaly Baranov
f9204315b5 Store columns.txt in backups for the Memory table engine too. 2022-07-05 19:03:20 +02:00
Kseniia Sumarokova
3e803006a2
Merge pull request #37171 from tchepavel/nats-integration 
Nats Integration
 2022-07-05 17:47:02 +02:00
Maksim Kita
bdc21737d5 MergeTree merge disable batch optimization 2022-07-05 16:15:00 +02:00
Vitaly Baranov
43d35eec1b Write unfinished mutations to backup. 2022-07-05 14:51:09 +02:00
Raúl Marín
9e17ae365b Deactivate mutations_finalizing_task during shutdown 2022-07-05 14:28:20 +02:00
vdimir
17903117d2
better 2022-07-05 10:46:08 +00:00
kssenii
7a4be3b748 Fix build, disable for freebsd 2022-07-05 12:10:49 +02:00
Kruglov Pavel
a6a9824261
Merge pull request #38362 from Avogar/better-exception-messages 
Better exception messages on wrong table engines/functions argument types
 2022-07-05 11:21:44 +02:00
lgbo-ustc
76813ee3c0 update codes 2022-07-05 11:49:55 +08:00
kssenii
b52084265c Merge master 2022-07-04 21:37:43 +02:00
Kseniia Sumarokova
6c3d648f49
Merge branch 'master' into nats-integration 2022-07-04 21:06:07 +02:00
Kseniia Sumarokova
0cd228cab3
Merge pull request #38365 from kssenii/better-rabbit-adddress-setting 
Fix RabbitMQ configuration via connection string setting
 2022-07-04 20:53:43 +02:00
vdimir
c0cb588e45
Fix build 2022-07-04 17:28:34 +00:00
vdimir
43afda408b
do not call setStorageJoin when setting isn't set 2022-07-04 17:28:15 +00:00
vdimir
65ac97ce89
direct join: handle nullable, add tests 2022-07-04 17:28:14 +00:00
vdimir
8359753a85
Move DirectJoin to separate file 2022-07-04 17:28:11 +00:00
vdimir
5ed0911ff4
Fix rocksdb::getSlicedKeys 2022-07-04 17:28:10 +00:00
vdimir
085e70c7a3
wip left kvjoin 2022-07-04 17:28:10 +00:00
vdimir
980000d04c
Change getByKeys, add StorageEmbeddedRocksDB::getByKeysImpl 2022-07-04 17:28:09 +00:00
vdimir
744c692be3
wip key value join 2022-07-04 17:28:09 +00:00
vdimir
8bce6451c6
add method StorageEmbeddedRocksDB::getByKeys 2022-07-04 17:27:37 +00:00
Anton Popov
8b356f0e1b better tests and comments 2022-07-04 15:06:43 +00:00
Dmitry Novik
c66e2c8095
Merge pull request #37155 from amosbird/projection-fix-two 
Fix intermediate header in aggregate projection
 2022-07-04 16:32:34 +02:00
FArthur-cmd
21df8ed12f refactoring 2022-07-04 17:21:50 +03:00
Kruglov Pavel
6c4b6fa834
Merge branch 'master' into better-exception-messages 2022-07-04 15:07:31 +02:00
kssenii
cfff7c4c28 Merge master 2022-07-04 14:13:26 +02:00
FArthur-cmd
21b99e7304 Merge branch 'master' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-07-04 14:54:21 +03:00
Vitaly Baranov
1dba3d9f9c Make temporary files in better places. 2022-07-04 11:02:58 +02:00
lgbo-ustc
932e596689 Merge remote-tracking branch 'ck/master' into storagehive_alter 2022-07-04 09:56:00 +08:00
Raúl Marín
838fd1e41e Fix replication after improper merge process 2022-07-03 15:11:25 +02:00
Vitaly Baranov
92e0ee0b6f More detailed error messages. 2022-07-03 14:20:19 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder 2022-07-02 15:39:44 +02:00
kssenii
831d2253ce Merge master 2022-07-01 18:53:20 +02:00
Anton Popov
ef87e1207c better support of read_in_order in case of fixed prefix of sorting key 2022-07-01 16:45:01 +00:00
Dmitry Novik
81dd90893e Merge remote-tracking branch 'origin/master' into group-by-use-nulls 2022-07-01 16:24:05 +00:00
alesapin
0d7298f3a9 Fix typos 2022-07-01 15:26:27 +02:00
alesapin
57284e6a9d Fix possible deadlocks with MergeTreeData::Transaction 2022-07-01 15:16:32 +02:00
Vitaly Baranov
cadb496894
Merge pull request #38537 from vitlibar/backup-improvements-8 
Backup Improvements 8
 2022-07-01 11:20:46 +02:00
Amos Bird
d73f975849
Fix intermediate header in aggregate projection 2022-07-01 16:47:32 +08:00
alesapin
09faa1641e Some comments 2022-06-30 23:09:32 +02:00
alesapin
979565edf0 Merge branch 'master' into better_data_part_storage_builder 2022-06-30 22:54:42 +02:00
alesapin
eb5046ab26 Simplify everything 2022-06-30 22:51:27 +02:00
Alexander Tokmakov
60dcae9af0
Merge pull request #38627 from Algunenano/rmarin_fix_replicated_part_2 
Adapt some more nodes to avoid issues with pre-22.4 replicas
 2022-06-30 21:11:35 +03:00
Alexander Tokmakov
d2cbdc7c53
Update ReplicatedMergeTreeQueue.cpp 2022-06-30 17:23:53 +03:00
Maksim Kita
a0e5768956
Merge pull request #38623 from loyispa/patch 
Fix typo
 2022-06-30 16:02:36 +02:00
Vitaly Baranov
e367d96964 Fix style. 2022-06-30 15:10:33 +02:00
Raúl Marín
cbcd740dc1 Adapt some more nodes to avoid issues with pre-22.4 replicas 2022-06-30 15:10:09 +02:00
avogar
ee54c4f9b7 Add some fixes and add settings in docs 2022-06-30 12:41:56 +00:00
alesapin
1363b478d4 Merge branch 'fix_disk_tx_write_op' into better_data_part_storage_builder 2022-06-30 14:38:32 +02:00
Anton Popov
7c721578c7
Merge pull request #38320 from CurtizJ/dynamic-columns-16 
Improve performace of insertion to columns of type JSON
 2022-06-30 14:18:20 +02:00
alesapin
cb90aca2ef Some changes in merge tree 2022-06-30 14:12:45 +02:00
loyispa
109c9bcbd5 Fix typo 2022-06-30 18:37:19 +08:00
Vitaly Baranov
5456bde4a2 Improve gathering metadata for storing ACL in backups. 2022-06-30 09:46:37 +02:00
Antonio Andelic
de264117fd
Merge pull request #38118 from bigo-sg/storagehive_struct_type 
Add struct type support in `StorageHive`
 2022-06-30 09:11:13 +02:00
Vitaly Baranov
031ca28fdc Add test for partition clause. More checks for data compatibility on restore. 2022-06-30 08:37:18 +02:00
Vitaly Baranov
11b51d2878 Implement storing UDF in backups. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
efbee5e723 Improve gathering metadata for backup - part 7. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
01921ce9a3 Move most code from setTableStructure() to a separate function. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
7689e0c36f Improve gathering metadata for backup - part 6. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
461a31f237 Improve gathering metadata for backup - part 2. 2022-06-30 08:37:17 +02:00