thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
87,112 Commits 9,326 Branches 1,784 Tags 2.1 GiB
e73d7fea31
Commit Graph

7683 Commits

Author SHA1 Message Date
lgbo-ustc
f4d8fb46c5 update codes 2022-03-07 09:26:05 +08:00
lgbo-ustc
62c1bd5ae9 hive read columns pruning 2022-03-07 09:26:05 +08:00
Azat Khuzhin
bc224dee36 Do not hide exceptions during mutations 
system.mutations includes only the message, but not stacktrace, and it
is not always obvious to understand the culprit w/o stacktrace.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-06 13:39:49 +03:00
Kseniia Sumarokova
3ec6cd3128
Update StorageFileLog.cpp 2022-03-06 11:03:22 +01:00
alexey-milovidov
f9b7df6ba1
Merge pull request #35050 from CurtizJ/fix-async-inserts-system-table 
Fix reading from `system.asynchronous_inserts` table
 2022-03-06 02:25:53 +03:00
Maksim Kita
7ae1f0fa3b
Merge pull request #34911 from larspars/master 
Allow LowCardinality strings for ngrambf_v1/tokenbf_v1 indexes. Fixes #21865
 2022-03-04 19:17:48 +01:00
Azat Khuzhin
2ef9d32448 Revert "Remove VERSION_DATE from system.build_options" 
As requested by @kitaisreal

This reverts commit 4a404532fb.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:32:54 +03:00
Azat Khuzhin
4a404532fb Remove VERSION_DATE from system.build_options 
It was set only bu utils/release/release_lib.sh, and seems that this
script is not used anymore, at least that part of it.

Also note, that GIT_DATE is the same, and it is date time, not only
date.

Plus VERSION_DATE is not installed for releases anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:37 +03:00
Azat Khuzhin
b0f964a14f Remove LIBRARY_ARCHITECTURE from system.build_options 
CMAKE_LIBRARY_ARCHITECTURE and it is useless, since it is reported only
if the compiler reports subdir arch triplet [1]

  [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1531678

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:32 +03:00
Azat Khuzhin
494fe91f86 Fix LINK_FLAGS in system.build_options 
Fixes: 79f6f5a202
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:32 +03:00
Azat Khuzhin
c426eef07d Fix generating USE_* for system.build_options 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:32 +03:00
Anton Popov
c836a57000 fix reading from system.asynchronous_inserts table 2022-03-04 11:46:15 +00:00
Anton Popov
df3b07fe7c Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-03 22:25:28 +00:00
Anton Popov
aea7bfb59a
Merge pull request #34992 from azat/fix-asynchronous_inserts-race 
Fix race between INSERT async_insert=1 and system.asynchronous_inserts
 2022-03-03 20:55:19 +03:00
kssenii
d19f199e93 Revert 2022-03-03 15:25:27 +01:00
Kseniia Sumarokova
b11b34dc8c
Merge pull request #34849 from kssenii/fix-too-many-columns 
Fix reading too many columns for s3 and url storages
 2022-03-03 13:57:22 +01:00
Kseniia Sumarokova
ad09554c4c
Merge pull request #34996 from kssenii/fix-filelog-assertion 
Fix possible segfault in filelog storage
 2022-03-03 13:52:54 +01:00
Frank Chen
b4829465d9
Improve the opentelemetry span logs for INSERT on distributed table (#34480) 2022-03-03 12:53:29 +01:00
mergify[bot]
e169813004
Merge branch 'master' into fix-too-many-columns 2022-03-02 18:46:35 +00:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
Maksim Kita
1f5837359e clang-tidy check performance-noexcept-move-constructor fix 2022-03-02 18:15:27 +00:00
mreddy017
f893002b69 Fix vulnerable code related to std::move and noexcept 
This commit fixes the vulnerable code related to std::move and noexcept identified by clangtidy tool.
 2022-03-02 18:15:27 +00:00
kssenii
d5952109fb Merge master 2022-03-02 18:15:25 +01:00
mergify[bot]
add225c83e
Merge branch 'master' into fix-filelog-assertion 2022-03-02 17:06:08 +00:00
Amos Bird
d4cdf04683
Add missing locks (#34025) 2022-03-02 16:23:29 +01:00
kssenii
5e84c75942 Fix 2022-03-02 15:31:34 +01:00
kssenii
ef344a581b Fix bug in FileLog storage 2022-03-02 15:28:17 +01:00
alesapin
b1f5805647
Merge pull request #34609 from ClickHouse/unrestricted-zk-reads 
allow unrestricted reads from zookeeper
 2022-03-02 14:53:12 +01:00
Azat Khuzhin
57f636a1e8 Fix race between INSERT async_insert=1 and system.asynchronous_inserts 
CI report [1]:

    [c190f600f8c6] 2022.03.02 01:07:34.553012 [ 23552 ] {76b6113b-1479-46c9-90ab-e78a3c9f3dbb}  executeQuery: Code: 60. DB::Exception: Both table name and UUID are empty. (UNKNOWN_TABLE) (version 22.3.1.1) (from [::1]:42040) (comment: '02015_async_inserts_stress_long.sh') (in query: SELECT * FROM system.asynchronous_inserts FORMAT Null), Stack trace (when copying this message, always include the lines below):

    0. ClickHouse/contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int) @ 0xf50e04c in /fasttest-workspace/build/programs/clickhouse
    1. ClickHouse/src/Common/Exception.cpp:58: DB::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int, bool) @ 0x663ebfa in /fasttest-workspace/build/programs/clickhouse
    2. DB::StorageID::assertNotEmpty() const @ 0xbc08591 in /fasttest-workspace/build/programs/clickhouse
    3. ClickHouse/contrib/libcxx/include/string:1444: DB::StorageID::getDatabaseName() const @ 0xe50d2b6 in /fasttest-workspace/build/programs/clickhouse
    4. ClickHouse/contrib/libcxx/include/string:1957: DB::StorageSystemAsynchronousInserts::fillData(std::__1::vector::mutable_ptr, std::__1::allocator::mutable_ptr > >&, std::__1::shared_ptr, DB::SelectQueryInfo const&) const @ 0xdac636c in /fasttest-workspace/build/programs/clickhouse

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/34973/e6fc6a22d5c018961c18247242dd3a40b8c54ff2/fast_test__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-02 15:28:06 +03:00
alesapin
9249c5d50e Use tryget instead of get 2022-03-02 13:09:12 +03:00
Maksim Kita
53116faeeb
Update MergeTreeIndexFullText.cpp 2022-03-02 11:08:35 +01:00
Kseniia Sumarokova
a9ab149b31
Merge pull request #34859 from Vxider/windowview-multi-column-groupby 
Fix bugs for multiple columns group by in WindowView
 2022-03-02 10:09:47 +01:00
Filatenkov Artur
f48f35cad0
Merge pull request #34975 from Vector-Similarity-Search-for-ClickHouse/fix-typo 
Fix typo
 2022-03-02 09:59:06 +03:00
Anton Popov
d7cd9aa69b fix reading of missed subcolumns 2022-03-02 03:31:40 +03:00
NikitaEvs
06f47673f4 Fix typo 2022-03-01 21:42:27 +00:00
alesapin
e2989c2b85 Fix storage system zookeeper 2022-03-01 21:23:26 +01:00
alesapin
ec3e4251e1 Fix style 2022-03-01 20:34:25 +01:00
kssenii
a594f388a4 Merge master 2022-03-01 19:43:45 +01:00
kssenii
aa1c71a877 Merge master 2022-03-01 19:25:04 +01:00
kssenii
755e63ed03 Keep compatibility 2022-03-01 19:21:59 +01:00
Anton Popov
c1fdcf7a64 Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-01 20:21:39 +03:00
Anton Popov
04a3a10148 minor fixes 2022-03-01 20:20:53 +03:00
alesapin
cba5fe44a9 Merge branch 'master' into unrestricted-zk-reads 2022-03-01 18:09:21 +01:00
Anton Popov
2758db5341 add more comments 2022-03-01 19:32:55 +03:00
taiyang-li
74bc819f12 fix conflict with master and update contrib/arrow version 2022-03-01 20:14:23 +08:00
kssenii
092ec45b47 Merge master 2022-03-01 12:06:56 +01:00
lgbo-ustc
ca470e1b94 lazy initialization about getting hive metadata in HiveStorage 2022-03-01 19:04:44 +08:00
Kseniia Sumarokova
781621eefe
Merge pull request #34946 from bigo-sg/hive_table_function 
Add hive table function
 2022-03-01 11:28:36 +01:00
alesapin
4b61e4795c
Merge pull request #34949 from nikitamikhaylov/system_log_tables_and_settings 
Recreate system.{*}_log table on settings changes
 2022-03-01 11:15:19 +01:00
lgbo-ustc
5ed41bda9b fixed code style 2022-03-01 17:20:32 +08:00
lgbo-ustc
5ae99df87c fxied code style 2022-03-01 15:35:57 +08:00
lgbo-ustc
6e568c1530 update codes 2022-03-01 15:24:40 +08:00
lgbo-ustc
91a45d799e optimization for first time to read a random access readbuffer 2022-03-01 15:22:07 +08:00
Nikita Mikhaylov
d6036f6da3 Better 
(cherry picked from commit 4ae445c9e227581ea9f1cbe9aa9d1ba82e1236c9)
 2022-02-28 15:27:52 +00:00
Kruglov Pavel
011813957d
Merge pull request #34938 from azat/create-as-ignore-ttl 
Ignore per-column TTL in CREATE TABLE AS if new table engine does not support it
 2022-02-28 16:58:15 +03:00
kssenii
9b64a8fe39 Fix odbc bridge 2022-02-28 14:29:05 +01:00
lgbo-ustc
99cd25d70e add new table function: hive() 2022-02-28 20:51:33 +08:00
mergify[bot]
b4574a777f
Merge branch 'master' into mvcc_prototype 2022-02-28 11:24:39 +00:00
lgbo-ustc
6473767c99 fixed code style 2022-02-28 17:10:56 +08:00
lgbo-ustc
5885cfd869 fixed bug : unexpected result when using in clause for filtering partitions 2022-02-28 16:47:50 +08:00
Hongbin
c9bc442114
fix comments 2022-02-28 16:44:35 +08:00
Azat Khuzhin
644f9168fa Ignore per-column TTL in CREATE TABLE AS if new table engine does not support it 
Follow-up for: #6968
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-28 10:29:26 +03:00
lgbo-ustc
c5e02be44e fixed code-style 2022-02-28 15:22:54 +08:00
lgbo-ustc
2176d74cd1 Use connection pool in HiveMetastoreClient 
1. remove lock for hive metastore client access
2. auo reconnect when connection is broken
 2022-02-28 15:11:38 +08:00
Hongbin
99bd56e2de
Fix some code comments style 2022-02-28 08:15:37 +08:00
mergify[bot]
8d84d22618
Merge branch 'master' into windowview-multi-column-groupby 2022-02-26 00:50:49 +00:00
kssenii
2ba9010a34 Fix 2022-02-25 17:53:19 +01:00
kssenii
6c8401bfbd Fix 2022-02-25 16:35:37 +01:00
Lars Eidnes
2629614dfe Allow LowCardinality strings for ngrambf_v1/tokenbf_v1 indexes. Fixes #21865 2022-02-25 15:36:36 +01:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
kssenii
cfad79bf74 Remove redundant 2022-02-25 09:37:48 +01:00
Alexander Tokmakov
11ae0d144b fix 2022-02-25 00:51:21 +03:00
Sergei Trifonov
2d25c79e37 analyze select queries from system.zookeeper table with LIKE pattern and fetch nodes using prefix recursively 2022-02-24 13:40:47 +03:00
Vxider
06469eb793 remove blank row 2022-02-24 14:16:24 +08:00
Vxider
43475f79bf windowview_multi_column_groupby 2022-02-24 14:06:37 +08:00
Alexander Tokmakov
711aad6953 fix 2022-02-24 01:31:21 +03:00
Alexander Tokmakov
aa6b9a2abc Merge branch 'master' into mvcc_prototype 2022-02-23 23:22:03 +03:00
kssenii
003b807b00 Fix 2022-02-23 20:33:05 +01:00
tavplubix
43626b3ffd
Update src/Storages/FileLog/StorageFileLog.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-02-23 21:07:37 +03:00
Alexander Tokmakov
5a26f856d9 remove trash that shouldn't have been merged 2022-02-22 23:41:33 +03:00
Dmitry Novik
2fd4baaa64
Merge pull request #34387 from nvartolomei/nv/move-part-settings-cleanup 
Remove useless setting experimental_query_deduplication_send_all_part_uuids
 2022-02-22 06:11:00 -08:00
Sergei Trifonov
7fe3bef866 add test for unrestricted zk reads 2022-02-22 16:51:30 +03:00
kssenii
c637385dd0 Merge master 2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove 
Some refactoring and improvement over async and remote buffer related stuff
 2022-02-22 11:36:40 +01:00
Dmitry Novik
1df43a7f57
Merge pull request #34385 from nvartolomei/nv/move-part-count 
Disable optimize_trivial_count when deduplication for part movement feature is enabled
 2022-02-21 08:53:09 -08:00
Anton Popov
065305ab65
Merge pull request #34764 from ucasfl/hints-index 
Add name hints for data skipping indices
 2022-02-21 16:50:59 +03:00
Mikhail f. Shiryaev
5ac8cdbc69
Merge pull request #34786 from ClickHouse/make_drop_column_metadata_only 
Make drop of alias column metadata only
 2022-02-21 14:11:55 +01:00
mergify[bot]
314ab73b11
Merge branch 'master' into nv/move-part-settings-cleanup 2022-02-21 10:18:44 +00:00
Dmitry Novik
4428e7aa1b
Merge branch 'master' into nv/move-part-count 2022-02-21 02:14:23 -08:00
alesapin
d7cae5ffb4 Fix build 2022-02-21 11:54:52 +03:00
alesapin
852757219f Make drop of alias column metadata only 2022-02-21 11:46:16 +03:00
Vitaly Baranov
aee67a6693
Merge pull request #31484 from eungenue/Implement-SSL-X509-certificate-authentication 
Implement ssl x509 certificate authentication
 2022-02-21 11:30:52 +03:00
Vitaly Baranov
0d377de5f0 Support syntax CREATE USER IDENTIFIED WITH ssl_certificate CN ... 2022-02-21 07:01:00 +03:00
Vitaly Baranov
7b97c986cb
Revert "Allow restrictive row policies without permissive" 2022-02-21 06:54:28 +03:00
feng lv
07280e0ab1 Add name hints for data skipping indices 
fix test
 2022-02-20 11:48:22 +00:00
Vitaly Baranov
874b2c8dcb
Merge pull request #34596 from vitlibar/allow-restrictive-without-permissive 
Allow restrictive row policies without permissive
 2022-02-19 21:45:28 +07:00
Azat Khuzhin
fef5f146e7 Fix ENOENT with fsync_part_directory and Vertical merge 
fsync of the temporary part directory is superfluous anyway, and besides
that directory is not exists at that time, that will lead to ENOENT
error:

    2022.02.18 17:02:51.634565 [ 35639 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/system/text_log/tmp_merge_202202_1864_3192_14/, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

    0. DB::Exception::Exception() @ 0xb26ecfa in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    1. DB::throwFromErrnoWithPath() @ 0xb2700ea in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    2. DB::LocalDirectorySyncGuard::LocalDirectorySyncGuard() @ 0x14905531 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    3. DB::DiskLocal::getDirectorySyncGuard() const @ 0x148af3e3 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    4. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x157bef13 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug

Note, that IMergeTreeDataPart::renameTo() anyway will have fsync for the
directory.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-19 07:50:59 +03:00
Nikolai Kochetov
e4d5db6161
Merge pull request #34717 from azat/merge-mutate-memory-tracker 
Fix possible memory_tracker use-after-free (for async s3 writes) for merges/mutations
 2022-02-18 19:28:43 +01:00
Vladimir C
9b7d011ee7
Merge pull request #34529 from vdimir/join-nullable-on-pipeline 
Apply join_use_nulls on types before join
 2022-02-18 18:34:44 +01:00
Azat Khuzhin
65e9b4879d Fix possible memory_tracker use-after-free for merges/mutations 
There are two possible cases for execution merges/mutations:
1) from background thread
2) from OPTIMIZE TABLE query

1) is pretty simple, it's memory tracking structure is as follow:

    current_thread::memory_tracker = level=Thread / description="(for thread)" ==
      background_thread_memory_tracker = level=Thread / description="(for thread)"
    current_thread::memory_tracker.parent = level=Global / description="(total)"

  So as you can see it is pretty simple and MemoryTrackerThreadSwitcher
  does not do anything icky for this case.

2) is complex, it's memory tracking structure is as follow:

    current_thread::memory_tracker = level=Thread / description="(for thread)"
    current_thread::memory_tracker.parent = level=Process / description="(for query)" ==
      background_thread_memory_tracker = level=Process / description="(for query)"

  Before this patch to track memory (and related things, like sampling,
  profiling and so on) for OPTIMIZE TABLE query dirty hacks was done to
  do this, since current_thread memory_tracker was of Thread scope, that
  does not have any limits.

  And so if will change parent for it to Merge/Mutate memory tracker
  (which also does not have some of settings) it will not be correctly
  tracked.

  To address this Merge/Mutate was set as parent not to the
  current_thread memory_tracker but to it's parent, since it's scope is
  Process with all settings.

  But that parent's memory_tracker is the memory_tracker of the
  thread_group, and so if you will have nested ThreadPool inside
  merge/mutate (this is the case for s3 async writes, which has been
  added in #33291) you may get use-after-free of memory_tracker.

  Consider the following example:

    MemoryTrackerThreadSwitcher()
      thread_group.memory_tracker.parent = merge_list_entry->memory_tracker
      (see also background_thread_memory_tracker above)

    CurrentThread::attachTo()
      current_thread.memory_tracker.parent = thread_group.memory_tracker

    CurrentThread::detachQuery()
      current_thread.memory_tracker.parent = thread_group.memory_tracker.parent
      # and this is equal to merge_list_entry->memory_tracker

    ~MemoryTrackerThreadSwitcher()
      thread_group.memory_tracker = thread_group.memory_tracker.parent

  So after the following we will get incorrect memory_tracker (from the
  mege_list_entry) when the next job in that ThreadPool will not have
  thread_group, since in this case it will not try to update the
  current_thread.memory_tracker.parent and use-after-free will happens.

So to address the (2) issue, settings from the parent memory_tracker
should be copied to the merge_list_entry->memory_tracker, to avoid
playing with parent memory tracker.

Note, that settings from the query (OPTIMIZE TABLE) is not available at
that time, so it cannot be used (instead of parent's memory tracker
settings).

v2: remove memory_tracker.setOrRaiseHardLimit() from settings

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-18 16:23:54 +03:00
Amos Bird
f459e8fc95
Less getMark calls 2022-02-18 19:55:19 +08:00
Alexander Tokmakov
f4a46a13fb fixes 2022-02-18 00:26:37 +03:00
Anton Popov
0a7895ebb9 add comments and small refactoring 2022-02-17 22:00:25 +03:00
zvonand
90c857c5e3 merge 2022-02-17 18:23:37 +03:00
tavplubix
0f5ee19d0b
Merge pull request #34633 from zhangjmruc/master 
For ReplatedMergeTree, early break for multiple leaders case when log has been updated by the other leader
 2022-02-17 14:01:50 +03:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
Kruglov Pavel
6dcb766879
Merge pull request #34465 from Avogar/fix-url-globs 
Improve schema inference with globs in FIle/S3/HDFS/URL engines
 2022-02-17 13:33:27 +03:00
Vitaly Baranov
2de6e8e575 Change type of RowPolicyKind: bool -> enum. 2022-02-17 14:18:10 +07:00
Amos Bird
d3bd8b5f93
Cosmetic fix 2022-02-17 14:31:22 +08:00
Amos Bird
ba19c7cf44
Slightly better interface of compressed buffer 2022-02-17 14:31:22 +08:00
Jianmei Zhang
ef0c3b99ff Merge remote-tracking branch 'upstream/master' 2022-02-17 14:02:27 +08:00
Sergei Trifonov
b6bb479c48 add setting to enable unrestricted reads from zookeeper 2022-02-16 23:03:44 +03:00
Sergei Trifonov
f342c497ef fix style 2022-02-16 20:22:03 +03:00
Azat Khuzhin
774744a86d Fix allow_experimental_projection_optimization with enable_global_with_statement 
allow_experimental_projection_optimization requires one more
InterpreterSelectQuery, which with enable_global_with_statement will
apply ApplyWithAliasVisitor if the query is not subquery.

But this should not be done for queries from
MergeTreeData::getQueryProcessingStage()/getQueryProcessingStageWithAggregateProjections()
since this will duplicate WITH statements over and over.

This will also fix scalar.xml perf tests, that leads to the following
error now:

    scalar.query0.prewarm0: DB::Exception: Stack size too large.

And since it has very long query in the log, this leads to the following
perf test error:

    _csv.Error: field larger than field limit (131072)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-16 19:14:47 +03:00
kssenii
47f94120da Merge master 2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0 Fix tests 2022-02-16 14:08:41 +01:00
Mikhail f. Shiryaev
4f84406136
Merge pull request #34641 from ClickHouse/version-and-release 
refactor version_helper, create release script
 2022-02-16 14:00:55 +01:00
Maksim Kita
d6e88f56cd
Merge pull request #34623 from CurtizJ/minor-subcolumns-fix 
Fix quadratic complexity while adding subcolumns
 2022-02-16 12:38:00 +01:00
Mikhail f. Shiryaev
c5db40f679
Deprecate sh script for StorageSystemContributors, update generated file 2022-02-16 12:16:43 +01:00
Nikolai Kochetov
f9d2dae88e
Merge pull request #34424 from yakov-olkhovskiy/ephemeral-column 
Ephemeral column issue #9436
 2022-02-16 12:04:57 +01:00
Kruglov Pavel
dd863ca2a0
Merge branch 'master' into fix-url-globs 2022-02-16 12:45:31 +03:00
Jianmei Zhang
25c761b3b6 Early break for multiple leaders case when log updated by other leader 2022-02-16 16:06:41 +08:00
taiyang-li
a4baec6d26 fix building 2022-02-16 15:12:43 +08:00
taiyang-li
afcb295273 fix compile error 2022-02-16 14:51:56 +08:00
taiyang-li
f19f0d847f fix code style 2022-02-16 12:23:06 +08:00
taiyang-li
dec083ab44 add minmax index for hivengine; remove libhdfspp dependency for hive engine 2022-02-16 12:19:16 +08:00
Anton Popov
e4fddaa03a fix quadratic complexity while adding subcolumns 2022-02-16 02:42:50 +03:00
Anton Popov
a661eaf39f better performance of getting storage snapshot 2022-02-16 02:17:22 +03:00
alesapin
bc2d0ee7c7
Merge pull request #34215 from ClickHouse/revert-34211-revert-34153-add_func_tests_over_s3 
Add func tests run with s3 and fix several bugs
 2022-02-15 19:07:11 +03:00
Alexander Tokmakov
e37ef4560c fix 2022-02-15 18:00:45 +03:00
Sergei Trifonov
a507f83d8d allow unrestricted reads from zookeeper 2022-02-15 17:12:37 +03:00
Nikolai Kochetov
ab288642f6 Merge branch 'master' into ephemeral-column 2022-02-15 10:03:34 +00:00
Nikolai Kochetov
d6cbac1ed3
Merge pull request #34577 from ClickHouse/alwasy-remove-unused-actions-for-add-missing-defaults 
Always remove unused actions from addMissingDefaults
 2022-02-15 11:01:29 +01:00
alesapin
447cd56cb9 Fix comments 2022-02-15 12:11:50 +03:00
李扬
f52b67b939
Merge branch 'master' into rocksdb_metacache 2022-02-15 02:16:29 -06:00
Alexander Tokmakov
1e4e569151 Merge branch 'master' into mvcc_prototype 2022-02-15 02:26:47 +03:00
Alexander Tokmakov
ae5aa8c12d write part version before other files 2022-02-15 02:24:51 +03:00
Alexander Tokmakov
cbd3b45646 add EXPLAIN CURRENT TRANSACTION 2022-02-14 22:47:17 +03:00
alesapin
e15396d90c Fix race condition: 2022-02-14 22:19:49 +03:00
Nikolai Kochetov
b3ea360cd2 Fix a little bit more 2022-02-14 19:05:30 +00:00
Kseniia Sumarokova
382b8e0388
Merge pull request #34432 from ClickHouse/static-files-disk-uploader-create-symlinks 
`static-files-disk-uploader`: add a mode to create symlinks
 2022-02-14 18:10:53 +01:00
vdimir
99ca89c0ca
Fix StorageJoin and Asof or join_use_nulls in pipeline 2022-02-14 14:14:27 +00:00
alesapin
bb69455395
Merge pull request #34504 from CurtizJ/ttl-move-if-exists 
Support `TTL TO [DISK|VOLUME] [IF EXISTS]`
 2022-02-14 14:56:18 +03:00
alesapin
b75d551281 Fix clang tidy and add check for master 2022-02-14 14:37:41 +03:00
alesapin
b2886a429b Fix lock during fetch 2022-02-14 12:20:27 +03:00
alesapin
beb4400978 Fix 'same local part' check 2022-02-13 23:08:29 +03:00
alesapin
89373155fc Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-13 21:07:54 +03:00
Yakov Olkhovskiy
579fe6c97a major rework, transform added to the insert pipe 2022-02-13 17:42:59 +00:00
zvonand
dcc0f53f8e updated Interval definitions 2022-02-13 17:54:03 +03:00
Maksim Kita
cb09aadc72
Merge pull request #34524 from FrankChen021/urlengine 
Fix compression support in URL engine
 2022-02-13 14:39:36 +01:00
Maksim Kita
ae1da31d19
Merge pull request #34558 from ucasfl/storage-log 
try fix data race in StorageLog
 2022-02-13 14:33:36 +01:00
mergify[bot]
8a47ed2b3f
Merge branch 'master' into static-files-disk-uploader-create-symlinks 2022-02-13 06:12:39 +00:00
alexey-milovidov
1774836d4f
Merge pull request #34555 from kitaisreal/table-functions-insert-partition-by-refactoring 
TableFunctionFile added performance test
 2022-02-13 09:08:51 +03:00
feng lv
4ccd1c3278 try fix data race in StorageLog 2022-02-13 04:52:16 +00:00
Maksim Kita
e2c8ba9ab2 Added performance test 2022-02-12 16:05:35 +00:00
李扬
daa27d0bda
Merge branch 'master' into rocksdb_metacache 2022-02-12 07:50:12 -06:00
alexey-milovidov
747b6b2058
Merge pull request #34528 from CurtizJ/key-condition-float 
Fix comparison between integers and floats in index analysis
 2022-02-12 10:19:09 +03:00
alexey-milovidov
ea71dc9d11
Merge pull request #34510 from kitaisreal/table-functions-insert-partition-by-refactoring 
Improve performance of insert into table functions URL, S3, File, HDFS
 2022-02-12 10:14:00 +03:00
alexey-milovidov
9bb2eba281
Merge pull request #34429 from ClickHouse/filimonov-SystemAsynchronousInserts 
asynchronous_inserts engine AsynchronousInserts -> SystemAsynchronousInserts
 2022-02-12 10:08:04 +03:00
Frank Chen
7b7bc8f6c3 Address review comment 2022-02-12 10:46:01 +08:00
Dmitry Novik
50131fd732 Fix cancelation for S3 and HDFS 2022-02-11 18:14:55 +00:00
Anton Popov
6a8e35930f fix comparison with integers and floats in index analysis 2022-02-11 18:20:37 +03:00
tavplubix
6d1fae834a
Merge pull request #34187 from qoega/default-table-engine 
Default table engine
 2022-02-11 18:14:29 +03:00
Anton Popov
2fcd69baf7 fix comparison with integers and floats in index analysis 2022-02-11 17:15:27 +03:00
Alexander Tokmakov
07e66e690d Merge branch 'master' into mvcc_prototype 2022-02-11 15:53:32 +03:00
mergify[bot]
f099048dd1
Merge branch 'master' into urlengine 2022-02-11 11:03:21 +00:00
Frank Chen
c8cddd7077 Fix compression for URL engine 2022-02-11 17:45:49 +08:00
Anton Popov
f012871a7c better caching of common types of object columns 2022-02-11 01:20:30 +03:00
Maksim Kita
13cbf79ecb Improve performance of insert into table functions URL, S3, File, HDFS 2022-02-10 20:06:23 +00:00
alesapin
705529ca03 Followup 2022-02-10 22:50:15 +03:00
alesapin
ef61c9b47c fix 2022-02-10 22:49:33 +03:00
alesapin
3af06b23f8 POC 2022-02-10 22:45:52 +03:00
alesapin
088f8cacf3 Small improvements 2022-02-10 19:50:21 +03:00
Anton Popov
70986a70a1 support TTL TO [DISK|VOLUME] [IF EXISTS] 2022-02-10 19:26:23 +03:00
Kruglov Pavel
bf8c028533
Merge pull request #34479 from Avogar/fix-filesystem-error 
Fix possible error 'file_size: Operation not supported'
 2022-02-10 14:17:05 +03:00
alesapin
f764da35ca Also zero copy mutations 2022-02-10 14:15:08 +03:00
Kruglov Pavel
a4f5610764
Merge pull request #34476 from CurtizJ/avoid-settings-copy 
Avoid unnecessary copying of `Settings`
 2022-02-10 14:13:46 +03:00
alesapin
70221b272b Better solution 2022-02-10 12:57:11 +03:00
Anton Popov
298838f891 avoid unnecessary copying of Settings 2022-02-10 12:13:51 +03:00
avogar
bfa96463ca Fix possible error 'file_size: Operation not supported' 2022-02-10 09:23:27 +03:00
Anton Popov
dcd7312d75 cache common type on objects in MergeTree 2022-02-09 23:47:53 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
alesapin
c587160308 Bad fix which will affect zk and should be changed 2022-02-09 23:06:44 +03:00
alesapin
57037465f5 Trying to fix tests blindly 2022-02-09 22:56:22 +03:00
Kruglov Pavel
538830931b
Merge branch 'master' into fix-url-globs 2022-02-09 20:09:56 +03:00
mergify[bot]
d55eadc3c9
Merge branch 'master' into default-table-engine 2022-02-09 17:08:31 +00:00
Kruglov Pavel
4ec8da73c4
Merge pull request #34448 from Avogar/fix-url-engine 
Fix bug in URL engine
 2022-02-09 19:19:05 +03:00
avogar
84087c1554 Improve schema inference with globs 2022-02-09 19:16:15 +03:00
Yakov Olkhovskiy
a134ab282b style fixed and minor optimization 2022-02-09 13:51:12 +00:00
Anton Popov
822b58247a
Merge pull request #34132 from CurtizJ/fix-distributed-inserts 
Fix inserts to distributed tables in case of change of native protocol
 2022-02-09 14:58:44 +03:00
Kruglov Pavel
e0e36c256e
Fix style 2022-02-09 14:44:46 +03:00
alesapin
10c3e6e546 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-09 14:14:58 +03:00
alesapin
72863cc4c3 Fix error 2022-02-09 13:57:10 +03:00
avogar
7baf23f35f Fix bug URL engine 2022-02-09 13:53:54 +03:00
alesapin
437940b29d
Merge pull request #34422 from ClickHouse/add_settings_to_part_size_in_s3 
Add settings for multipart upload part size in s3
 2022-02-09 12:51:20 +03:00
Kseniia Sumarokova
86956ca08b
Merge pull request #34392 from kssenii/http-buffer-skip-not-found-url-for-globs 
Allow to skip not found urls for globs
 2022-02-09 09:32:09 +01:00
taiyang-li
d04ccc0489 Merge branch 'master' into rocksdb_metacache 2022-02-09 11:54:10 +08:00
Yakov Olkhovskiy
b9777be271 insertion with ephemerals requires columns list, insertion without columns list omits ephemeral 2022-02-09 01:11:29 +00:00
Alexey Milovidov
742620209a static-files-disk-uploader: add mode to create symlinks 2022-02-09 03:50:50 +03:00
Anton Popov
587d7399ba support dynamic subcolumns for Memory engine 2022-02-09 03:18:53 +03:00
kssenii
e8a8f46864 Fix style check 2022-02-08 21:58:03 +01:00
alesapin
36909a986f Fix bug with files remove 2022-02-08 22:21:16 +03:00
filimonov
8d310e4ac3
AsynchronousInserts -> SystemAsynchronousInserts 2022-02-08 19:23:40 +01:00
alesapin
02a93cb852 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-08 19:42:27 +03:00
alesapin
f6e7cea1f8 Add settings for S3 multipart upload parts size 2022-02-08 19:38:04 +03:00
kssenii
9f2628b8f2 Fix 2022-02-08 16:41:55 +01:00
Kruglov Pavel
cae1517693
Fix build 2022-02-08 17:31:02 +03:00
Nikolai Kochetov
0f7c0c72bd
Merge pull request #34305 from amosbird/projection-fix27 
Fix various issues when projection is enabled by default
 2022-02-08 17:19:56 +03:00
Kruglov Pavel
720310fcf1
Merge branch 'master' into http-buffer-skip-not-found-url-for-globs 2022-02-08 16:56:22 +03:00
Kruglov Pavel
3e216adc17
Merge pull request #34405 from Avogar/fix-segfault-in-url 
Fix segfault in schema inference from url
 2022-02-08 16:48:37 +03:00
Anton Popov
7e9770dcf0 minor enhancements 2022-02-08 15:57:23 +03:00
kssenii
dc5f035265 Fix 2022-02-08 13:28:10 +01:00
Nicolae Vartolomei
50ee264223 Disable projects when allow_experimental_query_deduplication is in use 2022-02-08 12:16:10 +00:00
alesapin
8e9ccbd077
Merge pull request #33933 from sunny19930321/fix-substr-zk-metadata 
Better local metadata comparison with ZooKeeper metadata
 2022-02-08 14:58:46 +03:00
alesapin
b47b0eb1dc Revert accident change 2022-02-08 14:05:01 +03:00
avogar
e118c89bb2 Fix segfault in schema inference from url 2022-02-08 13:40:03 +03:00
Yatsishin Ilya
bcf4303286 Merge remote-tracking branch 'origin' into default-table-engine 2022-02-08 10:35:29 +00:00
kssenii
3834bdbae0 Fixes 2022-02-08 11:11:18 +01:00
zvonand
0633174b10 Merge branch 'master' of github.com:ClickHouse/ClickHouse into issue_33147 2022-02-08 11:42:31 +03:00
alesapin
3af6012cb4 Revert "Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"""" 
This reverts commit 2bc2ea485e.
 2022-02-08 11:01:26 +03:00
alesapin
2bc2ea485e Revert "Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer""" 
This reverts commit fb77d7a7d5.
 2022-02-08 10:56:29 +03:00
taiyang-li
b6132d490f merge master and solve conflict 2022-02-08 15:24:59 +08:00
alexey-milovidov
43ee8ddb5b
Merge pull request #34376 from CurtizJ/fix-reading-empty-arrays 
Fix consecutive backward seeks in seekable read buffers
 2022-02-08 02:27:57 +03:00
Yakov Olkhovskiy
4d5fb56c29 added EPHEMERAL default for column (preliminary) 2022-02-07 23:21:10 +00:00
kssenii
eba3011ada Fix 2022-02-07 20:40:47 +01:00
Nicolae Vartolomei
7d77678a9f Remove useless setting experimental_query_deduplication_send_all_part_uuids 
This setting made sense for testing deduplication before part movement was actually implemented.

allow_experimental_query_deduplication setting is enough and code is covered by test_part_moves_between_shards
 2022-02-07 19:03:20 +00:00
zvonand
9456d89fad Move current work from old laptop 
skip-checks: true
 2022-02-07 21:44:14 +03:00
Anton Popov
1b16db72c3 fix consecutive backward seeks in seekable read buffers 2022-02-07 17:20:26 +03:00
alesapin
523d1059ed
Update KeyDescription.cpp 2022-02-07 13:59:03 +03:00
alesapin
ba28c2c013 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-07 12:44:56 +03:00
alesapin
fb77d7a7d5 Revert "Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer"" 
This reverts commit 875e5413ad.
 2022-02-07 12:36:54 +03:00
alesapin
bb361e3b01 Fix bug with existing part 2022-02-07 12:33:37 +03:00
Yatsishin Ilya
e0803064e6 Merge remote-tracking branch 'origin' into default-table-engine 2022-02-07 08:45:32 +00:00
Amos Bird
2debfc922d
Better projection format and test fixes 2022-02-07 10:47:11 +08:00
alexey-milovidov
fb9ed9acad
Merge pull request #34175 from kitaisreal/bitset-sort-performance-check 
bitsetsort peformance check
 2022-02-07 05:34:57 +03:00
Alexander Tokmakov
45be75b4db Merge branch 'master' into mvcc_prototype 2022-02-06 23:36:08 +03:00
feng lv
6325d4d9b0 continue of #34317 
fix

fix
 2022-02-06 08:59:17 +00:00
Amos Bird
52aabf98fe
Revise and add more comments 2022-02-06 16:53:54 +08:00
Amos Bird
1ab773cc90
Fix aggregation_in_order with normal projection 2022-02-06 16:46:12 +08:00
Amos Bird
3fab7af541
Bug fix and improvement of minmax_count_projection 2022-02-06 16:46:11 +08:00
Amos Bird
a0ab7a01f1
Adapt minmax_count_projection with ModuleLegacy 2022-02-06 16:46:11 +08:00
Amos Bird
98857de82b
Disable projection for high-order storages 2022-02-06 16:46:10 +08:00
Amos Bird
27fcefd315
Disable projection when doing parallel replica reading 2022-02-06 16:46:10 +08:00
Amos Bird
7674bc986e
Disable projection when there is JOIN or SAMPLE 2022-02-06 16:46:09 +08:00
Maksim Kita
e626b7b2d3 Fixed tests 2022-02-04 19:52:18 +00:00
Maksim Kita
1cd1fb9658 Fixed tests 2022-02-04 19:52:18 +00:00
Maksim Kita
f301e50b23 Fixed tests 2022-02-04 19:52:18 +00:00
alexey-milovidov
e194bf3377
Merge pull request #34317 from ucasfl/lowcard 
use LowCardinality for _file and _path virtual columns in StorageFile
 2022-02-04 21:39:44 +03:00
Alexander Tokmakov
3956941aaf fixes 2022-02-04 21:18:20 +03:00
Nikolai Kochetov
daeeb6f3a2
Merge pull request #34316 from ClickHouse/probably-fix-data-race-in-WriteBufferFromS3 
Probably fix data race in WriteBufferFromS3 destructor.
 2022-02-04 21:04:46 +03:00
Nikolai Kochetov
6436024e08 Fix test with ttl. 2022-02-04 16:05:02 +00:00
alesapin
c028269e6f Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 18:02:25 +03:00
alesapin
d4d147abab Remove redundant 2022-02-04 18:01:49 +03:00
Azat Khuzhin
63e674280b Decrease severity for "Reading ... ranges ..." log message to Trace 
That way with send_logs_level='debug', you will not get verbose
information that you already has, since there is summary row:

    Selected ... parts by partition key, ... parts by primary key, ... marks by primary key, ... marks to read from ... ranges

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-04 18:00:59 +03:00
tavplubix
22de534fdc
Merge pull request #34297 from ClickHouse/fix_restarting_thread 
Try to fix race between pullLogsToQueue and RestartingThread
 2022-02-04 17:30:17 +03:00
feng lv
7bf2243431 use LowCardinality for _file and _path virtual columns in StorageFile 
fix
 2022-02-04 14:15:56 +00:00
alesapin
1582c4bf24 Fix mutaion status for retryable errors 2022-02-04 16:27:59 +03:00
Maksim Kita
ab696e6b59
Merge pull request #34310 from ClickHouse/fix-parallel-loading-parts 
Fix parallel loading of data parts
 2022-02-04 13:10:24 +01:00
alesapin
875e5413ad Revert "Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer" 
This reverts commit b92efed350, reversing
changes made to ecce006cb2.
 2022-02-04 14:30:33 +03:00
alesapin
af18905a33 Fixup 2022-02-04 13:03:13 +03:00
alesapin
2ed45b2a98 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 11:23:46 +03:00
Nikolai Kochetov
b92efed350
Merge pull request #34219 from ClickHouse/revert-34212-revert-33291-add-pool-to-s3-write-buffer 
Revert "Revert "Add pool to WriteBufferFromS3""
 2022-02-04 11:00:29 +03:00
Alexey Milovidov
78eebd5c7c Fix parallel loading of data parts 2022-02-04 02:29:46 +03:00
Alexander Tokmakov
897e94c16c make restarting thread less bad 2022-02-03 23:29:24 +03:00
Alexander Tokmakov
fe30e0f162 fixes 2022-02-03 21:57:09 +03:00
Alexander Tokmakov
ca5f951558 Merge branch 'master' into mvcc_prototype 2022-02-03 18:56:44 +03:00
Anton Popov
9b844c6b42
Merge pull request #32748 from CurtizJ/read-in-order-fixed-prefix 
Support `optimize_read_in_order` if prefix of sorting key is already sorted
 2022-02-03 18:17:08 +03:00
alesapin
2a9bc7cba8 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-03 16:20:06 +03:00
Nikolai Kochetov
a2a9c10617
Merge pull request #34221 from ClickHouse/revert-34201-revert-34189-less-logging-for-remote_fs_execute_merges_on_single_replica_time_threshold 
Revert "Revert "Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue""
 2022-02-03 13:09:44 +03:00
alesapin
763dfd7895 Fix write ahead log 2022-02-03 12:14:00 +03:00
alesapin
25375bc76b Remove unused param 2022-02-03 11:21:19 +03:00
alesapin
e0f640dd9f Fix 2022-02-02 19:44:29 +03:00
alesapin
80800e051e Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-02 19:42:04 +03:00
Nikolai Kochetov
ea044fc6b2 Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer 2022-02-02 19:40:41 +03:00
alesapin
b9c118524f Fix race condition on hardlink/erase/read metadata 2022-02-02 19:40:21 +03:00
Nikolai Kochetov
1117e6095d
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-02-02 18:34:05 +03:00
Azat Khuzhin
15993cb13b Add missing fmt::runtime() in MergeTreeBackgroundExecutor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-02 11:37:53 +03:00
alesapin
47d538b52f
Merge pull request #34232 from ClickHouse/cancelled-merging-parts-error-message 
Change severity of the "Cancelled merging parts" message in logs
 2022-02-02 10:29:07 +03:00
Sergei Trifonov
68bc456830
Merge pull request #34223 from azat/bump-fmt 
Bump fmtlib from 7.0.0 to 8.1.1
 2022-02-02 00:03:25 +03:00
Yatsishin Ilya
759574d6f0 improve 2022-02-01 16:59:41 +00:00
Nikolai Kochetov
dd81eff301 Fix tests. 2022-02-01 15:28:22 +00:00
mergify[bot]
d8bea598b2
Merge branch 'master' into fix-substr-zk-metadata 2022-02-01 14:10:12 +00:00
Yatsishin Ilya
2a698b8da3 Merge remote-tracking branch 'origin/master' into default-table-engine 2022-02-01 13:23:52 +00:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
mergify[bot]
a177ec91e5
Merge branch 'master' into read_in_order_max_rows_to_read 2022-02-01 12:12:20 +00:00
Alexander Tokmakov
d93c9f0be4 fix build after merge 2022-02-01 15:05:05 +03:00
alesapin
e87ed0ff02 Fix in-memory parts 2022-02-01 14:37:12 +03:00
Azat Khuzhin
5dfafd68a7 ReplicatedMergeTreeQueue: Fix fmt:: and reduce copy-paste of logging and out reason 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Azat Khuzhin
5be76bc969 Use proper fmt-like logging 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Azat Khuzhin
3b3635c6d5 Fix formatting error in logging messages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Azat Khuzhin
743096a883 Use proper fmt:: like Exception ctor in DataPartsExchange 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Nikolai Kochetov
a6cc61bd14
Revert "Revert "Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue"" 2022-02-01 14:17:46 +03:00
alesapin
0aebab50f5 Fix exception in destructor 2022-02-01 14:07:58 +03:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3"" 2022-02-01 13:36:51 +03:00
Alexander Tokmakov
2e4ae37d98 Merge branch 'master' into mvcc_prototype 2022-02-01 13:20:03 +03:00
alexey-milovidov
06477c2a7e Update ReplicatedMergeTreeSink.cpp 2022-02-01 09:22:40 +01:00
Igor Nikonov
f4c0b64420 Clean up: insert_deduplication_token setting for INSERT statement 
+ reduce number of allocations on replication merge tree path
+ bash test: move insert block settings into variable

Issue: ClickHouse#7461
 2022-02-01 09:22:33 +01:00
alexey-milovidov
99392b5ca7
Merge pull request #13544 from amosbird/mdha 
Multi-Disk auto-recovery.
 2022-02-01 06:13:26 +03:00
Alexey Milovidov
7dbf0dede5 Change severity of the "Cancelled merging parts" message in logs 2022-02-01 05:55:07 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3" 2022-02-01 05:49:40 +03:00
mergify[bot]
e229487817
Merge branch 'master' into mdha 2022-02-01 01:22:16 +00:00
Dmitry Novik
3a0ed0f8f5
Merge pull request #33324 from azat/buffer-different-structure 
Simplify different block structure (i.e. after ALTER) support for Buffer
 2022-02-01 02:19:49 +03:00
alexey-milovidov
15e4fe5c78
Revert "Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue" 2022-02-01 01:51:39 +03:00
Alexander Tokmakov
5fad3fdffc throw exception on non-transactional queries 2022-02-01 01:27:55 +03:00
Amos Bird
ec7d367814
DiskLocal checker 
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
 2022-02-01 05:55:27 +08:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines 
Fix benign race condition for storage HDFS, S3, URL
 2022-01-31 22:41:54 +03:00
alesapin
75d73d2785
Merge pull request #34139 from ClickHouse/fix_buf_s3_low_cardinality 
Fix bug with bounded S3 reads and LowCardinality
 2022-01-31 22:41:14 +03:00
Nikolai Kochetov
348d72266a
Merge pull request #34189 from ClickHouse/less-logging-for-remote_fs_execute_merges_on_single_replica_time_threshold 
Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue
 2022-01-31 21:39:04 +03:00
Nikolai Kochetov
a207cdf28f Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue. 2022-01-31 17:53:28 +00:00
Yatsishin Ilya
676fc5b20f Initial commit 2022-01-31 17:10:59 +00:00
Nikolai Kochetov
321fa4a9e8
Merge pull request #33291 from ClickHouse/add-pool-to-s3-write-buffer 
Add pool to WriteBufferFromS3
 2022-01-31 19:37:40 +03:00
Maksim Kita
8513f20cfd
Merge pull request #34145 from kitaisreal/bitset-sort-performance-check 
pdqsort performance check
 2022-01-31 12:35:13 +01:00
alesapin
5230c59f05 Smaller scope 2022-01-31 14:03:06 +03:00
alesapin
1adc7538fe Fix benign race condition for storage HDFS, S3, URL 2022-01-31 13:50:28 +03:00
tavplubix
d19e24f530
Merge pull request #34096 from ClickHouse/fix_race_merge_selecting_task 
Fix race between mergeSelectingTask and queue reinitialization
 2022-01-31 12:16:29 +03:00
Kruglov Pavel
a9d0beb7ae
Fix data race in StorageFile (#34113) 
* Fix data race in StorageFile

* Update StorageFile.h

* Fix
 2022-01-31 11:58:40 +03:00
alesapin
55c7936257 Fix incorrect range for index 2022-01-31 11:11:32 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
alesapin
4f1b902342 Fix compact parts as well 2022-01-30 22:36:19 +03:00
alesapin
4bedcc19b5 Better invariants 2022-01-30 20:40:09 +03:00
alesapin
c237c03c50 Fix 2022-01-30 18:39:26 +03:00
alesapin
bf918892ac More clear code with less getMark calls 2022-01-30 18:21:05 +03:00
alesapin
cb45a348f1 Merge branch 'master' into fix_buf_s3_low_cardinality 2022-01-30 17:30:55 +03:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
alesapin
3f3e90c8ba Remove redundant code 2022-01-29 23:55:45 +03:00
alexey-milovidov
2b43bad923
Update MergeTreeReaderStream.cpp 2022-01-29 19:28:39 +03:00
alesapin
7ada8227cf Fix bug with bounded S3 reads and LowCardinality 2022-01-29 18:28:40 +03:00
李扬
6d50d36405
Merge branch 'master' into rocksdb_metacache 2022-01-28 22:00:31 -06:00
Anton Popov
96a506c6fa fix inserts to distributed tables in case of change of native protocol 2022-01-29 03:23:25 +03:00
alexey-milovidov
6535b75322
Merge pull request #34001 from azat/memory-tracker-fix 
Fix memory accounting for queries that uses < max_untracker_memory
 2022-01-29 00:59:53 +03:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Alexander Tokmakov
e0304c2a58 review fixes, write tid into mutation entry 2022-01-28 20:47:37 +03:00
Azat Khuzhin
1519985c98 Fix possible "Can't attach query to the thread, it is already attached" 
After detachQueryIfNotDetached() had been removed it is not enough to
use attachTo() for ThreadPool (scheduleOrThrowOnError()) since the query
may be already attached, if the thread doing multiple jobs, so
CurrentThread::attachToIfDetached() should be used instead.

This should fix all the places from the failures on CI [1]:

    $ fgrep DB::CurrentThread::attachTo -A1 ~/Downloads/47.txt  | fgrep -v attachTo | cut -d' ' -f5,6 | sort | uniq -c
         92 --
          2 /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/deque:1393: DB::ParallelParsingInputFormat::parserThreadFunction(std::__1::shared_ptr<DB::ThreadGroupStatus>,
          4 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeData.cpp:1595: void
         87 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:993: void

  [1]: https://github.com/ClickHouse/ClickHouse/runs/4954466034?check_suite_focus=true

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Azat Khuzhin
b0c862c297 Fix memory accounting for queries that uses < max_untracker_memory 
MemoryTracker starts accounting memory directly only after per-thread
allocation exceeded max_untracker_memory (or memory_profiler_step).

But even memory under this limit should be accounted too, and there is
code to do this in ThreadStatus dtor, however due to
PullingAsyncPipelineExecutor detached the query from thread group that
memory was not accounted.

So remove CurrentThread::detachQueryIfNotDetached() from threads that
uses ThreadFromGlobalPool since it has ThreadStatus, and the query will
be detached using CurrentThread::defaultThreadDeleter.

Note, that before this patch memory accounting works for HTTP queries
due to it had been accounted from ParallelFormattingOutputFormat, but
not for TCP.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Nikolai Kochetov
1c9f026178 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-28 16:01:42 +03:00
Alexander Tokmakov
b3ddc601a5 fix race between mergeSelectingTask and queue reinitialization 2022-01-28 15:50:58 +03:00
liyang830
a69711ccff ExpressionList parse tuple and ttl 2022-01-28 19:07:59 +08:00
alexey-milovidov
f6684dbc62
Merge pull request #32304 from devcrafter/deduplication_token_7461 
insert_deduplication_token setting for INSERT statement
 2022-01-28 13:03:55 +03:00
Kruglov Pavel
efa8775ea6
Merge pull request #33960 from Avogar/autodetect-format 
Detect format and schema from stdin in clickhouse-local
 2022-01-28 12:02:15 +03:00
Azat Khuzhin
42ddf4d2b8 Fix negative StorageBufferBytes by proper accounting of empty blocks 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 11:10:00 +03:00
Azat Khuzhin
d77790182d Tiny cleanup in StorageBuffer::flushBuffer() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 11:10:00 +03:00
Azat Khuzhin
9948525816 Simplify different block sturcture (i.e. after ALTER) support for Buffer 
v2: fix empty block in case of flush

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 11:09:56 +03:00
Nikolai Kochetov
94999e86c2
Merge pull request #33392 from azat/_shard_num 
Replace old _shard_num implementation with shardNum() function
 2022-01-28 11:02:19 +03:00
taiyang-li
3de8bde7ce Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-28 09:58:52 +08:00
alexey-milovidov
63d8d75001
Merge pull request #33957 from azat/fix-getauxval 
Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work)
 2022-01-27 23:05:23 +03:00
mergify[bot]
ad146a67ff
Merge branch 'master' into _shard_num 2022-01-27 13:11:42 +00:00
liyang830
c2bdeb349c Merge branch 'fix-substr-zk-metadata' of github.com:sunny19930321/ClickHouse into fix-substr-zk-metadata 2022-01-27 16:35:02 +08:00
liyang830
eca0453564 fix local metadata differ zk metadata 2022-01-27 16:33:40 +08:00
Nikolai Kochetov
9b2998c639 Review fixes. 2022-01-26 18:08:01 +00:00
Nikolai Kochetov
a8171269a1 Review fixes. 2022-01-26 17:55:24 +00:00
Raúl Marín
5a59d976dd CurrentlyExecuting: Require mutex usage explicitly 2022-01-26 18:44:35 +01:00
Azat Khuzhin
08f4f45fd9 Fix Context leak (recursive context reference) in StorageSQLite 
CI founds after LSan had been fixed [1]:

    01889_sqlite_read_write:                                                [ FAIL ] 8.32 sec. - return code: 1

    =================================================================
    ==20649==ERROR: LeakSanitizer: detected memory leaks

    Indirect leak of 1968 byte(s) in 1 object(s) allocated from:
        0 0xc5c1ffd in operator new(unsigned long) (/usr/bin/clickhouse+0xc5c1ffd)
        1 0x25e32d0d in std::__1::__unique_if<DB::StorageInMemoryMetadata>::__unique_single std::__1::make_unique<DB::StorageInMemoryMetadata, DB::StorageInMemoryMetadata const&>(DB::StorageInMemoryMetadata c>
        2 0x25e32d0d in DB::IStorage::setInMemoryMetadata(DB::StorageInMemoryMetadata const&) obj-x86_64-linux-gnu/../src/Storages/IStorage.h:194:22
        3 0x29bdee98 in DB::StorageSQLite::StorageSQLite(DB::StorageID const&, std::__1::shared_ptr<sqlite3>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std:>
        4 0x25ee61d6 in std::__1::shared_ptr<DB::StorageSQLite> shared_ptr_helper<DB::StorageSQLite>::create<DB::StorageID, std::__1::shared_ptr<sqlite3> const&, std::__1::basic_string<char, std::__1::char_tr>
        5 0x25ee61d6 in DB::TableFunctionSQLite::executeImpl(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1:>

    SUMMARY: AddressSanitizer: 171256 byte(s) leaked in 130 allocation(s).

  [1]: https://github.com/ClickHouse/ClickHouse/runs/4929706698?check_suite_focus=true

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-26 12:56:27 +03:00
Nikolai Kochetov
fcc29dbd15 Try to fix integration tests. 2022-01-25 15:26:36 +00:00
taiyang-li
5398af7c05 update again 2022-01-25 18:59:29 +08:00
Kruglov Pavel
7873b4475f
Merge branch 'master' into autodetect-format 2022-01-25 10:56:52 +03:00
Kruglov Pavel
9f12f4af13
Merge pull request #33302 from Avogar/formats-with-suffixes 
Allow to create new files on insert for File/S3/HDFS engines
 2022-01-25 10:56:15 +03:00
avogar
a6740d2f9a Detect format and schema for stdin in clickhouse-local 2022-01-25 10:25:37 +03:00
Nikolai Kochetov
52b65c3454 Fix style 2022-01-24 16:21:27 +00:00
Nikolai Kochetov
506ee8c024 Refactor some code. 2022-01-24 15:55:29 +00:00
Nikolai Kochetov
4afa75e80b Fix deduplication. 2022-01-24 14:43:36 +00:00
zhongyuankai
a6254516e0 Fix Alter ttl modification unsupported table engine 2022-01-24 21:48:52 +08:00
Nikolai Kochetov
2882e46654 Remove debug logging. 2022-01-24 11:48:00 +00:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806 
Fix logical error on RESTORE REPLICA
 2022-01-24 13:28:37 +03:00
tavplubix
766786b570
Update StorageReplicatedMergeTree.h 2022-01-24 12:16:13 +03:00
tavplubix
34baca1298
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
 2022-01-24 12:12:08 +03:00
tavplubix
2f566d2af9
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
 2022-01-24 12:12:03 +03:00
taiyang-li
dfd199b047 fix integration test failed 2022-01-24 11:51:11 +08:00
taiyang-li
73def8b483 merge master and solve conflict 2022-01-24 11:01:43 +08:00
Nikolai Kochetov
f74cf1e152 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:55:42 +03:00
Nikolai Kochetov
a024ddd1d4 Merge branch 'add-pool-to-s3-write-buffer' of github.com:ClickHouse/ClickHouse into add-pool-to-s3-write-buffer 2022-01-21 18:42:24 +00:00
Nikolai Kochetov
b3cbb63487 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:41:54 +03:00
alexey-milovidov
43f2e2c178
Merge pull request #33849 from CurtizJ/fix-sparse-columns 
Fix sparse columns
 2022-01-21 21:13:02 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Alexander Tokmakov
e3b1397001 Merge branch 'master' into mvcc_prototype 2022-01-21 18:59:01 +03:00
Nikolai Kochetov
ae1b669fca Even more async writes. 2022-01-21 15:55:19 +00:00
Kruglov Pavel
daba57a933
Merge pull request #33737 from Avogar/bool-type 
Some improvements and fixes for Bool data type
 2022-01-21 15:24:58 +03:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
taiyang-li
22dc786dd8 optimize log level 2022-01-21 12:16:49 +08:00
Anton Popov
c2206c0b96 better storing serialization infos 2022-01-21 03:29:30 +03:00
Anton Popov
f78fb247a7 fix column ttl 2022-01-21 02:22:16 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts 
Fix flushing of in-memory parts
 2022-01-21 02:18:03 +03:00
Anton Popov
558eb7ea73 fix sparse columns 2022-01-21 00:06:32 +03:00
Maksim Kita
56a54af27b
Merge pull request #33833 from amosbird/keyconditionfix 
Fix KeyCondition with no common types
 2022-01-20 20:31:27 +01:00
李扬
9f6f84259d
Fix bug of check table when creating data part with wide format and projection (#33774) 2022-01-20 18:05:17 +03:00
Alexander Tokmakov
0ea0a8ccc7 Merge branch 'master' into mvcc_prototype 2022-01-20 17:05:55 +03:00
Amos Bird
e62f4e50a6
Fix KeyCondition with no common types 2022-01-20 21:56:12 +08:00
Kruglov Pavel
7bfb1231b9
Merge branch 'master' into formats-with-suffixes 2022-01-20 14:47:17 +03:00
avogar
ba54a5ff80 Some improvements and fixes for Bool data type 2022-01-20 14:16:11 +03:00
alexey-milovidov
28a9d569cc
Merge pull request #33690 from azat/remove-unbundled-support 
Remove unbundled support
 2022-01-20 12:29:07 +03:00
Kseniia Sumarokova
064c3dd654
Merge pull request #33724 from liuneng1994/fix_ignore_hdfs_error 
Throw exception when storage hdfs list directory failed
 2022-01-20 11:59:09 +03:00
Azat Khuzhin
b23053d1f6 Remove USE_XXHASH (reduntant) 2022-01-20 10:02:02 +03:00
Azat Khuzhin
e0e81b340d Fix w/o ODBC build 2022-01-20 10:02:02 +03:00
Azat Khuzhin
9426e6305b Fix USE_XXHASH in StorageSystemBuildOptions.generated.cpp.in 2022-01-20 10:02:02 +03:00
Azat Khuzhin
e160a20769 Remove USE_OPENCL (removed in #31744) 2022-01-20 10:02:02 +03:00
Azat Khuzhin
6dcb09f9e1 Cleanup llvm 2022-01-20 10:02:01 +03:00
Azat Khuzhin
614f86edbb Remove unbundled gsasl support 2022-01-20 10:02:00 +03:00
Azat Khuzhin
a773e7ff01 Remove unbundled libpqxx support 2022-01-20 10:01:59 +03:00
Azat Khuzhin
789dfd9f3b Remove unbundled re2 support 
v2: preserve re2_st name to make PVS check pass (since docker image
update fails)
 2022-01-20 10:00:49 +03:00
liuneng1994
59f3fe515b change null check position 
Signed-off-by: neng.liu <neng.liu@kyligence.io>
 2022-01-20 06:26:34 +00:00
Kseniia Sumarokova
19df6413dc
Merge pull request #33729 from Vxider/windowview-stream-table-join 
Enable stream to table join in WindowView
 2022-01-20 09:14:53 +03:00
Azat Khuzhin
aef6668cb4 Remove unbundled hdfs3 support 2022-01-20 08:47:16 +03:00
Azat Khuzhin
8485abd52b Remove unbundled lz4 support 2022-01-20 08:47:16 +03:00
alexey-milovidov
994584df09
Merge pull request #33753 from azat/mt-fix-count-race 
Fix tiny race between count() and INSERT/merges/... in MergeTree
 2022-01-20 06:33:54 +03:00
Alexander Tokmakov
e9a5a64a71 Merge branch 'master' into mvcc_prototype 2022-01-19 21:41:23 +03:00
Alexander Tokmakov
7458012103 some fixes, add test with restart 2022-01-19 21:29:31 +03:00
mergify[bot]
cbb87e762c
Merge branch 'master' into formats-with-suffixes 2022-01-19 17:21:18 +00:00
Amos Bird
62441f0a0f
Fix mutation when table contains projections (#33679) 2022-01-19 15:27:11 +03:00
taiyang-li
d73d4e831e fix bug of check table when create data part with wide format and projection 2022-01-19 17:13:41 +08:00
tavplubix
cdcd93330a
Merge pull request #33743 from ClickHouse/better_exception_message 
Better exception text on suspicious broken parts
 2022-01-19 11:25:59 +03:00
mergify[bot]
bfbcb69e20
Merge branch 'master' into windowview-stream-table-join 2022-01-19 07:46:00 +00:00
liuneng1994
4d3bb1584d fix Dereference of null pointer error 
Signed-off-by: neng.liu <neng.liu@kyligence.io>
 2022-01-19 07:41:46 +00:00
Nikolay Degterinsky
eaad8eef7f
Merge pull request #33731 from Vxider/remove_unused_var 
Remove unused variable
 2022-01-19 10:35:18 +03:00
Vxider
13cee6c184 update code style 2022-01-19 02:24:27 +00:00
mergify[bot]
f2da296990
Merge branch 'master' into windowview-stream-table-join 2022-01-19 02:08:52 +00:00
alexey-milovidov
d222cb9771
Merge pull request #33695 from amosbird/buildimprove 
Some build improvement
 2022-01-19 03:19:22 +03:00
Azat Khuzhin
c9e93b21c2 Fix tiny race between count() and INSERT/merges/... in MergeTree 
Before it was possible to return stale counter from
StorageMergeTree::totalRows() (that is used for
optimize_trivial_count_query) since the accounting is done in two steps:
- subtract old number of rows
  <-- here the number can be zero, even though there are rows -->
- add new number of rows

This was found by CI [1] in 01615_random_one_shard_insertion test:

Here you can see that INSERT went to both tables:

<details>

    2022.01.16 09:07:34.288252 [ 154369 ] {a1905be0-93da-460c-8c6f-9b5adace72a0} <Debug> DistributedBlockOutputStream: It took 0.035197041 sec. to insert 100
    blocks, 2841.1479249065287 rows per second. Insertion status:
    Wrote 54 blocks and 54 rows on shard 0 replica 0, localhost:9000 (average 0 ms per block, the slowest block 1 ms)
    Wrote 46 blocks and 46 rows on shard 1 replica 0, localhost:9000 (average 0 ms per block, the slowest block 1 ms)

</details>

But the test fails, since select from shard1.tbl returns 0, and the
problem was concurrent merge:

<details>

    2022.01.16 09:07:34.289470 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> executeQuery: (from [::1]:42082) (comment: 01615_random_one_shard_insertion.sql) select count() != 0 from shard_0.tbl;
    2022.01.16 09:07:34.289564 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Debug> MergeTask::MergeProjectionsStage: Merge sorted 54 rows, containing 1 columns (1 merged, 0 gathered) in 0.00171193 sec., 31543.345814373253 rows/sec., 246.43 KiB>
    2022.01.16 09:07:34.289810 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Trace> shard_0.tbl (c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab): Renaming temporary part tmp_merge_all_1_54_9 to all_1_54_9.
    2022.01.16 09:07:34.289858 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> ContextAccess (default): Access granted: SELECT(number) ON shard_0.tbl
    2022.01.16 09:07:34.289897 [ 375 ] {c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab::all_1_54_9} <Trace> shard_0.tbl (c7a885fa-4ef4-4dcf-a4de-1650d44fa0ab) (MergerMutator): Merged 6 parts: from all_1_49_8 to all_54_54_0
    2022.01.16 09:07:34.289920 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> InterpreterSelectQuery: WithMergeableState -> Complete
    2022.01.16 09:07:34.289987 [ 375 ] {} <Debug> MemoryTracker: Peak memory usage Mutate/Merge: 3.12 MiB.
    2022.01.16 09:07:34.290305 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> MergingAggregatedTransform: Reading blocks of partially aggregated data.
    2022.01.16 09:07:34.290332 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> MergingAggregatedTransform: Read 1 blocks of partially aggregated data, total 1 rows.
    2022.01.16 09:07:34.290343 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Merging partially aggregated single-level data.
    2022.01.16 09:07:34.290358 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Merged partially aggregated single-level data.
    2022.01.16 09:07:34.290366 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Trace> Aggregator: Converting aggregated data to blocks
    2022.01.16 09:07:34.290391 [ 154344 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> Aggregator: Converted aggregated data to blocks. 1 rows, 8.00 B in 1.0939e-05 sec. (91416.034 rows/sec., 714.19 KiB/sec.)
    2022.01.16 09:07:34.290709 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Information> executeQuery: Read 1 rows, 4.01 KiB in 0.001187722 sec., 841 rows/sec., 3.30 MiB/sec.
    2022.01.16 09:07:34.290774 [ 146495 ] {cd9d4cf2-7131-4179-b0b2-3aeec4045755} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/33675/7848ea7d609e4c720e8e4494eb6207c0751f5aea/stateless_tests__ubsan__actions_.html

This also fixes a race between DROP TABLE check and INSERT/merges.

v0: use Active parts instead.
v2: fix total counters accounting instead.
 2022-01-18 23:58:02 +03:00
avogar
4efadfad3c Fix tests 2022-01-18 22:26:13 +03:00
alesapin
127365658d Followup 2022-01-18 22:00:52 +03:00
alesapin
e7ee116358
Update src/Storages/MergeTree/MergeTreeData.cpp 
Co-authored-by: tavplubix <avtokmakov@yandex-team.ru>
 2022-01-18 22:00:23 +03:00
alesapin
24f4a5040d Better exception message 2022-01-18 21:59:54 +03:00
alesapin
358e3a165c Better exception text on suspicious broken parts 2022-01-18 19:28:58 +03:00
alesapin
e5ec1aa446
Merge pull request #33642 from ClickHouse/add_integration_test_for_zero_duplication 
Fix bug with background moves in zero copy replication
 2022-01-18 18:32:16 +03:00
liuneng1994
5e1e512bf2 fix check style 
Signed-off-by: neng.liu <neng.liu@kyligence.io>
 2022-01-18 14:42:28 +00:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl 
RFC: Split headers, move SystemLog into module, more forward declarations
 2022-01-18 17:22:49 +03:00
liuneng1994
15ea37c4ae ignore file not found exception 
Signed-off-by: neng.liu <neng.liu@kyligence.io>
 2022-01-18 12:49:40 +00:00
Vxider
fb62960836 remove unused variable 2022-01-18 09:34:26 +00:00
Vxider
ba0a702807 enable stream to table join in windowview 2022-01-18 09:20:15 +00:00
Vxider
f526ff2a51 remove unused variable 2022-01-18 09:03:16 +00:00
alesapin
eff85d3352 Review fixes 2022-01-18 11:27:01 +03:00
mergify[bot]
df869c5b9c
Merge branch 'master' into fix_ignore_hdfs_error 2022-01-18 07:05:24 +00:00
liuneng1994
c6e205b2ad add error check 2022-01-18 06:21:40 +00:00
Amos Bird
6d62060e16
Build improvement 2022-01-17 22:36:27 +08:00
taiyang-li
e64ba110ef fix path error 2022-01-17 19:55:44 +08:00
alesapin
31fb93330e Fix comment 2022-01-17 14:54:51 +03:00
alesapin
4caef03e6a Add ephemeral lock in zookeeper 2022-01-17 14:52:51 +03:00
alesapin
ccf990e887 Merge branch 'master' into add_integration_test_for_zero_duplication 2022-01-17 13:33:39 +03:00
taiyang-li
cfb18e7ea0 remove useless header 2022-01-17 10:05:36 +08:00
Nikolai Kochetov
6d49a62666 Some more async writes. 2022-01-14 19:53:55 +00:00
avogar
af54ca4866 Fix partition write 2022-01-14 21:21:36 +03:00
Alexander Tokmakov
c1c3f25c10 Merge branch 'master' into mvcc_prototype 2022-01-14 21:03:58 +03:00
Kruglov Pavel
a7df9cd53a
Merge branch 'master' into formats-with-suffixes 2022-01-14 21:03:49 +03:00
avogar
253035a5df Fix 2022-01-14 19:17:06 +03:00
alesapin
bf803472b1 Add a tests for a TTL bug in zero copy replication 2022-01-14 18:44:10 +03:00
Kruglov Pavel
d2e9f37bee
Merge branch 'master' into format-by-extention 2022-01-14 18:36:23 +03:00
avogar
89a181bd19 Make better 2022-01-14 18:16:18 +03:00
Alexander Tokmakov
4dd216ce8f Merge branch 'master' into mvcc_prototype 2022-01-14 17:11:32 +03:00
Alexander Tokmakov
e1a41fc694 add system log for introspection 2022-01-14 17:03:00 +03:00
Kruglov Pavel
5a908e8edd
Merge branch 'master' into formats-with-suffixes 2022-01-14 16:45:20 +03:00
avogar
8884c4a3fe Fix tests and build 2022-01-14 16:27:57 +03:00
Azat Khuzhin
c341b3b237 Add current database to table names in JOIN section for distributed queries 
This should fix JOIN w/o explicit database.

v2: rewrite only JOIN section, since there is old behavior that relies
on default_database for IN section, see [1]:
- 01487_distributed_in_not_default_db
- 01152_cross_replication

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/33611/d0ea3c76fa51131171b1825939680867eb1c04da/fast_test__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 11:23:38 +03:00
Azat Khuzhin
0a9b1ee803 Remove RestoreQualifiedNamesMatcher::Data::rename (always true) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 11:18:52 +03:00
avogar
2d7b1bfa5e Detect format in S3/HDFS/URL table engines 2022-01-13 16:14:18 +03:00
taiyang-li
964f92f2c1 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-13 18:29:38 +08:00
taiyang-li
82a3024712 Merge remote-tracking branch 'ck/master' into bigo_hive_table 2022-01-13 18:11:23 +08:00
taiyang-li
6567cd7abc fix all the stateless test 2022-01-13 15:27:41 +08:00
alexey-milovidov
6c71a7c40f
Merge pull request #33518 from hexiaoting/dev-limit-offset 
Fix bug for select from VIEW with format and limit settings
 2022-01-12 23:39:31 +03:00
Nikolai Kochetov
843983ea06 Add batch removing to IDiskl. 2022-01-12 15:58:04 +00:00
avogar
8390e9ad60 Detect format by file name in file/hdfs/s3/url table functions 2022-01-12 18:29:31 +03:00
taiyang-li
66813a3aa9 merge master 2022-01-12 16:56:29 +08:00
Nikolai Kochetov
28f2012b06 Add some more async writing. 2022-01-11 19:02:48 +00:00
hexiaoting
5c847264f7 Fix bug for select from VIEW with format and limit settings 2022-01-11 20:19:41 +08:00
Kseniia Sumarokova
6587cd0ec6
Merge pull request #33231 from kssenii/settings-changes-with-named-conf 
Pass settings as key value or config for storage with settings
 2022-01-11 12:30:41 +03:00
Kruglov Pavel
94cb98df28
Merge pull request #33177 from kssenii/materialized-postgresql-better-startup 
materialized postgresql better startup
 2022-01-11 12:18:29 +03:00
taiyang-li
6751c8552a merge master and resolve conflict 2022-01-11 15:21:56 +08:00
Maksim Kita
8465de9a4e
Merge pull request #33482 from zhongyuankai/materialized_view_support_virtual_column 
Materialized view support virtual column
 2022-01-10 23:13:31 +03:00
Mikhail f. Shiryaev
1cf46f378f
Merge pull request #33494 from msaf1980/rollup_rules_type 
GraphiteMergeTree Rollup rules type
 2022-01-10 20:53:20 +01:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog 
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
 2022-01-10 22:01:41 +03:00
Azat Khuzhin
1637c41d42 Remove leftovers of old _shard_num via identifier implementation 2022-01-10 21:21:24 +03:00
Azat Khuzhin
6671e86c89 Export IStorage::isVirtualColumn() 2022-01-10 21:21:24 +03:00
Alexander Tokmakov
158fbaaa29 fix 2022-01-10 20:07:01 +03:00
Alexander Tokmakov
bf2b6c2c37 Merge branch 'master' into mvcc_prototype 2022-01-10 19:53:50 +03:00
mergify[bot]
5857667453
Merge branch 'master' into materialized_view_support_virtual_column 2022-01-10 16:25:46 +00:00
Anton Popov
46b9279d81
Merge pull request #33300 from CurtizJ/support-prewhere-storage-merge 
Support `optimize_move_to_prewhere` with storage `Merge`
 2022-01-10 15:15:15 +03:00
taiyang-li
7e227040cb optimize getMergeTreeMetadataCache 2022-01-10 19:52:54 +08:00
msaf1980
780a1b2abe graphite: split tagged/plain rollup rules (for merges perfomance) 2022-01-10 16:34:16 +05:00
taiyang-li
2cae8d552c remove useless code 2022-01-10 19:17:12 +08:00
kssenii
21c34ad59b Add support for dictionary source 2022-01-10 14:00:03 +03:00
zhongyuankai
61ea0837d4 materialized view support virtual column 2022-01-10 15:53:41 +08:00
Kseniia Sumarokova
0a1300d32a
Merge pull request #33340 from Vxider/fix-windowview-inner-table-parser 
Fix inner table parser in window view
 2022-01-10 09:45:17 +03:00
taiyang-li
7c7f45bce2 fix comment 2022-01-10 12:32:38 +08:00
taiyang-li
3d409a4526 add metrics MergeTreeMetadataCacheSize 2022-01-10 12:20:39 +08:00
taiyang-li
5bff268f84 optimize code as advices 2022-01-10 11:45:06 +08:00
taiyang-li
1b4b727d79 fix conflict 2022-01-10 10:43:13 +08:00
Vxider
ab593d24f3 update code style 2022-01-10 02:16:07 +00:00
kssenii
1984a80750 Merge remote-tracking branch 'upstream/master' into materialized-postgresql-better-startup 2022-01-09 21:23:49 +03:00
Igor Nikonov
102b9cdcfb Merge remote-tracking branch 'origin/master' into deduplication_token_7461 2022-01-09 19:15:33 +01:00
Kseniia Sumarokova
d0a847befc
Merge pull request #33200 from kssenii/fix-materialized-postgresql-ddl-validation 
materialized postgresql fix ddl validation
 2022-01-09 21:12:13 +03:00
Igor Nikonov
0857a8d061 insert_deduplication_token support in non-replicated MergeTree 2022-01-09 19:06:09 +01:00
kssenii
bfaab1356f Merge remote-tracking branch 'upstream/master' into settings-changes-with-named-conf 2022-01-09 17:56:44 +03:00
kssenii
8f9920dd4d One more fix 2022-01-09 17:46:33 +03:00
kssenii
6639a93735 Better 2022-01-09 12:16:12 +03:00
kssenii
61f85e9afc More fixes 2022-01-08 20:44:17 +03:00
Kseniia Sumarokova
9b3493ba9a
Merge pull request #33468 from kssenii/fix-postgres-db-drop 
Fix drop materialized postgres db
 2022-01-08 19:41:24 +03:00
alesapin
733ed7cb54
Merge pull request #32061 from ianton-ru/MDB-15474 
Change ZooKeeper path for zero-copy locks for shared data
 2022-01-08 17:17:11 +03:00
kssenii
c94f4163f7 Merge remote-tracking branch 'upstream/master' into fix-materialized-postgresql-ddl-validation 2022-01-08 15:56:46 +03:00
kssenii
adaeb96b6d Fix 2022-01-08 15:46:22 +03:00
kssenii
64538cf20f Merge remote-tracking branch 'upstream/master' into materialized-postgresql-better-startup 2022-01-08 00:49:08 +03:00
kssenii
90972e3752 Fix race 2022-01-08 00:27:29 +03:00