thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
93,305 Commits 9,338 Branches 1,783 Tags 2.1 GiB
7f4a1b8bb8
Commit Graph

3870 Commits

Author SHA1 Message Date
Alexander Tokmakov
b0f8970d07
Merge pull request #38961 from Algunenano/reduce_zk_eexists 
Stop reporting Zookeeper "Node exists" exceptions in system.errors when they are expected
 2022-07-07 23:16:46 +03:00
Alexander Tokmakov
1d93dc1c9a
Merge pull request #38963 from ClickHouse/fix_38962 
Avoid weird exception in Keeper
 2022-07-07 22:01:35 +03:00
Robert Schulze
f15d9ca59c
Merge pull request #38774 from zvonand/zvonand-nnd 
Reintroduce nonNegativeDerivative()
 2022-07-07 20:39:13 +02:00
Alexander Tokmakov
7630fc76ae avoid weird exception 2022-07-07 17:19:05 +02:00
Raúl Marín
c3935000a8 Avoid reporting ZK EEXIST errors on checkPartChecksumsAndCommit 2022-07-07 16:10:05 +02:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
Vladimir Galunshchikov
cacc833da6 Rename NUMBER_OF_DIMENSIONS_MISMATHED const to NUMBER_OF_DIMENSIONS_MISMATCHED 2022-07-07 11:15:59 +03:00
Alexey Milovidov
02adad3f27
Merge pull request #37860 from amosbird/remove-duplicated-log 
Remove duplicate peak mem log
 2022-07-07 08:16:06 +03:00
Andrey Zvonov
7de39d9b15 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-nnd 2022-07-06 10:59:35 +03:00
HeenaBansal2009
d89ba2e5d9 Review Comments 2022-07-05 21:18:39 -07:00
Azat Khuzhin
98cd92017e Fix waiting of shared lock after exclusive lock failure 
When WRITE lock attemp fails (exclusive lock for ALTER/DELETE), and
there are multiple READ locks (shared lock for SELECT/INSERT), i.e. one
INSERT is in progress and one SELECT is queued after ALTER/DELETE
started but before it fails, this SELECT will wait until INSERT will
finishes.

This happens because in case of WRITE lock failure it does not notify
the next READ lock that can be acquired.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-05 19:12:47 +03:00
Kseniia Sumarokova
3e803006a2
Merge pull request #37171 from tchepavel/nats-integration 
Nats Integration
 2022-07-05 17:47:02 +02:00
alesapin
bd8a3ee841
Merge pull request #38338 from ClickHouse/zookeeper-add-extra-list-argument 
Extend ZooKeeper list request with support for filtering persistent or ephemeral nodes only
 2022-07-05 12:53:51 +02:00
Kseniia Sumarokova
6c3d648f49
Merge branch 'master' into nats-integration 2022-07-04 21:06:07 +02:00
zvonand
8a270c01e9 fix floating point in intervals 2022-07-04 20:45:05 +03:00
Kruglov Pavel
3d47416f8b
Merge pull request #38559 from ClickHouse/Avogar-patch-1 
Add logging in Epoll and TimerDescriptor in case of EINTR
 2022-07-04 15:05:08 +02:00
Alexey Milovidov
7adc87a37d
Merge pull request #38413 from nickitat/cache_values_inside_hash_table_grower 
Precalculate values inside `HashTableGrower`
 2022-07-02 23:30:24 +03:00
Kruglov Pavel
eaf8397c13
Fix style 2022-07-01 19:51:07 +02:00
Kruglov Pavel
fe0dad5429
Update src/Common/TimerDescriptor.cpp 2022-07-01 16:23:20 +02:00
Kruglov Pavel
4550acdc35
Update src/Common/Epoll.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-07-01 16:23:03 +02:00
Robert Schulze
2a1ede0f5a
Merge pull request #38589 from ClickHouse/fix-zero-bytes-in-haystack 
Fix countSubstrings() & position() on patterns with 0-bytes
 2022-07-01 16:15:43 +02:00
zvonand
3b5332d15e Revert "Revert "Non Negative Derivative window function"" 
This reverts commit dea3b5bfce.
 2022-07-01 18:59:07 +05:00
Vitaly Baranov
cadb496894
Merge pull request #38537 from vitlibar/backup-improvements-8 
Backup Improvements 8
 2022-07-01 11:20:46 +02:00
Alexey Milovidov
20841f0e1e
Merge pull request #38551 from ClickHouse/revert-37628-non-neg-deriv 
Revert "Non Negative Derivative window function"
 2022-07-01 02:46:28 +03:00
Robert Schulze
1e3b5bfcb7
Fix test 00233_position_function_family 2022-06-30 11:43:25 +00:00
Vitaly Baranov
461a31f237 Improve gathering metadata for backup - part 2. 2022-06-30 08:37:17 +02:00
Robert Schulze
81bb2242fd
Fix countSubstrings() & position() on patterns with 0-bytes 
SQL functions countSubstrings(), countSubstringsCaseInsensitive(),
countSubstringsUTF8(), position(), positionCaseInsensitive(),
positionUTF8() with non-const pattern argument use fallback sorters
LibCASCIICaseSensitiveStringSearcher and LibCASCIICaseInsensitiveStringSearcher
which call ::strstr(), resp. ::strcasestr(). These functions assume that
the haystack is 0-terminated and they even document that. However, the
callers did not check if the haystack contains 0-byte (perhaps because
its sort of expensive). As a consequence, if the haystack contained a
zero byte in it's payload, matches behind this zero byte were ignored.

    create table t (id UInt32, pattern String) engine = MergeTree() order by id;
    insert into t values (1, 'x');
    select countSubstrings('aaaxxxaa\0xxx', pattern) from t;

We returned 3 before this commit, now we return 6
 2022-06-29 21:41:18 +00:00
Nikita Taranov
7d2e8b0883 clean up 2022-06-29 22:15:12 +02:00
Kruglov Pavel
77873e75bf
Update TimerDescriptor.cpp 2022-06-29 13:01:14 +02:00
Kruglov Pavel
30fd086b5e
Add logging in Epoll and TimerDescriptor in case of EINTR 2022-06-29 12:59:18 +02:00
Antonio Andelic
7a8fbbf132 Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-29 08:31:03 +00:00
Antonio Andelic
1d26446f84 Define new list request type 2022-06-29 08:30:39 +00:00
Robert Schulze
bb441faa42
Merge pull request #38491 from ClickHouse/clang-tidy-in-branches 
Fix some clang-tidy warnings in headers
 2022-06-29 10:22:04 +02:00
Alexey Milovidov
dea3b5bfce
Revert "Non Negative Derivative window function" 2022-06-29 08:56:15 +03:00
HeenaBansal2009
608bfb8453 Fixed {..} regex parsing as shell does 2022-06-28 12:49:31 -07:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to 
Replace where possible by std::lock_guard which is more light-weight.
 2022-06-28 19:19:06 +00:00
Robert Schulze
c22038d48b
More clang-tidy fixes 2022-06-28 11:50:05 +00:00
HeenaBansal2009
6a5819a405 Fixed bug in parsing{..} regex 2022-06-27 22:51:31 -07:00
Robert Schulze
5e28580f11
Fix some clang-tidy warnings in headers 2022-06-27 23:49:07 +02:00
Dmitry Novik
b629d557ce
Merge pull request #38246 from ClickHouse/overcommit-deadlock-fix 
Fix deadlock in OvercommitTracker logging
 2022-06-27 21:57:31 +02:00
Nikita Taranov
986ead7358 use old HashTableGrower with *WithStackMemory data structures 2022-06-27 20:34:18 +02:00
Robert Schulze
53cd8820d3
Merge pull request #38434 from ClickHouse/multi-match-with-nonconst-patterns 
Refactoring to enable multi-match functions with non-const needles
 2022-06-27 19:35:48 +02:00
Antonio Andelic
c06776ce0d Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-27 06:53:29 +00:00
Antonio Andelic
0d906e311d PR review fixes 2022-06-27 06:53:09 +00:00
Robert Schulze
bb7c627964
Cosmetics: Pass patterns around as std::string_view instead of StringRef 
- The patterns are not used in hashing, there should not be a performance
  impact when we use stuff from the standard library instead.

- added forgotten .reserve() in FunctionsMultiStringPosition.h
 2022-06-26 15:32:19 +00:00
mergify[bot]
f63c959679
Merge branch 'master' into cleanup_garbage_in_store_dir 2022-06-26 13:35:10 +00:00
Nikita Taranov
95d413d7ec put the thing into its own cache line 2022-06-26 01:38:45 +02:00
Alexey Milovidov
25cc406201
Update src/Common/ThreadStatus.cpp 
Co-authored-by: Dmitry Novik <n0vik@clickhouse.com>
 2022-06-26 01:29:06 +03:00
Nikita Taranov
abcefeeb41 cleanup 2022-06-25 23:15:59 +02:00
mergify[bot]
469f24a12c
Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-25 12:01:06 +00:00
Alexey Milovidov
e33f236d50 Slight improvement 2022-06-25 03:59:01 +02:00
Alexey Milovidov
b89f01438e Lower mutex scope 2022-06-25 03:05:55 +02:00
Nikita Taranov
6639d94125 stash 2022-06-25 01:32:21 +02:00
Nikita Taranov
c476427422 stash 2022-06-25 01:27:24 +02:00
Nikita Taranov
f6ef78c0f1 stash 2022-06-25 01:26:46 +02:00
mergify[bot]
92eff431be
Merge branch 'master' into overcommit-deadlock-fix 2022-06-24 13:54:11 +00:00
Alexander Tokmakov
cfe0e434e5
Revert "Add support for io_uring read method" 2022-06-24 12:19:41 +03:00
mergify[bot]
a0eb831cd1
Merge branch 'master' into overcommit-deadlock-fix 2022-06-23 23:02:59 +00:00
Alexey Milovidov
812ab9bd6b
Merge pull request #36103 from sauliusvl/uring 
Add support for io_uring read method
 2022-06-24 00:34:29 +03:00
Alexander Tokmakov
74f38710a8 Merge branch 'master' into cleanup_garbage_in_store_dir 2022-06-23 21:43:28 +02:00
kssenii
6fbd49f554 Merge master 2022-06-23 21:40:01 +02:00
kssenii
468c98ff42 Better 2022-06-23 17:46:27 +02:00
Antonio Andelic
ae3d9fd962 Expose list request type in internal client 2022-06-23 10:28:12 +00:00
Antonio Andelic
568c957eb2 Fix TestKeeper 2022-06-23 10:28:12 +00:00
Antonio Andelic
3a71b63b5d Add list request type 2022-06-23 10:28:12 +00:00
Antonio Andelic
a5ee918bd2
Merge pull request #38047 from ClickHouse/keeper-sequential-consistency-reads 
Add `SYNC` command to internal ZooKeeper client
 2022-06-23 08:43:24 +02:00
Dmitry Novik
3544fbe5c4 cleanup 2022-06-22 14:05:20 +00:00
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa 
Support for Clang Thread Safety Analysis (TSA)
 2022-06-21 20:19:33 +02:00
kssenii
481644939a Add describe cache command 2022-06-21 16:07:39 +02:00
Dmitry Novik
099055c183 Explicitly forbid allocations in OvercommitTracker 2022-06-21 12:26:13 +00:00
Alexander Tokmakov
ba0fcec993 add background cleanup of store/ subdirs 2022-06-21 12:35:47 +02:00
Saulius Valatka
160c95a6b8 unpoisson memory read by io_uring, handle interrupted syscalls, do not throw from monitor thread 2022-06-21 13:20:21 +03:00
Saulius Valatka
7b5e13a9eb implement pending request queue to prevent CQ overflows, more error checking, reduce size of io_uring 2022-06-21 13:17:56 +03:00
Dmitry Novik
e10f079bd3 Get rid of allocations in OvercommitTracker 2022-06-21 10:15:33 +00:00
Kseniia Sumarokova
71ee5dbbb2
Merge pull request #38227 from kssenii/diff-reduce 
buffer's getFileSize small changes
 2022-06-21 12:12:22 +02:00
mergify[bot]
bdd5b94192
Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-21 08:13:04 +00:00
Saulius Valatka
0dc7854af1 add initial io_uring support 2022-06-21 09:59:46 +03:00
lingpeng0314
c18dd55400 Resolve comments 
- refine metric description
 2022-06-21 10:30:19 +08:00
Dmitry Novik
9ca368ac20 Use do while(false) in macro 2022-06-20 23:10:40 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Dmitry Novik
53d656d89f Fix deadlock in OvercommitTracker logging 2022-06-20 17:35:24 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Antonio Andelic
0c76fc1121 Fix style 2022-06-20 14:21:59 +00:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
Antonio Andelic
777ae72259 Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-20 14:06:45 +00:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
Antonio Andelic
6c225bb04d
Update src/Common/ZooKeeper/ZooKeeperImpl.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-06-20 14:55:59 +02:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6 
Backup Improvements 6
 2022-06-20 14:13:30 +02:00
mergify[bot]
4ed606a275
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 10:17:51 +00:00
Kseniia Sumarokova
a756b4be27
Merge pull request #37391 from azat/insert-profile-events-fix 
Send profile events for INSERT queries (previously only SELECT was supported)
 2022-06-20 12:16:29 +02:00
lingpeng0314
f06b19bdbf Add keeper monitoring data 2022-06-20 11:32:18 +08:00
Vitaly Baranov
a0c558a17e Implement backup/restore for ACL system tables (system.users, system.roles, etc.) 2022-06-17 18:14:31 +02:00
Alexander Tokmakov
e9c2157f01 Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-17 15:57:30 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache 
Extract some diff from pr #36171
 2022-06-17 12:07:31 +02:00
Alexander Tokmakov
39c0219c11 fixes 2022-06-16 19:41:32 +02:00
alesapin
ca33ff93cb
Merge pull request #37555 from ClickHouse/revert-37534-revert-37036-keeper-preprocess-operations 
Add support for preprocessing ZooKeeper operations and real-time digest in `clickhouse-keeper`
 2022-06-16 17:29:00 +02:00
kssenii
797482f132 Better 2022-06-16 16:40:29 +02:00
Azat Khuzhin
b3bf7589ef Fix possible concurrent access in ProgressIndication 
In case of all of the above:
- clickhouse-local
- input_format_parallel_parsing=true
- write_progress_on_update=true

It is possible concurrent access to the following:
- writeProgress() (class properties) (guarded with progress_mutex)
- thread_data/host_cpu_usage (guarded with profile_events_mutex)

v2: decrease number of rows for INSERT ProfileEvents test (10 times)
    CI: https://s3.amazonaws.com/clickhouse-test-reports/37391/4bd5c335182279dcc5020aa081b13c3044135951/stateless_tests__debug__actions__[1/3].html
v3: decrease number of rows for INSERT ProfileEvents test (10 times) and add a comment
    CI: https://s3.amazonaws.com/clickhouse-test-reports/37391/026d7f732cb166c90d6c287b02824b6c7fdebf0c/stateless_tests_flaky_check__address__actions_/runlog.log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

f
 2022-06-16 11:59:01 +03:00
Antonio Andelic
0b843d316f Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-16 07:52:56 +00:00
Alexey Milovidov
c26abbfdc5
Merge pull request #38093 from danlark1/master 
Optimize most important parts with NEON SIMD
 2022-06-16 08:26:56 +03:00