Alexey Milovidov
e33f236d50
Slight improvement
2022-06-25 03:59:01 +02:00
Alexey Milovidov
b89f01438e
Lower mutex scope
2022-06-25 03:05:55 +02:00
Nikita Taranov
6639d94125
stash
2022-06-25 01:32:21 +02:00
Nikita Taranov
c476427422
stash
2022-06-25 01:27:24 +02:00
Nikita Taranov
f6ef78c0f1
stash
2022-06-25 01:26:46 +02:00
mergify[bot]
92eff431be
Merge branch 'master' into overcommit-deadlock-fix
2022-06-24 13:54:11 +00:00
Alexander Tokmakov
cfe0e434e5
Revert "Add support for io_uring read method"
2022-06-24 12:19:41 +03:00
mergify[bot]
a0eb831cd1
Merge branch 'master' into overcommit-deadlock-fix
2022-06-23 23:02:59 +00:00
Alexey Milovidov
812ab9bd6b
Merge pull request #36103 from sauliusvl/uring
...
Add support for io_uring read method
2022-06-24 00:34:29 +03:00
Alexander Tokmakov
74f38710a8
Merge branch 'master' into cleanup_garbage_in_store_dir
2022-06-23 21:43:28 +02:00
kssenii
6fbd49f554
Merge master
2022-06-23 21:40:01 +02:00
kssenii
468c98ff42
Better
2022-06-23 17:46:27 +02:00
Antonio Andelic
ae3d9fd962
Expose list request type in internal client
2022-06-23 10:28:12 +00:00
Antonio Andelic
568c957eb2
Fix TestKeeper
2022-06-23 10:28:12 +00:00
Antonio Andelic
3a71b63b5d
Add list request type
2022-06-23 10:28:12 +00:00
Antonio Andelic
a5ee918bd2
Merge pull request #38047 from ClickHouse/keeper-sequential-consistency-reads
...
Add `SYNC` command to internal ZooKeeper client
2022-06-23 08:43:24 +02:00
Dmitry Novik
3544fbe5c4
cleanup
2022-06-22 14:05:20 +00:00
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa
...
Support for Clang Thread Safety Analysis (TSA)
2022-06-21 20:19:33 +02:00
kssenii
481644939a
Add describe cache command
2022-06-21 16:07:39 +02:00
Dmitry Novik
099055c183
Explicitly forbid allocations in OvercommitTracker
2022-06-21 12:26:13 +00:00
Alexander Tokmakov
ba0fcec993
add background cleanup of store/ subdirs
2022-06-21 12:35:47 +02:00
Saulius Valatka
160c95a6b8
unpoisson memory read by io_uring, handle interrupted syscalls, do not throw from monitor thread
2022-06-21 13:20:21 +03:00
Saulius Valatka
7b5e13a9eb
implement pending request queue to prevent CQ overflows, more error checking, reduce size of io_uring
2022-06-21 13:17:56 +03:00
Dmitry Novik
e10f079bd3
Get rid of allocations in OvercommitTracker
2022-06-21 10:15:33 +00:00
Kseniia Sumarokova
71ee5dbbb2
Merge pull request #38227 from kssenii/diff-reduce
...
buffer's getFileSize small changes
2022-06-21 12:12:22 +02:00
mergify[bot]
bdd5b94192
Merge branch 'master' into keeper-sequential-consistency-reads
2022-06-21 08:13:04 +00:00
Saulius Valatka
0dc7854af1
add initial io_uring support
2022-06-21 09:59:46 +03:00
lingpeng0314
c18dd55400
Resolve comments
...
- refine metric description
2022-06-21 10:30:19 +08:00
Dmitry Novik
9ca368ac20
Use do while(false) in macro
2022-06-20 23:10:40 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions
2022-06-20 18:11:30 +00:00
Dmitry Novik
53d656d89f
Fix deadlock in OvercommitTracker logging
2022-06-20 17:35:24 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa
2022-06-20 16:39:32 +02:00
Antonio Andelic
0c76fc1121
Fix style
2022-06-20 14:21:59 +00:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA)
...
- TSA is a static analyzer build by Google which finds race conditions
and deadlocks at compile time.
- It works by associating a shared member variable with a
synchronization primitive that protects it. The compiler can then
check at each access if proper locking happened before. A good
introduction are [0] and [1].
- TSA requires some help by the programmer via annotations. Luckily,
LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
std::shared_mutex and std::scoped_lock. This commit enables them
(--> contrib/libcxx-cmake/CMakeLists.txt).
- Further, this commit adds convenience macros for the low-level
annotations for use in ClickHouse (--> base/defines.h). For
demonstration, they are leveraged in a few places.
- As we compile with "-Wall -Wextra -Weverything", the required compiler
flag "-Wthread-safety-analysis" was already enabled. Negative checks
are an experimental feature of TSA and disabled
(--> cmake/warnings.cmake). Compile times did not increase noticeably.
- TSA is used in a few places with simple locking. I tried TSA also
where locking is more complex. The problem was usually that it is
unclear which data is protected by which lock :-(. But there was
definitely some weird code where locking looked broken. So there is
some potential to find bugs.
*** Limitations of TSA besides the ones listed in [1]:
- The programmer needs to know which lock protects which piece of shared
data. This is not always easy for large classes.
- Two synchronization primitives used in ClickHouse are not annotated in
libcxx:
(1) std::unique_lock: A releaseable lock handle often together with
std::condition_variable, e.g. in solve producer-consumer problems.
(2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
considered a design flaw + typically it is slower than a standard
mutex. In this commit, one std::recursive_mutex was converted to
std::mutex and annotated with TSA.
- For free-standing functions (e.g. helper functions) which are passed
shared data members, it can be tricky to specify the associated lock.
This is because the annotations use the normal C++ rules for symbol
resolution.
[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
2022-06-20 16:13:25 +02:00
Antonio Andelic
777ae72259
Merge branch 'master' into keeper-sequential-consistency-reads
2022-06-20 14:06:45 +00:00
kssenii
5dd1bb2fd8
improvements for getFileSize
2022-06-20 15:22:56 +02:00
Antonio Andelic
6c225bb04d
Update src/Common/ZooKeeper/ZooKeeperImpl.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2022-06-20 14:55:59 +02:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6
...
Backup Improvements 6
2022-06-20 14:13:30 +02:00
mergify[bot]
4ed606a275
Merge branch 'master' into fix_flaky_tests_with_transactions
2022-06-20 10:17:51 +00:00
Kseniia Sumarokova
a756b4be27
Merge pull request #37391 from azat/insert-profile-events-fix
...
Send profile events for INSERT queries (previously only SELECT was supported)
2022-06-20 12:16:29 +02:00
lingpeng0314
f06b19bdbf
Add keeper monitoring data
2022-06-20 11:32:18 +08:00
Vitaly Baranov
a0c558a17e
Implement backup/restore for ACL system tables (system.users, system.roles, etc.)
2022-06-17 18:14:31 +02:00
Alexander Tokmakov
e9c2157f01
Merge branch 'master' into fix_flaky_tests_with_transactions
2022-06-17 15:57:30 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache
...
Extract some diff from pr #36171
2022-06-17 12:07:31 +02:00
Alexander Tokmakov
39c0219c11
fixes
2022-06-16 19:41:32 +02:00
alesapin
ca33ff93cb
Merge pull request #37555 from ClickHouse/revert-37534-revert-37036-keeper-preprocess-operations
...
Add support for preprocessing ZooKeeper operations and real-time digest in `clickhouse-keeper`
2022-06-16 17:29:00 +02:00
kssenii
797482f132
Better
2022-06-16 16:40:29 +02:00
Azat Khuzhin
b3bf7589ef
Fix possible concurrent access in ProgressIndication
...
In case of all of the above:
- clickhouse-local
- input_format_parallel_parsing=true
- write_progress_on_update=true
It is possible concurrent access to the following:
- writeProgress() (class properties) (guarded with progress_mutex)
- thread_data/host_cpu_usage (guarded with profile_events_mutex)
v2: decrease number of rows for INSERT ProfileEvents test (10 times)
CI: https://s3.amazonaws.com/clickhouse-test-reports/37391/4bd5c335182279dcc5020aa081b13c3044135951/stateless_tests__debug__actions__[1/3].html
v3: decrease number of rows for INSERT ProfileEvents test (10 times) and add a comment
CI: https://s3.amazonaws.com/clickhouse-test-reports/37391/026d7f732cb166c90d6c287b02824b6c7fdebf0c/stateless_tests_flaky_check__address__actions_/runlog.log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
f
2022-06-16 11:59:01 +03:00
Antonio Andelic
0b843d316f
Merge branch 'master' into keeper-sequential-consistency-reads
2022-06-16 07:52:56 +00:00
Alexey Milovidov
c26abbfdc5
Merge pull request #38093 from danlark1/master
...
Optimize most important parts with NEON SIMD
2022-06-16 08:26:56 +03:00
Robert Schulze
b39b963733
Merge pull request #37993 from ClickHouse/stripping
...
Strip less aggressively to make the embedded hash survive
2022-06-16 00:08:36 +02:00
Vitaly Baranov
21f3bed435
Simplify path calculations in backup.
2022-06-15 20:32:34 +02:00
Vitaly Baranov
592f568f83
Move backup/restore code to storages and databases - part 2.
2022-06-15 20:32:31 +02:00
Danila Kutenin
048f56bf4d
Fix some tests and comments
2022-06-15 14:40:21 +00:00
kssenii
500f49972b
Extract diff from PR 36171
2022-06-15 16:40:18 +02:00
Danila Kutenin
08e3f77a9c
Optimize most important parts with NEON SIMD
...
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.
That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
2022-06-15 13:19:29 +00:00
mergify[bot]
d148299c20
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-15 07:11:31 +00:00
Alexey Milovidov
0957c885e2
Remove -0. from CPU usage in the client
2022-06-14 23:36:16 +02:00
mergify[bot]
4bd61950db
Merge branch 'master' into stripping
2022-06-14 20:48:52 +00:00
alesapin
af1cd745e1
Merge pull request #37975 from kssenii/clean-up-broken-detached
...
Clean up broken detached parts after timeout
2022-06-14 20:53:31 +02:00
Antonio Andelic
b7bd5a8eb1
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-14 12:51:35 +00:00
Antonio Andelic
c98c6566bf
Use std::pair
2022-06-14 12:49:42 +00:00
Maksim Kita
da8b1b1eba
Merge pull request #38025 from kitaisreal/use-base-sort-instead-of-standard
...
Use pdqsort instead of standard sort
2022-06-14 12:13:35 +02:00
Antonio Andelic
367df12626
Add sync support to client
2022-06-14 09:50:30 +00:00
Robert Schulze
bc46cef63c
Minor follow-up
...
- change ELF section name to ".clickhouse.hash" (lowercase seems
standard)
- more expressive/concise integrity check messages at startup
2022-06-14 08:52:13 +00:00
kssenii
74cffaf186
Merge master
2022-06-14 00:45:19 +02:00
Antonio Andelic
e5504f1b33
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-13 15:43:10 +00:00
Maksim Kita
98a89b50ff
Use pdqsort instead of standard sort
2022-06-13 15:31:08 +02:00
Kseniia Sumarokova
7fd1140e96
Merge pull request #38004 from xiedeyantu/optimize_max_cache
...
fix when open enable_filesystem_query_cache_limit, throw LOGICAL_ERROR
2022-06-13 11:54:09 +02:00
Robert Schulze
9ec56fa362
Merge pull request #37628 from zvonand/non-neg-deriv
...
Non Negative Derivative window function
2022-06-13 11:21:16 +02:00
Robert Schulze
bc6f30fd40
Move binary hash to ELF section ".ClickHouse.hash"
2022-06-13 08:46:23 +00:00
xiedeyantu
c93ef2b033
fix when open enable_filesystem_query_cache_limit, throw Reserved cache size exceeds the remaining cache size
2022-06-12 15:03:06 +08:00
kssenii
4859e11c27
Review fixes
2022-06-12 01:15:10 +02:00
Alexander Tokmakov
a1ca690afb
Merge pull request #37976 from ClickHouse/tavplubix-patch-1
...
Try fix `test_consistent_parts_after_clone_replica`
2022-06-10 21:04:56 +03:00
Alexander Tokmakov
2319ef7647
fix test
2022-06-10 17:48:14 +02:00
zvonand
5beb6e0c3b
Merge branch 'master' of github.com:ClickHouse/ClickHouse into non-neg-deriv
2022-06-10 17:40:16 +03:00
Kseniia Sumarokova
580a30c6ae
Merge pull request #37859 from KinderRiven/support_max_request_cache_size
...
Support to set max cache size for per query in local cache
2022-06-10 16:17:18 +02:00
zvonand
794dbe6b60
new stateless queries + minor fixes
2022-06-10 17:01:49 +03:00
Robert Schulze
5f5732a2c4
Merge pull request #37969 from ClickHouse/consistent-macro-usage
...
More consistent use of platform macros
2022-06-10 14:10:01 +02:00
kssenii
498f389c21
Merge master
2022-06-10 12:49:10 +02:00
kssenii
7a2676c7ab
Clean up broken detached parts with timeout
2022-06-10 12:27:57 +02:00
Kseniia Sumarokova
508676ab4e
Merge pull request #37804 from ClickHouse/metadata_storage
...
Metadata storage abstraction for Disks
2022-06-10 11:43:28 +02:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros
...
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.
__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
2022-06-10 10:22:31 +02:00
KinderRiven
360730bafe
fix
2022-06-10 12:28:22 +08:00
Alexander Tokmakov
780f7c87c7
Merge pull request #36113 from ClickHouse/remove-useless-code-2
...
Remove useless code in ReplicatedMergeTreeRestartingThread
2022-06-09 17:56:28 +03:00
KinderRiven
5c27244eb3
fix unit_test
2022-06-09 21:50:09 +08:00
alesapin
84ab147155
Merge branch 'master' into metadata_storage
2022-06-08 22:33:12 +02:00
Maksim Kita
a38c527e32
ProcfsMetricsProvider fix typo
2022-06-08 16:30:16 +02:00
KinderRiven
d5da970129
fix
2022-06-08 20:42:16 +08:00
KinderRiven
16ae349751
fix
2022-06-08 20:15:58 +08:00
KinderRiven
cec810ad29
fix
2022-06-08 20:13:20 +08:00
KinderRiven
6883ecda96
fix
2022-06-08 14:19:50 +08:00
KinderRiven
ff16d5555c
fix build
2022-06-08 03:23:40 +08:00
KinderRiven
1e47ae1026
fix build
2022-06-08 03:22:58 +08:00
KinderRiven
5e8a3c6352
fix build
2022-06-08 03:19:07 +08:00
KinderRiven
55c390c86f
fix
2022-06-08 01:42:46 +08:00
KinderRiven
6f8abd0328
fix
2022-06-07 19:53:24 +08:00
KinderRiven
de45f0a38c
fix unit_test
2022-06-07 19:30:39 +08:00
KinderRiven
f00914430d
fix holder
2022-06-07 19:10:57 +08:00
Vladimir C
1c184a7e18
Merge pull request #37840 from azat/join-query-fatal-errors
2022-06-07 12:07:37 +02:00
KinderRiven
3b2a427002
fix
2022-06-07 11:50:56 +08:00
alesapin
5a213c2396
Followup
2022-06-06 20:52:53 +02:00
alesapin
a459f2e43d
Fix bug
2022-06-06 20:51:58 +02:00
Antonio Andelic
2c37fe3d7b
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-06 12:39:55 +00:00
alesapin
daf49dca77
Remove logical error (legal case)
2022-06-06 13:32:58 +02:00
KinderRiven
bfe5150154
better
2022-06-06 16:48:30 +08:00
KinderRiven
4e78bbd414
fix
2022-06-06 03:21:56 +08:00
KinderRiven
13e74bef5e
support skip_download_if_exceeds_query_cache
2022-06-06 02:36:23 +08:00
KinderRiven
882104ab77
fix
2022-06-06 01:46:27 +08:00
KinderRiven
d9684daa6f
fix better
2022-06-06 01:29:49 +08:00
KinderRiven
ae86f7ceaa
fix
2022-06-06 00:00:59 +08:00
KinderRiven
c5fc826fc3
fix style
2022-06-05 17:45:45 +08:00
Amos Bird
a5afb73c52
Remove duplicated peak mem log
...
Also log "Processed in {} sec" a little bit earlier so that it's also
sent to client and have the log entry related to the query_id.
2022-06-05 17:02:58 +08:00
KinderRiven
cb9927cd00
fix
2022-06-05 16:52:01 +08:00
KinderRiven
3ffb0b3549
support_max_query_cache_size
2022-06-05 16:21:36 +08:00
Alexey Milovidov
f860407af5
Merge pull request #37777 from ClickHouse/avx512_tail_zero
...
Use multitarget framework for numZerosInTail implementation
2022-06-04 01:46:56 +03:00
Azat Khuzhin
c29768325c
Print query in one line on fatal errors
...
executeQuery() logging query without new lines in SQL, and it is useful,
since you can grep and see the whole query (usually).
So let's use the same in fatal error handler to make CI reports more
informative.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-03 18:55:53 +03:00
mergify[bot]
ddf7210ecc
Merge branch 'master' into remove-useless-code-2
2022-06-03 13:58:45 +00:00
alesapin
ac03e3af3e
Merge branch 'master' into metadata_storage
2022-06-03 14:42:21 +02:00
zvonand
5e4e349495
small fixes
2022-06-03 14:14:20 +03:00
Alexey Milovidov
1529d47207
Merge pull request #34754 from ClickHouse/llvm-14
...
Switch to clang/llvm 14
2022-06-03 14:07:34 +03:00
alesapin
46bccae078
Trying to create metadata layer
2022-06-02 18:09:40 +02:00
Antonio Andelic
7ee47fc5bb
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations
2022-06-02 13:38:52 +00:00
Nikolai Kochetov
edac3d6714
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-06-02 09:36:20 +00:00
Alexander Gololobov
6433fd6113
Added macros for AVX512BW target
2022-06-02 11:19:46 +02:00
Alexey Milovidov
cb8b0219ac
Merge branch 'master' of github.com:ClickHouse/ClickHouse into llvm-14
2022-06-02 08:46:31 +02:00
Alexander Gololobov
a0cf902d49
Merge pull request #37588 from yaqi-zhao/avx512_tail_zero
...
add avx512 support for mergetree reader
2022-06-02 08:51:52 +03:00
Alexander Gololobov
ec6e413f0b
Fixed runtime check for AVX512F
2022-06-01 23:00:49 +02:00
Alexey Milovidov
b5f48a7d3f
Merge branch 'master' of github.com:ClickHouse/ClickHouse into llvm-14
2022-06-01 22:09:58 +02:00
Alexey Milovidov
89638de521
Merge pull request #37738 from ClickHouse/fix-intersect-with-const
...
Fix `Intersect` with constant strings
2022-06-01 19:31:55 +03:00
Antonio Andelic
ded1398565
Fix intersect with const string
2022-06-01 11:13:33 +00:00
mergify[bot]
4aefcf29e3
Merge branch 'revert-37534-revert-37036-keeper-preprocess-operations' into keeper-real-time-digest
2022-06-01 10:17:49 +00:00
Alexander Gololobov
26609a1875
Style fixes
2022-05-31 21:41:10 +02:00
Nikolai Kochetov
86fbb74703
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-31 18:07:47 +00:00
mergify[bot]
ba49c6bb46
Merge branch 'master' into memory-overcommit-improvement
2022-05-31 13:17:06 +00:00
yaqi-zhao
a2857491c4
add avx512 support for mergetreereader
2022-05-30 20:53:00 -04:00
Kseniia Sumarokova
b1ba7b7027
Fix build
2022-05-30 17:30:59 +02:00
Kseniia Sumarokova
1869adfd7d
Update FileCache.cpp
2022-05-30 16:06:58 +02:00
Nikolai Kochetov
5b4658aa5e
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-30 09:47:35 +00:00
Andrey Zvonov
c79d87e629
fix style
2022-05-30 02:26:02 +03:00
Alexey Milovidov
5fda199dcf
Update src/Common/ErrorCodes.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2022-05-30 00:20:49 +03:00
Andrey Zvonov
2dbbf14de5
Merge branch 'master' into non-neg-deriv
2022-05-29 10:09:51 +03:00
Alexey Milovidov
c1169019d2
Merge branch 'master' into llvm-14
2022-05-29 02:29:02 +02:00
Alexey Milovidov
11788c8129
Fix clang-tidy-14
2022-05-29 02:28:46 +02:00
zvonand
032e54abbf
works now
2022-05-29 03:21:07 +03:00
Alexey Milovidov
73e2e63414
Merge pull request #37612 from ClickHouse/clang-tidy-14
...
Fix clang-tidy-14, part 1
2022-05-29 03:16:32 +03:00
Anton Popov
1d9b3be7da
Merge pull request #37536 from CurtizJ/profile-events-for-part-types
...
Add profile events for introspection of part types
2022-05-28 14:25:21 +02:00
Alexey Milovidov
eff285e24a
Update ThreadFuzzer.cpp
2022-05-28 05:13:16 +03:00
Alexey Milovidov
c50791dd3b
Fix clang-tidy-14, part 1
2022-05-27 22:52:14 +02:00
Alexey Milovidov
d2c6fd90cb
Fix clang-tidy-14, part 1
2022-05-27 22:51:37 +02:00