thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
99,229 Commits 9,328 Branches 1,783 Tags 2.1 GiB
9abf13bf78
Commit Graph

4528 Commits

Author SHA1 Message Date
Maksim Kita
f9326b00c8 PODArray assign empty array fix 2022-08-11 16:26:41 +02:00
Raúl Marín
11a274e990 Clean up constinit usage and add a comment 2022-08-11 13:27:53 +02:00
Nikolai Kochetov
971d94e492
Merge pull request #38682 from ClickHouse/separate-counter-for-rss-in-memory-tracker 
Use separate counter for RSS in global memory tracker.
 2022-08-11 10:11:18 +02:00
Duc Canh Le
84cd867aa8 materialize column instead of handling column in hash method 2022-08-11 10:46:06 +08:00
Alexey Milovidov
8374f31306
Merge pull request #39425 from arenadata/ADQM-485 
Add support of dates from year 1900 to 2299 for Date32 and DateTime64
 2022-08-11 05:01:53 +03:00
Raúl Marín
d7a545e30d Try to optimize CurrentMemoryTracker alloc and free 2022-08-10 22:05:09 +02:00
Nikolai Kochetov
c9353cbae8
Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-08-10 21:39:50 +02:00
Alexey Milovidov
0b24dbaec8 Merge 2022-08-10 19:41:44 +02:00
KinderRiven
1aa7bbcbbd fix unique_ptr 2022-08-11 00:11:41 +08:00
KinderRiven
9b7f87677d fix 2022-08-11 00:11:41 +08:00
KinderRiven
9ba94e64f9 fix 2022-08-11 00:11:41 +08:00
KinderRiven
2ae02a4921 fix style 2022-08-11 00:11:40 +08:00
KinderRiven
76e0aad69e fix 2022-08-11 00:11:40 +08:00
KinderRiven
fbaa70b313 fix 2022-08-11 00:11:40 +08:00
KinderRiven
f6a58bff4c fix build 2022-08-11 00:11:40 +08:00
KinderRiven
d2b5581632 fix 2022-08-11 00:11:40 +08:00
KinderRiven
1b01cc8ed9 fix 2022-08-11 00:11:40 +08:00
KinderRiven
164fa1ab0e fix build and style 2022-08-11 00:11:40 +08:00
KinderRiven
61b580aba4 add note 2022-08-11 00:11:40 +08:00
KinderRiven
081cd4938a fix style 2022-08-11 00:11:40 +08:00
KinderRiven
9d83b93e88 fix rebase 2022-08-11 00:11:40 +08:00
KinderRiven
50fd740ec3 fix 2022-08-11 00:11:40 +08:00
KinderRiven
c5f9022510 fix 2022-08-11 00:11:40 +08:00
KinderRiven
43cf771657 better 2022-08-11 00:11:40 +08:00
KinderRiven
ffaf44c1c1 fix style 2022-08-11 00:11:40 +08:00
KinderRiven
7f54fa726b Decoupling cache functions and algorithms 2022-08-11 00:11:40 +08:00
Kseniia Sumarokova
a4d513657c
Merge pull request #40010 from ClickHouse/catboost-further-preparation 
Further preparation for catboost integration into library-bridge
 2022-08-10 17:35:12 +02:00
kssenii
0ff5609738 Finish support for persistent files 2022-08-10 16:54:59 +02:00
Nikita Mikhaylov
8e8e845a89
Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-08-10 16:44:24 +02:00
Duc Canh Le
bf5e9173ad
fix hashMethodOneNumber constructor 2022-08-10 21:01:09 +08:00
Duc Canh Le
ec49f7b3a3 fix style 2022-08-10 19:28:08 +08:00
Duc Canh Le
17fa8076e3 better implementation 2022-08-10 18:59:13 +08:00
Duc Canh Le
7910a93b02 fix other hash methods and add tests 2022-08-10 18:59:13 +08:00
Duc Canh Le
1a08161dbf simplify solution 2022-08-10 18:59:13 +08:00
Duc Canh Le
cdd5b32d29 fix HashMethodOneNumber with const column 2022-08-10 18:59:13 +08:00
kssenii
7b66b96ec5 Minor change 2022-08-10 12:20:54 +02:00
kssenii
5c3227ba56 Merge master 2022-08-10 12:00:34 +02:00
Antonio Andelic
4023d4a37a Add support for drop/truncate and tests 2022-08-10 07:24:56 +00:00
Alexey Milovidov
33e43cc6f9
Merge branch 'master' into server-side-time 2022-08-10 07:28:15 +03:00
Robert Schulze
05ea33c897
Revert "Revert "ColumnVector: optimize filter with AVX512VBMI2 compress store"" 
This reverts commit 5524706b78.
 2022-08-09 17:00:13 +00:00
Kruglov Pavel
5ebf230dbb
Merge pull request #39426 from kitaisreal/shell-command-wait-pid-refactoring 
ShellCommand wait pid refactoring
 2022-08-09 11:48:20 +02:00
Roman Vasin
96598e3574 Replace normalizeDayNum() by std::clamp() 2022-08-09 08:52:02 +00:00
alexX512
21624708aa Fix tidy build 2022-08-09 08:36:49 +00:00
alexX512
c9c26d4898 Fix review 2022-08-08 21:34:24 +00:00
Robert Schulze
e0d5020a92
Add simple versioning to the *-bridge-to-server protocol 
- In general, it is expected that clickhouse-*-bridges and
  clickhouse-server were build from the same source version (e.g. are
  upgraded "atomically"). If that is not the case, we should at least
  be able to detect the mismatch and abort.

- This commit adds a URL parameter "version", defined in a header shared
  by the server and bridges. The bridge returns an error in case of
  mismatch.

- The version is *not* send and checked for "ping" requests (used for
  handshake), only for regular requests send after handshake. This is
  because the internally thrown server-side exception due to HTTP
  failure does not propagate the exact HTTP error (it only stores the
  error as text), and as a result, the server-side handshake code
  simply retries in case of error with exponential backoff and finally
  fails with a "timeout error". This is reasonable as pings typically
  fail due to time out. However, without a rework of HTTP exceptions,
  version mismatch during ping would also appear as "timeout" which is
  too misleading. The behavior may be changed later if needed.

- Note that introducing a version parameter does not represent a
  protocol upgrade itself. Bridges older than the server will simply
  ignore the field. Only servers older than the bridges receive an error
  but such a situation should never occur in practice.
 2022-08-08 19:40:37 +00:00
Alexey Milovidov
db85ebfaa7 Add server-side time to Progress 2022-08-08 06:55:41 +02:00
Alexey Milovidov
5524706b78
Revert "ColumnVector: optimize filter with AVX512VBMI2 compress store" 2022-08-08 05:03:15 +03:00
alexX512
f16c668a0c Delete Args from CacheBase 2022-08-07 20:44:26 +00:00
alexX512
1c9d40e4a8 Fix build error 2022-08-07 20:37:11 +00:00
alexX512
269631cfbe Fix typo 2022-08-07 20:02:32 +00:00
alexX512
62f84b88eb Review fixes 2022-08-07 20:02:21 +00:00
alexX512
8adf9f2448 fix compile error 2022-08-07 19:59:39 +00:00
alexX512
2be72560f4 Add loading of mark cache policy 2022-08-07 19:59:39 +00:00
alexX512
a057e1607a Fix remove operation, add evict ion i from cachef number of elements greater than max_elements_size 2022-08-07 19:59:39 +00:00
alexX512
8be62770aa Test with SLRU and debug logs 2022-08-07 19:59:39 +00:00
alexX512
c044fee7a0 Test with LRU 2022-08-07 19:59:39 +00:00
alexX512
a238e349cd Unblocking get max_size 2022-08-07 19:59:39 +00:00
alexX512
daab74925c Test with default cache policy = SLRU 2022-08-07 19:59:39 +00:00
alexX512
994a9fc005 Minor fixes 2022-08-07 19:59:38 +00:00
alexX512
50a8887ab9 Add [[maybe_unused]] locks and delete all objects from cache queue 2022-08-07 19:59:38 +00:00
alexX512
69cc89e36e Check with default cache policy = LRU 2022-08-07 19:59:38 +00:00
alexX512
a94950d3ac Test only with LRU 2022-08-07 19:59:38 +00:00
alexX512
9940b69bd7 Include fix in ICachePolicy.h 2022-08-07 19:59:38 +00:00
alexX512
910645a52a Include fix in SLRUCachePolicy.h 2022-08-07 19:59:38 +00:00
alexX512
4f52efd4bb Build fix 2022-08-07 19:59:38 +00:00
alexX512
6bf29cb610 Change class LRUCache to class CachBase. Check running CacheBase with default pcahce policy SLRU 2022-08-07 19:59:30 +00:00
alexX512
c606ccc75d Fix formatting problems 2022-08-07 19:16:40 +00:00
alexX512
1133e42367 Add CacheBase clas instead of LRUCache and SLRUCache fo simpler configuration oache policiesf 2022-08-07 19:16:40 +00:00
alexX512
a0d2da7261 Add tests for SLRU cache 2022-08-07 19:16:40 +00:00
AlPerevyshin
d7bdf9e601 Add SLRU cache 2022-08-07 19:16:38 +00:00
avogar
9b1a267203 Refactor, remove TTL, add size limit, add system table and system query 2022-08-05 16:20:15 +00:00
kssenii
91f4143b6e Merge master 2022-08-05 12:14:08 +02:00
Alexey Milovidov
3130aa7218
Merge pull request #39732 from HarryLeeIBM/hlee-s390x-unaligned 
Fix Endian issue in SipHash for s390x
 2022-08-05 07:24:57 +03:00
kssenii
7a9b0bc47f Merge master 2022-08-05 01:48:52 +02:00
Alexey Milovidov
24ee944986
Update TaskStatsInfoGetter.cpp 2022-08-04 07:34:36 +03:00
Fangyuan Deng
0bf5c3d1a1
Merge branch 'master' into fix_netlink_leak 2022-08-04 11:42:17 +08:00
pzhdfy
848a349e42 fix potential netlink leak when init fail in TaskStatsInfoGetter 2022-08-04 11:37:27 +08:00
Alexey Milovidov
f474eb957d
Merge pull request #39758 from ClickHouse/tsan_clang_15 
Try clang-15 for build with tsan
 2022-08-04 01:56:52 +03:00
Nikita Mikhaylov
f144eae388
Fix typo and extra dots in exception messages from OverCommitTracker (#39858) 2022-08-03 15:23:07 +02:00
Robert Schulze
00a7c8733b
Merge pull request #39633 from guowangy/filter-vbmi2 
ColumnVector: optimize filter with AVX512VBMI2 compress store
 2022-08-03 09:23:24 +02:00
Frank Chen
035896fa16 Fix build check 2022-08-02 20:18:42 +08:00
Frank Chen
1c9b462c80 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 20:18:22 +08:00
Alexander Tokmakov
b4c4448f0f fix 2022-08-02 12:07:38 +02:00
Alexander Tokmakov
82b50e79cf
Merge branch 'master' into tsan_clang_15 2022-08-02 13:00:55 +03:00
Frank Chen
57dde8c250 Clean up header inclusion 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-02 15:43:19 +08:00
Alexey Milovidov
316528817b
Merge pull request #39179 from azat/fsync-profile-events 
Add profile events for fsync
 2022-08-02 05:44:35 +03:00
Wangyang Guo
6a7213291b Merge master and resolve conflict 2022-08-02 10:40:40 +08:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
Alexander Tokmakov
ecf7ce1f74 Merge branch 'master' into zookeeper_client_fault_injection 2022-08-01 20:49:01 +02:00
Alexander Tokmakov
0d68b1c67f fix build with clang-15 2022-08-01 18:00:54 +02:00
Robert Schulze
bf574b9154
Merge pull request #39760 from ClickHouse/bit-fiddling 
Use std::popcount, ::countl_zero, ::countr_zero functions
 2022-08-01 17:04:51 +02:00
Alexander Tokmakov
80f9ba9186
Merge pull request #39690 from ClickHouse/show-addresses-in-stack-traces 
Configuration parameter to hide addresses in stack traces
 2022-08-01 14:20:37 +03:00
Wangyang Guo
b05be56eef ColumnVector: naming style fix 2022-08-01 10:16:15 +08:00
Robert Schulze
d39259a4c0
More conversions 2022-07-31 22:06:02 +00:00
Azat Khuzhin
3e627e2861 Add profile events for fsync 
The following new provile events had been added:

- FileSync - Number of times the F_FULLFSYNC/fsync/fdatasync function was called for files.
- DirectorySync - Number of times the F_FULLFSYNC/fsync/fdatasync function was called for directories.
- FileSyncElapsedMicroseconds - Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for files.
- DirectorySyncElapsedMicroseconds - Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for directories.

v2: rewrite test to sh with retries
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-31 23:19:30 +03:00
kssenii
e5f4a619ed Merge master 2022-07-31 20:24:40 +03:00
Robert Schulze
a7734672b9
Use std::popcount, ::countl_zero, ::countr_zero functions 
- Introduced with the C++20 <bit> header

- The problem with __builtin_c(l|t)z() is that 0 as input has an
  undefined result (*) and the code did not always check. The std::
  versions do not have this issue.

- In some cases, we continue to use buildin_c(l|t)z(), (e.g. in
  src/Common/BitHelpers.h) because the std:: versions only accept
  unsigned inputs (and they also check that) and the casting would be
  ugly.

(*) https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
 2022-07-31 15:16:51 +00:00
Robert Schulze
52d08d9db4
Merge pull request #39520 from ClickHouse/no-split-binary 
Remove SPLIT_BINARY
 2022-07-31 14:23:31 +02:00
Alexey Milovidov
d285648a37
Merge branch 'master' into shell-command-wait-pid-refactoring 2022-07-31 02:57:46 +03:00
Robert Schulze
85773e0926
Merge pull request #38615 from liyinsg/simplified_function_registration_interface 
Simplified function registration interface
 2022-07-31 00:18:37 +02:00
Alexey Milovidov
c0d7b6efc3 Suggestions from @tavplubix 2022-07-30 01:45:06 +02:00
Alexey Milovidov
4aa96767d5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into show-addresses-in-stack-traces 2022-07-30 01:44:04 +02:00
HarryLeeIBM
cc73b53116 Fix Endian issue in SipHash for s390x 2022-07-29 13:21:50 -07:00
Robert Schulze
3d1797f75f
Merge remote-tracking branch 'origin/master' into no-split-binary 2022-07-29 12:17:43 +00:00
Robert Schulze
a3c52f1140
Merge pull request #39679 from azat/fix-utils 
Fix utils build on CI
 2022-07-29 14:02:53 +02:00
Azat Khuzhin
498c8b3c52 Fix clang-tidy in utils/examples 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-29 11:36:51 +03:00
Li Yin
4088c0a7f3 Automated function registration 
Automated register all functions with below naming convention by
iterating through the symbols:
void DB::registerXXX(DB::FunctionFactory &)
 2022-07-29 15:39:50 +08:00
Alexey Milovidov
552b517e59 Allow to hide addresses in stack traces 2022-07-29 00:13:02 +02:00
Azat Khuzhin
e1f8cff3b4 Add duration_ms into system.zookeeper_log 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-28 22:34:51 +03:00
Robert Schulze
199e254777
Merge remote-tracking branch 'origin/master' into no-split-binary 2022-07-28 15:54:22 +00:00
Antonio Andelic
15e999f8bc
Merge pull request #39096 from ClickHouse/keeper-version-check 
Add version for Keeper API
 2022-07-28 10:10:49 +02:00
Sergei Trifonov
edf6b02723
Merge pull request #37558 from ClickHouse/concurrency-control 
add concurrency control for pipeline executors
 2022-07-28 08:56:24 +02:00
Yakov Olkhovskiy
f1bc767ccf
Merge pull request #39627 from HarryLeeIBM/hlee-s390x-wide_int 
Fix wide integer unit tests on s390x platform
 2022-07-27 21:30:12 -04:00
Alexander Tokmakov
8fc075a527
Merge pull request #39619 from ClickHouse/try-do-enable-global-allocations-for-attach 
Replace MemoryTrackerBlockerInThread to LockMemoryExceptionInThread
 2022-07-27 13:13:04 +03:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache 2022-07-27 11:35:36 +02:00
Antonio Andelic
904a05ac21
Merge pull request #39496 from azat/custom-tld-exclamation-asterisk 
Add support of !/* (exclamation/asterisk) in custom TLDs
 2022-07-27 08:55:49 +02:00
Wangyang Guo
e6752d687f TargetSpecific: add AVX512VBMI2 support 2022-07-27 13:30:43 +08:00
Wangyang Guo
50fdbcdc12 CpuId: add AVX512VBMI2 detection 2022-07-27 13:30:43 +08:00
HarryLeeIBM
d475086100 Fix wide integer unit test 2022-07-26 12:57:45 -07:00
Nikolai Kochetov
f1818a9103 Make only one possible public ctor in MemoryTrackerBlockerInThread 2022-07-26 19:04:20 +00:00
Nikolai Kochetov
dcb86eca0b Make only one possible public ctor in MemoryTrackerBlockerInThread 2022-07-26 19:03:26 +00:00
Nikolai Kochetov
fa0707b4e7 Review fixes. 2022-07-26 17:51:09 +00:00
Nikolai Kochetov
33ce662d3e Replace MemoryTrackerBlockerInThread to LockMemoryExceptionInThread in some places. Reduced MemoryTrackerBlockerInThread level to User. 2022-07-26 15:22:00 +00:00
Kruglov Pavel
c683cb252f
Merge pull request #39227 from amosbird/rename-log1 
Rename log when rename merge tree tables
 2022-07-26 17:12:44 +02:00
Sergei Trifonov
24ab5fbb86 fix finish() condition to account only active threads 2022-07-26 11:32:45 +02:00
Antonio Andelic
35b61cc94e Merge branch 'master' into keeper-version-check 2022-07-26 07:47:57 +00:00
Azat Khuzhin
1d4a7c7290 Add support of !/* (exclamation/asterisk) in custom TLDs 
Public suffix list may contain special characters (you may find format
here - [1]):
- asterisk (*)
- exclamation mark (!)

  [1]: https://github.com/publicsuffix/list/wiki/Format

It is easier to describe how it should be interpreted with an examples.

Consider the following part of the list:

    *.sch.uk
    *.kawasaki.jp
    !city.kawasaki.jp

And here are the results for `cutToFirstSignificantSubdomainCustom()`:

If you have only asterisk (*):

    foo.something.sheffield.sch.uk -> something.sheffield.sch.uk
    sheffield.sch.uk               -> sheffield.sch.uk

If you have exclamation mark (!) too:

    foo.kawasaki.jp                -> foo.kawasaki.jp
    foo.foo.kawasaki.jp            -> foo.foo.kawasaki.jp
    city.kawasaki.jp               -> city.kawasaki.jp
    some.city.kawasaki.jp          -> city.kawasaki.jp

TLDs had been verified wit the following script [2], to match with
python publicsuffix2 module.

  [2]: https://gist.github.com/azat/c1a7a9f1e3519793134ef4b1df5461a6

v2: fix StringHashTable padding requirements
Fixes: #39468
Follow-up for: #17748
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-26 08:34:30 +03:00
Roman Vasin
b462366415 Merge branch 'master' of github.com:ClickHouse/ClickHouse into ADQM-485 2022-07-25 17:55:47 +00:00
Roman Vasin
b412ea5f6d Improve generateRandom() for Date32; fix tests 01087_table_function_generate, 01277_fromUnixTimestamp64, 01691_DateTime64_clamp and 01702_toDateTime_from_string_clamping 2022-07-25 17:06:11 +00:00
Sergei Trifonov
d9d1381c3f Merge branch 'master' into concurrency-control 2022-07-25 14:16:46 +02:00
mergify[bot]
f667eff025
Merge branch 'master' into shell-command-wait-pid-refactoring 2022-07-25 11:42:05 +00:00
Robert Schulze
24c74da473
Merge pull request #39430 from azat/getauxval-fix 
Fix LSan by fixing getauxval() (resubmit v4)
 2022-07-25 10:43:09 +02:00
Alexey Milovidov
388d06fda1
Merge pull request #39535 from ClickHouse/stringref 
Less usage of StringRef
 2022-07-25 04:06:11 +03:00
Azat Khuzhin
93f876b58a Add a test for LSan 
(cherry picked from commit 51e7c41883)
v2: fix type check
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-25 01:21:40 +03:00
Robert Schulze
4333750985
Less usage of StringRef 
... replaced by std::string_view, see #39262
 2022-07-24 18:33:52 +00:00
Robert Schulze
73c04b64e3
Merge pull request #39493 from nicelulu/fix_build_osx 
Fix build on mac osx
 2022-07-24 18:11:34 +02:00
Robert Schulze
c788e05c77
Merge pull request #39292 from zvonand/zvonand-b58-datatype 
Simplify Base58 encoding/decoding
 2022-07-24 18:09:40 +02:00
Andrey Zvonov
032fcab70e set char to be explicitely signed 2022-07-24 14:23:22 +02:00
Alexey Milovidov
071374b152 Remove SPLIT_BINARY 2022-07-24 01:15:54 +02:00
Constantine Peresypkin
d645d03e89 fix -DENABLE_EXAMPLES=1 in master 2022-07-23 19:00:52 +02:00
Antonio Andelic
6aff87d4b5 Ignore system paths from snapshots and logstore 2022-07-23 13:46:59 +00:00
Antonio Andelic
46a6fbd7aa Small polish 2022-07-22 11:03:07 +00:00
zhangxiao871
0d4d1e66b5 Fix build 2022-07-22 18:53:44 +08:00
Alexander Tokmakov
bed2206ae9
Merge pull request #39460 from ClickHouse/remove_some_dead_and_commented_code 
Remove some dead and commented code
 2022-07-22 13:24:34 +03:00
Roman Vasin
0b102c6d1f Fix code style 2022-07-22 08:24:05 +00:00
Antonio Andelic
3040ff0959 Merge branch 'master' into keeper-version-check 2022-07-22 08:07:55 +00:00
Roman Vasin
74d2bf8d95 Fix gTest for DateLUTTest 2022-07-22 07:49:02 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache 2022-07-21 18:59:14 +02:00
Robert Schulze
ea0a3bf600
Merge branch 'master' into stringref-to-string_view 2022-07-21 18:33:06 +02:00
Maksim Kita
0159fe1274
Merge branch 'master' into shell-command-wait-pid-refactoring 2022-07-21 16:38:28 +02:00
Andrey Zvonov
e473606cd1
Merge branch 'master' into zvonand-b58-datatype 2022-07-21 15:55:05 +02:00
Alexander Tokmakov
9e9969cea7
Merge pull request #37827 from arthurpassos/host_regexp_multiple_domains 
Test host_regexp against all PTR records instead of only one
 2022-07-21 16:43:05 +03:00
Alexander Tokmakov
a8da5d96fc remove some dead and commented code 2022-07-21 15:05:48 +02:00
Maksim Kita
7f4a1b8bb8 ShellCommand wait pid refactoring 2022-07-21 11:45:09 +02:00
Nikolai Kochetov
e15967e9db
Merge pull request #38475 from ClickHouse/additional-filters 
Additional filters for a table (from setting)
 2022-07-21 07:52:04 +02:00
Alexey Milovidov
4305eacd92
Merge pull request #39434 from ClickHouse/revert-39299-fix-lsan-v3 
Revert "[RFC] Fix LSan by fixing getauxval()"
 2022-07-21 06:47:54 +03:00
Alexey Milovidov
f4f5a5e044
Merge pull request #39397 from azat/fix-extractAll 
Fix UB (stack-use-after-scope) in extactAll()
 2022-07-21 05:21:18 +03:00
Alexey Milovidov
72dda35108
Revert "[RFC] Fix LSan by fixing getauxval()" 2022-07-21 04:34:04 +03:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Kseniia Sumarokova
3759ee76a7
Merge pull request #39386 from ClickHouse/kssenii-patch-4 
Rethrow exception on cache initialisation on server startup, better error message
 2022-07-20 21:26:25 +02:00
Roman Vasin
e3192cf753 Correct docs to reflect new range 1900..2299 for Date32 and DateTime64; Cleanup code 2022-07-20 15:19:02 +00:00
zvonand
11b8d788ca small improvements 2022-07-20 13:36:47 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache 2022-07-20 13:30:42 +02:00
Kseniia Sumarokova
0840dfe2e0
Update LRUFileCache.cpp 2022-07-20 11:42:24 +02:00
Maksim Kita
d3ce4ffc88
Merge pull request #38929 from pkit/pkit/wait_pid_fix 
fix long wait for process exit in ShellCommand
 2022-07-20 11:26:38 +02:00
Azat Khuzhin
18eb8b6d48 Fix UB (stack-use-after-scope) in extactAll() 
After #37544 OptimizedRegularExpressionImpl started to be moved, but
StringSearcher is not copyable since it holds pointers that goes out of
scope after move (before Regexps::get() returns std::shared_ptr<Regexp>
but it had been replaced with Regexps::createRegexp() that returns
Regexp object).

<details>

<summary>ASan report</summary>

    ==48348==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7fff577239a9 at pc 0x00001518209b bp 0x7fff57723820 sp 0x7fff57723818
    READ of size 1 at 0x7fff577239a9 thread T0
        0 0x1518209a in char8_t const* DB::StringSearcher<true, true>::search<char8_t>(char8_t const*, char8_t const*) const /bld/./src/Common/StringSearcher.h:730:41
        1 0x1518dd3f in char8_t const* DB::StringSearcher<true, true>::search<char8_t>(char8_t const*, unsigned long) const /bld/./src/Common/StringSearcher.h:751:16
        2 0x1518dd3f in OptimizedRegularExpressionImpl<false>::match(char const*, unsigned long, std::__1::vector<OptimizedRegularExpressionDetails::Match, std::__1::allocator<OptimizedRegularExpressionDetails::Match> >&, unsigned int) const /bld/./src/Common/OptimizedRegularExpression.cpp:463:54
        3 0x1811cb42 in DB::ExtractAllImpl::get(char const*&, char const*&) /bld/./src/Functions/FunctionsStringArray.h:588:18
        4 0x1811aa62 in DB::FunctionTokens<DB::ExtractAllImpl>::executeImpl(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const /bld/./src/Functions/FunctionsStringArray.h:704:30
        5 0x14fe17b4 in DB::IFunction::executeImplDryRun(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const /bld/./src/Functions/IFunction.h:409:16

    Address 0x7fff577239a9 is located in stack of thread T0 at offset 201 in frame
        0 0x1518d98f in OptimizedRegularExpressionImpl<false>::match(char const*, unsigned long, std::__1::vector<OptimizedRegularExpressionDetails::Match, std::__1::allocator<OptimizedRegularExpressionDetails::Match> >&, unsigned int) const /bld/./src/Common/OptimizedRegularExpression.cpp:439

</details>

CI: https://s3.amazonaws.com/clickhouse-test-reports/39342/c6f7698f9ad6ae22199182ebf7c3b2dac77d69d8/fuzzer_astfuzzerasan,actions//report.html
Fixes: #37544 (cc @rschu1ze)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-20 10:18:44 +03:00
Roman Vasin
5f9c293963 Fix addDays() and addWeeks() in upper and lower limits of Date and Date32 2022-07-19 17:29:08 +00:00
Kseniia Sumarokova
891dbccdc7
Update LRUFileCache.cpp 2022-07-19 19:18:42 +02:00
Robert Schulze
02ed126533
Update src/Common/TraceSender.cpp 
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
 2022-07-19 18:44:31 +02:00
Sergei Trifonov
499818751e
Merge pull request #39280 from ClickHouse/avg-cpu-progress 
fix cpu usage metric in client
 2022-07-19 15:17:44 +02:00
Kseniia Sumarokova
2e3ceb351e
Update LRUFileCache.cpp 2022-07-19 14:44:56 +02:00
Kseniia Sumarokova
f7bb69ef7a
Update LRUFileCache.cpp 2022-07-19 14:43:11 +02:00
Kseniia Sumarokova
af7acb1782
Update LRUFileCache.cpp 2022-07-19 14:41:31 +02:00
Kseniia Sumarokova
5ad4e9d28f
Update LRUFileCache.cpp 2022-07-19 14:33:50 +02:00
Robert Schulze
7146685f9d
Merge pull request #39299 from azat/fix-lsan-v3 
[RFC] Fix LSan by fixing getauxval()
 2022-07-19 14:27:41 +02:00
Constantine Peresypkin
246614229f fix long wait for process exit in ShellCommand 
fixes #38889
 2022-07-19 14:07:27 +02:00
kssenii
21ebf8874e Merge remote-tracking branch 'upstream/master' into make-cache-composable 2022-07-19 13:36:42 +02:00
Nikolai Kochetov
fc6982f9e2 Fixing test. 2022-07-19 10:10:02 +00:00
Antonio Andelic
ce570b6ee3 Add logs and 4LW for api version 2022-07-19 09:02:57 +00:00
Antonio Andelic
e6ded88ea3 Small refactoring 2022-07-19 08:51:12 +00:00
Robert Schulze
81ef1099cc
Even less usage of StringRef 
--> see #39300
 2022-07-19 07:01:06 +00:00
Nikolai Kochetov
5a3cb07711 Fix rss 2022-07-18 21:24:41 +00:00
Nikolai Kochetov
3ee3144708 Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-07-18 23:20:38 +02:00
Nikolai Kochetov
87e5b31598 Do not include memory buffered by allocator into drift. 2022-07-18 21:16:16 +00:00
Robert Schulze
6df3c9d799
Merge pull request #39300 from ClickHouse/stringref-to-stringview 
First try at reducing the use of StringRef
 2022-07-18 19:20:16 +02:00
Roman Vasin
28fd774df8 Correct 01821_to_date_time_ubsan and 01921_datatype_date32 tests 2022-07-18 16:44:14 +00:00
Antonio Andelic
cfc741030f Revert to the version with path 2022-07-18 14:30:15 +00:00
kssenii
0aed5a602a Merge master 2022-07-18 16:16:28 +02:00
Azat Khuzhin
51e7c41883 Add a test for LSan 
v2: fix type check
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-18 16:49:49 +03:00
Sergei Trifonov
30cb4e168e cleanup 2022-07-18 13:31:32 +02:00
Sergei Trifonov
0ffe5bd39c fix typo and style 2022-07-18 13:26:33 +02:00
Sergei Trifonov
2593dbcaa6 reuse Common/ExponentiallySmoothedCounter.h 2022-07-18 12:21:44 +02:00
Antonio Andelic
758a61a36f Close socket if ApiVersion wasn't processed 2022-07-18 09:43:21 +00:00
Antonio Andelic
5170ac0898 Merge branch 'master' into keeper-version-check 2022-07-18 08:57:50 +00:00
Antonio Andelic
b8ffb151dc Some small polishing 2022-07-18 08:52:52 +00:00
Antonio Andelic
742e3d7643 Merge branch 'master' into keeper-version-check 2022-07-18 08:45:00 +00:00
Antonio Andelic
7d7c4ce4cd Add ApiVersion request 2022-07-18 08:44:38 +00:00
Alexey Milovidov
f1256c3aef
Merge pull request #37891 from guowangy/lz4-decompress-vbmi 
LZ4 decompress: add VBMI optimized copyOverlap32Shuffle
 2022-07-18 00:37:59 +03:00
Robert Schulze
13482af4ee
First try at reducing the use of StringRef 
- to be replaced by std::string_view
- suggested in #39262
 2022-07-17 17:26:02 +00:00
Robert Schulze
fffeab2e86
Merge pull request #39262 from ClickHouse/stringref-by-value 
Pass const StringRef by value, not by reference
 2022-07-17 16:53:15 +02:00
kssenii
3ccbf5ced0 Review fixes 2022-07-17 15:13:28 +02:00
Wangyang Guo
c07c65615d fix ENABLE_MULTITARGET_CODE not defined warning in some builds 2022-07-17 12:21:10 +08:00
kssenii
ca09c6c2c0 Fix some review comments, fix after merge 2022-07-16 23:30:54 +02:00
kssenii
8ad26b3111 Merge master 2022-07-16 22:50:03 +02:00
zvonand
3089da16be fix identation 2022-07-16 19:07:42 +02:00
zvonand
d07a652883 merge 2022-07-16 19:05:07 +02:00
zvonand
4ab52b6873 added new DataType + fixes 2022-07-16 18:58:47 +02:00
Alexey Milovidov
e37caa5bd6
Merge pull request #39216 from Algunenano/rotational_error 
Don't report system.errors when the disk is not rotational
 2022-07-16 16:16:52 +03:00
Sergei Trifonov
82f98cfe6b remove initial linear growth, increase avg period to 3 sec 2022-07-16 02:17:18 +02:00
Sergei Trifonov
8a5286ea4c fix cpu usage metric in client 2022-07-16 00:27:26 +02:00
Arthur Passos
abb550d2ff minor style fixes 2022-07-15 12:42:07 -03:00
Alexander Tokmakov
5ef7f5f494
Merge pull request #39229 from azat/fix-noexcept-scope 
[RFC] Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception)
 2022-07-15 18:14:56 +03:00
Robert Schulze
deda29b46b
Pass const StringRef by value, not by reference 
See #39224
 2022-07-15 11:34:56 +00:00
Kruglov Pavel
a944a92d4c
Merge pull request #39224 from Avogar/string-view-by-value 
Pass const std::string_view by value, not by reference
 2022-07-15 13:05:56 +02:00
Roman Vasin
1d0818d9cf Set max year to 2299; Code cleanup; Make working 02245_make_datetime64 test 2022-07-15 10:33:52 +00:00
Antonio Andelic
73e0c35ab0 Use 4LW for api version 2022-07-15 10:13:39 +00:00
Robert Schulze
8e44c2c375
Merge pull request #39223 from ClickHouse/warnings 
Turn some warnings on
 2022-07-15 10:06:59 +02:00
Wangyang Guo
397a4ac888 TargetSpecific: add AVX512VBMI support 2022-07-15 15:07:09 +08:00
Sergei Trifonov
62f975da60
Merge pull request #39095 from ClickHouse/remote-io-bandwidth-limits 
add max_remote_{read,write}_network_bandwidth_for_server settings
 2022-07-14 23:14:53 +02:00
Roman Vasin
266039ea64 Correct gTests for DateLUT 2022-07-14 19:00:17 +00:00
Raúl Marín
3fd327b23a Use std::filesystem::path::operator/ 2022-07-14 19:25:40 +02:00
avogar
9291d33080 Pass const std::string_view & by value, not by reference 2022-07-14 16:11:57 +00:00
Azat Khuzhin
b20407fab9 Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception) 
Current implementation of NOEXCEPT_SCOPE will not work, you cannot
rethrow exception outside the catch block, this will simply terminate
(via std::terminate) the program.
In other words NOEXCEPT_SCOPE macro will simply call std::terminate on
exception and will lost original exception.

But if NOEXCEPT_SCOPE will accept the code that should be runned w/o
exceptions, then it can catch exception and log it, rewrite it in this
way.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 17:16:18 +03:00
Amos Bird
96bb6e0cd2
Rename log when rename merge tree tables 2022-07-14 21:22:46 +08:00
Robert Schulze
3d734a0640
Enable -Wnested-anon-types 2022-07-14 12:36:55 +00:00
Robert Schulze
add83f7836
Merge pull request #39213 from ClickHouse/enable-wdeprecated-dynamic-exception-spec 
Enable warning "-Wdeprecated-dynamic-exception-spec"
 2022-07-14 13:09:18 +02:00
Sergei Trifonov
8a693469f0 Merge branch 'master' into remote-io-bandwidth-limits 2022-07-14 12:38:35 +02:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache 2022-07-14 12:29:54 +02:00
Raúl Marín
35d3a0dd1f Don't report system.errors when the disk is not rotational 2022-07-14 11:08:26 +02:00
Antonio Andelic
0e25dbbbeb Fix stateless test 2022-07-14 08:01:20 +00:00
Robert Schulze
110862a11e
Merge pull request #38502 from ClibMouse/Issue_38498 
Fixed bug in parsing {..} regex
 2022-07-14 08:27:31 +02:00
Nikolai Kochetov
812143c76b
Merge pull request #39160 from azat/threadpool-fix 
ThreadPool fixes
 2022-07-13 22:54:57 +02:00
Arthur Passos
7a09aa478a Add missing include header for osx builds 2022-07-13 17:45:30 -03:00
kssenii
615221f73b Minor changes 2022-07-13 19:22:47 +02:00
Sergei Trifonov
f854507729 merge master and resolve conflicts 2022-07-13 19:07:28 +02:00
Heena Bansal
76cbfedd2a
Merge branch 'master' into Issue_38498 2022-07-13 11:50:26 -04:00
Antonio Andelic
f05f22aa13 Merge branch 'master' into keeper-version-check 2022-07-13 15:25:48 +00:00
Kseniia Sumarokova
71aba5b4c6
Merge pull request #38860 from kssenii/object-storages-refactor 
Refactoring of code around object storages, added LocalObjectStorage (extracted this diff from PR #36171)
 2022-07-13 16:51:56 +02:00
Arthur Passos
7f4043a3ed Force use of c-ares inet_net_pton and style adjustments 2022-07-13 11:51:43 -03:00
kssenii
0944183eec Fix merge 2022-07-13 16:50:31 +02:00
Robert Schulze
87197f8f60
Enable -Wdeprecated-dynamic-exception-spec 2022-07-13 13:49:40 +00:00
Antonio Andelic
93d8b8bb61 small fixes 2022-07-13 13:01:13 +00:00
Azat Khuzhin
68789895a4 ThreadPool: incapsulate Poco::Event and std:🧵:id into State struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 15:51:49 +03:00
Azat Khuzhin
7ffa15b94b ThreadPool: fix thread_id assignment 
As found by @KochetovNicolai before this patch, lambda in
ThreadFromGlobalPool() ctor assigns value only to a copy of the
thread_id value, and so check in joinable() had been working
incorrectly, fix this by changing the value not the shared_ptr itself.

Also it is not safe to assign thread_id w/o atomics, since this can be
racy, so wrap id with std::atomic<>

Fixes: #28431
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 15:42:46 +03:00
Azat Khuzhin
25eb82f120 ThreadPool: do not use joinable() internally 
joinable() should be used only outside, since internally it is enough to
know `state` to know that something is wrong.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 15:42:46 +03:00
Azat Khuzhin
3473b80077 ThreadPool: add some clarification comments 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 15:42:46 +03:00
Arthur Passos
395dada988 minor style adjustments 2022-07-13 09:40:56 -03:00
kssenii
0d2d3ac2b4 Merge master 2022-07-13 13:26:15 +02:00
Yakov Olkhovskiy
334b3f3c73
Merge pull request #38864 from azat/fix-rwlock 
Fix waiting of shared lock after exclusive lock failure
 2022-07-13 02:10:35 -04:00
Arthur Passos
d306c1b6f4 Use Strings alias instead of std::vec<std::string>> 2022-07-12 20:37:39 -03:00
Arthur Passos
cb349c05c5 minor adjustments 2022-07-12 19:27:44 -03:00
Arthur Passos
381d60c510 minor adjustments 2022-07-12 17:55:49 -03:00
Arthur Passos
397d05eb9c Add DNSPTRResolverProvider docs 2022-07-12 14:31:33 -03:00
kssenii
6f29eada80 Merge master 2022-07-12 19:28:18 +02:00
Arthur Passos
d48690d455 Make CaresPTRResolver a singleton through DNSPTRResolverProvider, add comments and address minor comments 2022-07-12 14:21:10 -03:00
mergify[bot]
48688da835
Merge branch 'master' into concurrency-control 2022-07-12 10:57:13 +00:00
Yakov Olkhovskiy
5887deac99
Merge branch 'master' into fix-rwlock 2022-07-11 16:01:16 -04:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_" 
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
 2022-07-11 19:36:18 +02:00
Arthur Passos
dcc8c646fd Throw proper DB::Exception in case c-ares initialization fails 2022-07-11 14:29:29 -03:00
Arthur Passos
0c6329bdd3 ares_channel unique_ptr instead of shared_ptr 2022-07-11 10:19:25 -03:00
Arthur Passos
5c221464ba Forward declare ares_channel and apply -wno-reserved-identifier to CARESPTRResolver unit 2022-07-11 10:10:31 -03:00
Sergei Trifonov
43779ec280 add max_remote_{read,write}_network_bandwidth_for_server settings 2022-07-11 14:59:39 +02:00
Antonio Andelic
197388e29c Fix list request 2022-07-11 12:13:50 +00:00
Antonio Andelic
41dd4e13f4 Merge branch 'master' into keeper-version-check 2022-07-11 09:38:20 +00:00
Antonio Andelic
37f799550b
Merge pull request #38072 from lingpeng0314/master 
Add Keeper related monitoring data
 2022-07-10 09:39:57 +02:00
kssenii
4c88527c8e Review fixes 2022-07-09 19:35:07 +02:00
kssenii
bf1c99daab Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
Frank Chen
da57a993e4 Fix CI 2022-07-09 13:43:10 +08:00
Arthur Passos
4f4acfabe5 c-ares draft wrapper and draft build rules 2022-07-08 17:52:14 -03:00
Dmitry Novik
99cef119e1 Remove logging from OvercommitTracker 2022-07-08 15:34:23 +00:00
Frank Chen
6ae2504e57 Fix code 2022-07-08 18:55:59 +08:00
Roman Vasin
12f4a48957 Extend LUT range to 1900..2300 2022-07-08 06:48:05 +00:00
Nikolay Degterinsky
5cd06ac488
Merge pull request #38947 from soyayaos/master 
Rename NUMBER_OF_DIMENSIONS_MISMATHED const to NUMBER_OF_DIMENSIONS_MISMATCHED
 2022-07-07 22:25:42 +02:00
Alexander Tokmakov
b0f8970d07
Merge pull request #38961 from Algunenano/reduce_zk_eexists 
Stop reporting Zookeeper "Node exists" exceptions in system.errors when they are expected
 2022-07-07 23:16:46 +03:00
Alexander Tokmakov
1d93dc1c9a
Merge pull request #38963 from ClickHouse/fix_38962 
Avoid weird exception in Keeper
 2022-07-07 22:01:35 +03:00
Robert Schulze
f15d9ca59c
Merge pull request #38774 from zvonand/zvonand-nnd 
Reintroduce nonNegativeDerivative()
 2022-07-07 20:39:13 +02:00
Alexander Tokmakov
7630fc76ae avoid weird exception 2022-07-07 17:19:05 +02:00
Raúl Marín
c3935000a8 Avoid reporting ZK EEXIST errors on checkPartChecksumsAndCommit 2022-07-07 16:10:05 +02:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
Antonio Andelic
e96af48e28 Polish 2022-07-07 11:35:02 +00:00
Frank Chen
47da972277 Allows re-entering on thread context to make code more robust. 2022-07-07 17:45:08 +08:00
Frank Chen
4ef85164aa Fix file name 2022-07-07 17:44:50 +08:00
Frank Chen
93dc109e36 Fix code 2022-07-07 17:44:19 +08:00
Frank Chen
c0a67dc6b9 Add API to simplify user code 2022-07-07 17:44:05 +08:00
Frank Chen
57a7e4a7c9 Remove old API reference 2022-07-07 17:42:35 +08:00
Frank Chen
32931857f0 Propagate tracing context across threads 2022-07-07 17:41:43 +08:00
Frank Chen
2e8c530bed Move thread trace context out of ThreadStatus 2022-07-07 17:41:10 +08:00
Vladimir Galunshchikov
cacc833da6 Rename NUMBER_OF_DIMENSIONS_MISMATHED const to NUMBER_OF_DIMENSIONS_MISMATCHED 2022-07-07 11:15:59 +03:00
Alexey Milovidov
02adad3f27
Merge pull request #37860 from amosbird/remove-duplicated-log 
Remove duplicate peak mem log
 2022-07-07 08:16:06 +03:00
Andrey Zvonov
7de39d9b15 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-nnd 2022-07-06 10:59:35 +03:00
HeenaBansal2009
d89ba2e5d9 Review Comments 2022-07-05 21:18:39 -07:00
Azat Khuzhin
98cd92017e Fix waiting of shared lock after exclusive lock failure 
When WRITE lock attemp fails (exclusive lock for ALTER/DELETE), and
there are multiple READ locks (shared lock for SELECT/INSERT), i.e. one
INSERT is in progress and one SELECT is queued after ALTER/DELETE
started but before it fails, this SELECT will wait until INSERT will
finishes.

This happens because in case of WRITE lock failure it does not notify
the next READ lock that can be acquired.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-05 19:12:47 +03:00
Kseniia Sumarokova
3e803006a2
Merge pull request #37171 from tchepavel/nats-integration 
Nats Integration
 2022-07-05 17:47:02 +02:00
kssenii
2281f16c15 Fixes 2022-07-05 13:17:44 +02:00
alesapin
bd8a3ee841
Merge pull request #38338 from ClickHouse/zookeeper-add-extra-list-argument 
Extend ZooKeeper list request with support for filtering persistent or ephemeral nodes only
 2022-07-05 12:53:51 +02:00
kssenii
b52084265c Merge master 2022-07-04 21:37:43 +02:00
Kseniia Sumarokova
6c3d648f49
Merge branch 'master' into nats-integration 2022-07-04 21:06:07 +02:00
zvonand
8a270c01e9 fix floating point in intervals 2022-07-04 20:45:05 +03:00
Arthur Passos
5c00dcd884 Link cpp-dns/udns unconditionally and update submodules to clickhouse fork 2022-07-04 10:07:03 -03:00
Arthur Passos
dcda1c5eb9 Fix style case and DNS error message 2022-07-04 10:06:27 -03:00
Arthur Passos
d66154e697 Test host_regexp against all PTR records instead of only one 2022-07-04 10:05:28 -03:00
Kruglov Pavel
3d47416f8b
Merge pull request #38559 from ClickHouse/Avogar-patch-1 
Add logging in Epoll and TimerDescriptor in case of EINTR
 2022-07-04 15:05:08 +02:00
Alexey Milovidov
7adc87a37d
Merge pull request #38413 from nickitat/cache_values_inside_hash_table_grower 
Precalculate values inside `HashTableGrower`
 2022-07-02 23:30:24 +03:00
Kruglov Pavel
eaf8397c13
Fix style 2022-07-01 19:51:07 +02:00
Kruglov Pavel
fe0dad5429
Update src/Common/TimerDescriptor.cpp 2022-07-01 16:23:20 +02:00
Kruglov Pavel
4550acdc35
Update src/Common/Epoll.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-07-01 16:23:03 +02:00
Robert Schulze
2a1ede0f5a
Merge pull request #38589 from ClickHouse/fix-zero-bytes-in-haystack 
Fix countSubstrings() & position() on patterns with 0-bytes
 2022-07-01 16:15:43 +02:00
zvonand
3b5332d15e Revert "Revert "Non Negative Derivative window function"" 
This reverts commit dea3b5bfce.
 2022-07-01 18:59:07 +05:00
Nikolai Kochetov
b7529986c0 Use separate counter for RSS in global memory tracker. 2022-07-01 12:09:32 +00:00
Vitaly Baranov
cadb496894
Merge pull request #38537 from vitlibar/backup-improvements-8 
Backup Improvements 8
 2022-07-01 11:20:46 +02:00
Alexey Milovidov
20841f0e1e
Merge pull request #38551 from ClickHouse/revert-37628-non-neg-deriv 
Revert "Non Negative Derivative window function"
 2022-07-01 02:46:28 +03:00
Robert Schulze
1e3b5bfcb7
Fix test 00233_position_function_family 2022-06-30 11:43:25 +00:00
Vitaly Baranov
461a31f237 Improve gathering metadata for backup - part 2. 2022-06-30 08:37:17 +02:00
Robert Schulze
81bb2242fd
Fix countSubstrings() & position() on patterns with 0-bytes 
SQL functions countSubstrings(), countSubstringsCaseInsensitive(),
countSubstringsUTF8(), position(), positionCaseInsensitive(),
positionUTF8() with non-const pattern argument use fallback sorters
LibCASCIICaseSensitiveStringSearcher and LibCASCIICaseInsensitiveStringSearcher
which call ::strstr(), resp. ::strcasestr(). These functions assume that
the haystack is 0-terminated and they even document that. However, the
callers did not check if the haystack contains 0-byte (perhaps because
its sort of expensive). As a consequence, if the haystack contained a
zero byte in it's payload, matches behind this zero byte were ignored.

    create table t (id UInt32, pattern String) engine = MergeTree() order by id;
    insert into t values (1, 'x');
    select countSubstrings('aaaxxxaa\0xxx', pattern) from t;

We returned 3 before this commit, now we return 6
 2022-06-29 21:41:18 +00:00
Nikita Taranov
7d2e8b0883 clean up 2022-06-29 22:15:12 +02:00
Kruglov Pavel
77873e75bf
Update TimerDescriptor.cpp 2022-06-29 13:01:14 +02:00
Kruglov Pavel
30fd086b5e
Add logging in Epoll and TimerDescriptor in case of EINTR 2022-06-29 12:59:18 +02:00
Antonio Andelic
7a8fbbf132 Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-29 08:31:03 +00:00
Antonio Andelic
1d26446f84 Define new list request type 2022-06-29 08:30:39 +00:00
Robert Schulze
bb441faa42
Merge pull request #38491 from ClickHouse/clang-tidy-in-branches 
Fix some clang-tidy warnings in headers
 2022-06-29 10:22:04 +02:00
Alexey Milovidov
dea3b5bfce
Revert "Non Negative Derivative window function" 2022-06-29 08:56:15 +03:00
HeenaBansal2009
608bfb8453 Fixed {..} regex parsing as shell does 2022-06-28 12:49:31 -07:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to 
Replace where possible by std::lock_guard which is more light-weight.
 2022-06-28 19:19:06 +00:00
Robert Schulze
c22038d48b
More clang-tidy fixes 2022-06-28 11:50:05 +00:00
HeenaBansal2009
6a5819a405 Fixed bug in parsing{..} regex 2022-06-27 22:51:31 -07:00
Robert Schulze
5e28580f11
Fix some clang-tidy warnings in headers 2022-06-27 23:49:07 +02:00
Dmitry Novik
b629d557ce
Merge pull request #38246 from ClickHouse/overcommit-deadlock-fix 
Fix deadlock in OvercommitTracker logging
 2022-06-27 21:57:31 +02:00
Nikita Taranov
986ead7358 use old HashTableGrower with *WithStackMemory data structures 2022-06-27 20:34:18 +02:00
Robert Schulze
53cd8820d3
Merge pull request #38434 from ClickHouse/multi-match-with-nonconst-patterns 
Refactoring to enable multi-match functions with non-const needles
 2022-06-27 19:35:48 +02:00
kssenii
5c26f99bc5 Merge master 2022-06-27 13:36:41 +02:00
Antonio Andelic
c06776ce0d Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-27 06:53:29 +00:00
Antonio Andelic
0d906e311d PR review fixes 2022-06-27 06:53:09 +00:00
Robert Schulze
bb7c627964
Cosmetics: Pass patterns around as std::string_view instead of StringRef 
- The patterns are not used in hashing, there should not be a performance
  impact when we use stuff from the standard library instead.

- added forgotten .reserve() in FunctionsMultiStringPosition.h
 2022-06-26 15:32:19 +00:00
mergify[bot]
f63c959679
Merge branch 'master' into cleanup_garbage_in_store_dir 2022-06-26 13:35:10 +00:00
Nikita Taranov
95d413d7ec put the thing into its own cache line 2022-06-26 01:38:45 +02:00
Alexey Milovidov
25cc406201
Update src/Common/ThreadStatus.cpp 
Co-authored-by: Dmitry Novik <n0vik@clickhouse.com>
 2022-06-26 01:29:06 +03:00
Nikita Taranov
abcefeeb41 cleanup 2022-06-25 23:15:59 +02:00
mergify[bot]
469f24a12c
Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-25 12:01:06 +00:00
Alexey Milovidov
e33f236d50 Slight improvement 2022-06-25 03:59:01 +02:00
Alexey Milovidov
b89f01438e Lower mutex scope 2022-06-25 03:05:55 +02:00
Nikita Taranov
6639d94125 stash 2022-06-25 01:32:21 +02:00
Nikita Taranov
c476427422 stash 2022-06-25 01:27:24 +02:00
Nikita Taranov
f6ef78c0f1 stash 2022-06-25 01:26:46 +02:00
kssenii
708b93a03b Merge master 2022-06-25 00:08:24 +02:00
kssenii
082c306ee8 Fix test, fix style 2022-06-25 00:04:21 +02:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache 2022-06-24 16:10:25 +02:00
mergify[bot]
92eff431be
Merge branch 'master' into overcommit-deadlock-fix 2022-06-24 13:54:11 +00:00
Alexander Tokmakov
cfe0e434e5
Revert "Add support for io_uring read method" 2022-06-24 12:19:41 +03:00
mergify[bot]
a0eb831cd1
Merge branch 'master' into overcommit-deadlock-fix 2022-06-23 23:02:59 +00:00
Alexey Milovidov
812ab9bd6b
Merge pull request #36103 from sauliusvl/uring 
Add support for io_uring read method
 2022-06-24 00:34:29 +03:00
Alexander Tokmakov
74f38710a8 Merge branch 'master' into cleanup_garbage_in_store_dir 2022-06-23 21:43:28 +02:00
kssenii
6fbd49f554 Merge master 2022-06-23 21:40:01 +02:00
kssenii
468c98ff42 Better 2022-06-23 17:46:27 +02:00
kssenii
f7b329ee57 Merge master 2022-06-23 14:56:48 +02:00
Antonio Andelic
ae3d9fd962 Expose list request type in internal client 2022-06-23 10:28:12 +00:00
Antonio Andelic
568c957eb2 Fix TestKeeper 2022-06-23 10:28:12 +00:00
Antonio Andelic
3a71b63b5d Add list request type 2022-06-23 10:28:12 +00:00
Antonio Andelic
a5ee918bd2
Merge pull request #38047 from ClickHouse/keeper-sequential-consistency-reads 
Add `SYNC` command to internal ZooKeeper client
 2022-06-23 08:43:24 +02:00
kssenii
e40d9bcf55 Merge master 2022-06-22 23:28:52 +02:00
mergify[bot]
9688388be2
Merge branch 'master' into concurrency-control 2022-06-22 15:07:56 +00:00
Dmitry Novik
3544fbe5c4 cleanup 2022-06-22 14:05:20 +00:00
kssenii
4178abc0a5 Merge master 2022-06-21 22:38:51 +02:00
kssenii
90ce0b44f0 Fixes 2022-06-21 21:03:18 +02:00
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa 
Support for Clang Thread Safety Analysis (TSA)
 2022-06-21 20:19:33 +02:00
Sergei Trifonov
dc59d208de unittest style/perf minor fixes 2022-06-21 18:38:51 +02:00
kssenii
481644939a Add describe cache command 2022-06-21 16:07:39 +02:00
avogar
d37ad2e6de Implement cache for schema inference for file/s3/hdfs/url 2022-06-21 13:02:48 +00:00
Dmitry Novik
099055c183 Explicitly forbid allocations in OvercommitTracker 2022-06-21 12:26:13 +00:00
Alexander Tokmakov
ba0fcec993 add background cleanup of store/ subdirs 2022-06-21 12:35:47 +02:00
Saulius Valatka
160c95a6b8 unpoisson memory read by io_uring, handle interrupted syscalls, do not throw from monitor thread 2022-06-21 13:20:21 +03:00
Saulius Valatka
7b5e13a9eb implement pending request queue to prevent CQ overflows, more error checking, reduce size of io_uring 2022-06-21 13:17:56 +03:00
Dmitry Novik
e10f079bd3 Get rid of allocations in OvercommitTracker 2022-06-21 10:15:33 +00:00
Kseniia Sumarokova
71ee5dbbb2
Merge pull request #38227 from kssenii/diff-reduce 
buffer's getFileSize small changes
 2022-06-21 12:12:22 +02:00
mergify[bot]
bdd5b94192
Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-21 08:13:04 +00:00
Saulius Valatka
0dc7854af1 add initial io_uring support 2022-06-21 09:59:46 +03:00
lingpeng0314
c18dd55400 Resolve comments 
- refine metric description
 2022-06-21 10:30:19 +08:00
Dmitry Novik
9ca368ac20 Use do while(false) in macro 2022-06-20 23:10:40 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Dmitry Novik
53d656d89f Fix deadlock in OvercommitTracker logging 2022-06-20 17:35:24 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Antonio Andelic
0c76fc1121 Fix style 2022-06-20 14:21:59 +00:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
Antonio Andelic
777ae72259 Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-20 14:06:45 +00:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
Antonio Andelic
6c225bb04d
Update src/Common/ZooKeeper/ZooKeeperImpl.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-06-20 14:55:59 +02:00
kssenii
90be49faa4 Merge master 2022-06-20 14:26:56 +02:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6 
Backup Improvements 6
 2022-06-20 14:13:30 +02:00
kssenii
84a2bb031c Fix 2022-06-20 14:02:06 +02:00
mergify[bot]
4ed606a275
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 10:17:51 +00:00
Kseniia Sumarokova
a756b4be27
Merge pull request #37391 from azat/insert-profile-events-fix 
Send profile events for INSERT queries (previously only SELECT was supported)
 2022-06-20 12:16:29 +02:00
lingpeng0314
f06b19bdbf Add keeper monitoring data 2022-06-20 11:32:18 +08:00
Vitaly Baranov
a0c558a17e Implement backup/restore for ACL system tables (system.users, system.roles, etc.) 2022-06-17 18:14:31 +02:00
Alexander Tokmakov
e9c2157f01 Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-17 15:57:30 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache 
Extract some diff from pr #36171
 2022-06-17 12:07:31 +02:00
Alexander Tokmakov
39c0219c11 fixes 2022-06-16 19:41:32 +02:00
alesapin
ca33ff93cb
Merge pull request #37555 from ClickHouse/revert-37534-revert-37036-keeper-preprocess-operations 
Add support for preprocessing ZooKeeper operations and real-time digest in `clickhouse-keeper`
 2022-06-16 17:29:00 +02:00
kssenii
797482f132 Better 2022-06-16 16:40:29 +02:00
Azat Khuzhin
b3bf7589ef Fix possible concurrent access in ProgressIndication 
In case of all of the above:
- clickhouse-local
- input_format_parallel_parsing=true
- write_progress_on_update=true

It is possible concurrent access to the following:
- writeProgress() (class properties) (guarded with progress_mutex)
- thread_data/host_cpu_usage (guarded with profile_events_mutex)

v2: decrease number of rows for INSERT ProfileEvents test (10 times)
    CI: https://s3.amazonaws.com/clickhouse-test-reports/37391/4bd5c335182279dcc5020aa081b13c3044135951/stateless_tests__debug__actions__[1/3].html
v3: decrease number of rows for INSERT ProfileEvents test (10 times) and add a comment
    CI: https://s3.amazonaws.com/clickhouse-test-reports/37391/026d7f732cb166c90d6c287b02824b6c7fdebf0c/stateless_tests_flaky_check__address__actions_/runlog.log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

f
 2022-06-16 11:59:01 +03:00
Antonio Andelic
0b843d316f Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-16 07:52:56 +00:00
Alexey Milovidov
c26abbfdc5
Merge pull request #38093 from danlark1/master 
Optimize most important parts with NEON SIMD
 2022-06-16 08:26:56 +03:00