thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
96,892 Commits 9,326 Branches 1,783 Tags 2.1 GiB
83514fa2ef
Commit Graph

4320 Commits

Author SHA1 Message Date
Azat Khuzhin
ec2e671d20 Remove ThreadStatus::untracked_memory_limit_increase 
It looks useless nowadays, because operator new cannot throw
MEMORY_LIMIT_EXCEEDED today, and so any code that works on Exception is
likely safe.

Refs: #40249
Refs: #24483
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-22 13:26:18 +03:00
kssenii
74087d57c4 Less notify 2022-08-22 12:00:14 +02:00
kssenii
26384b2543 Fix 2022-08-22 00:02:05 +02:00
Alexey Milovidov
996aa2d126 Remove some trash 2022-08-21 20:46:33 +02:00
Robert Schulze
e8b3f56733
Limit suppression to a specific warning 2022-08-21 18:24:17 +00:00
Robert Schulze
4caf2c4c33
Reduce some usage of StringRef 
See #39535 and #39300
 2022-08-21 18:10:32 +00:00
alesapin
78ba732b31 Merge branch 'master' into data_source_description 2022-08-21 12:11:21 +02:00
Alexey Milovidov
0db45afb08
Merge branch 'master' into alexey-milovidov-patch-4 2022-08-20 19:58:31 +03:00
Alexey Milovidov
3649aac593 Fix style 2022-08-20 17:59:51 +02:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
Frank Chen
54a530a4e0 Make sure right toString is linked 2022-08-20 14:22:33 +08:00
Frank Chen
0569608192 Use toString 2022-08-20 12:30:39 +08:00
Azat Khuzhin
28f2607d50 Fix rare race in updating peak of memory tracker 
Previously it was possible that allocation, that had been done with
locked MEMORY_LIMIT_EXCEEDED (LockMemoryExceptionInThread), will update
the peak to the amount that will include some other allocations.

Consider the following example (T1/T2 - different threads):

Initial state:

    MemoryTracker::amount = 0

Possible race:

    LockMemoryExceptionInThread locker;
T1: MemoryTracker::allocImpl(1MiB)      T2: MemoryTracker::allocImpl(1PiB)
                                            will_be = 1T, amount = 1T
    will_be = 1T+1MiB, amount=1T+1MiB
    peak = will_be
                                            amount = 0 // revert due to exceeding limit
                                            throw(MEMORY_LIMIT_EXCEEDED)

So as you can see T1 includes allocations from T2 into the peak.

Note, that almost every place that has memory_usage column uses
MemoryTracker::peak (not MemoryTracker::amount), this is true for
query_log (for part_log column name is peak_memory_usage, and obviously
it uses MemoryTracker::peak).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-19 20:57:12 +02:00
kssenii
e641664115 Refactor 2022-08-19 18:15:18 +02:00
alesapin
d8664c3227 Add shortcut for backups 2022-08-19 16:58:30 +02:00
Antonio Andelic
5e2c8869a9 Merge branch 'master' into replicate-tables-readonly 2022-08-19 09:41:11 +00:00
Frank Chen
a3b6ad2a65
Merge branch 'master' into tracing_context_propagation 2022-08-18 20:59:07 +08:00
Nikolai Kochetov
5a85531ef7
Merge pull request #38286 from Avogar/schema-inference-cache 
Add schema inference cache for s3/hdfs/file/url
 2022-08-18 13:07:50 +02:00
Kseniia Sumarokova
d8fd1f9c37
Merge pull request #36171 from kssenii/make-cache-composable 
Make cache composable, allow not to evict certain files (regarding .idx, .mrk, ..), delete old cache version
 2022-08-18 12:37:34 +02:00
Vladimir C
7ac0eb1af1
Merge pull request #40035 from ClickHouse/vdimir/cluster-discovery-observer 
Add observer mode to (zoo)keeper cluster discovery feature
 2022-08-18 10:53:49 +02:00
kssenii
d63d34f7bb Merge master 2022-08-17 15:49:32 +02:00
avogar
8dd54c043d Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache 2022-08-17 11:47:40 +00:00
Nikolai Kochetov
8ec3f0821c
Merge pull request #40249 from azat/fix-memory-account-on-error 
Fix memory accounting in case of MEMORY_LIMIT_EXCEEDED errors
 2022-08-17 12:51:40 +02:00
Alexander Tokmakov
4482465ce3
Merge pull request #40134 from arthurpassos/fix_cares_reverse_dns_race_condition 
Fix CaresPTRResolver race condition
 2022-08-16 21:06:17 +03:00
Nikolai Kochetov
feab1eb0a3
Update CurrentMemoryTracker.cpp 2022-08-16 18:40:06 +02:00
Nikolai Kochetov
8e4c967fc9 Try to account untracked memory (per thread) more correctly. 2022-08-16 18:09:05 +02:00
Azat Khuzhin
ded70fbd53 Fix memory accounting in case of MEMORY_LIMIT_EXCEEDED errors 
In case of memory allocation had been failed, it should not be take into
account anywhere, otherwise it will report incorrect memory usage for
query and in logs.

I found this while was digging into OOM in stress tests, and saw memory
allocations that had been significantly greater the memory limit (even
with overcommit).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-16 18:08:56 +02:00
kssenii
43991729ef Fix 2022-08-16 17:08:12 +02:00
kssenii
93816e7366 Review fixes 2022-08-16 15:11:55 +02:00
kssenii
3901778d49 Add more current metrics for cache 2022-08-16 11:32:23 +02:00
Antonio Andelic
2ed42bca7e Merge branch 'master' into replicate-tables-readonly 2022-08-16 08:17:53 +00:00
kssenii
eb26b219b9 Merge master 2022-08-16 00:56:27 +02:00
Alexander Tokmakov
c9bb91c31f
Merge pull request #40232 from ClickHouse/fix_a_bug_with_symlinks_detection 
Fix a bug with symlinks detection
 2022-08-15 20:28:30 +03:00
Robert Schulze
70f63d2aae
Merge pull request #40033 from ClickHouse/reenable-avx512-vbmi-columnvector-filter 
Revert the revert of "ColumnVector: optimize filter with AVX512 VBMI2 compress store"
 2022-08-15 18:39:57 +02:00
Alexander Tokmakov
0e22b4b800
Update src/Common/filesystemHelpers.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-08-15 14:25:25 +03:00
Alexander Tokmakov
6f5a7c3bf7 fix a bug with symlinks detection 2022-08-15 12:30:47 +02:00
Alexey Milovidov
903021f35a
Merge branch 'master' into profile-events-incomplete-result 2022-08-14 11:04:19 +03:00
Alexey Milovidov
1a8ddf2956 Addition to prev. revision 2022-08-14 09:35:22 +02:00
Alexey Milovidov
001aca3b47 ProfileEvents for incomplete data due to query complexity settings 2022-08-14 09:17:02 +02:00
Alexey Milovidov
216c1d8efa
Revert "Use separate counter for RSS in global memory tracker." 2022-08-14 09:50:12 +03:00
Alexey Milovidov
6b8f4f1d59 Merge branch 'master' into server-side-time 2022-08-12 21:36:40 +02:00
Maksim Kita
098436f012
Merge pull request #40129 from kitaisreal/pod-array-assign-empty-array-fix 
PODArray assign empty array fix
 2022-08-12 18:34:19 +02:00
Kseniia Sumarokova
a6cfc7bc3b
Merge pull request #34651 from alexX512/master 
New caching strategies
 2022-08-12 17:23:37 +02:00
Alexey Milovidov
8c3d1f2b5b
Merge branch 'master' into server-side-time 2022-08-12 17:40:02 +03:00
Arthur Passos
07163eb79a Remove c-ares lib de-initialization 2022-08-12 10:49:39 -03:00
vdimir
cef6c4f8ce
Add observer mode to (zoo)keeper cluster discovery feature 2022-08-12 13:26:12 +00:00
Kruglov Pavel
4c7222d938
Merge pull request #40020 from canhld94/ch_canh_fix_hash 
fix HashMethodOneNumber with const column
 2022-08-12 14:40:24 +02:00
Antonio Andelic
1c0f85ccdb Attach table in background 2022-08-12 09:32:13 +00:00
kssenii
fd6362d2f2 Merge master 2022-08-12 11:22:39 +02:00
Kseniia Sumarokova
09ad1115d7
Merge pull request #40138 from ClickHouse/kssenii-patch-4 
fs cache: minor change
 2022-08-12 11:18:03 +02:00
Maksim Kita
e2ae0d585b
Merge pull request #40078 from Algunenano/threadlocal_perf 
Optimize CurrentMemoryTracker alloc and free
 2022-08-12 10:24:54 +02:00
Maksim Kita
5dd4f4b10f
Merge branch 'master' into pod-array-assign-empty-array-fix 2022-08-12 10:20:33 +02:00
alexX512
06cb8737a8 Delete TODO 2022-08-11 22:38:13 +00:00
kssenii
b7c91b9cdb Merge remote-tracking branch 'origin/make-cache-composable' into make-cache-composable 2022-08-11 23:09:35 +02:00
kssenii
768a207176 Merge master 2022-08-11 22:25:53 +02:00
Kseniia Sumarokova
004a4d4947
Update FileCache.cpp 2022-08-11 22:15:33 +02:00
Kseniia Sumarokova
687ea5dd76
Update FileCache.cpp 2022-08-11 22:13:22 +02:00
Arthur Passos
59108345e9 Use one CaresPTRResolver per thread 2022-08-11 13:28:07 -03:00
Kseniia Sumarokova
be69169f97
Merge pull request #38048 from KinderRiven/decoupling_cache_function_and_algorithm 
Decoupling local cache function and cache algorithm
 2022-08-11 16:49:56 +02:00
Maksim Kita
f9326b00c8 PODArray assign empty array fix 2022-08-11 16:26:41 +02:00
Raúl Marín
11a274e990 Clean up constinit usage and add a comment 2022-08-11 13:27:53 +02:00
Nikolai Kochetov
971d94e492
Merge pull request #38682 from ClickHouse/separate-counter-for-rss-in-memory-tracker 
Use separate counter for RSS in global memory tracker.
 2022-08-11 10:11:18 +02:00
Duc Canh Le
84cd867aa8 materialize column instead of handling column in hash method 2022-08-11 10:46:06 +08:00
Alexey Milovidov
8374f31306
Merge pull request #39425 from arenadata/ADQM-485 
Add support of dates from year 1900 to 2299 for Date32 and DateTime64
 2022-08-11 05:01:53 +03:00
Raúl Marín
d7a545e30d Try to optimize CurrentMemoryTracker alloc and free 2022-08-10 22:05:09 +02:00
Nikolai Kochetov
c9353cbae8
Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-08-10 21:39:50 +02:00
Alexey Milovidov
0b24dbaec8 Merge 2022-08-10 19:41:44 +02:00
KinderRiven
1aa7bbcbbd fix unique_ptr 2022-08-11 00:11:41 +08:00
KinderRiven
9b7f87677d fix 2022-08-11 00:11:41 +08:00
KinderRiven
9ba94e64f9 fix 2022-08-11 00:11:41 +08:00
KinderRiven
2ae02a4921 fix style 2022-08-11 00:11:40 +08:00
KinderRiven
76e0aad69e fix 2022-08-11 00:11:40 +08:00
KinderRiven
fbaa70b313 fix 2022-08-11 00:11:40 +08:00
KinderRiven
f6a58bff4c fix build 2022-08-11 00:11:40 +08:00
KinderRiven
d2b5581632 fix 2022-08-11 00:11:40 +08:00
KinderRiven
1b01cc8ed9 fix 2022-08-11 00:11:40 +08:00
KinderRiven
164fa1ab0e fix build and style 2022-08-11 00:11:40 +08:00
KinderRiven
61b580aba4 add note 2022-08-11 00:11:40 +08:00
KinderRiven
081cd4938a fix style 2022-08-11 00:11:40 +08:00
KinderRiven
9d83b93e88 fix rebase 2022-08-11 00:11:40 +08:00
KinderRiven
50fd740ec3 fix 2022-08-11 00:11:40 +08:00
KinderRiven
c5f9022510 fix 2022-08-11 00:11:40 +08:00
KinderRiven
43cf771657 better 2022-08-11 00:11:40 +08:00
KinderRiven
ffaf44c1c1 fix style 2022-08-11 00:11:40 +08:00
KinderRiven
7f54fa726b Decoupling cache functions and algorithms 2022-08-11 00:11:40 +08:00
Kseniia Sumarokova
a4d513657c
Merge pull request #40010 from ClickHouse/catboost-further-preparation 
Further preparation for catboost integration into library-bridge
 2022-08-10 17:35:12 +02:00
kssenii
0ff5609738 Finish support for persistent files 2022-08-10 16:54:59 +02:00
Nikita Mikhaylov
8e8e845a89
Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-08-10 16:44:24 +02:00
Duc Canh Le
bf5e9173ad
fix hashMethodOneNumber constructor 2022-08-10 21:01:09 +08:00
Duc Canh Le
ec49f7b3a3 fix style 2022-08-10 19:28:08 +08:00
Duc Canh Le
17fa8076e3 better implementation 2022-08-10 18:59:13 +08:00
Duc Canh Le
7910a93b02 fix other hash methods and add tests 2022-08-10 18:59:13 +08:00
Duc Canh Le
1a08161dbf simplify solution 2022-08-10 18:59:13 +08:00
Duc Canh Le
cdd5b32d29 fix HashMethodOneNumber with const column 2022-08-10 18:59:13 +08:00
kssenii
7b66b96ec5 Minor change 2022-08-10 12:20:54 +02:00
kssenii
5c3227ba56 Merge master 2022-08-10 12:00:34 +02:00
Alexey Milovidov
33e43cc6f9
Merge branch 'master' into server-side-time 2022-08-10 07:28:15 +03:00
Robert Schulze
05ea33c897
Revert "Revert "ColumnVector: optimize filter with AVX512VBMI2 compress store"" 
This reverts commit 5524706b78.
 2022-08-09 17:00:13 +00:00
Kruglov Pavel
5ebf230dbb
Merge pull request #39426 from kitaisreal/shell-command-wait-pid-refactoring 
ShellCommand wait pid refactoring
 2022-08-09 11:48:20 +02:00
Roman Vasin
96598e3574 Replace normalizeDayNum() by std::clamp() 2022-08-09 08:52:02 +00:00
alexX512
21624708aa Fix tidy build 2022-08-09 08:36:49 +00:00
alexX512
c9c26d4898 Fix review 2022-08-08 21:34:24 +00:00
Robert Schulze
e0d5020a92
Add simple versioning to the *-bridge-to-server protocol 
- In general, it is expected that clickhouse-*-bridges and
  clickhouse-server were build from the same source version (e.g. are
  upgraded "atomically"). If that is not the case, we should at least
  be able to detect the mismatch and abort.

- This commit adds a URL parameter "version", defined in a header shared
  by the server and bridges. The bridge returns an error in case of
  mismatch.

- The version is *not* send and checked for "ping" requests (used for
  handshake), only for regular requests send after handshake. This is
  because the internally thrown server-side exception due to HTTP
  failure does not propagate the exact HTTP error (it only stores the
  error as text), and as a result, the server-side handshake code
  simply retries in case of error with exponential backoff and finally
  fails with a "timeout error". This is reasonable as pings typically
  fail due to time out. However, without a rework of HTTP exceptions,
  version mismatch during ping would also appear as "timeout" which is
  too misleading. The behavior may be changed later if needed.

- Note that introducing a version parameter does not represent a
  protocol upgrade itself. Bridges older than the server will simply
  ignore the field. Only servers older than the bridges receive an error
  but such a situation should never occur in practice.
 2022-08-08 19:40:37 +00:00
Alexey Milovidov
db85ebfaa7 Add server-side time to Progress 2022-08-08 06:55:41 +02:00
Alexey Milovidov
5524706b78
Revert "ColumnVector: optimize filter with AVX512VBMI2 compress store" 2022-08-08 05:03:15 +03:00
alexX512
f16c668a0c Delete Args from CacheBase 2022-08-07 20:44:26 +00:00
alexX512
1c9d40e4a8 Fix build error 2022-08-07 20:37:11 +00:00
alexX512
269631cfbe Fix typo 2022-08-07 20:02:32 +00:00
alexX512
62f84b88eb Review fixes 2022-08-07 20:02:21 +00:00
alexX512
8adf9f2448 fix compile error 2022-08-07 19:59:39 +00:00
alexX512
2be72560f4 Add loading of mark cache policy 2022-08-07 19:59:39 +00:00
alexX512
a057e1607a Fix remove operation, add evict ion i from cachef number of elements greater than max_elements_size 2022-08-07 19:59:39 +00:00
alexX512
8be62770aa Test with SLRU and debug logs 2022-08-07 19:59:39 +00:00
alexX512
c044fee7a0 Test with LRU 2022-08-07 19:59:39 +00:00
alexX512
a238e349cd Unblocking get max_size 2022-08-07 19:59:39 +00:00
alexX512
daab74925c Test with default cache policy = SLRU 2022-08-07 19:59:39 +00:00
alexX512
994a9fc005 Minor fixes 2022-08-07 19:59:38 +00:00
alexX512
50a8887ab9 Add [[maybe_unused]] locks and delete all objects from cache queue 2022-08-07 19:59:38 +00:00
alexX512
69cc89e36e Check with default cache policy = LRU 2022-08-07 19:59:38 +00:00
alexX512
a94950d3ac Test only with LRU 2022-08-07 19:59:38 +00:00
alexX512
9940b69bd7 Include fix in ICachePolicy.h 2022-08-07 19:59:38 +00:00
alexX512
910645a52a Include fix in SLRUCachePolicy.h 2022-08-07 19:59:38 +00:00
alexX512
4f52efd4bb Build fix 2022-08-07 19:59:38 +00:00
alexX512
6bf29cb610 Change class LRUCache to class CachBase. Check running CacheBase with default pcahce policy SLRU 2022-08-07 19:59:30 +00:00
alexX512
c606ccc75d Fix formatting problems 2022-08-07 19:16:40 +00:00
alexX512
1133e42367 Add CacheBase clas instead of LRUCache and SLRUCache fo simpler configuration oache policiesf 2022-08-07 19:16:40 +00:00
alexX512
a0d2da7261 Add tests for SLRU cache 2022-08-07 19:16:40 +00:00
AlPerevyshin
d7bdf9e601 Add SLRU cache 2022-08-07 19:16:38 +00:00
avogar
9b1a267203 Refactor, remove TTL, add size limit, add system table and system query 2022-08-05 16:20:15 +00:00
kssenii
91f4143b6e Merge master 2022-08-05 12:14:08 +02:00
Alexey Milovidov
3130aa7218
Merge pull request #39732 from HarryLeeIBM/hlee-s390x-unaligned 
Fix Endian issue in SipHash for s390x
 2022-08-05 07:24:57 +03:00
kssenii
7a9b0bc47f Merge master 2022-08-05 01:48:52 +02:00
Alexey Milovidov
24ee944986
Update TaskStatsInfoGetter.cpp 2022-08-04 07:34:36 +03:00
Fangyuan Deng
0bf5c3d1a1
Merge branch 'master' into fix_netlink_leak 2022-08-04 11:42:17 +08:00
pzhdfy
848a349e42 fix potential netlink leak when init fail in TaskStatsInfoGetter 2022-08-04 11:37:27 +08:00
Alexey Milovidov
f474eb957d
Merge pull request #39758 from ClickHouse/tsan_clang_15 
Try clang-15 for build with tsan
 2022-08-04 01:56:52 +03:00
Nikita Mikhaylov
f144eae388
Fix typo and extra dots in exception messages from OverCommitTracker (#39858) 2022-08-03 15:23:07 +02:00
Robert Schulze
00a7c8733b
Merge pull request #39633 from guowangy/filter-vbmi2 
ColumnVector: optimize filter with AVX512VBMI2 compress store
 2022-08-03 09:23:24 +02:00
Frank Chen
035896fa16 Fix build check 2022-08-02 20:18:42 +08:00
Frank Chen
1c9b462c80 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 20:18:22 +08:00
Alexander Tokmakov
82b50e79cf
Merge branch 'master' into tsan_clang_15 2022-08-02 13:00:55 +03:00
Frank Chen
57dde8c250 Clean up header inclusion 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-02 15:43:19 +08:00
Alexey Milovidov
316528817b
Merge pull request #39179 from azat/fsync-profile-events 
Add profile events for fsync
 2022-08-02 05:44:35 +03:00
Wangyang Guo
6a7213291b Merge master and resolve conflict 2022-08-02 10:40:40 +08:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
Alexander Tokmakov
0d68b1c67f fix build with clang-15 2022-08-01 18:00:54 +02:00
Robert Schulze
bf574b9154
Merge pull request #39760 from ClickHouse/bit-fiddling 
Use std::popcount, ::countl_zero, ::countr_zero functions
 2022-08-01 17:04:51 +02:00
Alexander Tokmakov
80f9ba9186
Merge pull request #39690 from ClickHouse/show-addresses-in-stack-traces 
Configuration parameter to hide addresses in stack traces
 2022-08-01 14:20:37 +03:00
Wangyang Guo
b05be56eef ColumnVector: naming style fix 2022-08-01 10:16:15 +08:00
Robert Schulze
d39259a4c0
More conversions 2022-07-31 22:06:02 +00:00