Commit Graph

589 Commits

Author SHA1 Message Date
Azat Khuzhin
72d7b6117e Use total_memory_tracker when there is no other MemoryTracker object.
This should significantly reduce the MemoryTracking drift, test shows
that there is 0 drift after query storm (100 queries, via http/tcp/tcp
in one session).

TL;DR;

To track memory, clickhouse creates memory tracker object for each
thread **explicitly**, but until it is not created the memory
allocations are not under account.
There should not be lot of allocations w/o memory tracker, since most of
the time it is created early enough, but even this maybe enough to
trigger some problems.

Plus sometimes it is not possible to create it, for example some 3d
party library does not allow to do this explicitly:
- for example before  allocations from librdkafka threads,
- or even worse, poco threads, they don't have any routines to do this.
This won't be a problem for `MemoryTracking` metric if the deallocation
will be done from the same thread w/o memory tracker (or vise versa),
but this is not always true.

NOTE, that this will slow down per-thread allocations w/o memory
tracker, since before this patch there were no memory tracking for them
while now they will be accounted in total_memory_tracker, and for
total_memory_tracker max_untracked_memory is always reached.
But this should not be significant.
2020-10-23 21:07:52 +03:00
Alexander Kuzmenkov
a1a7bc0217
Merge pull request from filimonov/clickhouse-local-segfault
Add setTemporaryStorage to clickhouse-local to make OPTIMIZE work
2020-10-22 20:47:28 +03:00
Alexander Kuzmenkov
ef39def7ff
Apply suggestions from code review
Co-authored-by: filimonov <1549571+filimonov@users.noreply.github.com>
2020-10-22 20:46:17 +03:00
Mikhail Filimonov
111b553ee5
Better 2020-10-22 09:37:03 +02:00
Mikhail Filimonov
67cbb55d63
Fix compilation 2020-10-21 22:16:13 +02:00
Mikhail Filimonov
748ff404f9
Attempt to fix the race 2020-10-21 20:36:01 +02:00
alexey-milovidov
db481a33bd
Merge pull request from ClickHouse/simplify-init-script
Simplify init script
2020-10-21 03:37:50 +03:00
Alexey Milovidov
b26e979e5c Added TODOs + minor adjustments 2020-10-21 01:13:41 +03:00
alexey-milovidov
2b00b5391a
Merge pull request from Enmk/AES_encrypt_decrypt
encrypt and decrypt functions
2020-10-20 22:57:35 +03:00
Alexey Milovidov
104019bd87 Set capabilities with caution 2020-10-20 21:09:15 +03:00
Mikhail Filimonov
d7ea9b6d93
Add setTemporaryStorage to clickhouse-local to make OPTIMIZE work 2020-10-20 17:57:53 +02:00
alexey-milovidov
b4f0e08369
Merge pull request from ClickHouse/minimal-ui
Added minimal web UI
2020-10-20 11:57:56 +03:00
Alexey Milovidov
3be41e55a7 Add comments 2020-10-20 04:08:30 +03:00
Alexey Milovidov
9ae144dc69 Edit comment 2020-10-20 04:05:29 +03:00
Alexey Milovidov
9b57f7cdc9 Minor changes 2020-10-20 03:39:24 +03:00
Alexey Milovidov
f8fef7eff6 Minor changes 2020-10-20 03:32:50 +03:00
Alexey Milovidov
a554ca5e62 Added /play handler; added a test 2020-10-20 03:28:33 +03:00
Alexey Milovidov
0af87e9598 Advancements and adjustments 2020-10-20 02:37:10 +03:00
Vitaly Baranov
235a493a25
Merge pull request from traceon/ldap-any-user-authentication
Add LDAP user directory support for locally non-existent users
2020-10-19 23:58:35 +03:00
Alexey Milovidov
f229185bee Comments and corrections 2020-10-19 22:25:58 +03:00
Alexey Milovidov
7b4a537538 Added minimal web UI 2020-10-19 21:29:51 +03:00
Alexander Kuzmenkov
036cdd33c5 reset error code in ast fuzzer 2020-10-16 17:27:47 +03:00
Alexey Milovidov
5e502c52b3 Merge branch 'master' into AES_encrypt_decrypt 2020-10-14 16:30:23 +03:00
Denis Glazachev
6eec447a64 Merge branch 'master' into ldap-any-user-authentication
* master: (189 commits)
  Fix ugly links in docs
  Update Internals.cpp
  Whitespace
  Fix ugliness in docs
  better
  better backticks
  Try fix tests.
  Fetch helper image before run
  Fetch helper image before run
  Fetch helper image before run
  Update version_date.tsv after release 20.7.4.11
  Update version_date.tsv after release 20.8.4.11
  Update 01515_with_global_and_with_propagation.sql
  fix compilation error
  Supress pvs
  Update version_date.tsv after release 20.9.3.45
  Add missed file
  fix thread restart for parallel quorum inserts
  Update openssl
  Fix race in openssl
  ...
2020-10-14 16:46:37 +04:00
Amos Bird
e2643d881e
Prettify 2020-10-12 00:12:36 +08:00
alexey-milovidov
1aafa19146
Update Client.cpp 2020-10-11 13:27:09 +03:00
Amos Bird
ff665c362f
Add print query id support to client 2020-10-11 10:09:08 +08:00
Nikita Mikhaylov
dbb607573b
Update Internals.cpp 2020-10-10 02:31:33 +03:00
nikitamikhaylov
b45a4a2cd1 better 2020-10-09 20:44:54 +03:00
nikitamikhaylov
d0b610c6b2 better backticks 2020-10-09 19:58:45 +03:00
nikitamikhaylov
88a5302759 Merge branch 'master' of github.com:ClickHouse/ClickHouse into copier-backquotes 2020-10-09 16:26:31 +03:00
Alexey Milovidov
7d97c8b053 Remove useless code 2020-10-07 11:21:32 +03:00
alexey-milovidov
674d8d44b9
Update main.cpp 2020-10-05 09:28:52 +03:00
alexey-milovidov
8ff92a16ab
Update main.cpp 2020-10-05 08:01:08 +03:00
Azat Khuzhin
9e3ff349eb Ensure that there will be no strlen() calls for SSE checks 2020-10-04 17:37:31 +03:00
Azat Khuzhin
2a6874e065 Run MADV_DONTNEED after SSE check to fix 01103_check_cpu_instructions_at_startup 2020-10-04 11:20:29 +03:00
Azat Khuzhin
caf3156fb7 Check MADV_DONTNEED (for jemalloc), maybe broken under qemu
jemalloc relies on working MADV_DONTNEED (that fact that after
madvise(MADV_DONTNEED) returns success, after subsequent access to those
pages they will be zeroed).
However qemu does not support this, yet [1], and you can get very tricky
assert if you will run clickhouse-server under qemu:

    <jemalloc>: ../contrib/jemalloc/src/extent.c:1195: Failed assertion: "p[i] == 0"

  [1]: https://patchwork.kernel.org/patch/10576637/

But after this patch you will get pretty error:

  $ qemu-x86_64-static programs/clickhouse
  MADV_DONTNEED does not zeroed page. jemalloc will be broken
2020-10-04 11:20:29 +03:00
Azat Khuzhin
9c09050e7c Cleanup server instructions startup checking code 2020-10-04 01:47:44 +03:00
Denis Glazachev
44916af70a Merge branch 'master' into ldap-any-user-authentication
* master: (279 commits)
  Convert to python3 ()
  update en and ru docs: add 128/256 types ()
  Drop unused code for numeric_limits<int128> in MergeTreeDataSelectExecutor ()
  Fix comment
  More defaults on defaults
  Recursive defaults
  Fix bash
  Update ActionsVisitor.cpp
  Don't add memcpy for sanitizers builds
  Fix gRPC build scripts.
  Neutrialize thinlto's memcpy libcall gen.
  Revert "Add fno-builtin-memcpy"
  Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it
  Update SECURITY.md
  Update tsan_suppressions.txt
  Update ActionsVisitor.cpp
  remove unrelated changes
  Update SECURITY.md
  fixes
  Update ActionsVisitor.cpp
  ...
2020-10-03 00:45:57 +04:00
alesapin
17b86dec96 Recursive defaults 2020-10-02 15:38:50 +03:00
Alexey Milovidov
e65465ac16 Fix install script 2020-09-29 00:11:08 +03:00
tavplubix
4578d89819
Merge pull request from ClickHouse/replicate_default_args
Continuation of 
2020-09-28 18:53:02 +03:00
Nikita Mikhaylov
a4832ed5ce
Merge branch 'master' into HyaZz-master 2020-09-28 13:55:45 +03:00
Alexander Tokmakov
34addcf61f add review suggestions 2020-09-26 22:18:28 +03:00
alexey-milovidov
bbbe51033d
Merge pull request from ClickHouse/query-obfuscator
Query obfuscator
2020-09-26 19:43:36 +03:00
alesapin
adf8c94d70
Merge pull request from ClickHouse/fix_race_on_default_database
Fix for default database race on server startup
2020-09-26 10:43:17 +03:00
Alexey Milovidov
57b480e65d Fix PVS warning 2020-09-26 07:03:55 +03:00
Alexey Milovidov
3a2c809173 Add missing dependencies 2020-09-26 04:59:25 +03:00
Alexey Milovidov
ed4ff51ed8 Integrate into clickhouse-format; add test and comments 2020-09-26 03:38:59 +03:00
Denis Glazachev
8f2c23e6de Merge branch 'master' into ldap-any-user-authentication
* master: (224 commits)
  Update in.md
  Add a test for embedded configs
  Allow to run without /proc/self/maps
  Update adopters.md
  More convenient
  Enable embedded configs for AArch64
  More results
  More handy
  More handy
  Added a comment
  Adjustments
  Adjustments
  Adjustments
  Added new results
  Step 1: make adding hardware benchmark results more convenient
  Revert "Avoid deadlocks in Log/TinyLog"
  Fix MSan report in QueryLog
  add some disabled tests from arcadia to skip_list.json
  Update skip_list.json
  Bump CI. [2]
  ...

# Conflicts:
#	docker/test/testflows/runner/Dockerfile
2020-09-25 17:33:50 +04:00