* Fix test "max_memory_usage_for_user"
* Update test
* Update annoying boilerplate
* Support symlinks in tarballs
* Fix Fuzzer
* Remove "secondary" user
* Remove "secondary" user
* Use only |name_parts| as primary name source
* Restore legacy logic for table restoration
* Fix build
* Fix tests
* Add pytest server config
* Fix tests
* Fixes due to review
This should significantly reduce the MemoryTracking drift, test shows
that there is 0 drift after query storm (100 queries, via http/tcp/tcp
in one session).
TL;DR;
To track memory, clickhouse creates memory tracker object for each
thread **explicitly**, but until it is not created the memory
allocations are not under account.
There should not be lot of allocations w/o memory tracker, since most of
the time it is created early enough, but even this maybe enough to
trigger some problems.
Plus sometimes it is not possible to create it, for example some 3d
party library does not allow to do this explicitly:
- for example before #15740 allocations from librdkafka threads,
- or even worse, poco threads, they don't have any routines to do this.
This won't be a problem for `MemoryTracking` metric if the deallocation
will be done from the same thread w/o memory tracker (or vise versa),
but this is not always true.
NOTE, that this will slow down per-thread allocations w/o memory
tracker, since before this patch there were no memory tracking for them
while now they will be accounted in total_memory_tracker, and for
total_memory_tracker max_untracked_memory is always reached.
But this should not be significant.
jemalloc relies on working MADV_DONTNEED (that fact that after
madvise(MADV_DONTNEED) returns success, after subsequent access to those
pages they will be zeroed).
However qemu does not support this, yet [1], and you can get very tricky
assert if you will run clickhouse-server under qemu:
<jemalloc>: ../contrib/jemalloc/src/extent.c:1195: Failed assertion: "p[i] == 0"
[1]: https://patchwork.kernel.org/patch/10576637/
But after this patch you will get pretty error:
$ qemu-x86_64-static programs/clickhouse
MADV_DONTNEED does not zeroed page. jemalloc will be broken
* master: (279 commits)
Convert to python3 (#15007)
update en and ru docs: add 128/256 types (#15515)
Drop unused code for numeric_limits<int128> in MergeTreeDataSelectExecutor (#15519)
Fix comment
More defaults on defaults
Recursive defaults
Fix bash
Update ActionsVisitor.cpp
Don't add memcpy for sanitizers builds
Fix gRPC build scripts.
Neutrialize thinlto's memcpy libcall gen.
Revert "Add fno-builtin-memcpy"
Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it
Update SECURITY.md
Update tsan_suppressions.txt
Update ActionsVisitor.cpp
remove unrelated changes
Update SECURITY.md
fixes
Update ActionsVisitor.cpp
...
* master: (224 commits)
Update in.md
Add a test for embedded configs
Allow to run without /proc/self/maps
Update adopters.md
More convenient
Enable embedded configs for AArch64
More results
More handy
More handy
Added a comment
Adjustments
Adjustments
Adjustments
Added new results
Step 1: make adding hardware benchmark results more convenient
Revert "Avoid deadlocks in Log/TinyLog"
Fix MSan report in QueryLog
add some disabled tests from arcadia to skip_list.json
Update skip_list.json
Bump CI. [2]
...
# Conflicts:
# docker/test/testflows/runner/Dockerfile
* master: (375 commits)
Update type-conversion-functions.md
Update maxmap.md
Update maxmap.md
Update maxmap.md
Update single_fixed_string_groupby.xml
Alter remove column properties and TTLs (#14742)
better fixed string group by support
Fix incorrect key condition of fixed strings.
constant output order
more tests for #14646
Maybe fix MSan report in base64
Proper exception message for wrong number of arguments of CAST
Added a test
Fix buffer overflow in "bar" function
Update convertMySQLDataType.cpp
Fix clang-tidy
Remove obsolete code from performance test
Slightly better code
Even more
Even more
...
# Conflicts:
# src/Interpreters/Context.cpp
With '\n...' after the query [1] clickhouse-client prefer data from the
INSERT over from stdin, and produce very tricky message:
Code: 27. DB::Exception: Cannot parse input: expected '\n' before: ' ': (at row 1)
Well for TSV it is ok, but for RowBinary:
Code: 33. DB::Exception: Cannot read all data. Bytes read: 1. Bytes expected: 4.
So improve error message by adding the source of data for INSERT.
[1]: clickhouse-client -q "INSERT INTO data FORMAT TSV\n " <<<2
Add inter-server cluster secret, it is used for Distributed queries
inside cluster, you can configure in the configuration file:
<remote_servers>
<logs>
<shard>
<secret>foobar</secret> <!-- empty -- works as before -->
...
</shard>
</logs>
</remote_servers>
And this will allow clickhouse to make sure that the query was not
faked, and was issued from the node that knows the secret. And since
trust appeared it can use initial_user for query execution, this will
apply correct *_for_user (since with inter-server secret enabled, the
query will be executed from the same user on the shards as on initator,
unlike "default" user w/o it).
v2: Change user to the initial_user for Distributed queries if secret match
v3: Add Protocol::Cluster package
v4: Drop Protocol::Cluster and use plain Protocol::Hello + user marker
v5: Do not use user from Hello for cluster-secure (superfluous)
Add distributed_ddl.pool_size to control maximum parallel to handle
distributed DDL.
Also:
- convert Exception constructors to fmt-like
- use sleepFor* over std::this_thread::sleep_for()
Process query until the stage where the aggregate functions were
calculated and finalized.
It will be used for optimize_distributed_group_by_sharding_key.
v2: fix aliases
v3: Fix protocol ABI breakage due to WithMergeableStateAfterAggregation
Conditions >= for QueryProcessingStage::Enum has been verified, and they
are ok (in InterpreterSelectQuery).
Functions to encrypt/decrypt any input data with OpenSSL's ciphers
with custom key, iv, and add (-gcm mode only).
_mysql versions are 100% compatitable with corresponding MySQL functions
Supported modes depend on OpenSSL version, but generally are:
aes-{128,192,56}-{ecb,cbc,cfb1,cfb8,cfb128,ofb,gcm}
Please note that in a -gcm mode a 16-byte tag is appended to the ciphertext
on encryption and is expected to be found at the end of ciphertext on decryption.
Added tests that verify compatibility with MySQL functions,
and test vectors for GCM mode from OpenSSL.
Added masking rules for aes_X funtions
Rules are installed by default to config.d/query_masking_rules.xml
* master: (414 commits)
Add .reference
Update 01460_mark_inclusion_search_crash.sql
Fix bug in mark inclusion search.
cosmetic fixes for performance report
cosmetic fixes in performance report
Add more docs about functional tests
Add ability to specify Default codec for columns (#14049)
better
done
Disable force TTL on optimise
Update CMakeLists.txt
Update custom parts of storage AST only if it has extended definition
DOCSUP-712: Documented the ttl_only_drop_parts setting (#13823)
Small fixes
gix comment and useDefault*(), add tests for nullables
Fix a build for old some OS with old find
Update hdfs.md
Fix 01085_max_distributed_connections flackiness
Fix 00974_distributed_join_on flackiness (by allow retries and hide logs)
Update adopters.md
...
* master: (43 commits)
Publish list of tests that failed the concurrent fast test
fix test
Use gnu++2a instead of c++2a for unbundled build to fix numeric_limits<__int128>
fix test
Update 01453_fixsed_string_sort.sql
Added test.
Fix ColumnString::updatePermutationWithCollation.
Add support for extended precision integers and decimals (#13097)
remove retries
Check that ya.make files are auto-generated
Fix "Arcadia"
ISSUES-4006 trigger CI again
fixed typo arrayCompact
Stratify nans comparison in arrayCompact function
Remove even more useless code
Remove useless code around zkutil
Fix 00956_sensitive_data_masking flackiness
Update docker/test/stress/run.sh
add exclusive DDLGuard for database
Fix handling embedded config.
...
* master: (30 commits)
Documentation improve: Translate [select] section into Chinese to improve readability (#13814)
Update adopters.md
partially disable test with MaterializeMySQL
one more unroll for arerage.cpp
Update adopters.md
Update adopters.md
Update adopters.md
Update adopters.md
fix#13819
DOCSUP-1888: Documented the input_format_avro_allow_missing_fields setting (#13671)
Fix "Arcadia" and "Unbundled" builds
DOCSUP-928: Documented the groupArraySample function (#13791)
Documentation about ReplacingMergeTree extended with type DateTime64 for column (#13498)
doc: update quotas.md (#13400)
Fix 01356_initialize_aggregation in unbundled build (change topKWeighted order)
Fix style
Add normalizedQueryHash function with tests
Fix topK/topKWeighted merge (wtih non-default parameters)
Add test
Add function "normalizeQuery"
...
# Conflicts:
# programs/server/Server.cpp
# src/Access/AccessControlManager.cpp
# src/Access/AccessControlManager.h
* Experiment with start/stop in stress test
* Support other installation options in "clickhouse start"
* Better diagnostics in install script
* Fixup
* Fixup
* Better diagnostics in stress test
* Update run.sh
* master: (70 commits)
Fix terrible markup due to wrong @blinkov's script.
Update extended-roadmap.md
Update extended-roadmap.md
Fix bug on the website
Suppress Shellcheck
Fix typos
Fix typos
Install codespell
Fix typo
Update gtest_thread_pool_schedule_exception.cpp
Fix flaky unit test
Fix set index with const column pred
Updating healthcheck parameters.
Fix step overflow in range()
* Moving to standard healthcheck parameters. * Adding docker-compose pull and docker-compose down before executing docker-compose up. * Adding docker-compose ps if any of the containers are unhealthy.
Fix topK test
Adding missing certificates and dhparam.pem for openldap4.
Keep "metadata_loading" test only for release build
Fix error in test
Fix typo
...
* master: (219 commits)
Fix link in the documentation (#12769)
add mapAdd and mapSubtract functions (#11735)
DOCSUP-1916 split reference.md into separate files (#12963)
DOCSUP-790: docs for mutations_sync setting (#12639)
Adding extra xfails for some ldap tests.
Fix JSON
Additions
Added instruction
Add benchmark instructions
Add benchmark results for OmniSci
[docs] maybe fix build
Fix terrible unbearable data rot. CC @blinkov
Block structure mismatch is a program error
Merging #12548 - Correction to `merge_with_ttl_timeout` logic by @excitoon (#12982)
Update entrypoint.sh
Added test.
Fix crash in ColumnTuple::updatePermutation
DOCS-590: parallel_distributed_insert_select (#12980)
Added test.
Fix header for totals and extremes in QueryPipeline::unitePipelines.
...
* commit 'ceac649c01b0158090cd271776f3219f5e7ff57c': (75 commits)
[docs] split misc statements (#12403)
Update 00405_pretty_formats.reference
Update PrettyCompactBlockOutputFormat.cpp
Update PrettyBlockOutputFormat.cpp
Update DataTypeNullable.cpp
Update 01383_remote_ambiguous_column_shard.sql
add output_format_pretty_grid_charset setting in docs
add setting output_format_pretty_grid_charset
Added a test for #11135
Update index.md
RIGHT and FULL JOIN for MergeJoin (#12118)
Update MergeTreeIndexFullText.cpp
restart the tests
[docs] add syntax highlight (#12398)
query fuzzer
Fix std::bad_typeid when JSON functions called with argument of wrong type.
Allow typeid_cast() to cast nullptr to nullptr.
fix another context-related segfault
[security docs] actually, only admins can create advisories
query fuzzer
...
