Commit Graph

848 Commits

Author SHA1 Message Date
alesapin
4d40d9be3f Merge branch 'master' into complete_zk_api 2020-11-25 10:31:16 +03:00
Vitaly Baranov
eab3006a4d Support SSL and compression when using gRPC protocol. 2020-11-24 17:55:02 +03:00
Vitaly Baranov
c2edd9f8ce Rework async server: switch to coroutine-like approach, no using statuses anymore for control flow. 2020-11-24 17:53:34 +03:00
Vitaly Baranov
ff62fd4967 Add an adapter for protocol servers. 2020-11-24 17:53:34 +03:00
Vitaly Baranov
44717797c9 Fix code style and compilation. 2020-11-24 17:53:34 +03:00
Vitaly Baranov
13f2352c57 Move files to right places. 2020-11-24 17:53:34 +03:00
mnkonkova
6cd1557d67 Implement GRPC protocol. 2020-11-24 17:53:34 +03:00
alesapin
c1a7e4f5fa Merge branch 'master' into complete_zk_api 2020-11-24 13:37:51 +03:00
Denis Glazachev
a367abb88f Merge branch 'master' into ldap-cache-login
* master: (70 commits)
  Update documentation-issue.md
  Add an option to use existing tables to perf.py
  DOCSUP-4280: Update the SELECT query (#17231)
  DOCSUP-3584 edit and translate (#17176)
  Fixed flaky test_storage_s3::test_custom_auth_headers
  Update 01560_merge_distributed_join.sql
  Minor improvements
  Slightly more correct
  Auto version update to [20.13.1.1] [54444]
  Auto version update to [20.12.1.5236] [54443]
  Update roadmap
  Add favicon; add loading indicator
  Fix race condition; history and sharing capabilities
  Update bitmap-functions.md
  Fix exception message
  Use default value for read-only flag in metadata for Disk3.
  ISSUES-16605 try fix review comment
  trigger CI
  ISSUES-16605 try fix integration failure
  ISSUES-16605 try fix integration test failure
  ...
2020-11-24 02:55:20 +04:00
Denis Glazachev
b02f2cf1ca Merge branch 'master' into ldap-role-mapping
* master: (50 commits)
  Update documentation-issue.md
  Add an option to use existing tables to perf.py
  DOCSUP-4280: Update the SELECT query (#17231)
  DOCSUP-3584 edit and translate (#17176)
  Fixed flaky test_storage_s3::test_custom_auth_headers
  Update 01560_merge_distributed_join.sql
  Minor improvements
  Slightly more correct
  Auto version update to [20.13.1.1] [54444]
  Auto version update to [20.12.1.5236] [54443]
  Update roadmap
  Add favicon; add loading indicator
  Fix race condition; history and sharing capabilities
  Update bitmap-functions.md
  Fix exception message
  Use default value for read-only flag in metadata for Disk3.
  ISSUES-16605 try fix review comment
  trigger CI
  ISSUES-16605 try fix integration failure
  ISSUES-16605 try fix integration test failure
  ...
2020-11-24 02:53:26 +04:00
alesapin
1de175257b More const 2020-11-23 23:22:04 +03:00
alesapin
5a48076802 Trying to get rid of leak 2020-11-23 13:55:00 +03:00
alesapin
b8fb44a7b9 Merge with master 2020-11-23 11:05:50 +03:00
Alexey Milovidov
fdc11cb739 Minor improvements 2020-11-23 11:05:13 +03:00
Alexey Milovidov
56651a94c4 Slightly more correct 2020-11-23 10:52:33 +03:00
Alexey Milovidov
f3e567b4b4 Update roadmap 2020-11-23 10:15:52 +03:00
Alexey Milovidov
5058c56ecf Add favicon; add loading indicator 2020-11-23 10:15:33 +03:00
Alexey Milovidov
2c982b4ccf Fix race condition; history and sharing capabilities 2020-11-23 09:35:08 +03:00
Denis Glazachev
1a587b0c21 Merge branch 'master' into ldap-role-mapping
* master: (159 commits)
  Review fix.
  Update version_date.tsv after release 20.8.7.15
  wrong translation
  Update version_date.tsv after release 20.9.6.14
  Update version_date.tsv after release 20.10.5.10
  Update version_date.tsv after release 20.11.4.13
  Improvements in coverage images
  Fixed a problem with the translation of the document
  final_parallel
  final_parallel
  DOCSUP-4162: Document the system.replicated_fetches system table (#16900)
  Update settings.md
  Update settings.md
  Less verbose logging when fetch is impossible
  Don't add tons of client coverage files in stateful tests with coverage
  More compatible watches in TestKeeper
  Trying to make read_in_order_many_parts more stable
  trigger CI
  Update version_date.tsv after release 20.6.10.2
  Update visibleWidth.cpp
  ...
2020-11-22 00:58:55 +04:00
Denis Glazachev
c12e6ae7c5 Actualize
Add role_mapping documentation (comments)
2020-11-21 00:49:29 +04:00
Tai White
43fce34eec Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into ldap-cache-login 2020-11-20 20:56:36 +01:00
alesapin
0208751a7d Add dirty hack for connections close 2020-11-20 15:46:05 +03:00
vdimir
36544a45b7
Merge remote-tracking branch 'upstream/master' into ip-dict-no-trie 2020-11-19 18:56:24 +03:00
alesapin
991386e37b Fix stupid error 2020-11-19 13:56:36 +03:00
Azat Khuzhin
dd0fccae47 Disable rotation of the logs in tests (on CI)
Right now due to rotation the archive with the clickhouse-server.log
maybe not full, for example:
- not full - https://clickhouse-test-reports.s3.yandex.net/16947/caf5f98db2ae39dd911f7d1fc0a7c2cc382b1c53/functional_stateless_tests_(ubsan)/clickhouse-server.log (no messages about server start)
- full     - https://clickhouse-test-reports.s3.yandex.net/16993/d1f52dc72d417580c4088cf3880593176416bea2/functional_stateless_tests_(thread).html

And sometimes rotated part may include relative part of the log, so it
is better to disable it.
2020-11-19 01:08:47 +03:00
Azat Khuzhin
8a7376247b Add a comment about rotation policy into server config 2020-11-19 01:07:09 +03:00
alesapin
2cd66f23a3 Merge branch 'master' into complete_zk_api 2020-11-18 17:50:50 +03:00
Alexey Milovidov
3df04ce0c2 Follow Arcadia ya.make rules 2020-11-16 21:24:58 +03:00
alesapin
4aad6a597c Merge branch 'correctly_send_close_request' into complete_zk_api 2020-11-12 17:46:02 +03:00
tavplubix
058aa8f85e
Merge pull request #16824 from ClickHouse/replace_stringstreams_with_buffers
Replace std::*stringstreams with DB::*Buffers
2020-11-12 01:11:44 +03:00
alesapin
6398271432 Merge branch 'master' into complete_zk_api 2020-11-11 16:57:24 +03:00
alesapin
d83c68fca8 Fix timeouts 2020-11-11 16:55:28 +03:00
alesapin
66236d6ebb Refactored a little 2020-11-11 16:07:06 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
alesapin
932ea0ef8b Merge branch 'master' into complete_zk_api 2020-11-10 13:56:11 +03:00
Amos Bird
aa8e6db786
reload auxiliary zookeepers configuration 2020-11-10 15:56:37 +08:00
Denis Glazachev
0e704ec278
Merge branch 'master' into ldap-cache-login 2020-11-10 00:10:22 +04:00
vdimir
a67f5b780f
Use sorted ip array instead of trie in TrieDictionary 2020-11-08 19:19:04 +03:00
alexey-milovidov
2d427eb1e8
Merge pull request #16496 from ClickHouse/register-formats
Fix inconsistency in FormatFactory
2020-11-08 14:27:33 +03:00
Azat Khuzhin
b2e17916e4 Mask password in data_path in the system.distribution_queue 2020-11-06 00:10:25 +03:00
Alexey Milovidov
6757fb6996 Add development roadmap for web UI 2020-11-05 01:17:30 +03:00
alexey-milovidov
fa929b0312
Replace duplicate development config files with symlinks. (#16486)
* Fix test "max_memory_usage_for_user"

* Update test

* Update annoying boilerplate

* Support symlinks in tarballs

* Fix Fuzzer

* Remove "secondary" user

* Remove "secondary" user
2020-11-04 23:45:06 +03:00
Vitaliy Zakaznikov
e6d04b4780 Merge branch 'master' of https://github.com/traceon/ClickHouse into ldap-cache-login 2020-11-03 16:53:23 -05:00
alesapin
c2525ef211 Server and client pinging each other 2020-10-30 17:16:47 +03:00
Alexey Milovidov
2e0a979e3a Fix inconsistency in FormatFactory 2020-10-29 06:39:43 +03:00
Alexander Kuzmenkov
6509f1eb39
Merge branch 'master' into minumum 2020-10-27 21:38:23 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell 2020-10-27 12:04:03 +01:00
Alexander Kuzmenkov
5a70e26c70 Merge remote-tracking branch 'origin/master' into HEAD 2020-10-26 19:21:13 +03:00
Denis Glazachev
f2a6696362 Implement verification_cooldown LDAP server connection param 2020-10-26 16:44:36 +04:00
Alexey Milovidov
b472bd35b7 Merge branch 'master' into sigaltstack 2020-10-26 07:42:33 +03:00
Alexey Milovidov
068235c2d7 Disable alt stack for sanitizers 2020-10-26 05:18:36 +03:00
alexey-milovidov
bfe17d7dca
Merge pull request #16358 from azat/dist-insert-fix
Fix async Distributed INSERT w/ prefer_localhost_replica=0 and internal_replication
2020-10-26 05:07:35 +03:00
Azat Khuzhin
edc8d6e5e7 Fix async Distributed INSERT w/ prefer_localhost_replica=0 and internal_replication 2020-10-25 14:27:06 +03:00
Azat Khuzhin
72d7b6117e Use total_memory_tracker when there is no other MemoryTracker object.
This should significantly reduce the MemoryTracking drift, test shows
that there is 0 drift after query storm (100 queries, via http/tcp/tcp
in one session).

TL;DR;

To track memory, clickhouse creates memory tracker object for each
thread **explicitly**, but until it is not created the memory
allocations are not under account.
There should not be lot of allocations w/o memory tracker, since most of
the time it is created early enough, but even this maybe enough to
trigger some problems.

Plus sometimes it is not possible to create it, for example some 3d
party library does not allow to do this explicitly:
- for example before #15740 allocations from librdkafka threads,
- or even worse, poco threads, they don't have any routines to do this.
This won't be a problem for `MemoryTracking` metric if the deallocation
will be done from the same thread w/o memory tracker (or vise versa),
but this is not always true.

NOTE, that this will slow down per-thread allocations w/o memory
tracker, since before this patch there were no memory tracking for them
while now they will be accounted in total_memory_tracker, and for
total_memory_tracker max_untracked_memory is always reached.
But this should not be significant.
2020-10-23 21:07:52 +03:00
Alexander Kuzmenkov
5cbf645de4 Merge remote-tracking branch 'origin/master' into tmp 2020-10-22 19:48:02 +03:00
Alexander Kuzmenkov
145e2b012f cleanup 2020-10-22 19:47:20 +03:00
Alexey Milovidov
b26e979e5c Added TODOs + minor adjustments 2020-10-21 01:13:41 +03:00
alexey-milovidov
2b00b5391a
Merge pull request #11844 from Enmk/AES_encrypt_decrypt
encrypt and decrypt functions
2020-10-20 22:57:35 +03:00
alexey-milovidov
b4f0e08369
Merge pull request #16158 from ClickHouse/minimal-ui
Added minimal web UI
2020-10-20 11:57:56 +03:00
Alexey Milovidov
3be41e55a7 Add comments 2020-10-20 04:08:30 +03:00
Alexey Milovidov
9ae144dc69 Edit comment 2020-10-20 04:05:29 +03:00
Alexey Milovidov
9b57f7cdc9 Minor changes 2020-10-20 03:39:24 +03:00
Alexey Milovidov
f8fef7eff6 Minor changes 2020-10-20 03:32:50 +03:00
Alexey Milovidov
a554ca5e62 Added /play handler; added a test 2020-10-20 03:28:33 +03:00
Alexey Milovidov
0af87e9598 Advancements and adjustments 2020-10-20 02:37:10 +03:00
Alexey Milovidov
f229185bee Comments and corrections 2020-10-19 22:25:58 +03:00
Alexey Milovidov
7b4a537538 Added minimal web UI 2020-10-19 21:29:51 +03:00
Alexey Milovidov
5e502c52b3 Merge branch 'master' into AES_encrypt_decrypt 2020-10-14 16:30:23 +03:00
Denis Glazachev
6eec447a64 Merge branch 'master' into ldap-any-user-authentication
* master: (189 commits)
  Fix ugly links in docs
  Update Internals.cpp
  Whitespace
  Fix ugliness in docs
  better
  better backticks
  Try fix tests.
  Fetch helper image before run
  Fetch helper image before run
  Fetch helper image before run
  Update version_date.tsv after release 20.7.4.11
  Update version_date.tsv after release 20.8.4.11
  Update 01515_with_global_and_with_propagation.sql
  fix compilation error
  Supress pvs
  Update version_date.tsv after release 20.9.3.45
  Add missed file
  fix thread restart for parallel quorum inserts
  Update openssl
  Fix race in openssl
  ...
2020-10-14 16:46:37 +04:00
Alexander Kuzmenkov
6200433e12 Merge origin/master into tmp (using imerge) 2020-10-12 21:15:31 +03:00
John Skopis
1f9331198d [WIP] Introduce dynamic TLS server context
This is a first pass at setting an x509 keypair dynamically.

I'm not so sure it should be a singleton (especaially because it depends
on the SSLManager singleton).

Needs more better error handling.

Functionally, it works. Simply change the cert/key path in config, or
touch cert; touch config.
2020-10-08 16:34:17 +00:00
Alexey Milovidov
7d97c8b053 Remove useless code 2020-10-07 11:21:32 +03:00
Denis Glazachev
44916af70a Merge branch 'master' into ldap-any-user-authentication
* master: (279 commits)
  Convert to python3 (#15007)
  update en and ru docs: add 128/256 types (#15515)
  Drop unused code for numeric_limits<int128> in MergeTreeDataSelectExecutor (#15519)
  Fix comment
  More defaults on defaults
  Recursive defaults
  Fix bash
  Update ActionsVisitor.cpp
  Don't add memcpy for sanitizers builds
  Fix gRPC build scripts.
  Neutrialize thinlto's memcpy libcall gen.
  Revert "Add fno-builtin-memcpy"
  Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it
  Update SECURITY.md
  Update tsan_suppressions.txt
  Update ActionsVisitor.cpp
  remove unrelated changes
  Update SECURITY.md
  fixes
  Update ActionsVisitor.cpp
  ...
2020-10-03 00:45:57 +04:00
Alexander Kuzmenkov
244687bfc1 Merge remote-tracking branch 'origin/master' into HEAD 2020-10-01 18:41:48 +03:00
Alexander Kuzmenkov
626c2a3e28 microsecond precision for start/finish time 2020-09-30 19:16:33 +03:00
tavplubix
4578d89819
Merge pull request #15348 from ClickHouse/replicate_default_args
Continuation of #14791
2020-09-28 18:53:02 +03:00
Alexander Kuzmenkov
218d86de7e Merge remote-tracking branch 'origin/master' into tmp 2020-09-28 16:29:51 +03:00
Alexander Tokmakov
34addcf61f add review suggestions 2020-09-26 22:18:28 +03:00
Denis Glazachev
8f2c23e6de Merge branch 'master' into ldap-any-user-authentication
* master: (224 commits)
  Update in.md
  Add a test for embedded configs
  Allow to run without /proc/self/maps
  Update adopters.md
  More convenient
  Enable embedded configs for AArch64
  More results
  More handy
  More handy
  Added a comment
  Adjustments
  Adjustments
  Adjustments
  Added new results
  Step 1: make adding hardware benchmark results more convenient
  Revert "Avoid deadlocks in Log/TinyLog"
  Fix MSan report in QueryLog
  add some disabled tests from arcadia to skip_list.json
  Update skip_list.json
  Bump CI. [2]
  ...

# Conflicts:
#	docker/test/testflows/runner/Dockerfile
2020-09-25 17:33:50 +04:00
alesapin
97671a1771 Ugly fix for default database race 2020-09-25 14:27:00 +03:00
Alexey Milovidov
1f6e55ff35 Allow to run without /proc/self/maps 2020-09-25 05:03:58 +03:00
Alexey Milovidov
f5eead5b2a Enable embedded configs for AArch64 2020-09-25 04:45:47 +03:00
Alexander Kuzmenkov
679c0988a5 Merge remote-tracking branch 'origin/master' into HEAD 2020-09-22 16:25:52 +03:00
Alexander Kuzmenkov
dde4cf70e1 Merge remote-tracking branch 'origin/master' into HEAD 2020-09-22 14:03:59 +03:00
Alexander Kuzmenkov
7b64ca33b1 Merge remote-tracking branch 'origin/master' into tmp 2020-09-22 14:02:20 +03:00
Alexander Kuzmenkov
bfc3be9e43 Merge remote-tracking branch 'origin/aku/tcp-versio' into tmp 2020-09-22 14:02:15 +03:00
Alexander Kuzmenkov
478c7309d4
Merge pull request #14843 from ClickHouse/aku/global-pool
Exception on double init of global thread pool
2020-09-22 13:44:39 +03:00
alexey-milovidov
3e0509af69
Merge pull request #13573 from amosbird/rms
specific ReplicatedMergeTree settings
2020-09-21 01:36:33 +03:00
Denis Glazachev
442b1407cf Merge branch 'master' into ldap-any-user-authentication
* master: (375 commits)
  Update type-conversion-functions.md
  Update maxmap.md
  Update maxmap.md
  Update maxmap.md
  Update single_fixed_string_groupby.xml
  Alter remove column properties and TTLs (#14742)
  better fixed string group by support
  Fix incorrect key condition of fixed strings.
  constant output order
  more tests for  #14646
  Maybe fix MSan report in base64
  Proper exception message for wrong number of arguments of CAST
  Added a test
  Fix buffer overflow in "bar" function
  Update convertMySQLDataType.cpp
  Fix clang-tidy
  Remove obsolete code from performance test
  Slightly better code
  Even more
  Even more
  ...

# Conflicts:
#	src/Interpreters/Context.cpp
2020-09-21 00:31:04 +04:00
alexey-milovidov
1474b80ecc
Merge pull request #15008 from ClickHouse/clang-tidy-11-without-enabling
Fix terribly wrong code
2020-09-20 02:00:54 +03:00
alexey-milovidov
c461d782e4
Merge pull request #14929 from ClickHouse/mlock-code-segment
Only mlock code segment
2020-09-19 13:46:49 +03:00
Alexey Milovidov
ef5363b87f Fix terribly wrong code 2020-09-19 02:05:13 +03:00
alexey-milovidov
988b20a32c
Merge pull request #14684 from azat/parallel-distributed_ddl
Allow parallel execution of distributed DDL
2020-09-18 22:18:17 +03:00
Alexander Kuzmenkov
449189dcda Initialize global thread pool before we fetch configs from ZK 2020-09-18 13:47:09 +03:00
Alexey Milovidov
799a675b5d Merge branch 'master' into AES_encrypt_decrypt 2020-09-17 21:42:51 +03:00
Alexey Milovidov
eb523f1a7c Merge branch 'master' into mlock-code-segment 2020-09-17 21:15:09 +03:00
Alexander Kuzmenkov
fb64cf210a straighten the protocol version 2020-09-17 17:37:29 +03:00
Vitaly Baranov
3356d75b23
Merge pull request #13156 from azat/cluster-secure
Secure inter-cluster query execution (with initial_user as current query user) [v3]
2020-09-17 17:11:00 +03:00
Alexey Milovidov
068e8576b7 Corrections 2020-09-17 15:53:52 +03:00
Alexey Milovidov
8bce20076c Only mlock code segment 2020-09-17 15:39:37 +03:00
Alexander Kuzmenkov
a374541214 straighten the protocol version 2020-09-17 15:15:05 +03:00
Alexander Kuzmenkov
6217dd77b5 Merge remote-tracking branch 'origin/master' into tmp 2020-09-17 12:35:10 +03:00
sundy-li
544b2cb20d add configChanged method for zookeeper
fix logic error && skip reload testkeeper
2020-09-17 13:33:45 +08:00
alexey-milovidov
3d9ec963e9
Update query_masking_rules.xml 2020-09-15 15:49:16 +03:00
alexey-milovidov
018f596d21
Merge pull request #14685 from ClickHouse/remap-executable
Attempt to make performance test more reliable
2020-09-15 15:09:47 +03:00
Azat Khuzhin
0159c74f21 Secure inter-cluster query execution (with initial_user as current query user) [v3]
Add inter-server cluster secret, it is used for Distributed queries
inside cluster, you can configure in the configuration file:

  <remote_servers>
      <logs>
          <shard>
              <secret>foobar</secret> <!-- empty -- works as before -->
              ...
          </shard>
      </logs>
  </remote_servers>

And this will allow clickhouse to make sure that the query was not
faked, and was issued from the node that knows the secret. And since
trust appeared it can use initial_user for query execution, this will
apply correct *_for_user (since with inter-server secret enabled, the
query will be executed from the same user on the shards as on initator,
unlike "default" user w/o it).

v2: Change user to the initial_user for Distributed queries if secret match
v3: Add Protocol::Cluster package
v4: Drop Protocol::Cluster and use plain Protocol::Hello + user marker
v5: Do not use user from Hello for cluster-secure (superfluous)
2020-09-15 01:36:28 +03:00
Alexey Milovidov
7aa3f86ab9 Added config.d file for development 2020-09-14 22:44:20 +03:00
Alexey Milovidov
2ac88ab47d Added config option 2020-09-14 21:08:09 +03:00
Alexey Milovidov
0e73b8acf3 Merge branch 'master' into remap-executable 2020-09-14 20:26:16 +03:00
Vasily Nemkov
30b1831752 Moved default values for query_masking rules for encrypt/decrypt to config.xml 2020-09-14 18:15:07 +03:00
Denis Glazachev
c202364f01
Merge branch 'master' into ldap-any-user-authentication 2020-09-14 18:53:46 +04:00
Vitaly Baranov
421eeeccef Add the section user_directories to the default config. 2020-09-12 23:04:28 +03:00
Azat Khuzhin
dd867b787f Allow parallel execution of distributed DDL
Add distributed_ddl.pool_size to control maximum parallel to handle
distributed DDL.

Also:
- convert Exception constructors to fmt-like
- use sleepFor* over std::this_thread::sleep_for()
2020-09-12 02:32:08 +03:00
Alexey Milovidov
a644733139 Attempt to make performance test more reliable 2020-09-10 12:05:57 +03:00
Alexander Kuzmenkov
c62bf555d2 Merge origin/master into tmp (using imerge) 2020-09-08 16:26:38 +03:00
Alexander Kuzmenkov
d0a9926e7d fixes and some docs 2020-09-08 16:19:27 +03:00
Alexey Milovidov
e3924b8057 Fix "Arcadia" 2020-09-08 01:14:13 +03:00
Denis Glazachev
59e0c10aaf Merge branch 'master' into ldap-any-user-authentication 2020-09-03 19:36:19 +04:00
Yatsishin Ilya
45e54f81c7 better resolv.conf, add DNSCacheUpdater logs 2020-09-02 11:07:46 +03:00
Vasily Nemkov
b147ffcd43 encrypt, aes_encrypt_mysql, decrypt, aes_decrypt_mysql functions
Functions to encrypt/decrypt any input data with OpenSSL's ciphers
with custom key, iv, and add (-gcm mode only).

_mysql versions are 100% compatitable with corresponding MySQL functions

Supported modes depend on OpenSSL version, but generally are:
aes-{128,192,56}-{ecb,cbc,cfb1,cfb8,cfb128,ofb,gcm}

Please note that in a -gcm mode a 16-byte tag is appended to the ciphertext
on encryption and is expected to be found at the end of ciphertext on decryption.

Added tests that verify compatibility with MySQL functions,
and test vectors for GCM mode from OpenSSL.

Added masking rules for aes_X funtions
Rules are installed by default to config.d/query_masking_rules.xml
2020-09-01 12:42:15 +03:00
Denis Glazachev
9b4cd06051 Merge branch 'master' into ldap-any-user-authentication
* master: (414 commits)
  Add .reference
  Update 01460_mark_inclusion_search_crash.sql
  Fix bug in mark inclusion search.
  cosmetic fixes for performance report
  cosmetic fixes in performance report
  Add more docs about functional tests
  Add ability to specify Default codec for columns (#14049)
  better
  done
  Disable force TTL on optimise
  Update CMakeLists.txt
  Update custom parts of storage AST only if it has extended definition
  DOCSUP-712: Documented the ttl_only_drop_parts setting (#13823)
  Small fixes
  gix comment and useDefault*(), add tests for nullables
  Fix a build for old some OS with old find
  Update hdfs.md
  Fix 01085_max_distributed_connections flackiness
  Fix 00974_distributed_join_on flackiness (by allow retries and hide logs)
  Update adopters.md
  ...
2020-08-30 11:56:38 +04:00
Amos Bird
05a5a13e08
specific ReplicatedMergeTree settings 2020-08-27 22:37:41 +08:00
alexey-milovidov
0f706c01ca
Merge pull request #13888 from vladimir-golovchenko/add-date-trunc-function
Added date_trunc function
2020-08-27 02:12:27 +03:00
vladimir golovchenko
f3cfd39003 Fixed .gitignore to exclude tests-generated data. 2020-08-25 16:41:23 -07:00
Alexander Kuzmenkov
05ad9b9fff opentelemetry wip 2020-08-20 23:59:58 +03:00
Denis Glazachev
ec52a165af Style fixes 2020-08-20 12:46:42 +04:00
Denis Glazachev
bdfea652c1 Change user_template to roles
Change top_enclosing_storage to access_control_manager
Simplify the lookup in peer storages
2020-08-20 11:39:27 +04:00
Denis Glazachev
58f73ff041 Merge branch 'master' into ldap-any-user-authentication
* master: (30 commits)
  Documentation improve:  Translate [select] section into Chinese to improve readability (#13814)
  Update adopters.md
  partially disable test with MaterializeMySQL
  one more unroll for arerage.cpp
  Update adopters.md
  Update adopters.md
  Update adopters.md
  Update adopters.md
  fix #13819
  DOCSUP-1888: Documented the input_format_avro_allow_missing_fields setting (#13671)
  Fix "Arcadia" and "Unbundled" builds
  DOCSUP-928: Documented the groupArraySample function (#13791)
  Documentation about ReplacingMergeTree extended with type DateTime64 for column (#13498)
  doc: update quotas.md (#13400)
  Fix 01356_initialize_aggregation in unbundled build (change topKWeighted order)
  Fix style
  Add normalizedQueryHash function with tests
  Fix topK/topKWeighted merge (wtih non-default parameters)
  Add test
  Add function "normalizeQuery"
  ...

# Conflicts:
#	programs/server/Server.cpp
#	src/Access/AccessControlManager.cpp
#	src/Access/AccessControlManager.h
2020-08-18 14:54:02 +04:00
Denis Glazachev
fbbaf645b9 Merge branch 'master' into ldap-any-user-authentication
* master: (70 commits)
  Fix terrible markup due to wrong @blinkov's script.
  Update extended-roadmap.md
  Update extended-roadmap.md
  Fix bug on the website
  Suppress Shellcheck
  Fix typos
  Fix typos
  Install codespell
  Fix typo
  Update gtest_thread_pool_schedule_exception.cpp
  Fix flaky unit test
  Fix set index with const column pred
  Updating healthcheck parameters.
  Fix step overflow in range()
  * Moving to standard healthcheck parameters. * Adding docker-compose pull and docker-compose down before executing docker-compose up. * Adding docker-compose ps if any of the containers are unhealthy.
  Fix topK test
  Adding missing certificates and dhparam.pem for openldap4.
  Keep "metadata_loading" test only for release build
  Fix error in test
  Fix typo
  ...
2020-08-17 18:08:24 +04:00
Vitaly Baranov
0759dff12b Support <user_directories> section in the main config. 2020-08-16 19:15:38 +03:00
Vitaly Baranov
2909ed1bc0 Better initialization of access storages. Make list of access storages dynamic. 2020-08-16 19:15:34 +03:00
Alexey Milovidov
e43746395e Merge branch 'master' into codespell-2 2020-08-16 14:57:21 +03:00
Alexey Milovidov
6bc8da633c Minor modification 2020-08-16 14:52:55 +03:00
Alexey Milovidov
8e36bfe54d Merge branch 'master' into watchdog 2020-08-16 14:37:55 +03:00
Denis Glazachev
7375dc5d66 Merge branch 'master' into ldap-any-user-authentication
* master: (956 commits)
  Remove ZooKeeper from unit tests
  Revert check location of Docker compose files @qoega.
  Update developer-instruction.md
  trigger the CI
  Update settings.md
  Update settings.md
  Fix
  fix sync 2
  Fix tests.
  Add docker for style check
  fix sync 1
  passwd and group location error
  Hotfix for pushdown with StorageMerge (#13679)
  Fix error with batch aggregation and -Array combinator
  Fix cassandra build on macos
  Update adopters.md
  Update adopters.md
  Fix build after merge
  Fix shared build
  log error message
  ...

# Conflicts:
#	tests/testflows/ldap/docker-compose/clickhouse-service.yml
2020-08-15 14:18:40 +04:00
Alexander Tokmakov
a6ff049eec use Atomic for system database 2020-08-12 23:40:13 +03:00
Alexey Milovidov
c7a6a18a75 Preparation to enable clickhouse install on Mac and FreeBSD 2020-08-10 02:58:09 +03:00
Alexey Milovidov
4129ad5bbf Merge branch 'embed-configs' into clickhouse-install 2020-08-09 05:17:12 +03:00
Alexey Milovidov
aa3b4bbfe0 Fix build 2020-08-09 05:16:55 +03:00
Alexey Milovidov
8d616444e4 Support for AArch64 2020-08-09 02:54:38 +03:00
Alexey Milovidov
182e2929f3 Merge branch 'master' into embed-configs 2020-08-09 02:41:44 +03:00
Alexey Milovidov
f459640269 Fix build 2020-08-09 02:41:34 +03:00
Alexey Milovidov
6c4df0f27a Better tool 2020-08-08 17:10:49 +03:00
Alexey Milovidov
ae716e13e0 Watchdog (experimental) 2020-08-08 07:52:09 +03:00
Alexey Milovidov
3b8020168f Add simple watchdog 2020-08-08 07:44:04 +03:00
Alexey Milovidov
0f79eb3cc5 Embed configs into binary 2020-08-08 06:42:42 +03:00
Alexey Milovidov
0cbbe153cd Fix typos, the last 1% 2020-08-08 04:21:04 +03:00
Vitaly Baranov
1a4a8a219c
Merge pull request #13305 from vitlibar/correct-error-message-if-setting-not-found-in-users_xml
Correct error message if setting not found in users.xml
2020-08-07 23:31:23 +03:00
alexey-milovidov
9cb0914bbd
Merge pull request #13447 from vladimir-golovchenko/fix-server-gitignore
Updated gitignore-files.
2020-08-07 05:01:33 +03:00
vladimir golovchenko
cb153d2605 Updated gitignore-files. 2020-08-06 18:05:32 -07:00
Vitaly Baranov
a804f9499d Use references while iterating through settings. 2020-08-04 04:00:38 +03:00
Vitaly Baranov
dadebadcac Print correct error message in log for unknown settings in users.xml 2020-08-04 00:20:33 +03:00
Alexey Milovidov
778abb346f Merge branch 'master' into merge-tree-settings-sanity-check 2020-08-02 17:13:17 +03:00
Vitaly Baranov
18b21511a9
Merge pull request #13013 from vitlibar/implement-custom-settings
Implement custom settings
2020-08-02 05:01:14 +03:00
Alexey Milovidov
53450b1b09 Merge branch 'master' into merge-tree-settings-sanity-check 2020-08-02 02:47:57 +03:00
Vitaly Baranov
7c4ae5ee65 Add the parameter custom_settings_prefixes to the server config. 2020-07-31 20:57:49 +03:00
Vitaly Baranov
56665a15f7 Rework and rename the template class SettingsCollection => BaseSettings. 2020-07-31 20:54:18 +03:00
Alexey Milovidov
c3ad710b84 Merge branch 'master' into crash-log 2020-07-31 16:12:53 +03:00
Alexey Milovidov
b9f49d31df Sanity checks for MergeTreeSettings 2020-07-30 22:08:13 +03:00
Denis Glazachev
479fa4c325 Improve LDAP-related comments 2020-07-27 14:24:56 +04:00
Denis Glazachev
90a064c7a6 Fix compilation 2020-07-24 19:39:18 +04:00
Denis Glazachev
3b3404c326 Style fix
Remove unused declarations
2020-07-23 22:10:57 +04:00
Denis Glazachev
5d6b5101fe Implement LDAPAccessStorage and integrate it into AccessControlManager
Rename ExternalAuthenticators::setConfig to setConfiguration
Revisit LDAP servers config section comments
Add user_directories config section with comments (only for ldap)
Fix bug in MemoryAccessStorage::insertImpl
2020-07-23 21:55:24 +04:00
Denis Glazachev
8688a1f5d0 Recreate ExternalAuthenticators (LDAP) on config update 2020-07-10 15:59:48 +04:00
Alexey Milovidov
31cbdd1a56 system.crash_log: development 2020-07-09 07:15:45 +03:00
alesapin
1aa45f203b
bump CI 2020-07-08 13:40:02 +03:00
Denis Glazachev
9effacfbc1 Merge branch 'master' into ldap-per-user-authentication
* master: (1102 commits)
  Update README.md
  Update README.md
  Update README.md
  Update index.md
  [docs] add intrdocution for statements page (#12189)
  Revert "Run perf tests with memory sampling (for allocations >1M)"
  Sanitize LINK_LIBRARIES property for the directories (#12160)
  [docs] refactor Domains overview (#12186)
  DOCS-647: toStartOfSecond (#12190)
  [docs] add intrdocution for commercial page (#12187)
  DOCSUP-1348 Russian translation for new functions (#133) (#12194)
  changelog fixes
  Update index.md (#12191)
  Update zh kafka.md title (#12192)
  Added test for #3767
  style fix for #12152
  Tests for fixed issues #10846 and #7347
  changelog fixes
  [docs] introduction for special table engines (#12170)
  [docs] introduction for third-party interfaces (#12175)
  ...

# Conflicts:
#	src/Access/ya.make
#	src/Common/ErrorCodes.cpp
2020-07-08 00:42:09 +04:00
Nikita Mikhaylov
5c6d6bdf54
Merge pull request #11995 from azat/load_balancing-priority
Load balancing manual priority
2020-07-06 13:32:20 +04:00
alexey-milovidov
f1151ed474
Merge pull request #12133 from ClickHouse/normalize-pid-file
Normalize "pid" file handling #3501
2020-07-04 21:36:27 +03:00
Alexey Milovidov
176a7f2f72 Normalize "pid" file handling #3501 2020-07-04 16:54:24 +03:00
alesapin
c57edd2018 Tiny fixes 2020-07-03 11:02:35 +03:00
alesapin
64583ceb22 Merge branch 'master' into add-storage-rabbitmq-read-only 2020-07-02 16:08:29 +03:00
Azat Khuzhin
ebff4eae7d Add replica priority for load_balancing
Make default 1, to match with <weight>
2020-06-29 23:03:28 +03:00
Azat Khuzhin
e547f6b6d7 Add <weight> example in <remote_servers> into dist config 2020-06-29 23:03:28 +03:00
Alexey Milovidov
44f2742a51 Fix bad log message at server startup 2020-06-27 15:56:06 +03:00
Alexander Kuzmenkov
d77f397b38 review fixes 2020-06-26 03:18:33 +03:00
alexey-milovidov
a34032cace
Merge pull request #11300 from blinkov/sentry
Opt-in support for sending crash reports
2020-06-25 17:20:50 +03:00
Alexander Kuzmenkov
593a0181bd
Merge pull request #11616 from ClickHouse/aku/perf-benchmark
Add concurrent benchmark to performance test
2020-06-25 09:36:52 +03:00
alesapin
3fc65b3269 Merge branch 'master' into kssenii-rabbit-mq 2020-06-24 20:14:28 +03:00
Alexander Kuzmenkov
f0fbba5522 allow implicit initialization 2020-06-23 20:31:46 +03:00
Denis Glazachev
d17d72b075 More generic DN in the example 2020-06-23 18:23:34 +04:00
Alexander Kuzmenkov
e0bdbe73d2 Merge remote-tracking branch 'origin/master' into HEAD 2020-06-23 15:31:09 +03:00
Alexander Kuzmenkov
ab809f59b9 memory usage settings 2020-06-23 15:30:45 +03:00
Alexander Kuzmenkov
96d2e9c997 Initialize GlobalThreadPool explicitly 2020-06-22 22:04:12 +03:00
Alexander Kuzmenkov
bd5ab9c686 Merge remote-tracking branch 'origin/master' into bnc/config-max-num-threads 2020-06-22 15:55:39 +03:00
Ivan Blinkov
2c0ff29c48 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-22 10:06:36 +03:00
alexey-milovidov
4ee623ccac
Merge pull request #10242 from MovElb/movelb-postgresql-wire-protocol-impl
PostgreSQL wire protocol implementation
2020-06-21 14:39:22 +03:00
Denis Glazachev
5db60202b6 Merge branch 'master' into ldap-per-user-authentication 2020-06-19 00:11:08 +04:00
Ivan Blinkov
8b50e3450b move the default endpoint to config 2020-06-16 23:01:15 +03:00
Denis Glazachev
6338225c50 Fix tls_cipher_suite example 2020-06-16 17:49:18 +04:00
Ivan Blinkov
7e2bb1fd83 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-16 16:02:55 +03:00
Ivan Blinkov
0e77692a27 improvements after review comments 2020-06-16 15:56:28 +03:00
Bharat Nallan
c43bd228ab make max global thread pool setting configurable
This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.
2020-06-15 22:04:30 -07:00
Alexey Milovidov
bb6c0743fc Change the level of log message about failure to listen, to warning #4406 2020-06-15 23:30:36 +03:00
Alexey Milovidov
a421e7e4b4 Added a test 2020-06-14 00:13:52 +03:00
Denis Glazachev
276fcd8903 Add/rename parameters that control TLS 2020-06-12 21:59:47 +04:00
Ivan Blinkov
e98a91e9f7 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-12 16:53:00 +03:00
Alexander Kuzmenkov
e354bbc78c
Merge pull request #11588 from ClickHouse/aku/async-log
Add system.asynchronous_metric_log
2020-06-11 16:10:35 +03:00
Ivan Blinkov
b70ed921ea Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-11 12:22:13 +03:00
kssenii
18820814f5 Merge 2020-06-10 23:25:33 +00:00
Denis Glazachev
9e3a28a6b8 Merge branch 'master' into ldap-per-user-authentication
* master: (414 commits)
  Update file.md
  Update merge.md
  Update dictionary.md
  Update external-data.md
  Update distributed.md
  Update null.md
  Update set.md
  Update join.md
  Update url.md
  Update view.md
  Update materializedview.md
  Update memory.md
  Update buffer.md
  Update generate.md
  removed a sentence about global lock during rename (#11577)
  greatCircleAngle en translation (#11584)
  Update configuration-files.md
  try fix flacky test
  Update why.html
  Update rich.html
  ...

# Conflicts:
#	src/Common/ErrorCodes.cpp
#	utils/ci/jobs/quick-build/run.sh
2020-06-11 03:06:17 +04:00
Denis Glazachev
848330b37a Expect <ldap_servers> in main config.xml 2020-06-11 02:48:15 +04:00
Alexander Kuzmenkov
a6648516f4 Add files; flush more often 2020-06-11 00:16:58 +03:00
Alexey Milovidov
fd3ff19868 Fix trivial error in log message #11399 2020-06-10 22:34:23 +03:00
Alexander Kuzmenkov
e4be52f35f Add system.asynchronous_metric_log 2020-06-10 22:17:54 +03:00
Ivan Blinkov
5b3fef526e Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-09 11:51:29 +03:00
Alexey Milovidov
df19db1509 Added a test for history in clickhouse-client 2020-06-07 20:29:34 +03:00
kssenii
68b94c5c20 Fixes 2020-06-05 13:46:20 +00:00
Alexey Milovidov
3d68cd4df6 Also check for users.xml elements in config.xml 2020-06-04 23:34:33 +03:00
Alexey Milovidov
02e14f9fe8 Also check for users.xml elements in config.xml 2020-06-04 23:33:19 +03:00
Alexey Milovidov
a89ce20d38 Added a check for incorrect settings 2020-06-04 22:30:30 +03:00
Denis Glazachev
c61cbe8e98 Merge branch 'master' into ldap-per-user-authentication
* master: (114 commits)
  Update PushingToViewsBlockOutputStream.cpp
  Update PushingToViewsBlockOutputStream.cpp
  make clang-10 happy
  Fix sync_async test (remove timeout)
  CLICKHOUSEDOCS-631: temporary_files_codec, join_on_disk_max_files_to_merge settings. (#11242)
  Suppress output of cancelled queries in clickhouse-client #9473
  Better log messages in ConfigReloader
  fix select from StorageJoin
  Fix unit tests under MSan
  Added test.
  Fix build.
  Fix arguments for AggregateFunctionQuantile/
  Update style.md
  Add a guide on error messages.
  Report dictionary name on dictionary load errors.
  more types in ASOF JOIN (#11301)
  Fix part_log test
  Update test.
  Add perftest.
  Parallel processing for PushingToViewsBlockOutputStream::writeSuffix
  ...
2020-06-02 12:42:29 +04:00
Ivan Blinkov
6bd4282727 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-01 21:49:56 +03:00
alexey-milovidov
19c6e0e5ae
Merge pull request #11307 from ClickHouse/remove-system-tables-lazy-load
Remove "system_tables_lazy_load" option
2020-06-01 16:26:23 +03:00
Ivan Blinkov
fcc7ff0ef9 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-01 08:42:03 +03:00
Alexey Milovidov
87cf123ef8 Merge branch 'master' into remove-system-tables-lazy-load 2020-05-31 18:06:08 +03:00
Denis Glazachev
e8144976fe Merge branch 'master' into ldap-per-user-authentication 2020-05-31 15:09:16 +04:00
Alexey Milovidov
25f941020b Remove namespace pollution 2020-05-31 00:57:37 +03:00
MovElb
142c63e487 done rebase 2020-05-30 23:02:11 +03:00
MovElb
b4b5c90343 squash 2020-05-30 20:05:05 +03:00
Alexey Milovidov
496fed5ed3 Remove "system_tables_lazy_load" option 2020-05-30 17:24:09 +03:00
Ivan Blinkov
d154415a5b adjust comments 2020-05-30 11:13:04 +03:00
Alexey Milovidov
b79020d0a8 Emit a warning if server was build in debug or with sanitizers 2020-05-30 00:41:33 +03:00
Ivan Blinkov
d033941399 try to fix merge issues 2020-05-29 23:20:28 +03:00
Ivan Blinkov
0386e526b2 grammar 2020-05-29 23:03:59 +03:00
Ivan Blinkov
95ee742ee7 merge master 2020-05-29 22:56:10 +03:00
Ivan Blinkov
4ef322274d Add integration test 2020-05-29 22:53:16 +03:00
Denis Glazachev
2863de750e Merge branch 'master' into ldap-per-user-authentication 2020-05-28 01:30:52 +04:00
Denis Glazachev
d74f1357d4 Add LDAP authentication support 2020-05-28 01:06:33 +04:00
Ivan Blinkov
709b4f42c8 Prototype sending crash reports on segfaults 2020-05-27 22:11:04 +03:00
Ivan
5b858de374
Split programs/server into actual program and library (#11186)
* Split programs/server into actual program and library
2020-05-27 20:52:52 +03:00
Alexey Milovidov
7e1813825b Return old names of macros 2020-05-24 01:24:01 +03:00
Alexey Milovidov
61cc605ee7 Remove old method 2020-05-24 01:21:29 +03:00
Alexey Milovidov
cd6d3c7509 Progress on task 2020-05-23 22:55:06 +03:00
Alexey Milovidov
f13ccfc8e8 Progress on task 2020-05-23 22:43:13 +03:00
Alexey Milovidov
7e2fb9ad65 Apply all transformations again 2020-05-23 22:38:30 +03:00
Alexey Milovidov
29762240de Remove duplicate whitespaces (preparation) 2020-05-23 22:31:54 +03:00
Alexey Milovidov
86b4d5a86f Remove duplicate whitespaces (preparation) 2020-05-23 22:09:06 +03:00
Alexey Milovidov
9d2a0d2dd7 Apply all transformations again 2020-05-23 21:59:49 +03:00
Alexey Milovidov
a2ad11897f Remove duplicate whitespaces (preparation) 2020-05-23 21:53:58 +03:00
Alexey Milovidov
1f13515a65 Make all LOG in single line (preparation) 2020-05-23 21:31:37 +03:00
Alexey Milovidov
3a09f9ca1c find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_[^\W_]+\([^,]+, [^<+)"]+\);' | xargs sed -i -r -e 's/(LOG_[^\W_]+)\(([^,]+), ([^<+)"]+)\);/\1_FORMATTED(\2, \3);/' 2020-05-23 20:29:56 +03:00
Alexey Milovidov
f69cbdcbfc find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" \+ [^+]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" \+ ([^+]+)\);/\1_FORMATTED(\2, "\3{}", \4);/' 2020-05-23 20:09:37 +03:00
Alexey Milovidov
533f86278a find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5{}\7", \4, \6);/' 2020-05-23 20:00:41 +03:00
Alexey Milovidov
8042e5febe find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}\5{}", \4, \6);/' 2020-05-23 19:58:15 +03:00
Alexey Milovidov
e391b77d81 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/' 2020-05-23 19:56:05 +03:00
Alexey Milovidov
ee4ffbc332 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}", \4);/' 2020-05-23 19:47:56 +03:00
Alexey Milovidov
8d2e80a5e2 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+"\)' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+, "[^"]+")\)/\1_FORMATTED(\2)/' 2020-05-23 19:42:39 +03:00
Alexey Milovidov
1a67c89c26 Enable mlock by default 2020-05-22 14:31:27 +03:00
Nikolai Kochetov
f7456f8320
Merge pull request #11004 from ClickHouse/remove-experimental-use-processors-flag-2
Remove allow_processors flag from executeQuery()
2020-05-22 10:47:15 +03:00
alexey-milovidov
2d9433242e
Merge pull request #11097 from ClickHouse/azat-memory-tracking-2
Merging #11022
2020-05-21 20:19:59 +03:00
alexey-milovidov
765510d5c3
Merge pull request #10991 from ClickHouse/enable-clang-tidy-for-programs-and-utils
Enable clang-tidy for programs and utils
2020-05-21 15:12:07 +03:00
Azat Khuzhin
09474ebea5 Fix memory tracking for two-level GROUP BY when not all rows read from Aggregator (TCP)
Example of such cases:
- SELECT GROUP BY LIMIT
- SELECT GROUP BY with subsequent MEMORY_LIMIT_EXCEEDED error

And it should be two-level aggregation, since otherwise there will be
only one hashtable which will be cleared correctly, only if you have
two-level GROUP BY some of hashtables will not be cleared since nobody
consume rows.

Before this patch:
    09:39.015292 [ 3070801 ] {609a0610-e377-4132-9cf3-f49454cf3c96} <Information> executeQuery: Read 1000000 rows, 7.63 MiB in 0.707 sec., 1413826 rows/sec., 10.79 MiB/sec.
    09:39.015348 [ 3070801 ] {609a0610-e377-4132-9cf3-f49454cf3c96} <Debug> MemoryTracker: Peak memory usage (for query): 51.93 MiB.
    09:39.015942 [ 3070801 ] {} <Trace> Aggregator: Destroying aggregate states <-- **problem**
    09:39.017057 [ 3070801 ] {} <Trace> Aggregator: Destroying aggregate states <--
    09:39.017961 [ 3070801 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 51.93 MiB.
    09:39.018029 [ 3070801 ] {} <Information> TCPHandler: Processed in 0.711 sec.

After this patch:
    16:24.544030 [ 3087333 ] {79da208a-b3c0-48d4-9943-c974a3cbb6ea} <Information> executeQuery: Read 1000000 rows, 7.63 MiB in 0.599 sec., 1670199 rows/sec., 12.74 MiB/sec.
    16:24.544084 [ 3087333 ] {79da208a-b3c0-48d4-9943-c974a3cbb6ea} <Debug> MemoryTracker: Peak memory usage (for query): 72.11 MiB.
    16:24.544398 [ 3087333 ] {79da208a-b3c0-48d4-9943-c974a3cbb6ea} <Trace> Aggregator: Destroying aggregate states
    16:24.545485 [ 3087333 ] {79da208a-b3c0-48d4-9943-c974a3cbb6ea} <Trace> Aggregator: Destroying aggregate states
    16:24.547053 [ 3087333 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 72.11 MiB.
    16:24.547093 [ 3087333 ] {} <Information> TCPHandler: Processed in 0.603 sec.
2020-05-21 02:06:55 +03:00
Nikolai Kochetov
1183bad4d0 Fix build. 2020-05-20 22:01:36 +03:00
alexey-milovidov
07924f0ae1
Merge pull request #11029 from azat/system_tables_lazy_load
Disable system_tables_lazy_load by default
2020-05-20 21:15:09 +03:00
alesapin
a416813597
Merge pull request #10777 from NanoBjorn/refactor-reservations
VolumePtr instead of DiskPtr in MergeTreeData*
2020-05-19 13:27:11 +03:00
Azat Khuzhin
ef01eb954f Add note about system_tables_lazy_load into config.xml 2020-05-19 02:41:18 +03:00
Artem Zuikov
97c7447130
Bitonic sort improvements (#10934) 2020-05-19 00:41:23 +03:00
alexey-milovidov
551fbf8519
Update HTTPHandlerFactory.cpp 2020-05-18 22:46:32 +03:00
Artem Zuikov
bfcbc08bba arcadia sync fixes 2020-05-18 15:50:23 +03:00
Artem Zuikov
0fe723acc9
Try to fix arcadia sync (#10959) 2020-05-18 13:26:23 +03:00
alesapin
07cb21ccb7
Merge branch 'master' into refactor-reservations 2020-05-18 11:43:48 +03:00
Alexey Milovidov
f54435e7fd Fix clang-tidy 2020-05-18 11:08:55 +03:00
Alexey Milovidov
f4ac1eec04 Merge branch 'master' into quota-key-in-client 2020-05-17 08:15:13 +03:00
Gleb Novikov
1a25ac6e1f Merge branch 'master' into refactor-reservations 2020-05-16 23:34:45 +03:00
Nikolai Kochetov
caab379602
Merge pull request #10932 from ClickHouse/pulling-executor
Add PullingPipelineExecutor.
2020-05-16 23:09:18 +03:00
Nikolai Kochetov
6ac93e9a61 Try simplify code. 2020-05-15 22:20:02 +03:00
Nikolai Kochetov
83a35a7390 try fix tests. 2020-05-15 20:41:12 +03:00
alesapin
9ae37a054c Fix build without openCL 2020-05-15 11:31:32 +03:00
Artem Zuikov
8e8a2a17d6 build fixes 2020-05-15 03:01:31 +03:00
Ri
fc7afaa639
Bitonic sort on GPU (OpenCL) (#10232) 2020-05-15 03:01:14 +03:00
Nikolai Kochetov
681955661f Add PullingPipelineExecutor. 2020-05-15 00:03:38 +03:00
alexey-milovidov
ddc84163a7
Merge pull request #10486 from azat/dist-send-on-INSERT
Fix distributed send that are scheduled by INSERT query
2020-05-11 06:28:35 +03:00
Gleb Novikov
57b6fb6d80 Merge branch 'master' into refactor-reservations 2020-05-10 13:01:45 +03:00
Alexey Milovidov
894a2ccb4b Merge branch 'master' into even-more-warnings 2020-05-10 06:07:50 +03:00
Alexey Milovidov
9872964290 Enable "allow_introspection_functions" for development 2020-05-10 04:18:16 +03:00
Alexey Milovidov
4a13aba8de Fix build 2020-05-10 02:34:24 +03:00
Alexey Milovidov
a576a4fbcd Enable extra warnings for base, utils, programs 2020-05-10 01:59:34 +03:00
Gleb Novikov
390b39b272 VolumePtr instead of DiskPtr in MergeTreeData* 2020-05-10 00:24:15 +03:00
Ivan
85d783c247
Poco contrib refactoring (#10396)
* Remove config_common.h
* Refactor libcpuid contrib
* Remove support for libcpuinfo
* Define USE_CPUID in Arcadia
* Refactor Poco libraries
2020-05-08 17:11:19 +03:00
Alexander Tokmakov
d35150ecf3 fix http readonly request stucks on client close 2020-05-05 23:32:34 +03:00
alexey-milovidov
114f09cfca
Merge pull request #10666 from NanoBjorn/refactor-volume
Volumes and storages refactoring
2020-05-05 16:25:26 +03:00
Gleb Novikov
c637d99e07 Volumes and storages refactoring:
1. Moved Volume to separate file
  2. Created IVolume interface and implemented current behaviour in implementation of new interface — VolumeJBOD
  3. Replaced all old volume usages with new VolumeJBOD. Where it is unnecessary to have JBOD — left just IVolume.
  4. Removed old Volume completely
  5. Moved StoragePolicy to separated files
  6. Moved DiskSelector to separated files
  7. Removed DiskSpaceMonitor file
2020-05-04 23:15:38 +03:00
Alexey Milovidov
99c18c5a09 Check for suspicious codecs #4966 2020-05-04 03:11:49 +03:00
Alexey Milovidov
bb6eabd82d Fix HTTP response code for some parse errors #10636 2020-05-03 22:47:07 +03:00
Azat Khuzhin
8ccaf1bf32 Add internal_replication example config.xml 2020-05-03 14:46:45 +03:00
Alexey Milovidov
c40ffb88c6 Comments in config.xml; reset values for production use case 2020-05-01 22:18:35 +03:00
Alexey Milovidov
600b396f1b Merge remote-tracking branch 'origin/master' into sampling-memory-profiler 2020-05-01 21:48:20 +03:00
Alexander Kazakov
e9baaa439b
Implementation of new system metrics provider (Procfs) (#10544)
* New metrics provider (Procfs) + Refactored TasksStatsCounters

* Trivial statless test that ProcFS is provided

* Trivial perf test for ProcfsMetricsProvider

Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
2020-05-01 21:47:41 +03:00
Alexey Milovidov
293ae88e7f Add sampling memory profiler 2020-04-30 16:25:17 +03:00
alexey-milovidov
cef5f12186
Merge pull request #10542 from azat/logger.levels.logger
Ability to set logger level for loggers that have non allowed in xml tags symbols
2020-04-29 00:13:52 +03:00
tavplubix
11c3493676
Merge pull request #10547 from ClickHouse/zhang2014/feature/ISSUES-5436
Merging #7572
2020-04-28 12:40:48 +03:00
Alexander Tokmakov
aeac8cb621 fix 2020-04-28 10:19:37 +03:00
Alexander Tokmakov
9d43c2f6c2 save compiled regex 2020-04-28 02:05:15 +03:00
Alexander Tokmakov
e72a484beb minor fixes 2020-04-28 02:05:15 +03:00
Azat Khuzhin
f767309574 Ability to set logger level for loggers that have non allowed in xml tags symbols 2020-04-27 21:30:29 +03:00
Azat Khuzhin
9ff0aaac9c Add example of logger.levels usage into config.xml 2020-04-27 21:21:01 +03:00
zhang2014
d56002b81f ISSUES-5436 fix ya make 2020-04-27 20:17:16 +08:00
zhang2014
81230944a4 ISSUES-5436 add integration test 2020-04-27 18:49:32 +08:00
Alexander Tokmakov
04d6b59ac0 Merge branch 'master' into database_atomic 2020-04-23 17:31:37 +03:00
zhang2014
0070f75218 ISSUES-5436 fix integration test failure & add test 2020-04-23 17:11:20 +08:00
Alexey Milovidov
07dcf40a6a Simple server wide memory profiler 2020-04-22 20:52:21 +03:00
Alexander Tokmakov
fefbbd37df Merge branch 'master' into database_atomic 2020-04-22 17:02:30 +03:00
Alexey Milovidov
0482e2e3ea Implement suggestion from Sergey Veletsky 2020-04-22 01:40:18 +03:00
alexey-milovidov
8c2839d3c9
Merge pull request #10308 from ClickHouse/shared-context-lifetime
Shared context lifetime
2020-04-21 00:12:45 +03:00
Alexander Tokmakov
b29bddac12 Merge branch 'master' into database_atomic 2020-04-20 14:09:09 +03:00
zhang2014
318ab3b51e ISSUES-5436 try fix build failure & pvs & style 2020-04-20 12:49:17 +08:00
Alexey Milovidov
1d885b3ff9 Merge branch 'master' into quota-key-in-client 2020-04-20 02:15:37 +03:00
Alexey Milovidov
2987e70aab Introduce "max_server_memory_usage" setting 2020-04-20 01:59:21 +03:00
Alexey Milovidov
ec6b176f3b Rework total memory tracker #10293 2020-04-20 01:59:21 +03:00
zhang2014
57cbecf935 ISSUES-5436 reworking predefine http 2020-04-20 03:18:36 +08:00
zhang2014
835dc4c445 After merge upsream master fix 2020-04-20 02:26:42 +08:00
zhang2014
b82eee5255 ISSUES-5436 fix review suggestions & add some '?' re2 test 2020-04-20 02:26:26 +08:00
zhang2014
8105a9bbe0 ISSUES-5436 fix review suggestions 2020-04-20 02:26:06 +08:00
tai
dafba9d66a ISSUES-5436 add integration test for custom http 2020-04-20 02:25:43 +08:00
zhang2014
847f7ab439 ISSUES-5436 fix build failure & fix test failure 2020-04-20 02:25:43 +08:00
zhang2014
6b716e511b ISSUES-5436 support custom http [part 6] 2020-04-20 02:25:12 +08:00
zhang2014
7aef95b082 ISSUES-5436 support custom http [part 5] 2020-04-20 02:23:18 +08:00
zhang2014
fd00757178 ISSUES-5436 support custom http [part 4] 2020-04-20 02:23:18 +08:00
zhang2014
60abff3341 ISSUES-5436 support custom http [part 3] 2020-04-20 02:23:18 +08:00
zhang2014
183eb82b7c ISSUES-5436 support custom http [part 2] 2020-04-20 02:23:18 +08:00
zhang2014
159ba24fca ISSUES-5436 support custom http 2020-04-20 02:23:18 +08:00
Nikolai Kochetov
84faa9af26 Merge branch 'master' into shared-context-lifetime 2020-04-17 19:13:13 +03:00
Nikolai Kochetov
ce11662ab1 Fix build. 2020-04-17 15:58:52 +03:00
Nikolai Kochetov
d51a9c551b Remove shared_ptr to SharedContext from Context. 2020-04-17 12:47:40 +03:00
alexey-milovidov
17e7d4d88a
Merge pull request #10307 from abyss7/arcadia-4
Changes for auto-sync with Arcadia
2020-04-17 05:08:34 +03:00
Nikolai Kochetov
025093d354 Fix build. 2020-04-16 17:51:33 +03:00
Nikolai Kochetov
d6db27e372 Modify context. 2020-04-16 17:37:38 +03:00
Ivan Lezhankin
e230632645 Changes required for auto-sync with Arcadia 2020-04-16 15:31:57 +03:00
alexey-milovidov
a92a6d8e63
Merge pull request #10274 from ClickHouse/avoid-error-message-on-rst
Avoid error message on RST
2020-04-15 17:33:35 +03:00
alexey-milovidov
2a48b0ecd7
Merge pull request #10184 from xzkostyan/feature-zookeeper-ssl
Optional secured communication between ClickHouse and Zookeeper
2020-04-15 07:09:14 +03:00
Alexey Milovidov
ddd840a81b Avoid error message on RST 2020-04-15 06:56:42 +03:00
Alexey Milovidov
fe59524481 Implemented in a different way 2020-04-15 04:59:14 +03:00
Alexey Milovidov
c22f91db01 Allow to pass quota_key in clickhouse-client 2020-04-15 04:15:05 +03:00
Alexander Tokmakov
6c0d9723e0 Merge branch 'master' into database_atomic 2020-04-13 18:47:47 +03:00
Alexander Tokmakov
5e6d4b9449 Merge branch 'master' into database_atomic 2020-04-12 16:35:44 +03:00
BohuTANG
e7f399f652 MySQLHandler: max_allowed_packet returned by default when server setup with select variables 2020-04-12 13:37:39 +08:00
Vitaly Baranov
ca634f603a Revert resetting the settings to their defaults after each query in TCPHandler. 2020-04-10 21:09:59 +03:00
Alexander Tokmakov
033d7f9b32 Merge branch 'master' into database_atomic 2020-04-10 17:12:09 +03:00
Vitaly Baranov
8a19f6d322
Merge pull request #10115 from azat/path.xml-access_control_path
Rewrite access_control_path in path.xml
2020-04-09 11:38:08 +03:00
Konstantin Lebedev
b666f60af8 Optional secured communication between ClickHouse and Zookeeper 2020-04-09 01:40:54 +03:00
BohuTANG
f48fdda678
Enhanced compatibility with native mysql-connector-java(JDBC) (#10021)
* Skip the `/* comments */ SELECT @@variables ...` from mysql-connector-java setup for MySQL Handler #9336

mysql-connector setup query:
/* mysql-connector-java-5.1.38 ( Revision: ${revinfo.commit} ) */SELECT  @@session.auto_increment_increment AS auto_increment_increment, @@character_set_client AS character_set_client, @@character_set_connection AS character_set_connection, @@character_set_results AS character_set_results, @@character_set_server AS character_set_server, @@init_connect AS init_connect, @@interactive_timeout AS interactive_timeout...

ClickHouse side Error:
{} <Error> executeQuery: Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 74: @@session.auto_increment_increment AS auto_increment_increment, @@character_set_client AS character_set_client, @@character_set_connection AS character_set_conn. Expected one of: CAST, NULL...

Client side Exception:
java.sql.SQLException: Syntax error: failed at position 74: @@session.auto_increment_increment AS auto_increment_increment, @@character_set_client AS character_set_client, @@character_set_connection AS character_set_conn. Expected one of: CAST...

* add repalce 'SHOW VARIABLES' for mysql-connector-java-5.1.34 #9336

* Add java client(JDBC) integration test to test_mysql_protocol

* shift out java tests from dbms

* Update MySQLHandler.cpp

* Update MySQLHandler.cpp

* test_mysql_protocol: add Test.java exit code 1 when expection

Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
2020-04-09 00:52:19 +03:00
Alexander Tokmakov
dd1590830b Merge branch 'master' into database_atomic 2020-04-08 22:00:46 +03:00
Azat Khuzhin
ed25ac8b12 Rewrite access_control_path in path.xml
Follow-up for: #9811
2020-04-08 10:45:14 +03:00
Vitaly Baranov
bc3e2cb48c Add the option "access_control_path" to server config. 2020-04-07 23:31:59 +03:00
Vitaly Baranov
b71ff6160a Add the "access_management" option to users.xml which enables a user to manage access rights.
The option is turned off by default.
2020-04-07 23:31:54 +03:00
Alexander Tokmakov
9c67d2716e Merge branch 'master' into database_atomic 2020-04-03 03:19:11 +03:00
Ivan
97f2a2213e
Move all folders inside /dbms one level up (#9974)
* Move some code outside dbms/src folder
* Fix paths
2020-04-02 02:51:21 +03:00