Commit Graph

573 Commits

Author SHA1 Message Date
Alexander Tokmakov
13e41cdd20 try enable atomic database by default 2020-09-21 13:55:34 +03:00
alexey-milovidov
3e0509af69
Merge pull request #13573 from amosbird/rms
specific ReplicatedMergeTree settings
2020-09-21 01:36:33 +03:00
Denis Glazachev
442b1407cf Merge branch 'master' into ldap-any-user-authentication
* master: (375 commits)
  Update type-conversion-functions.md
  Update maxmap.md
  Update maxmap.md
  Update maxmap.md
  Update single_fixed_string_groupby.xml
  Alter remove column properties and TTLs (#14742)
  better fixed string group by support
  Fix incorrect key condition of fixed strings.
  constant output order
  more tests for  #14646
  Maybe fix MSan report in base64
  Proper exception message for wrong number of arguments of CAST
  Added a test
  Fix buffer overflow in "bar" function
  Update convertMySQLDataType.cpp
  Fix clang-tidy
  Remove obsolete code from performance test
  Slightly better code
  Even more
  Even more
  ...

# Conflicts:
#	src/Interpreters/Context.cpp
2020-09-21 00:31:04 +04:00
alexey-milovidov
1474b80ecc
Merge pull request #15008 from ClickHouse/clang-tidy-11-without-enabling
Fix terribly wrong code
2020-09-20 02:00:54 +03:00
myrrc
c78d9f9f38 moved the generators to docs/ folder 2020-09-19 19:42:36 +03:00
alexey-milovidov
c461d782e4
Merge pull request #14929 from ClickHouse/mlock-code-segment
Only mlock code segment
2020-09-19 13:46:49 +03:00
Alexey Milovidov
ef5363b87f Fix terribly wrong code 2020-09-19 02:05:13 +03:00
alexey-milovidov
988b20a32c
Merge pull request #14684 from azat/parallel-distributed_ddl
Allow parallel execution of distributed DDL
2020-09-18 22:18:17 +03:00
myrrc
f6bce8bd1e reverted USE to ENABLE 2020-09-18 17:18:59 +03:00
myrrc
3953ed485b added sub-tables in report, replaced some USE_ -> ENABLE_ for
consistency
2020-09-18 16:09:22 +03:00
Alexander Kuzmenkov
449189dcda Initialize global thread pool before we fetch configs from ZK 2020-09-18 13:47:09 +03:00
alexey-milovidov
cf5db5e4dc
Merge pull request #14888 from azat/client-imporove-INSERT-error-message
Improve error message for INSERT via clickhouse-client
2020-09-18 01:13:22 +03:00
Alexey Milovidov
eb523f1a7c Merge branch 'master' into mlock-code-segment 2020-09-17 21:15:09 +03:00
myrrc
a8fa23fb9d updated multi-line command 2020-09-17 19:27:51 +03:00
myrrc
507acdb5e7 fix 2020-09-17 19:17:56 +03:00
myrrc
7cd5c0d34f Merge remote-tracking branch 'upstream/master' into feature/cmake-flags-doc-generator 2020-09-17 19:17:09 +03:00
myrrc
7eff17cd40 removed extra dots in generated list 2020-09-17 18:46:55 +03:00
myrrc
acff0feb58 slighly updated CMake files 2020-09-17 18:09:59 +03:00
Alexey Milovidov
08668de2c1 Remove previous modification 2020-09-17 17:39:22 +03:00
Alexey Milovidov
5ba1ad73d0 Merge branch 'back_quote_in_copier' into HyaZz-master 2020-09-17 17:38:48 +03:00
Alexey Milovidov
4cf131256a Merge branch 'master' of https://github.com/HyaZz/ClickHouse into HyaZz-master 2020-09-17 17:38:41 +03:00
Alexey Milovidov
c1e9ce2cf2 Remove questionable code 2020-09-17 17:38:06 +03:00
Alexander Kuzmenkov
fb64cf210a straighten the protocol version 2020-09-17 17:37:29 +03:00
alexey-milovidov
e101c912aa
Update Internals.cpp 2020-09-17 17:19:10 +03:00
Vitaly Baranov
3356d75b23
Merge pull request #13156 from azat/cluster-secure
Secure inter-cluster query execution (with initial_user as current query user) [v3]
2020-09-17 17:11:00 +03:00
Alexey Milovidov
068e8576b7 Corrections 2020-09-17 15:53:52 +03:00
Alexey Milovidov
8bce20076c Only mlock code segment 2020-09-17 15:39:37 +03:00
sundy-li
544b2cb20d add configChanged method for zookeeper
fix logic error && skip reload testkeeper
2020-09-17 13:33:45 +08:00
20018712
88738cd610 when using copier,column name ,as sort key,with digital number as beginning,will cause a syntax error 2020-09-17 07:27:55 +08:00
Azat Khuzhin
7d046b24e6 Improve error message for INSERT via clickhouse-client
With '\n...' after the query [1] clickhouse-client prefer data from the
INSERT over from stdin, and produce very tricky message:

    Code: 27. DB::Exception: Cannot parse input: expected '\n' before: ' ': (at row 1)

Well for TSV it is ok, but for RowBinary:

    Code: 33. DB::Exception: Cannot read all data. Bytes read: 1. Bytes expected: 4.

So improve error message by adding the source of data for INSERT.

  [1]: clickhouse-client -q "INSERT INTO data FORMAT TSV\n " <<<2
2020-09-17 00:16:51 +03:00
myrrc
29ac26f0f2 separated ch parts table 2020-09-16 00:12:37 +03:00
alexey-milovidov
018f596d21
Merge pull request #14685 from ClickHouse/remap-executable
Attempt to make performance test more reliable
2020-09-15 15:09:47 +03:00
tavplubix
1d8ee50769
Merge pull request #14628 from ClickHouse/enable_some_tests_with_database_atomic
Enable more tests with Atomic database
2020-09-15 12:13:40 +03:00
Azat Khuzhin
0159c74f21 Secure inter-cluster query execution (with initial_user as current query user) [v3]
Add inter-server cluster secret, it is used for Distributed queries
inside cluster, you can configure in the configuration file:

  <remote_servers>
      <logs>
          <shard>
              <secret>foobar</secret> <!-- empty -- works as before -->
              ...
          </shard>
      </logs>
  </remote_servers>

And this will allow clickhouse to make sure that the query was not
faked, and was issued from the node that knows the secret. And since
trust appeared it can use initial_user for query execution, this will
apply correct *_for_user (since with inter-server secret enabled, the
query will be executed from the same user on the shards as on initator,
unlike "default" user w/o it).

v2: Change user to the initial_user for Distributed queries if secret match
v3: Add Protocol::Cluster package
v4: Drop Protocol::Cluster and use plain Protocol::Hello + user marker
v5: Do not use user from Hello for cluster-secure (superfluous)
2020-09-15 01:36:28 +03:00
Alexander Tokmakov
da6beb1a91 Merge branch 'master' into enable_some_tests_with_database_atomic 2020-09-14 23:00:14 +03:00
Alexey Milovidov
7aa3f86ab9 Added config.d file for development 2020-09-14 22:44:20 +03:00
Alexey Milovidov
2ac88ab47d Added config option 2020-09-14 21:08:09 +03:00
Alexey Milovidov
0e73b8acf3 Merge branch 'master' into remap-executable 2020-09-14 20:26:16 +03:00
Denis Glazachev
c202364f01
Merge branch 'master' into ldap-any-user-authentication 2020-09-14 18:53:46 +04:00
Vitaly Baranov
421eeeccef Add the section user_directories to the default config. 2020-09-12 23:04:28 +03:00
alexey-milovidov
cc3d1a3922
Merge pull request #14471 from ClickHouse/git-to-clickhouse
Added a script to import git repository to ClickHouse
2020-09-12 03:56:31 +03:00
Alexey Milovidov
f6e9a9bdfb Merge branch 'git-to-clickhouse' of github.com:yandex/ClickHouse into git-to-clickhouse 2020-09-12 03:56:13 +03:00
Alexey Milovidov
e12ae99bf7 Added review suggestion 2020-09-12 03:55:54 +03:00
Azat Khuzhin
dd867b787f Allow parallel execution of distributed DDL
Add distributed_ddl.pool_size to control maximum parallel to handle
distributed DDL.

Also:
- convert Exception constructors to fmt-like
- use sleepFor* over std::this_thread::sleep_for()
2020-09-12 02:32:08 +03:00
Alexander Kuzmenkov
d227f1f706
Merge pull request #13192 from ClickHouse/aku/semicolon-simple
Simpler version of semicolon-terminated VALUES #12999 w/o `pos` changes
2020-09-11 11:51:52 +03:00
Alexey Milovidov
a644733139 Attempt to make performance test more reliable 2020-09-10 12:05:57 +03:00
Alexander Kuzmenkov
1e8e8ea6f9
Merge pull request #14632 from ClickHouse/aku/reset-db
Fuzzer: reset default database before reconnect
2020-09-10 09:56:35 +03:00
Alexander Kuzmenkov
83ec93dec8 Fuzzer: reset default database before reconnect 2020-09-09 16:46:59 +03:00
Alexander Tokmakov
b68782d285 enable more tests with Atomic database 2020-09-09 16:32:50 +03:00
Vasily Nemkov
3973a17530
MySql datatypes dateTime64 and decimal (#11512) 2020-09-09 15:18:02 +03:00
Alexey Milovidov
e3924b8057 Fix "Arcadia" 2020-09-08 01:14:13 +03:00
alexey-milovidov
3942cc615f
Update git-import.cpp 2020-09-07 10:09:42 +03:00
Alexey Milovidov
ee54971c3d Fix build 2020-09-07 07:11:03 +03:00
Alexey Milovidov
d18e7adbc0 Add git-import as a tool 2020-09-07 06:22:47 +03:00
Dmitry
ace61fa67e Uint128 logic and tests 2020-09-06 00:35:24 +03:00
Dmitry
61a570f361 Merge branch 'obfuscator_uuid' of github.com:dimarub2000/ClickHouse into obfuscator_uuid 2020-09-04 16:56:57 +03:00
Dmitry
0d7fbe863a Merge branch 'master' into obfuscator_uuid 2020-09-04 16:56:29 +03:00
alexey-milovidov
4f9df21d3e
Merge pull request #10373 from azat/dist-SELECT-optimization
Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key
2020-09-04 01:38:56 +03:00
Denis Glazachev
59e0c10aaf Merge branch 'master' into ldap-any-user-authentication 2020-09-03 19:36:19 +04:00
Azat Khuzhin
4043be3121 Add new query processing stage WithMergeableStateAfterAggregation
Process query until the stage where the aggregate functions were
calculated and finalized.

It will be used for optimize_distributed_group_by_sharding_key.

v2: fix aliases
v3: Fix protocol ABI breakage due to WithMergeableStateAfterAggregation
    Conditions >= for QueryProcessingStage::Enum has been verified, and they
    are ok (in InterpreterSelectQuery).
2020-09-03 00:51:08 +03:00
Ilya Yatsishin
a5e9449bc3
Merge branch 'master' into try-dns-fix 2020-09-02 19:01:30 +03:00
alexey-milovidov
4c039002f9
Update Obfuscator.cpp 2020-09-02 15:27:15 +03:00
Dmitry
c32c83f653 added include 2020-09-02 14:20:20 +03:00
Dmitry
a04c8bb095 fixes 2020-09-02 14:18:24 +03:00
Dmitry
0e4c9ff9a8 Added UUID to Obfuscator 2020-09-02 14:14:49 +03:00
Yatsishin Ilya
45e54f81c7 better resolv.conf, add DNSCacheUpdater logs 2020-09-02 11:07:46 +03:00
Alexander Kuzmenkov
73d433a7dd
Merge pull request #14316 from ClickHouse/aku/html-fuzzer
Add a HTML report for AST Fuzzer
2020-09-01 16:46:30 +03:00
Alexander Kuzmenkov
9cf2a38eb8 fixup 2020-09-01 03:22:06 +03:00
Alexander Kuzmenkov
33c5815eb3 Add a HTML report for AST Fuzzer 2020-09-01 02:33:42 +03:00
Denis Glazachev
9b4cd06051 Merge branch 'master' into ldap-any-user-authentication
* master: (414 commits)
  Add .reference
  Update 01460_mark_inclusion_search_crash.sql
  Fix bug in mark inclusion search.
  cosmetic fixes for performance report
  cosmetic fixes in performance report
  Add more docs about functional tests
  Add ability to specify Default codec for columns (#14049)
  better
  done
  Disable force TTL on optimise
  Update CMakeLists.txt
  Update custom parts of storage AST only if it has extended definition
  DOCSUP-712: Documented the ttl_only_drop_parts setting (#13823)
  Small fixes
  gix comment and useDefault*(), add tests for nullables
  Fix a build for old some OS with old find
  Update hdfs.md
  Fix 01085_max_distributed_connections flackiness
  Fix 00974_distributed_join_on flackiness (by allow retries and hide logs)
  Update adopters.md
  ...
2020-08-30 11:56:38 +04:00
Vitaly Baranov
90dd31fef1 Fix crash in SchemaAllowedHandler. 2020-08-29 15:53:15 +03:00
alesapin
10c7a6c45e
Add ability to specify Default codec for columns (#14049)
* Add ability to specify DefaultCompression codec which correspond to settings specified in config.xml

* Fix style

* Rename DefaultCompression to simple Default

* Fix compression codec

* Better codec description representation

* Less strange code and one method

* Fix delta
2020-08-28 20:40:45 +03:00
Amos Bird
05a5a13e08
specific ReplicatedMergeTree settings 2020-08-27 22:37:41 +08:00
alexey-milovidov
8c1d33cca1
Merge pull request #14154 from amosbird/betterstart
Only use sudo when we are not `user`
2020-08-27 17:06:00 +03:00
Amos Bird
b968f0c339
Only use sudo when we are not user 2020-08-27 14:22:49 +08:00
alexey-milovidov
0f706c01ca
Merge pull request #13888 from vladimir-golovchenko/add-date-trunc-function
Added date_trunc function
2020-08-27 02:12:27 +03:00
alexey-milovidov
318f14b95e
Merge pull request #13691 from ClickHouse/client-progress-clobber-data
Fix visible data clobbering by progress bar in client in interactive mode
2020-08-26 13:25:25 +03:00
vladimir golovchenko
f3cfd39003 Fixed .gitignore to exclude tests-generated data. 2020-08-25 16:41:23 -07:00
Azat Khuzhin
0a34ae85a3 Use terminal width for the formatted help text of the clickhouse-copier
Like other utils, clickhouse-client and so on (since copier uses Poco,
while other uses boost).
Otherwise clickhouse-copier --help is unreadable.
2020-08-25 21:17:21 +03:00
Alexey Milovidov
c5984ca1c5 Merge branch 'master' into client-progress-clobber-data 2020-08-24 23:44:43 +03:00
alexey-milovidov
6cf1a61735
Merge pull request #13841 from azat/unknown-packet-fix
Unknown packet fix (for Protocol::Server::Log for distributed queries)
2020-08-20 20:25:58 +03:00
Denis Glazachev
ec52a165af Style fixes 2020-08-20 12:46:42 +04:00
Denis Glazachev
cbc9285bd6 Merge branch 'master' into ldap-any-user-authentication
* master: (43 commits)
  Publish list of tests that failed the concurrent fast test
  fix test
  Use gnu++2a instead of c++2a for unbundled build to fix numeric_limits<__int128>
  fix test
  Update 01453_fixsed_string_sort.sql
  Added test.
  Fix ColumnString::updatePermutationWithCollation.
  Add support for extended precision integers and decimals (#13097)
  remove retries
  Check that ya.make files are auto-generated
  Fix "Arcadia"
  ISSUES-4006 trigger CI again
  fixed typo arrayCompact
  Stratify nans comparison in arrayCompact function
  Remove even more useless code
  Remove useless code around zkutil
  Fix 00956_sensitive_data_masking flackiness
  Update docker/test/stress/run.sh
  add exclusive DDLGuard for database
  Fix handling embedded config.
  ...
2020-08-20 11:41:43 +04:00
Denis Glazachev
bdfea652c1 Change user_template to roles
Change top_enclosing_storage to access_control_manager
Simplify the lookup in peer storages
2020-08-20 11:39:27 +04:00
Denis Glazachev
58f73ff041 Merge branch 'master' into ldap-any-user-authentication
* master: (30 commits)
  Documentation improve:  Translate [select] section into Chinese to improve readability (#13814)
  Update adopters.md
  partially disable test with MaterializeMySQL
  one more unroll for arerage.cpp
  Update adopters.md
  Update adopters.md
  Update adopters.md
  Update adopters.md
  fix #13819
  DOCSUP-1888: Documented the input_format_avro_allow_missing_fields setting (#13671)
  Fix "Arcadia" and "Unbundled" builds
  DOCSUP-928: Documented the groupArraySample function (#13791)
  Documentation about ReplacingMergeTree extended with type DateTime64 for column (#13498)
  doc: update quotas.md (#13400)
  Fix 01356_initialize_aggregation in unbundled build (change topKWeighted order)
  Fix style
  Add normalizedQueryHash function with tests
  Fix topK/topKWeighted merge (wtih non-default parameters)
  Add test
  Add function "normalizeQuery"
  ...

# Conflicts:
#	programs/server/Server.cpp
#	src/Access/AccessControlManager.cpp
#	src/Access/AccessControlManager.h
2020-08-18 14:54:02 +04:00
alexey-milovidov
3e6acf1329
Experiment with start/stop in stress test (#13779)
* Experiment with start/stop in stress test

* Support other installation options in "clickhouse start"

* Better diagnostics in install script

* Fixup

* Fixup

* Better diagnostics in stress test

* Update run.sh
2020-08-18 12:43:02 +03:00
Azat Khuzhin
ad7379a37e Improve error message on unknown packet for client suggestion 2020-08-17 21:38:34 +03:00
Azat Khuzhin
13f6af3834 Improve error message on unknown packet in client 2020-08-17 21:38:34 +03:00
Denis Glazachev
fbbaf645b9 Merge branch 'master' into ldap-any-user-authentication
* master: (70 commits)
  Fix terrible markup due to wrong @blinkov's script.
  Update extended-roadmap.md
  Update extended-roadmap.md
  Fix bug on the website
  Suppress Shellcheck
  Fix typos
  Fix typos
  Install codespell
  Fix typo
  Update gtest_thread_pool_schedule_exception.cpp
  Fix flaky unit test
  Fix set index with const column pred
  Updating healthcheck parameters.
  Fix step overflow in range()
  * Moving to standard healthcheck parameters. * Adding docker-compose pull and docker-compose down before executing docker-compose up. * Adding docker-compose ps if any of the containers are unhealthy.
  Fix topK test
  Adding missing certificates and dhparam.pem for openldap4.
  Keep "metadata_loading" test only for release build
  Fix error in test
  Fix typo
  ...
2020-08-17 18:08:24 +04:00
Vitaly Baranov
0759dff12b Support <user_directories> section in the main config. 2020-08-16 19:15:38 +03:00
Vitaly Baranov
2909ed1bc0 Better initialization of access storages. Make list of access storages dynamic. 2020-08-16 19:15:34 +03:00
Alexey Milovidov
e43746395e Merge branch 'master' into codespell-2 2020-08-16 14:57:21 +03:00
Denis Glazachev
7375dc5d66 Merge branch 'master' into ldap-any-user-authentication
* master: (956 commits)
  Remove ZooKeeper from unit tests
  Revert check location of Docker compose files @qoega.
  Update developer-instruction.md
  trigger the CI
  Update settings.md
  Update settings.md
  Fix
  fix sync 2
  Fix tests.
  Add docker for style check
  fix sync 1
  passwd and group location error
  Hotfix for pushdown with StorageMerge (#13679)
  Fix error with batch aggregation and -Array combinator
  Fix cassandra build on macos
  Update adopters.md
  Update adopters.md
  Fix build after merge
  Fix shared build
  log error message
  ...

# Conflicts:
#	tests/testflows/ldap/docker-compose/clickhouse-service.yml
2020-08-15 14:18:40 +04:00
Alexander Tokmakov
128b19fdba Merge branch 'master' into use_atomic_engine_for_system_db 2020-08-14 13:47:44 +03:00
Amos Bird
fe6696390c
Fix shared build 2020-08-14 10:49:40 +08:00
Alexey Milovidov
f5b8f508f0 Better 2020-08-14 05:42:23 +03:00
Alexey Milovidov
2359759b0e Better 2020-08-14 05:35:42 +03:00
Alexey Milovidov
cd095b7671 Fix data visible data clobbering with progress bar in client in interactive mode 2020-08-14 05:19:25 +03:00
Alexander Tokmakov
d1ef34adff fix 2020-08-13 22:41:06 +03:00
Alexander Tokmakov
a6ff049eec use Atomic for system database 2020-08-12 23:40:13 +03:00