Commit Graph

801 Commits

Author SHA1 Message Date
youennL-cs
6526c2a8ab
[RFC] Replacing merge tree new engine (#41005)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add keyword 'CLEANUP' when OPTIMIZE

* Cleanup uniquely when it's a replacingMergeTree

* Propagate CLEANUP information and change from 'with_cleanup' to 'cleanup'

* Cleanup data flagged as 'is_deleted'

* Fix merge when optimize and add a test

* Fix OPTIMIZE and INSERT + add tests

* New fix for cleanup at the merge

* Cleanup debug logs

* Add the SETTINGS option 'clean_deleted_rows' that can be 'never' or 'always'

* Fix regression bug; Now REplicatedMergeTree can be called as before without 'is_deleted'

* Add Replicated tests

* Disable tag 'long' for our test and cleanup some white spaces

* Update tests

* Fix tests and remove additional useless whitespace

* Fix replica test

* Style clean && add condition check for is_deleted values

* clean_deleted_rows settings is nom an enum

* Add valid default value to the clean_deleted_rows settings

* Update cleanup checkers to use the enum and fix typos in the test

* Fix submodule contrib/AMQP-CPP pointer

* Add missing messages in test reference and remove a print with non derterministic order

* fix replica test reference

* Fix edge case

* Fix a typo for the spell checker

* Fix reference

* Fix a condition to raise an error if is_deleted differ from 0/1 and cleanup

* Change tests file name and update number

* This should fix the ReplacingMergeTree parameter set

* Fix replicated parameters

* Disable allow_deprecated_syntax_for_merge_tree for our new column

* Fix a test

* Remove non deterministic order print in the test

* Test on replicas

* Remove a condition, when checking optional parameters, that should not be sueful since we disabled the deprected_syntaxe

* Revert "Remove a condition, when checking optional parameters, that should not be useful since we disabled the deprected_syntaxe"

This reverts commit b65d64c05e.

* Fix replica management and limit the number of argument to two maximum, due to the possiblity of deprecated table create/attach failing otherwise

* Test a fix for replicated log information error

* Try to add sync to have consistent results

* Change path of replicas that should cause one issue and add few prints in case it's not that

* Get cleanup info on replicas only if information found

* Fix style issues

* Try to avoid replication error 'cannot select parts...' and and replica read/write field order

* Cleanup according to PR reviews
 and add tests on error raised.

* Update src/Storages/MergeTree/registerStorageMergeTree.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* Select ... FINAL don't show rows with is_deleted = true

* Update and fix SELECT ... FINAL merge parameter

* Remove is_deleted rows only on the version inserted when merge

* Fix (master) updates issues

* Revert changes that should not be commited

* Add changes according to review

* Revert changes that should not be commited - part 2

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-02-16 16:03:16 +03:00
Antonio Andelic
8d16fe5793
Merge branch 'master' into add-support-for-settings-alias 2023-02-13 08:46:00 +01:00
Kruglov Pavel
e84baa9106
Merge pull request #43179 from tonickkozlov/tonickkozlov/add-interserver-dns-retries
CLICKHOUSE-2375 Add interserver DNS retries
2023-02-11 14:35:40 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias 2023-02-09 16:07:45 +01:00
Azat Khuzhin
911fabd0dc Fix applying settings for FORMAT on the client
Previously the following query does not works correctly:

    SELECT number FROM numbers(5) SETTINGS output_format_json_array_of_rows = 1 FORMAT JSONEachRow

While this one works OK:

    SELECT number FROM numbers(5) FORMAT JSONEachRow SETTINGS output_format_json_array_of_rows = 1

The problem is in which AST those settings are stored, use the logic as
executeQuery() has to apply them:

  c83f701696/src/Interpreters/executeQuery.cpp (L467-L497)

Note, the only problem should be with the settings for FORMAT, since
client applies thoes settings (and formats) locally w/o server, while in
case of i.e. HTTP it will be applied on the server and everything will
works fine.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-07 18:29:40 +01:00
Antonio Andelic
12569da984 Merge branch 'master' into add-support-for-settings-alias 2023-02-05 16:08:57 +00:00
Alexey Milovidov
22bd0b6f69
Merge pull request #38983 from CurtizJ/randomize-mt-settings
Allow to randomize merge tree settings in tests
2023-02-04 02:59:52 +01:00
Nikita Mikhaylov
33877b5e00
Parallel replicas. Part [2] (#43772) 2023-02-03 14:34:18 +01:00
Azat Khuzhin
ef2bc589fa Avoid client prompt overlap after query fuzzy search
Under certain conditions it is possible for skim to overlap the prompt,
well, not overlap, but not re-render it, and so the client does not have
a nice prompt.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-02 19:31:45 +01:00
Anton Popov
d8fe2bcbaa
Merge branch 'master' into randomize-mt-settings 2023-01-27 16:03:00 +01:00
Anton Popov
92fa39798b Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-26 13:29:23 +00:00
Alexander Tokmakov
3f6594f4c6 forbid old ctor of Exception 2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449)
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
2023-01-24 00:13:58 +03:00
Anton Popov
08b2e524c7 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-23 15:34:53 +00:00
Azat Khuzhin
2a8f116c18 Forward declaration of ConcurrentBoundedQueue in ThreadStatus
ThreadStatus is the header that recomplies almost all ClickHouse
modules.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-21 16:02:09 +01:00
Alexander Tokmakov
870cfcc36a less fmt::runtime usages 2023-01-17 00:11:59 +01:00
Alexander Gololobov
263a3f10c9 Also copy or move PREWHERE to WHERE 2023-01-12 14:11:30 +01:00
Alexander Gololobov
2a92ae8d8e Fuzz PREWHERE clause 2023-01-12 13:14:09 +01:00
Anton Popov
ca80240d84 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-12 12:13:45 +00:00
Alexey Milovidov
1f45845e70 Fix (benign) data race in clickhouse-client 2023-01-09 08:49:21 +01:00
Kruglov Pavel
966f57ef68
Merge pull request #42777 from Avogar/improve-streaming-engines
Refactor and Improve streaming engines Kafka/RabbitMQ/NATS and data formats
2023-01-02 15:59:06 +01:00
Nikolay Degterinsky
dfe93b5d82
Merge pull request #42284 from Algunenano/perf_experiment
Performance experiment
2022-12-30 03:14:22 +01:00
Kruglov Pavel
894726bd8f
Merge branch 'master' into improve-streaming-engines 2022-12-29 22:59:45 +01:00
Raúl Marín
5de11979ce
Unify query elapsed time measurements (#43455)
* Unify query elapsed time reporting

* add-test: Make shell tests executable

* Add some tests around query elapsed time

* Style and ubsan
2022-12-28 21:01:41 +01:00
Azat Khuzhin
1f9a6076b9 Use DB::ReadBufferFromFile over std::ifstream in ReplxxLineReader
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-27 14:23:51 +01:00
Azat Khuzhin
b7f92454bf Move LineReader/ReplxxLineReader out from base
This will simplify linking dependencies, and anyway this classes are not
base in any sense.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-27 14:23:51 +01:00
Azat Khuzhin
83cad3625b Remove unused setTerminalEcho()
It had been replaced with readpassphrase in #5092

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-27 14:23:51 +01:00
Raúl Marín
45d27f461b
Merge branch 'master' into perf_experiment 2022-12-20 09:07:48 +00:00
Anton Popov
33dc0d8283 fix more tests 2022-12-16 17:50:01 +00:00
Anton Popov
b6f2754afb Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-16 14:37:34 +00:00
Kruglov Pavel
c5b2e4cc23
Merge branch 'master' into improve-streaming-engines 2022-12-15 18:44:35 +01:00
Anton Popov
87e02a1744 Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-14 02:08:10 +00:00
Nikolay Degterinsky
443bc3e656
Merge pull request #43719 from evillique/password_complexity_rules
Add password complexity rules
2022-12-13 17:56:45 +01:00
Nikolay Degterinsky
9b6d31b95d
Merge branch 'master' into perf_experiment 2022-12-13 17:15:07 +01:00
Nikolay Degterinsky
19e95e8cd8 Better implementation of password rules 2022-12-11 23:59:04 +00:00
Nikolay Degterinsky
f2c07c0337 Merge remote-tracking branch 'upstream/master' into password_complexity_rules 2022-12-10 00:46:05 +00:00
Anton Popov
0a951db1d0 fix more tests with random merge tree settings 2022-12-02 14:16:28 +00:00
Anton Popov
5d9077cdb6 Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-02 12:06:45 +00:00
HarryLeeIBM
304e4a73d7 Fix setting TCP_KEEPAIDLE for s390x 2022-11-30 13:22:04 -08:00
Joanna Hulboj
d587a52afe FIXUP: PR comment 2022-11-27 11:38:43 +00:00
Joanna Hulboj
e360f85798 Added client option to display the number of locally processed rows 2022-11-27 11:38:43 +00:00
Nikolay Degterinsky
3fd10e0011 Better solution 2022-11-25 22:38:28 +00:00
Raúl Marín
4d3217cb18
Merge branch 'master' into perf_experiment 2022-11-24 10:32:12 +01:00
Kseniia Sumarokova
c81257c7e2
Merge pull request #43442 from kssenii/fix-progress-from-stdin
Show read rows for reading from stdin in client
2022-11-23 11:23:16 +01:00
Kseniia Sumarokova
b9b1d22d72
Update ClientBase.cpp 2022-11-21 15:04:37 +01:00
kssenii
10a258242d Fix progress from stdin 2022-11-21 14:11:51 +01:00
Azat Khuzhin
da8fe9c398 Fix possible heap-use-after-free in local if history file cannot be created
ASAN report:

    Code: 586. DB::ErrnoException: Cannot create file: /src/.clickhouse_history, errno: 2, strerror: No such file or directory. (CANNOT_CREATE_FILE)
    =================================================================
    ==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x6240000208f0 at pc 0x000030d22ade bp 0x7ffff2ff3f70 sp 0x7ffff2ff3f68
    READ of size 8 at 0x6240000208f0 thread T2
        #0 0x30d22add in DB::ProcessList::insert() build_docker/../src/Interpreters/ProcessList.cpp:89:36
        #1 0x31411018 in DB::executeQueryImpl() build_docker/../src/Interpreters/executeQuery.cpp:516:60
        #2 0x3140e1ab in DB::executeQuery() build_docker/../src/Interpreters/executeQuery.cpp:1083:30
        #3 0x3364391e in DB::LocalConnection::sendQuery() build_docker/../src/Client/LocalConnection.cpp:119:21
        #4 0x3367bab0 in DB::Suggest::fetch() build_docker/../src/Client/Suggest.cpp:141:16
        #5 0x336820eb in void DB::Suggest::load<DB::LocalConnection>()::'lambda'()::operator()() const build_docker/../src/Client/Suggest.cpp:118:17

    0x6240000208f0 is located 2032 bytes inside of 7056-byte region [0x624000020100,0x624000021c90)
    freed by thread T0 here:

        #0 0xe381ef2 in operator delete(void*, unsigned long) (/wrk/clickhouse-asan+0xe381ef2) (BuildId: 6ea6d1a5d2d5a164f60f0fd8230936305bc8d9d0)
        #1 0x335509fe in DB::ClientBase::~ClientBase() build_docker/../src/Client/ClientBase.cpp:293:25
        #2 0x1f809bd5 in mainEntryClickHouseLocal(int, char**) build_docker/../programs/local/LocalServer.cpp:804:5
        #3 0xe3856ad in main build_docker/../programs/main.cpp:482:12
        #4 0x7ffff7dc0082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16

    previously allocated by thread T0 here:
        #0 0xe38128d in operator new(unsigned long) (/wrk/clickhouse-asan+0xe38128d) (BuildId: 6ea6d1a5d2d5a164f60f0fd8230936305bc8d9d0)
        #1 0x2f34a7f3 in std::__1::__unique_if<DB::ContextSharedPart>::__unique_single std::__1::make_unique[abi:v15003]<DB::ContextSharedPart>() build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:714:28
        #2 0x2f34a7f3 in DB::Context::createShared() build_docker/../src/Interpreters/Context.cpp:603:32
        #3 0x1f7f901d in DB::LocalServer::processConfig() build_docker/../programs/local/LocalServer.cpp:535:22
        #4 0x1f7f4d92 in DB::LocalServer::main() build_docker/../programs/local/LocalServer.cpp:419:5
        #5 0x3af24ffe in Poco::Util::Application::run() build_docker/../contrib/poco/Util/src/Application.cpp:334:8
        #6 0x1f809bca in mainEntryClickHouseLocal(int, char**) build_docker/../programs/local/LocalServer.cpp:803:20
        #7 0xe3856ad in main build_docker/../programs/main.cpp:482:12
        #8 0x7ffff7dc0082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16

    Thread T2 created by T0 here:
        #0 0xe32fedc in pthread_create (/wrk/clickhouse-asan+0xe32fedc) (BuildId: 6ea6d1a5d2d5a164f60f0fd8230936305bc8d9d0)
        #1 0x336806df in std::__1::__libcpp_thread_create[abi:v15003](unsigned long*, void* (*)(void*), void*) build_docker/../contrib/libcxx/include/__threading_support:376:10
        #2 0x336806df in std::__1:🧵:thread<void DB::Suggest::load<DB::LocalConnection>()::'lambda'(), void>() build_docker/../contrib/libcxx/include/thread:311:16
        #3 0x3367ff5b in void DB::Suggest::load<DB::LocalConnection>(std::__1::shared_ptr<DB::Context const>, DB::ConnectionParameters const&, int) build_docker/../src/Client/Suggest.cpp:110:22
        #4 0x3357fee9 in DB::ClientBase::runInteractive() build_docker/../src/Client/ClientBase.cpp:2066:22
        #5 0x1f7f5264 in DB::LocalServer::main() build_docker/../programs/local/LocalServer.cpp
        #6 0x3af24ffe in Poco::Util::Application::run() build_docker/../contrib/poco/Util/src/Application.cpp:334:8
        #7 0x1f809bca in mainEntryClickHouseLocal(int, char**) build_docker/../programs/local/LocalServer.cpp:803:20
        #8 0xe3856ad in main build_docker/../programs/main.cpp:482:12
        #9 0x7ffff7dc0082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-20 22:51:27 +01:00
Raúl Marín
97d6fc3071 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-11-17 11:48:46 +01:00
Nikolay Degterinsky
d48d4cf87e
Merge branch 'master' into query_parameters_inserts 2022-11-15 01:05:40 +01:00
Alexey Milovidov
127631ee47
Merge branch 'master' into perf_experiment 2022-11-12 18:58:25 +01:00