thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-29 11:02:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
67,924 Commits 9,189 Branches 1,782 Tags 2.2 GiB
eefb798e33
Commit Graph

15461 Commits

Author SHA1 Message Date
Maksim Kita
f88d8ccb1f Compile AggregateFunctionAvg 2021-07-01 22:56:35 +03:00
Maksim Kita
b491e75a56 Fix compile functions data type offset 2021-07-01 22:56:35 +03:00
Maksim Kita
a5ef0067b8 Compile AggregateFunctionIf 2021-07-01 22:56:35 +03:00
Maksim Kita
9b71b1040a Aggregate functions update compile interface 2021-07-01 22:56:35 +03:00
Maksim Kita
3fe559b31f Compile aggregate functions 2021-07-01 22:56:35 +03:00
Aleksei Semiglazov
9a5365fc41
CLICKHOUSE-784: reset merge tree setting 
Add an ability to reset custom setting to default and remove it from table's metadata.
This will allow to rollback the change without knowing the system/config's default.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>
 2021-07-01 20:09:54 +01:00
Aleksei Semiglazov
f47e1ff102
CLICKHOUSE-1194: add skipping index to the beginning of the list 
add the FIRST keyword to the ADD INDEX command to be able to add index in the beginning of the list.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>
 2021-07-01 19:37:48 +01:00
Kruglov Pavel
e23cf79338
Merge pull request #25596 from ildus/map_add_for_maps 
Support Map type in mapAdd and mapSubtract
 2021-07-01 21:08:07 +03:00
kssenii
4c8d82001f Fix checks 2021-07-01 17:52:21 +00:00
kssenii
9977629c89 Fix 2021-07-01 17:26:15 +00:00
kssenii
4d4ba4b5b1 More types for json functions 2021-07-01 16:46:34 +00:00
feng lv
9ad405cae8 update 
update

fix test

fix test
 2021-07-01 15:19:36 +00:00
neng.liu
17bbfd7722 fix some test error 2021-07-01 08:13:43 -07:00
alesapin
7792b09911 Reset counter if background pool is full 2021-07-01 16:10:07 +03:00
feng lv
fe22563465 fix conflict 2021-07-01 13:09:33 +00:00
alesapin
1a4ccab8e6 Fix style 2021-07-01 15:12:27 +03:00
Ildus Kurbangaliev
c86739794f Change error code in mapAdd 2021-07-01 13:41:12 +02:00
Ildus Kurbangaliev
f56ae8f9cd
Apply suggestions from code review 
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
 2021-07-01 13:38:13 +02:00
mergify[bot]
98fa9f7951
Merge branch 'master' into better_remove_empty_parts 2021-07-01 11:32:43 +00:00
kssenii
4ad0f45f0e Some more tests 2021-07-01 08:35:28 +00:00
kssenii
f01c8edbff Fix access to tables after restart 2021-07-01 07:33:58 +00:00
tavplubix
afbc6bf17d
Merge pull request #25684 from ClickHouse/cancel_merges_on_drop_partition 
Cancel merges on DROP PARTITION
 2021-06-30 23:41:04 +03:00
alesapin
c845a205f9 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 22:42:02 +03:00
alesapin
0193a9d087 Fix locking 2021-06-30 22:41:25 +03:00
alesapin
45edc8c861 Fix tidy 2021-06-30 22:11:00 +03:00
Azat Khuzhin
fc46effe16 Fix race between DETACH and merges 
CI reports failure of the 01442_merge_detach_attach test [1]:

<details>

    2021-06-21 02:25:43 01442_merge_detach_attach:                                              [ FAIL ] 122.37 sec. - result differs with reference:
    2021-06-21 02:25:43 --- /usr/share/clickhouse-test/queries/0_stateless/01442_merge_detach_attach.reference	2021-06-21 00:43:12.000000000 +0300
    2021-06-21 02:25:43 +++ /tmp/clickhouse-test/0_stateless/01442_merge_detach_attach.stdout	2021-06-21 02:25:43.211212197 +0300
    2021-06-21 02:25:43 @@ -0,0 +1 @@
    2021-06-21 02:25:43 +2

    2021.06.21 02:25:08.930896 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Debug> executeQuery: (from [::1]:36540, using production parser) (comment: '/usr/share/clickhouse-test/queries/0_stateless/01442_merge_detach_attach.sh') ALTER TABLE t DETACH PARTITION tuple()
    2021.06.21 02:25:08.931245 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> executeQuery: (from [::1]:36542, using production parser) (comment: '/usr/share/clickhouse-test/queries/0_stateless/01442_merge_detach_attach.sh') OPTIMIZE TABLE t FINAL
    2021.06.21 02:25:08.931826 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Trace> ContextAccess (default): Access granted: ALTER DELETE ON test_89nl0v.t
    2021.06.21 02:25:08.932159 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Trace> ContextAccess (default): Access granted: OPTIMIZE ON test_89nl0v.t
    2021.06.21 02:25:08.932889 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Information> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249): Detaching all_143_143_0
    2021.06.21 02:25:08.932921 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Selected 2 parts from all_143_143_0 to all_144_144_0
    2021.06.21 02:25:08.933530 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 4.60 TiB.
    2021.06.21 02:25:08.933705 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Information> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249): Detaching all_144_144_0
    2021.06.21 02:25:08.934215 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Merging 2 parts: from all_143_143_0 to all_144_144_0 into Compact
    2021.06.21 02:25:08.934280 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Information> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249): Detached 2 parts.
    2021.06.21 02:25:08.934948 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Selected MergeAlgorithm: Horizontal
    2021.06.21 02:25:08.936090 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_143_143_0, total 1 rows starting from the beginning of the part, column x
    2021.06.21 02:25:08.937621 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_144_144_0, total 1 rows starting from the beginning of the part, column x
    2021.06.21 02:25:08.938124 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.
    2021.06.21 02:25:08.939928 [ 100543 ] {} <Debug> TCPHandler: Processed in 0.018537432 sec.
    2021.06.21 02:25:08.942140 [ 100543 ] {} <Debug> TCPHandler: Done processing connection.
    2021.06.21 02:25:08.948343 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Merge sorted 2 rows, containing 1 columns (1 merged, 0 gathered) in 0.014203821 sec., 140.80718139154246 rows/sec., 140.81 B/sec.
    2021.06.21 02:25:08.952021 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Trace> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249): Renaming temporary part tmp_merge_all_143_144_1 to all_143_144_1.
    2021.06.21 02:25:08.952869 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Warning> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Unexpected number of parts removed when adding all_143_144_1: 0 instead of 2
    2021.06.21 02:25:08.953264 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Trace> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Merged 2 parts: from all_143_143_0 to all_144_144_0
    2021.06.21 02:25:08.953913 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> MemoryTracker: Peak memory usage: 4.01 MiB.
    2021.06.21 02:25:08.958369 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> MemoryTracker: Peak memory usage (for query): 4.01 MiB.
    ...
    2021.06.21 02:25:09.216075 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> executeQuery: (from [::1]:36544, using production parser) (comment: '/usr/share/clickhouse-test/queries/0_stateless/01442_merge_detach_attach.sh') SELECT count() FROM t HAVING count() > 0
    2021.06.21 02:25:09.229491 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> ContextAccess (default): Access granted: SELECT(x) ON test_89nl0v.t
    2021.06.21 02:25:09.232000 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
    2021.06.21 02:25:09.239907 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (SelectExecutor): Key condition: unknown
    2021.06.21 02:25:09.240358 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (SelectExecutor): Selected 1/1 parts by partition key, 1 parts by primary key, 1/1 marks by primary key, 1 marks to read from 1 ranges
    2021.06.21 02:25:09.241560 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> MergeTreeSelectProcessor: Reading 1 ranges from part all_143_144_1, approx. 2 rows starting from 0
    2021.06.21 02:25:09.256053 [ 58403 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> AggregatingTransform: Aggregating
    2021.06.21 02:25:09.256410 [ 58403 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> Aggregator: Aggregation method: without_key
    2021.06.21 02:25:09.257576 [ 58403 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> AggregatingTransform: Aggregated. 2 to 1 rows (from 2.00 B) in 0.013910412 sec. (143.777 rows/sec., 143.78 B/sec.)
    2021.06.21 02:25:09.257911 [ 58403 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> Aggregator: Merging aggregated data
    2021.06.21 02:25:09.262595 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Information> executeQuery: Read 2 rows, 2.00 B in 0.045805192 sec., 43 rows/sec., 43.66 B/sec.
    2021.06.21 02:25:09.263337 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.

</details>

  [1]: https://clickhouse-test-reports.s3.yandex.net/25513/b96df135aee40b1a54a4fc3f0d4db89e11385564/functional_stateless_tests_(memory).html#fail1

The problem here is that there is a tiny race window between DETACH
and OPTIMIZE.

DETACH cancel merges, and checks that there no merges for parts
currently under lock, and later changes the state of parts. After it
allows merges.

However OPTIMIZE get parts for processing under lock and only when it
starts writing temporary part it checks that merges are not canceled.

So suppose the following scenario:

      T#OPTIMIZE                       T#DETACH

                                       - cancel merges
                                           - lock
                                             - check that no merges are in progress
                                           - unlock
    - lock
      - get parts

          *<--Here, state of the parts are not changed yet,
              and lock is not held,
              so OPTIMIZE may (and will) get those parts. -->*

      - write temporary part
        - check merges not canceled
    - unlock
                                           - change the parts state
                                       - allows merges

Plus this patch will also fail merges earlier in case of concurrent
DETACH.

Refs: #13746
Refs: #23315
 2021-06-30 21:13:31 +03:00
Kseniia Sumarokova
360f772762
Merge pull request #25795 from sand6255/MySQL-column-comments-support 
MySQL Engine column comments support
 2021-06-30 20:18:47 +03:00
Kseniia Sumarokova
ed687f45a1
Merge branch 'master' into truncate 2021-06-30 20:08:42 +03:00
alesapin
89a401a84f Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 18:53:17 +03:00
alesapin
6a73c8b49e Review fixes 2021-06-30 18:24:51 +03:00
tavplubix
c958f876b8
Update MergeList.h 2021-06-30 15:55:26 +03:00
alesapin
21f15be990 Merge branch 'better_remove_empty_parts' into fix_drop_part_in_queue 2021-06-30 15:30:32 +03:00
alesapin
2eb27540b2 Some test version 2021-06-30 15:29:09 +03:00
Maksim Kita
b47bb086e7
Merge pull request #25817 from ianton-ru/fix_start_client_in_wrong_dir 
Fix start clickhouse-client with unreadable working directory
 2021-06-30 14:14:33 +03:00
Maksim Kita
53dc831a60
Update configReadClient.cpp 2021-06-30 12:42:44 +03:00
NengLiu
00377e7c9b fix style error 2021-06-30 17:29:13 +08:00
alesapin
94d4d32e4c
Merge pull request #25783 from ClickHouse/followup_drop_part_fix 
Relax check during DROP PART
 2021-06-30 12:25:42 +03:00
mergify[bot]
1799804243
Merge branch 'master' into fix_aggregation_ttl 2021-06-30 09:15:46 +00:00
Anton Ivashkin
8aae56f24a Minor fixes based on code review result 2021-06-30 12:06:11 +03:00
mergify[bot]
8182b6e890
Merge branch 'master' into better_remove_empty_parts 2021-06-30 08:42:26 +00:00
Maksim Kita
1e9e073b0a
Merge pull request #25618 from vdimir/slow-dict-join-fix 
Fix unnecessary dictionary re-reads in KeyGetterForDict
 2021-06-30 10:54:48 +03:00
kssenii
1a5c30e249 Merge branch 'pg-ch-replica' of github.com:kssenii/ClickHouse into pg-ch-replica 2021-06-29 23:20:33 +00:00
kssenii
da70f85d4e More tests 2021-06-29 23:11:46 +00:00
Maksim Kita
5b3fc9b49e Added comment 2021-06-30 01:46:57 +03:00
kssenii
9d5847a1a8 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-29 21:30:31 +00:00
Maksim Kita
b2177ab723 ExpressionCache destruction fix 2021-06-29 23:40:15 +03:00
alesapin
0af53ace55 Fix comment 2021-06-29 22:49:41 +03:00
alesapin
ca1d21b663
Update src/Storages/MergeTree/MergeTreeData.cpp 
Co-authored-by: tavplubix <avtokmakov@yandex-team.ru>
 2021-06-29 22:48:46 +03:00
alesapin
a6834213a1 Add tests 2021-06-29 22:47:54 +03:00
Maksim Kita
567f6a2b6f
Merge pull request #25819 from egatov/master 
METR-41529
 2021-06-29 21:35:16 +03:00
alexey-milovidov
48348e6375
Update configReadClient.cpp 2021-06-29 21:12:10 +03:00
Maksim Kita
4bb3fd53f7
Merge pull request #24129 from kitaisreal/date-time-timezone-fix 
Date time timezone fix
 2021-06-29 20:13:37 +03:00
Nikolai Kochetov
d5bd2b1fa9
Merge pull request #25719 from ClickHouse/remove-PrewhereDAGInfo 
Remove PrewhereDAGInfo.
 2021-06-29 18:58:03 +03:00
alesapin
5822f0ba29 Replace lost parts with empty parts instead of hacking replication queue 2021-06-29 18:14:44 +03:00
kssenii
6816e8c5aa Revert change, fix injectiveness 2021-06-29 14:55:29 +00:00
Evgeniy Gatov
739966dc99 METR-41529 2021-06-29 17:51:53 +03:00
Anton Ivashkin
955c031474 Fix start clickhouse-client with unreadable working directory 2021-06-29 17:37:12 +03:00
mergify[bot]
c6cda5ebce
Merge branch 'master' into followup_drop_part_fix 2021-06-29 12:43:54 +00:00
Nikolai Kochetov
bf827936b7 Rename PrewhereActions 2021-06-29 14:53:34 +03:00
alesapin
7680eab0e4 one more fix 2021-06-29 13:03:00 +03:00
alesapin
cdc95fa98a
Merge pull request #25428 from ClickHouse/zookeeper_snapshots 
Tool for conversion of ZooKeeper data into clickhouse-keeper snapshot
 2021-06-29 12:58:45 +03:00
kssenii
e49930fde3 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into inconsisteny 2021-06-29 09:14:12 +00:00
Maksim Kita
41620a5661 Fixed tests 2021-06-29 11:05:07 +03:00
alesapin
941a6e539f Fix 2021-06-29 10:36:20 +03:00
Kostiantyn Storozhuk
a9510f25f9 Style fix 2021-06-29 15:22:08 +08:00
alesapin
2b3d50f0e6
Merge pull request #25771 from azat/version-revision-comment 
Add comments for VERSION_REVISION vs DBMS_TCP_PROTOCOL_VERSION
 2021-06-29 10:07:13 +03:00
Kostiantyn Storozhuk
296593716d Reverted constructor change 2021-06-29 14:37:53 +08:00
alexey-milovidov
9a33dd8f9f
Merge pull request #25796 from Algunenano/storages_typos 
Fix some typos in Storage classes
 2021-06-29 04:25:02 +03:00
alexey-milovidov
5b21ca72ac
Merge pull request #25801 from azat/storage-merge-no-col-fix 
Fix "No available columns" for Merge() storage
 2021-06-29 04:23:52 +03:00
alexey-milovidov
878a159538
Merge pull request #25773 from ClickHouse/system-query-log-map-type-merge 
Merging #23934
 2021-06-29 04:19:53 +03:00
Alexey Milovidov
a7ee0b91b1 Fix strange whitespaces @abyss7 2021-06-29 01:52:22 +03:00
Ildus Kurbangaliev
5d89907ae9 Fix style errors 2021-06-29 00:34:45 +02:00
alexey-milovidov
2243aed4c5
Merge pull request #25787 from ClickHouse/tavplubix-patch-2 
Remove dead code from VirtualColumnUtils.cpp
 2021-06-29 01:25:58 +03:00
Azat Khuzhin
15a14d6da2 Fix "No available columns" for Merge() storage 2021-06-29 00:39:38 +03:00
Azat Khuzhin
e590d9c8bb Add comments for VERSION_REVISION vs DBMS_TCP_PROTOCOL_VERSION 2021-06-28 22:01:14 +03:00
Kseniia Sumarokova
d9c46003cb
Merge pull request #25781 from sand6255/MaterializeMySQL-Engine-column-comments-improvement 
MaterializeMySQL: Improved column comments support
 2021-06-28 21:57:24 +03:00
Raúl Marín
bfc122df64 Fix some typos in Storage classes 2021-06-28 19:03:56 +02:00
Maksim Kita
764380f701 Updated tests 2021-06-28 19:56:15 +03:00
mergify[bot]
da09856a94
Merge branch 'master' into followup_drop_part_fix 2021-06-28 16:30:55 +00:00
Kseniia Sumarokova
9d02af7d8b
Merge pull request #25676 from sand6255/MaterializeMySQL-Support-Enum-Data-Type 
MaterializeMySQL: support ENUM data type
 2021-06-28 19:15:00 +03:00
Kostiantyn Storozhuk
c2c78929cb Implemented MySQL column comments support 2021-06-28 22:44:38 +08:00
Kseniia Sumarokova
ff8c44179c
Update DatabaseAtomic.cpp 2021-06-28 17:37:22 +03:00
Kseniia Sumarokova
a96c7a3a70
Update DatabaseAtomic.cpp 2021-06-28 17:32:03 +03:00
mergify[bot]
04814114e7
Merge branch 'master' into zookeeper_snapshots 2021-06-28 14:17:32 +00:00
alesapin
72b281987e Add more debug 2021-06-28 17:15:51 +03:00
Kseniia Sumarokova
54a7e2158c
Update ActionsDAG.cpp 2021-06-28 17:09:26 +03:00
tavplubix
3afb7aa97e
Update VirtualColumnUtils.cpp 2021-06-28 16:47:47 +03:00
kssenii
48d21bb03b Fix inconsistency 2021-06-28 13:20:23 +00:00
Anton Popov
03507f548e
Merge branch 'master' into storage-merge-aliases 2021-06-28 16:03:10 +03:00
alesapin
d20feeb42c Relax check during DROP PART 2021-06-28 15:05:22 +03:00
Maksim Kita
352e1f27ff Update using Map datatyle in system log tables before merge 2021-06-28 14:42:21 +03:00
alesapin
b11254d191
Merge branch 'master' into fix_aggregation_ttl 2021-06-28 13:31:29 +03:00
alesapin
71603a7d13
Merge pull request #25772 from ClickHouse/fix_mutation_test 
Fix flaky test and wrong message
 2021-06-28 13:30:36 +03:00
alesapin
4c213b639f
Merge pull request #25548 from nikitamikhaylov/background-processing 
A little improvement in BackgroundJobsExecutor
 2021-06-28 13:24:21 +03:00
Zijie Lu
f20a1f708b Merge branch 'master' into support-distinct-on 2021-06-28 16:38:29 +08:00
neng.liu
ec7604b013 add parquet test and fix add interval error when type is date32 2021-06-28 16:36:54 +08:00
alesapin
7e73762b48 Fix flaky test and wrong message 2021-06-28 11:28:45 +03:00
Maksim Kita
294b937a39
Merge branch 'system-query-log-map-type-merge' into system-querylog-map-continue 2021-06-28 11:27:59 +03:00
alesapin
1b56b0a020 Fix flaky test 2021-06-28 11:10:38 +03:00
Kostiantyn Storozhuk
ca4783d854 Fixed typo and casting 2021-06-28 15:22:13 +08:00
Kseniia Sumarokova
b6b7924413
Update DatabaseAtomic.cpp 2021-06-28 10:20:33 +03:00
Zijie Lu
b408c7d7e6 Merge branch 'master' into support-distinct-on 
Signed-off-by: Zijie Lu <wslzj40@gmail.com>
 2021-06-28 14:53:55 +08:00
robot-clickhouse
648801760e Auto version update to [21.8.1.1] [54453] 2021-06-28 00:50:20 +03:00
alexey-milovidov
a11351a473
Merge pull request #25484 from vdimir/issue-19589 
Fix assert with non uint8 in prewhere
 2021-06-28 00:03:12 +03:00
alexey-milovidov
16f88fd985
Merge pull request #25693 from novikd/master 
Introduce system.data_skipping_indices table
 2021-06-27 23:24:53 +03:00
kssenii
aa7133c239 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-27 20:08:00 +00:00
kssenii
c3a2fc0584 Review fixes 2021-06-27 19:17:17 +00:00
Maksim Kita
61b116332d Fixed tests 2021-06-27 20:53:02 +03:00
Maksim Kita
e6adc405d8 DateTime timezone fix 2021-06-27 20:53:02 +03:00
alexey-milovidov
fb89505656
Merge pull request #25538 from kssenii/pg-dix 
Fix postrges array conversion
 2021-06-27 19:35:24 +03:00
alexey-milovidov
0a26687115
Merge pull request #23864 from azat/dist-split-batch-and-retry 
Add ability to split distributed batch on failures (i.e. due to memory limits)
 2021-06-27 19:28:26 +03:00
kssenii
db998c3f6c More robust 2021-06-27 16:28:10 +00:00
alesapin
c977c33d6d Fix bug in execution of TTL GROUP BY 2021-06-27 19:18:15 +03:00
Alexey Milovidov
b11f9223de Merge branch 'master' into fix-receive-time-metric 2021-06-27 19:00:39 +03:00
alexey-milovidov
1b644b9a31
Merge pull request #25663 from azat/dist-startup 
Improve startup time of Distributed engine.
 2021-06-27 18:22:45 +03:00
alexey-milovidov
f6e67d3dc1
Update StorageDistributed.cpp 2021-06-27 18:22:34 +03:00
alexey-milovidov
6e25d11e1f
Merge pull request #25582 from azat/atomic-unlink-symlink 
Remove only symlinks during force_restore_data of Atomic engine
 2021-06-27 18:19:59 +03:00
alexey-milovidov
6e0c8f0122
Merge pull request #25737 from amosbird/dataracereloadcluster 
Fix data race in getClusters()
 2021-06-27 17:55:28 +03:00
Alexey Milovidov
9ab463efdd utils/generate-ya-make/generate-ya-make.sh 2021-06-27 17:49:26 +03:00
Alexey Milovidov
4dbd659d64 Change error code in LIVE VIEW 2021-06-27 15:40:07 +03:00
Alexey Milovidov
4b994fc3c7 Change error code in LIVE VIEW 2021-06-27 15:38:08 +03:00
alexey-milovidov
40095a9bfe
Update StorageSystemDataSkippingIndices.cpp 2021-06-27 15:03:56 +03:00
feng lv
06242f85e6 fix 2021-06-27 06:09:23 +00:00
alexey-milovidov
ec738c6170
Update StorageSystemDataSkippingIndices.cpp 2021-06-27 03:25:04 +03:00
alexey-milovidov
80d9346080
Update StorageSystemDataSkippingIndices.h 2021-06-27 03:20:28 +03:00
mergify[bot]
8ca52ea2df
Merge branch 'master' into master 2021-06-27 00:17:31 +00:00
kssenii
3d90f0710c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-26 22:04:47 +00:00
Amos Bird
4da1b8154a
Fix data race in getClusters() 2021-06-26 22:15:57 +08:00
Kseniia Sumarokova
28c5a14922
Fix clang-tidy 2021-06-26 15:50:15 +03:00
Maksim Kita
44122118de Fixed ASTRenameQuery for EXCHANGE DICTIONARIES 2021-06-26 14:28:13 +03:00
Maksim Kita
3c9ae7b5ba Added more tests 2021-06-26 14:28:13 +03:00
Maksim Kita
7b03150b90 Fixed tests 2021-06-26 14:28:13 +03:00
Maksim Kita
008adabec2 Support REPLACE DICTIONARY, CREATE OR REPLACE DICTIONARY queries 2021-06-26 14:28:13 +03:00
alexey-milovidov
6662e7ab0f
Merge pull request #25732 from matwey/missed_include_optional_2 
Fix some more missed includes
 2021-06-26 14:04:06 +03:00
Matwey V. Kornilov
a29bd2c4b3 Add missed #include <limits> 
Signed-off-by: Matwey V. Kornilov <matwey.kornilov@gmail.com>
 2021-06-26 11:25:54 +03:00
Matwey V. Kornilov
9306b598cf Merge branch 'missed_include_optional' into missed_include_optional_2 2021-06-26 11:07:59 +03:00
Matwey V. Kornilov
2f73469629 Merge branch 'missed_include_condition_variable' into missed_include_optional_2 2021-06-26 11:07:50 +03:00
alexey-milovidov
420839a4c0
Merge pull request #25697 from sundy-li/mysql-fix 
[mysql] fix mysql select user() return empty
 2021-06-26 03:31:46 +03:00
l1tsolaiki
7892e44467 Fix style yet again 😎 2021-06-25 21:14:08 +03:00
l1tsolaiki
6981eb64ac Fixes 2021-06-25 19:24:22 +03:00
l1tsolaiki
f8b1a6d185 WriteBuffer is canceled 😳 2021-06-25 18:33:31 +03:00
Ivan
14e43cfafa
Add CI check for darwin-aarch64 (#25560) 
* Add support for darwin-aarch64

* Fix PVS warnings

* Add build configuration

* Fix packager args
 2021-06-25 18:21:56 +03:00
feng lv
61550f91f2 fix and update test 2021-06-25 15:01:57 +00:00
Nikolai Kochetov
3544b059ce Merge branch 'master' into remove-PrewhereDAGInfo 2021-06-25 17:52:03 +03:00
Nikolai Kochetov
6bc0a628cd Remove PrewhereDAGInfo. 2021-06-25 17:49:28 +03:00
Kseniia Sumarokova
1eb005850c
Merge branch 'master' into storage-merge-aliases 2021-06-25 17:33:31 +03:00
kssenii
c2ac9b6027 Better 2021-06-25 14:31:56 +00:00
Kostiantyn Storozhuk
ca04b077ef Improved column comments support 2021-06-25 22:30:11 +08:00
Nikita Mikhaylov
06bad99720
Update src/Functions/geoToH3.cpp 
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
 2021-06-25 17:07:46 +03:00
Nikita Mikhaylov
20d20a5389
Merge pull request #25666 from georgthegreat/murmurhash 
Enable MurmurHash in ArcadiaBuild
 2021-06-25 17:05:02 +03:00
feng lv
07a47b4bd3 fix 2021-06-25 13:51:17 +00:00
Nikita Mikhaylov
930a2aa0b7
Merge pull request #24095 from vdimir/view-join-settings 
Use global settings for query to VIEW
 2021-06-25 16:44:28 +03:00
Anton Popov
4a250dc14e
Merge pull request #25371 from CurtizJ/add-compatibility-setting 
Add compatibility setting for name of tuple literals
 2021-06-25 15:08:13 +03:00
vdimir
2949cd1e6f
Support ALIASed columns in JOIN ON expression 2021-06-25 14:50:02 +03:00
Kostiantyn Storozhuk
3c71e067fa Update docs and style 2021-06-25 19:27:03 +08:00
Kostiantyn Storozhuk
7516d39091 Style fixes 2021-06-25 16:44:28 +08:00
Storozhuk Kostiantyn
4a3145f586 Materialize my sql support enum data type 
* Implemented Enum for MaterializeMySQL
 2021-06-25 16:42:25 +08:00
alesapin
f6afc381be
Merge pull request #25665 from ClickHouse/fix_24869 
Fix REPLACE_RANGE entry removal
 2021-06-25 11:11:25 +03:00
feng lv
228e06628a fix conflict 2021-06-25 08:06:06 +00:00
alesapin
d138ba79c7
Merge pull request #25549 from ClickHouse/add_test_for_concurrent_merges 
Fix bug in TTL merges assignment
 2021-06-25 11:02:29 +03:00
feng lv
b54287c8da fix 2021-06-25 08:00:30 +00:00
kssenii
ded9007ca0 Fix 2021-06-25 07:49:38 +00:00
kssenii
ca1f336c46 Aliases for storageMerge fix 2021-06-25 05:28:44 +00:00
sundy-li
d423d07a87 [mysql] fix mysql select user() return empty 2021-06-25 10:20:22 +08:00
Dmitry Novik
960d0de73d Introduce system.data_skipping_indices table 2021-06-25 02:05:45 +03:00
tavplubix
a14724ca7b
Merge pull request #25667 from ClickHouse/fix_rename_distributed2 
Fix rename Distributed table
 2021-06-25 00:05:06 +03:00
alexey-milovidov
89007a8b6a
Merge pull request #25654 from azat/altstack-fix-osx 
Fix alternative stack (MINSIGSTKSZ) and stack size check under osx
 2021-06-24 19:22:14 +03:00
vdimir
241b64d02c
Support ALIASed columns for right joined table 2021-06-24 17:57:21 +03:00
vdimir
af7776554b
Fix space in ExpressionAnalyzer.cpp 2021-06-24 17:16:57 +03:00
vdimir
84e02911cf
Materialized columns for joined table, don't add to asterisk select 2021-06-24 17:14:36 +03:00
Alexander Tokmakov
76156af5cc cancel merges on drop partition 2021-06-24 17:07:43 +03:00
alesapin
230286776d Merge branch 'master' into add_test_for_concurrent_merges 2021-06-24 16:37:25 +03:00
Ivan
e9e49a9085
Fix ANTLR parser and enable it back in CI (#25638) 
* Fix

* Skip more tests
 2021-06-24 14:45:29 +03:00
Alexander Tokmakov
3a25b05765 fix rename Distributed table 2021-06-24 13:00:33 +03:00
Yuriy Chernyshov
9dcd37b94d Enable MurmurHash in ArcadiaBuild 2021-06-24 12:04:40 +03:00
Alexander Tokmakov
7e6963bb39 fix REPLACE_RANGE entry removal 2021-06-24 11:16:54 +03:00
Azat Khuzhin
8262e251c4 Fix getStackSize() stack address on osx 
Should fix the following exception under osx:

  Stack size too large. Stack address: 0x700001a58000, frame address: 0x700001a53f40, stack size: 540864, maximum stack size: 524288 (version 21.7.1.1)
 2021-06-24 10:48:43 +03:00
tavplubix
b1263c18ee
Fix pcg deserialization (#24538) 
* fix pcg deserialization

* Update 01156_pcg_deserialization.sh

* Update 01156_pcg_deserialization.sh

* Update 01156_pcg_deserialization.sh

* fix another bug

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
 2021-06-24 10:40:00 +03:00
Azat Khuzhin
a616ae8861 Improve startup time of Distributed engine. 
- create directory monitors in parallel (this also includes rmdir in
  case of directory is empty, since even if the directory is empty it
  may take some time to remove it, due to waiting for journal or if the
  directory is large, i.e. it had lots of files before, since remember
  ext4 does not truncate the directory size on each unlink [1])
- initialize increment in parallel too (since it does readdir())

  [1]: https://lore.kernel.org/linux-ext4/930A5754-5CE6-4567-8CF0-62447C97825C@dilger.ca/
 2021-06-24 10:27:51 +03:00
Azat Khuzhin
3c76986311 Fix alternative stack under osx (MINSIGSTKSZ is 32K) 
Fixes: #25632
 2021-06-24 00:33:33 +03:00
alexey-milovidov
bbb216f4c8
Merge pull request #25604 from ClickHouse/roaring-bitmap-small-change 
Small change in Roaring Bitmaps
 2021-06-24 00:29:38 +03:00
alesapin
e6da53dc46 better 2021-06-24 00:09:06 +03:00
alesapin
24876291d1 Followup fix 2021-06-24 00:01:07 +03:00
alesapin
4be4bc21e2 Fix for fix 2021-06-23 23:57:49 +03:00
alexey-milovidov
99f60a7c1d
Merge pull request #25513 from azat/use_compact_format_in_distributed_parts_names-2 
Drop replicas from dirname for internal_replication=true
 2021-06-23 22:51:07 +03:00
alexey-milovidov
6adf0955c2
Merge pull request #25400 from georgthegreat/murmurhash 
Rename & reimport murmurhash sources from smhasher repo
 2021-06-23 22:48:29 +03:00
alesapin
d744136c0d Merge branch 'master' into add_test_for_concurrent_merges 2021-06-23 22:26:15 +03:00
alesapin
81c74435a3 Fix drop part bug 2021-06-23 22:25:30 +03:00
Nikolai Kochetov
8bfa26102b
Merge pull request #25631 from ClickHouse/fix-explain-ast-no-query-crash 
Fix crash in EXPLAIN AST without query.
 2021-06-23 19:56:59 +03:00
Kruglov Pavel
4913b18532
Merge pull request #25531 from Avogar/fix-arrow-decimal 
Fix segfault in Arrow format, add arrow Decimal256 support in Arrow and Parquet.
 2021-06-23 17:28:16 +03:00
vdimir
b00efaf3d1
Add materialized columns to joined columns 2021-06-23 17:03:39 +03:00
Ildus Kurbangaliev
541c09d7b3 Fix tests errors 2021-06-23 15:03:58 +02:00
Nikolai Kochetov
c92c352d9a Fix crash in EXPLAIN AST without query. 2021-06-23 15:29:10 +03:00
l1tsolaiki
abe7e4195e . 2021-06-23 15:02:47 +03:00
vdimir
4ce829d7c1
Create KeyGetter outside of joinRightColumns 2021-06-23 14:27:39 +03:00
vdimir
8f9166df4e
Add read flag to KeyGetterForDict 2021-06-23 14:15:31 +03:00
Anton Popov
a15ee72d0f Merge remote-tracking branch 'upstream/master' into HEAD 2021-06-23 13:53:01 +03:00
Anton Popov
93f15e443d
Merge pull request #25539 from CurtizJ/with-fill-totals 
Fix usage of `WITH FILL` together with `WITH TOTALS`
 2021-06-23 13:42:27 +03:00
l1tsolaiki
ec7ec63a40 Fix style 2021-06-23 13:22:38 +03:00
l1tsolaiki
a5d3600f20 Fix compile errors with WriteBuffer 2021-06-23 12:41:38 +03:00
Kseniia Sumarokova
3e238c1767
Merge pull request #25601 from ianton-ru/fix_s3_restore 
Fix DiskS3 restore
 2021-06-23 12:34:50 +03:00
mergify[bot]
fddf47a900
Merge branch 'master' into zookeeper_snapshots 2021-06-23 08:36:14 +00:00
Azat Khuzhin
50d9d3efd5 Remove only symlinks during force_restore_data of Atomic engine 2021-06-23 10:17:31 +03:00
Azat Khuzhin
3bd53c68f9 Try to split the batch in case of broken batch too 
Broken batches may be because of abnormal server shutdown (and lack of
fsync), and ignoring the whole batch is not great in this case, so apply
the same split logic here too.

v2: rename exception
v3: catch missing exception
v4: fix marking the file as broken multiple times (fixes
test_insert_distributed_async_send with setting enabled)
 2021-06-23 02:48:47 +03:00
Azat Khuzhin
a0209178cc Add ability to split distributed batch on failures 
Add distributed_directory_monitor_split_batch_on_failure setting (OFF by
default), that will split the batch and send files one by one in case of
retriable errors.

v2: more error codes
 2021-06-23 02:48:47 +03:00
Azat Khuzhin
e148ef739d Drop replicas from dirname for internal_replication=true 
Under use_compact_format_in_distributed_parts_names=1 and
internal_replication=true the server encodes all replicas for the
directory name for async INSERT into Distributed, and the directory name
looks like:

    shard1_replica1,shard1_replica2,shard3_replica3

This is required for creating connections (to specific replicas only),
but in case of internal_replication=true, this can be avoided, since
this path will always includes all replicas.

This patch replaces all replicas with "_all_replicas" marker.

Note, that initial problem was that this path may overflow the NAME_MAX
if you will have more then 15 replicas, and the server will fail to
create the directory.

Also note, that changed directory name should not be a problem, since:
- empty directories will be removed since #16729
- and replicas encoded in the directory name is also supported anyway.
 2021-06-23 02:47:38 +03:00
Nikita Mikhaylov
000c937ebe fix pvs check 2021-06-22 23:24:47 +00:00
Nikita Mikhaylov
be4aff0c19 delete test 2021-06-22 23:24:47 +00:00
Nikita Mikhaylov
f4d6a6e128 support dynamic reloading of server config 2021-06-22 23:24:47 +00:00
Nikita Mikhaylov
a39bca0acf better 2021-06-22 23:24:47 +00:00
Nikita Mikhaylov
c66a3b22b5 done 2021-06-22 23:24:47 +00:00
Alexey Milovidov
afeb562744 Small change in Roaring Bitmaps 2021-06-23 00:29:20 +03:00
alesapin
a92bd49229 Supress PVS 2021-06-22 23:23:26 +03:00
Alexey Milovidov
99e08b7406 Make network receive time metric to correctly include the time spent waiting for data from the client to INSERT #9958 2021-06-22 23:22:13 +03:00
Anton Ivashkin
e0199d8006 Fix DiskS3 restore 2021-06-22 20:05:20 +03:00
Anton Popov
18fad6d322 Merge remote-tracking branch 'upstream/master' into HEAD 2021-06-22 19:20:59 +03:00
mergify[bot]
c15493f0df
Merge branch 'master' into add_test_for_concurrent_merges 2021-06-22 15:49:35 +00:00
alexey-milovidov
d6b93f3af8
Merge pull request #25584 from kelvich/patch-1 
Fix typo in hardware failure error message
 2021-06-22 18:38:00 +03:00
Ildus Kurbangaliev
0a937e7377 Support Map type in mapAdd and mapSubtract 2021-06-22 17:02:42 +02:00
Yatsishin Ilya
3578a79e08 fix style 2021-06-22 17:41:20 +03:00
vdimir
80a001ab0c
Use settings from global context in StorageView 2021-06-22 17:39:09 +03:00
Yatsishin Ilya
2b1f3d4681 Merge remote-tracking branch 'origin' into update-h3 2021-06-22 17:38:59 +03:00
vdimir
85bcf8a1dc
Merge branch 'master' into view-join-settings 2021-06-22 17:36:44 +03:00
mergify[bot]
f3512f15d5
Merge branch 'master' into zookeeper_snapshots 2021-06-22 14:29:36 +00:00
tavplubix
a6a43ce3ee
Revert "Implement h3ToGeo function" 2021-06-22 17:25:21 +03:00
alesapin
47b29092b4 Fix bug in MergerMutator parts selector 2021-06-22 16:47:42 +03:00
Anton Popov
ac25b55e87
Merge pull request #25546 from CurtizJ/fix-index-array-join 
Fix usage of index with array columns and ARRAY JOIN
 2021-06-22 16:23:00 +03:00
Nikita Mikhaylov
2ec3bc2d96
Merge pull request #25520 from azat/not-parser-fix 
Fix NOT parsing
 2021-06-22 16:19:39 +03:00
Anton Popov
447fef702d fix fuzzer in query with 'WITH FILL' and 'WITH TOTALS' 2021-06-22 16:08:12 +03:00
Nikita Mikhaylov
19b3659517
Merge pull request #24867 from bharatnc/ncb/h3-indexing-funcs 
Implement h3ToGeo function
 2021-06-22 15:59:09 +03:00
Zijie Lu
63db58710d Support for DISTINCT ON (columns) 
Signed-off-by: Zijie Lu <wslzj40@gmail.com>
 2021-06-22 19:25:14 +08:00
Neng Liu
aadc2af5cb add some functional test 2021-06-22 19:10:56 +08:00
alesapin
6b815b4f17 Merge branch 'master' into zookeeper_snapshots 2021-06-22 13:50:27 +03:00
alesapin
bf0a4864ac Add support for set/get ACL commands 2021-06-22 13:49:35 +03:00
Stas Kelvich
9a84033db5
Fix typo in hardware failure error message 2021-06-22 12:48:15 +03:00
Yatsishin Ilya
f2486ac8e9 changes 2021-06-22 12:23:27 +03:00
alesapin
a8ce3bed2b
Merge pull request #25421 from ClickHouse/update_buffer_size_in_nuraft 
Update NuRaft buffer to 64 bit size
 2021-06-22 10:55:31 +03:00
Kseniia Sumarokova
155bb03239
Merge branch 'master' into pg-dix 2021-06-22 10:14:52 +03:00
Neng Liu
7ed1728a37 Merge branch 'master' into datatype-date32 
# Conflicts:
#	src/DataTypes/IDataType.h
#	src/Functions/CustomWeekTransforms.h
 2021-06-22 10:00:18 +08:00
Kseniia Sumarokova
f3b0f11b59
Update StorageHDFS.cpp 2021-06-21 23:04:58 +03:00
alesapin
71e9689ba6 Fix PVS warning 2021-06-21 22:59:19 +03:00
Kseniia Sumarokova
41c65a9965
Merge pull request #25528 from kssenii/something 
Fix convertion of datetime with timezone for mysql, odbc, etc..
 2021-06-21 22:55:22 +03:00
alesapin
731edc9a6d Fixes in ACLs 2021-06-21 18:45:45 +03:00
kssenii
ac0f86cdbf Truncate for s3 2021-06-21 15:44:36 +00:00
Anton Popov
880ed24565 move comment 2021-06-21 18:31:18 +03:00
Anton Popov
662d5b8495 fix ALTER MODIFY COLUMN of columns, that participates in TTL expressions 2021-06-21 18:20:18 +03:00
Kseniia Sumarokova
77a3a1416d
Update fetchPostgreSQLTableStructure.cpp 2021-06-21 17:26:14 +03:00
kssenii
4688f9e038 hdfs truncate table 2021-06-21 13:59:57 +00:00
alesapin
76cee4e3cf Debugging 2021-06-21 16:58:39 +03:00
Amos Bird
0adad2425a
json extract string or raw 2021-06-21 21:36:42 +08:00
vdimir
46a5dd6701
Fix MergeTreeBaseSelectProcessor::executePrewhereActions 2021-06-21 16:01:02 +03:00
Anton Popov
ffa56bde24 fix usage of index with array columns and ARRAY JOIN 2021-06-21 15:34:05 +03:00
vdimir
eb85c5a8e7
Fix assert with non uint8 in prewhere 2021-06-21 15:32:29 +03:00
kssenii
7d3dee172c FIx progress 2021-06-21 12:19:25 +00:00
Anton Popov
82f0a5f2dd fix usage of WITH FILL together with WITH TOTALS 2021-06-21 14:43:38 +03:00
kssenii
0f9fc33a4e Fix postgres arrays 2021-06-21 11:27:10 +00:00
alesapin
743600a359 Merge branch 'update_buffer_size_in_nuraft' into zookeeper_snapshots 2021-06-21 13:46:22 +03:00
kssenii
fb9c92cb6c Fix 2021-06-21 10:32:40 +00:00
alesapin
2f24ad6165 Merge branch 'fix_some_tests' into update_buffer_size_in_nuraft 2021-06-21 13:20:27 +03:00
Anton Popov
c2754c34f4 Merge remote-tracking branch 'upstream/master' into HEAD 2021-06-21 12:42:48 +03:00
alexey-milovidov
8ba6a5393f
Merge pull request #25509 from azat/fix-altstack 
Fix alternative stack for SIGSEGV handling
 2021-06-21 12:10:00 +03:00
alexey-milovidov
d2c86e52e6
Update checkStackSize.cpp 2021-06-21 12:09:14 +03:00
alexey-milovidov
f37faf02fa
Update checkStackSize.cpp 2021-06-21 12:08:30 +03:00
alexey-milovidov
9a4125dd7c
Update ThreadStatus.cpp 2021-06-21 12:06:41 +03:00
Pavel Kruglov
556ec9e533 Fix segfault in arrow and support Decimal256 2021-06-21 12:06:28 +03:00
kssenii
c0732ddc12 Fix datetime with timezone 2021-06-21 08:22:12 +00:00
Kruglov Pavel
a54cbef5b7
Merge pull request #24341 from Avogar/arrow 
Support complex types in Arrow/Parquet/ORC
 2021-06-21 11:05:57 +03:00
Azat Khuzhin
fabff86788 Fix NOT parsing 2021-06-20 23:27:24 +03:00
Anton Popov
e409ead64a fix tests 2021-06-20 23:18:05 +03:00
Anton Popov
d8b6f15ef4
Merge pull request #23027 from azat/distributed-push-down-limit 
Add ability to push down LIMIT for distributed queries
 2021-06-20 23:08:50 +03:00
Azat Khuzhin
12eadc2b5e Add a guard page for alternative signal handling stack 2021-06-20 20:08:50 +03:00
Azat Khuzhin
4dcbf3a5c5 Replace __pthread_get_minstack() with a const 2021-06-20 14:03:04 +03:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652) 
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
 2021-06-20 11:24:43 +03:00
Azat Khuzhin
9a3a1397e6 Fix alternative stack for SIGSEGV handling 
Previous stack size of 4096 wasn't enough, and this lead to that on
SIGSEGV signal handler may overwrite some TLS data (like in the
following example [1]) and in real binary "current_thread" was
overwritten, and this leads to lack of query-id on SIGSEGV, like here
[2].

  [1]: https://gist.github.com/azat/2ee360fd4f2828d363b0926431afacc6
  [2]: https://clickhouse-test-reports.s3.yandex.net/24411/79563953201ce6249a8fd49e22be3902ca4ee43a/fuzzer_ubsan/report.html#fail1

Fix this by using __pthread_get_minstack() that takes TLS block into
account. Yes it is private, but let's try (like rust tried).

This patch will also decrease TLS, since now the per-thread stack will
be allocated in the heap, not on the stack of each thread.

Refs: https://sourceware.org/legacy-ml/libc-alpha/2012-06/msg00335.html
Refs: https://maskray.me/blog/2021-02-14-all-about-thread-local-storage
Refs: https://sourceware.org/bugzilla/show_bug.cgi?id=11787
 2021-06-19 22:47:01 +03:00
Azat Khuzhin
57960746e2 Add getStackSize() helper 2021-06-19 20:09:01 +03:00
Azat Khuzhin
c6339b4c08 Stable NOT chain formatting 2021-06-19 08:03:36 +03:00
Vladimir
1c91a8d2c2
Merge pull request #25454 from vdimir/issue-23351-v2 2021-06-18 21:56:46 +03:00
alesapin
b5dae909dd Add some tests 2021-06-18 21:36:19 +03:00
Kseniia Sumarokova
4fe722d1a4
Merge pull request #25431 from vitlibar/no-underscore_before-preprocessed_configs 
Fix underscore before preprocessed configs.
 2021-06-18 17:53:03 +03:00
Kseniia Sumarokova
5974f96f0c
Merge pull request #23656 from kssenii/progress-bar 
Progress indication for File table engine in clickhouse-local and insert from file in clickhouse-client
 2021-06-18 17:46:54 +03:00
mergify[bot]
c723dd7d40
Merge branch 'master' into arrow 2021-06-18 12:18:57 +00:00
vdimir
0ebca58e86
Check column in removeColumnNullability 2021-06-18 14:06:55 +03:00
Vladimir
4de0d1a593
Merge pull request #24885 from Jokser/disk-s3-read-error-fix 2021-06-18 13:38:59 +03:00
alesapin
2d8f45a098 Add some initialization 2021-06-18 11:55:59 +03:00
Vladimir
6af272891a
Merge pull request #24870 from azat/mv-dist-join-fix 2021-06-18 11:18:43 +03:00
vdimir
b25f6ba5b6
Remove assumeMutable from removeColumnNullability 2021-06-18 11:13:53 +03:00
Vladimir
fab7c9c7f6
Merge pull request #24061 from vdimir/issue-23351 2021-06-18 11:07:28 +03:00
Vladimir
51e5a019ed
Merge pull request #25410 from vdimir/issue-21730 2021-06-18 11:00:26 +03:00
kssenii
f7029b168b Generate ya make 2021-06-18 10:56:24 +03:00
kssenii
bdc8fe1ca0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into progress-bar 2021-06-18 09:33:29 +03:00
Nikolai Kochetov
b2620e1def
Merge pull request #24405 from amosbird/replacemorealias 
Rewrite more alias columns
 2021-06-18 09:10:31 +03:00
Vitaly Baranov
fc3d77bd66 Fix underscore before preprocessed configs. 2021-06-18 06:14:11 +03:00
l1tsolaiki
1863a9beb0 Change stringstream to WriteBuffer 2021-06-18 02:15:15 +03:00
l1tsolaiki
e2765991b0 Separate star from ranges for better code 2021-06-18 01:53:10 +03:00
Alexey Milovidov
3e71181451 Whitespaces 2021-06-18 01:42:33 +03:00
Maksim Kita
b34b66c55d
Merge pull request #25394 from ClickHouse/function-bit-position-to-array-merge 
Merging #23843
 2021-06-18 01:31:33 +03:00
Maksim Kita
2c4c2680f7
Merge pull request #25430 from kitaisreal/database-mysql-rename 
DatabaseMySQL rename
 2021-06-18 01:30:52 +03:00
Kseniia Sumarokova
47cb288834
Update src/Common/ProgressIndication.h 
Co-authored-by: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com>
 2021-06-18 00:24:28 +03:00
Nikolai Kochetov
669b8a8b96
Merge pull request #25370 from ClickHouse/disable-query_plan_filter_push_down-because-of-25368 
Fix lost WHERE condition in expression-push-down optimization of query plan
 2021-06-17 23:07:32 +03:00
Nikolai Kochetov
5227a76e3a Add comment. 2021-06-17 23:05:39 +03:00
Maksim Kita
ca672b0a0d Function bitPositionsToArray added support for big integers 2021-06-17 22:32:39 +03:00
Pavel Kovalenko
ce8e2f7f28 Avoid double initializing ReadBufferFromS3 2021-06-17 22:31:50 +03:00
Maksim Kita
cf77e9f2c6 DatabaseMySQL rename 2021-06-17 22:26:34 +03:00
Vladimir
9e5ef95677
Merge pull request #25288 from amosbird/fixjoingetornull 2021-06-17 21:54:21 +03:00
alesapin
1747c254dc Remove unused flag 2021-06-17 21:36:50 +03:00
alexey-milovidov
35411d1085
Merge pull request #24427 from azat/formatting-fixes 
Fix formatting of negative values
 2021-06-17 21:21:47 +03:00
alexey-milovidov
496aff2118
Merge pull request #25300 from ClickHouse/simpla-agg-func-for-summing-mt 
Support simple aggregate functions for SummingMergeTree.
 2021-06-17 20:18:51 +03:00
alesapin
dbe4ba8c2c Merge branch 'update_buffer_size_in_nuraft' into zookeeper_snapshots 2021-06-17 19:51:11 +03:00
alexey-milovidov
c48402d997
Merge pull request #23434 from amosbird/preferglobal 
Add prefer_global_in_and_join setting
 2021-06-17 19:48:28 +03:00
alesapin
dc893f9644 Update NuRaft buffer to 64 bit size 2021-06-17 19:37:08 +03:00
alesapin
1a6abb4db4 Better 2021-06-17 19:32:50 +03:00
l1tsolaiki
0c2d0b55a4 Merge branch 'jsonpath' of github.com:l1tsolaiki/ClickHouse into jsonpath 2021-06-17 18:26:14 +03:00
l1tsolaiki
f2d451e0ec
Merge branch 'ClickHouse:master' into jsonpath 2021-06-17 18:25:26 +03:00
neng.liu
45e9e9f8bb fix some compile problem 2021-06-17 08:08:29 -07:00
Amos Bird
0f7728cd5e
Fix RBAC test again 2021-06-17 23:00:03 +08:00
Kseniia Sumarokova
f9c0bfb769
Merge pull request #25199 from sand6255/MySQL-column-comments-support 
MaterializeMySQL Engine column comments support
 2021-06-17 17:08:53 +03:00
alesapin
d513d14b2c Add some functions for data conversion 2021-06-17 16:29:11 +03:00
vdimir
23fb229dec
Merge branch 'update_01049_join_low_card_bug' into issue-23351 2021-06-17 15:38:40 +03:00
vdimir
e34a6525ef
Support Interval for LowCardinality 2021-06-17 15:28:08 +03:00
l1tsolaiki
f44d6792b0 Fix ext::range 2021-06-17 14:48:05 +03:00
Matwey V. Kornilov
aa1ac2a506 Add missed #include <optional> 
This is required for std::optional<std::string> language;

Signed-off-by: Matwey V. Kornilov <matwey.kornilov@gmail.com>
 2021-06-17 14:41:28 +03:00
Maksim Kita
6d44aa3034 Fixed style check 2021-06-17 13:26:40 +03:00
vdimir
0cee5ad8e8
Use IColumn::mutate in JoinCommon::convertColumnToNullable 2021-06-17 13:24:17 +03:00
l1tsolaiki
8bac279cb8 Merge branch 'jsonpath' of github.com:l1tsolaiki/ClickHouse into jsonpath 2021-06-17 13:21:13 +03:00
l1tsolaiki
ff3857fbe7 Add root parsing 2021-06-17 13:18:43 +03:00
vdimir
ebdce8e1a6
Fix JoinCommon::convertColumnToNullable for LC(Nullable) 2021-06-17 13:17:28 +03:00
Matwey V. Kornilov
37180e0842 Add missed #include <optional> 
std::optional<> is used multiple times here.

Signed-off-by: Matwey V. Kornilov <matwey.kornilov@gmail.com>
 2021-06-17 13:14:56 +03:00
Nikolai Kochetov
4a496f1374 Fix lost considion in predicate push down. 2021-06-17 12:47:12 +03:00
Maksim Kita
bc86b2b738 Updated function bitPositionsToArray before merge 2021-06-17 12:20:53 +03:00
Vladimir
7a56ab8b1d
Merge pull request #25299 from mga-chka/equals_op_in_sequence_functions 2021-06-17 11:45:04 +03:00
mergify[bot]
13fdefebf5
Merge branch 'master' into fixjoingetornull 2021-06-17 08:26:13 +00:00
Neng Liu
b6eab5fd2f add datetype32 function support 2021-06-17 16:05:30 +08:00
Yuriy Chernyshov
2bb1c75fb5 Rename & reimport murmurhash sources from smhasher repo 2021-06-17 10:58:42 +03:00
Maksim Kita
5b2380d541
Merge branch 'function-bit-position-to-array-merge' into feature-function-bitpositionToArray 2021-06-17 10:23:41 +03:00
Maksim Kita
37770f20fc
Merge pull request #25372 from ClickHouse/add-date-name-function-merge 
Merging #23085
 2021-06-17 10:16:32 +03:00
Azat Khuzhin
9242654231 negate: replace special case with parens 
v0: negate: add parens for negative values
    Before this patch -(-1) was formatted to --1, but this is not valid
    syntax.
v2: replace special case with parens, otherwise formatting of -(-(-1))
    will be unstable, in v0 it was formatted to -(1) but -(1) is
    formatted to -1 [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/24427/4512460aeb02fec557c558eee3f8291a6297efb2/fuzzer_asan/fuzzer.log
 2021-06-17 10:07:03 +03:00
Amos Bird
31815f4a68
Proper documentation of the setting 2021-06-17 14:27:33 +08:00
Amos Bird
c8ea6527cb
Add prefer_global_in_and_join setting 2021-06-17 14:27:29 +08:00
alexey-milovidov
fffd69b023
Merge pull request #25283 from ClickHouse/if-support-int-decimal 
Remove code and fix #20549
 2021-06-17 09:17:42 +03:00
alexey-milovidov
f7707b4409
Merge pull request #25391 from amosbird/ctf 
Fix column transformer replace format issue
 2021-06-17 09:06:57 +03:00
mergify[bot]
41d007a1b3
Merge branch 'master' into MySQL-column-comments-support 2021-06-17 05:40:15 +00:00
tavplubix
2b62a09aa3
Merge pull request #25373 from ClickHouse/improve_replicated_database_tests2 
Improve Replicated database tests 2
 2021-06-17 08:34:06 +03:00
Amos Bird
f9d00d6dfb
Fix column transformer replace format issue 2021-06-17 11:12:26 +08:00
Alexey Milovidov
0f1a94bdef Merge branch 'master' into if-support-int-decimal 2021-06-17 01:53:58 +03:00
alexey-milovidov
04be5437d9
Merge pull request #25296 from abyss7/http-issues 
Add settings for HTTP header limitations
 2021-06-17 01:50:48 +03:00
alexey-milovidov
16931a661c
Update HTTPServerRequest.h 2021-06-17 01:50:06 +03:00
alexey-milovidov
43cee9018e
Merge pull request #25281 from ClickHouse/client-update-prompt-on-reconnect 
Update prompt in client when reconnecting
 2021-06-17 01:40:08 +03:00
alexey-milovidov
0e16ddd7d9
Merge pull request #25313 from ClickHouse/fix-quantile-determinism 
Fix behaviour of quantileDeterministic function
 2021-06-17 01:24:52 +03:00
alexey-milovidov
5de9f6cdca
Merge pull request #25314 from ClickHouse/fix-ubsan-quantile-timing 
Fix UBSan report in quantileTiming
 2021-06-17 01:24:24 +03:00
Alexander Tokmakov
c235539620 minor fixes 2021-06-17 00:20:35 +03:00
Maksim Kita
c2ecdb7269 Fixed build issues 2021-06-16 23:28:41 +03:00
Maksim Kita
67e9b85951 Merge ext into common 2021-06-16 23:28:41 +03:00
Kseniia Sumarokova
60d220bd14
Merge pull request #24413 from kssenii/postgres-fixes-schema 
postgres non-default schema for insert query
 2021-06-16 22:18:23 +03:00
Kseniia Sumarokova
d6240f2bd4
Merge pull request #25309 from azat/fs-fix 
Catch ErrnoException during parts cleaning
 2021-06-16 22:03:54 +03:00
Maksim Kita
ac2f9dd15a Updated function dateName before merge 2021-06-16 21:19:05 +03:00
Anton Popov
ea9b890618
Merge branch 'master' into add-compatibility-setting 2021-06-16 21:10:56 +03:00
Anton Popov
8bd7cbd925 add compatibility setting for name of columns 2021-06-16 21:10:07 +03:00
Nikolai Kochetov
3d7100ca60 Disable query_plan_filter_push_down. 2021-06-16 20:51:16 +03:00
Maksim Kita
d2df23a959
Function formatDateTime fix code comments (#25334) 2021-06-16 17:59:58 +03:00
Ivan Lezhankin
ba08a580f8 Add test 2021-06-16 17:33:14 +03:00
Maksim Kita
5191fecb09
Merge pull request #23085 from dankondr/add-datename-function 
[WIP] Add dateName function
 2021-06-16 16:13:50 +03:00
tavplubix
76348edc71
Merge pull request #25305 from ClickHouse/improve_replicated_database_tests 
Improve Replicated database tests
 2021-06-16 15:58:46 +03:00
Nikolai Kochetov
16675a4f4e
Merge pull request #25290 from ClickHouse/fix-lc-in-tuple-header 
Fix invalid header for (LowCardinality IN (tuple)).
 2021-06-16 15:37:25 +03:00
Maksim Kita
0414d1b39e Setting min_count_to_compile_expression fix 2021-06-16 15:18:55 +03:00
Nikolai Kochetov
192fe1fc5b Add test. 2021-06-16 15:05:31 +03:00
tavplubix
07ce932282
Merge pull request #25306 from ClickHouse/tavplubix-patch-2 
Forcefully stop queue tasks on restarting
 2021-06-16 14:22:36 +03:00
Kruglov Pavel
bf36f5a977
Merge pull request #25000 from vitlibar/fix-protobuf-serialization-of-splitted-nested-messages 
Fix serialization of splitted nested messages in Protobuf format.
 2021-06-16 14:04:14 +03:00
tavplubix
710546b9e8
Merge branch 'master' into improve_replicated_database_tests 2021-06-16 13:24:32 +03:00
Anton Popov
6b264618aa
Merge pull request #25298 from Avogar/fix-min-map 
Fix Logical Error in min/maxMap
 2021-06-16 12:26:29 +03:00
Kseniia Sumarokova
661524e3d0
Update StoragePostgreSQL.cpp 2021-06-16 12:11:31 +03:00
Nikolai Kochetov
8bf55ae2b5
Merge pull request #25166 from ClickHouse/always-detach-parts-with-wrong-partition-id 
Always detach parts with wrong partition ID
 2021-06-16 11:06:07 +03:00
Alexey Milovidov
d1bba10be0 Fix UBSan report in quantileTiming 2021-06-16 10:38:36 +03:00
Kseniia Sumarokova
e08c05cdf5
Merge pull request #25295 from ClickHouse/remove-some-code-from-key-condition 
Remove some code from KeyCondition.
 2021-06-16 10:12:12 +03:00
Alexey Milovidov
ea7b5497c0 Fix behaviour of quantileDeterministic function 2021-06-16 09:50:21 +03:00
Storozhuk Kostiantyn
46b230ddd7 Update InterpretersMySQLDDLQuery.cpp 2021-06-16 14:22:07 +08:00
alexey-milovidov
7180cf2367
Merge pull request #25265 from ClickHouse/minor-changes-3 
Minor change
 2021-06-16 08:33:12 +03:00
Azat Khuzhin
de9689379f Catch ErrnoException during parts cleaning 
Avoids noisy messages like:

    2021.06.15 09:27:35.546982 [ 58176 ] {} <Error> auto DB::IBackgroundJobExecutor::jobExecutingTask()::(anonymous class)::operator()() const: Code: 481, e.displayText() = DB::ErrnoException: Cannot check modification time for file: /var/lib/clickhouse/store/c52/c52619f8-99bd-435e-8526-19f899bdb35e/tmp_insert_202106_7010_7010_0/, errno: 2, strerror: No such file or directory, Stack trace (when copying this message, always include the lines below):

    0. DB::Exception::Exception() @ 0x903209a in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    1. DB::throwFromErrnoWithPath() @ 0x9033089 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    2. FS::getModificationTime() @ 0x907b166 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    3. DB::DiskLocal::getLastModified() @ 0xf9a6669 in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
    4. DB::MergeTreeData::clearOldTemporaryDirectories(long) @ 0x1069a6de in /usr/lib/debug/.build-id/be/ea088a1830cb2f.debug
 2021-06-16 06:56:55 +03:00
alexey-milovidov
cbf154cf0c
Merge pull request #25301 from kitaisreal/cmake-dictionaries-disable-debug-info-fix 
CMake dictionaries disable debug info fix
 2021-06-16 05:57:02 +03:00
Maksim Kita
0a057773f1
Merge pull request #24814 from kitaisreal/expression-jit-simplify-loop 
ExpressionJIT simplify loop
 2021-06-16 01:42:15 +03:00
tavplubix
60c8dc1c0b
Update ReplicatedMergeTreeRestartingThread.cpp 2021-06-16 00:48:38 +03:00
Alexander Tokmakov
c873714698 improve replicated database tests 2021-06-15 23:52:29 +03:00
Maksim Kita
f8e95abaf6 CMake dictionaries disable debug info fix 2021-06-15 22:18:17 +03:00
Nikolai Kochetov
96d98ff020 Add comment 2021-06-15 21:42:26 +03:00
Nikolai Kochetov
c8cbb44455 Support simple aggregate functions for SummingMergeTree. 2021-06-15 21:33:46 +03:00
Nikolai Kochetov
03b92dc1b7 Support simple aggregate functions for SummingMergeTree. 2021-06-15 21:32:27 +03:00
mergify[bot]
7959d92029
Merge branch 'master' into minor-changes-3 2021-06-15 18:07:24 +00:00
christophe.kalenzaga
2dae69a40b add == operators in sequence*() aggregate fuctions 2021-06-15 18:58:20 +02:00
Pavel Kruglov
7b9c8edaf5 Fix Logical Error in min/maxMap 2021-06-15 19:24:18 +03:00
Nikolai Kochetov
d94b9c592d
Update in.cpp 2021-06-15 18:51:06 +03:00
Nikolai Kochetov
80a13c489b Revert back moduloLegacy check for canConstantBeWrappedByFunctions. 2021-06-15 18:21:31 +03:00
Storozhuk Kostiantyn
74849bbe0c
Update InterpretersMySQLDDLQuery.cpp 
Added space after if
 2021-06-15 18:21:02 +03:00
Ivan Lezhankin
b182d87d9c Add settings for HTTP header limitations 2021-06-15 17:33:46 +03:00
alesapin
d68338a792
Merge pull request #24979 from nvartolomei/nv/parts-uuid-move-shard-acl 
Part movement between shards ACL
 2021-06-15 17:19:10 +03:00
alesapin
ca4628d31c
Merge pull request #25197 from ClickHouse/disable-paln-optimizations-for-mutations 
Do not optimize query plan for mutations.
 2021-06-15 17:15:36 +03:00
Nikolai Kochetov
481b87b37a Remove some code from keyCondition. 2021-06-15 16:47:37 +03:00
Kostiantyn Storozhuk
d96478c9ff Integration test added and small fixes 2021-06-15 21:45:30 +08:00
Pavel Kruglov
48cd8f2207 Fix build 2021-06-15 16:15:27 +03:00
Pavel Kruglov
692150b578 Fix tests 2021-06-15 16:15:27 +03:00
Pavel Kruglov
931e05ab04 Minor refactoring 2021-06-15 16:15:27 +03:00
Pavel Kruglov
a4ef60e230 Remove Impl including from .h file 2021-06-15 16:15:27 +03:00