thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
77,231 Commits 9,328 Branches 1,783 Tags 2.1 GiB
af56b20da7
Commit Graph

6026 Commits

Author SHA1 Message Date
Nikolai Kochetov
8986de7e50
Merge pull request #30384 from amosbird/projection-fix21 
Fix crash when minmax_count projection is used with primary key in partition expr
 2021-10-20 16:17:13 +03:00
Kseniia Sumarokova
c692155c7e
Merge pull request #30309 from kssenii/fix-file-symlinks 
Fix symlinks in file table function
 2021-10-20 11:09:51 +03:00
tavplubix
5e24f337a0
Update StorageReplicatedMergeTree.cpp 2021-10-20 09:29:41 +03:00
Kseniia Sumarokova
5324cc8359
Merge pull request #30282 from kssenii/fix-local-less-threads 
Less threads in clickhouse-local, fix Ok. printing
 2021-10-19 22:54:28 +03:00
Amos Bird
05d93796dc
Fix minmax_count projection with primary key in partition expr 2021-10-19 23:36:07 +08:00
Maksim Kita
382c82805f
Merge pull request #30352 from kitaisreal/storage-executable-fix-small-issues 
StorageExecutable fix small issues
 2021-10-19 17:43:11 +03:00
Maksim Kita
0f2e23d775
Update StorageExecutable.cpp 2021-10-19 17:41:59 +03:00
feng lv
05fd6f7caf fix comflict 2021-10-19 12:50:22 +00:00
Nikita Mikhaylov
75487be899 White list of storages that supports final 2021-10-19 12:16:33 +00:00
Alexander Tokmakov
caf972db48 fix 2021-10-19 14:56:04 +03:00
Nikita Mikhaylov
4fbd332bf1 Done 2021-10-19 11:53:55 +00:00
Nikita Mikhaylov
ea2a647724
Merge pull request #30192 from azat/granula-adjust-LOGICAL_ERROR-fix 
Fix reading from MergeTree with max_read_buffer_size=0
 2021-10-19 13:25:55 +03:00
kssenii
e53335bc6f Better way 2021-10-19 08:21:38 +00:00
Nikolai Kochetov
cc7a3ea9ff
Merge pull request #30322 from ClickHouse/update-mergine-sorted-memory-optimization 
Update memory optimisation for MergingSorted.
 2021-10-19 11:12:50 +03:00
kssenii
df5dc4e991 Adjust MergeTreeIndexReader 2021-10-19 10:51:45 +03:00
kssenii
49106f407f Fix some checks 2021-10-19 09:34:27 +03:00
Maksim Kita
fb91b1d92b StorageExecutable fix small issues 2021-10-19 01:04:07 +03:00
Alexander Tokmakov
89fe606d15 try fix 'some fetches may stuck' 2021-10-18 23:16:02 +03:00
Nikita Mikhaylov
d004fd96ed
Merge pull request #30318 from azat/fix-shared-build 
Move SquashingTransform to Interpreters (to fix split build)
 2021-10-18 22:41:01 +03:00
Azat Khuzhin
e576fd17bd Do not allow reading to empty buffer in MergeTreeReaderStream 
Changelog:
- Verify that all rows was read in MergeTreeReaderWide
- Ignore some exceptions for Wide parts
- Take max_rows_to_read into account
- Verify that requested rows is not zero (otherwise it is too tricky)
- Simply verify that buffer is not empty
 2021-10-18 21:19:34 +03:00
Azat Khuzhin
cd4b33c8c9 Verify that all rows was read in MergeTreeReaderCompact 
v0: Use fmt-like style exception in MergeTreeReaderCompact
v2: Update the check
 2021-10-18 21:19:34 +03:00
Azat Khuzhin
41e6df0f4b Remove unused offset_columns from MergeTreeReaderWide::readRows() 2021-10-18 21:19:34 +03:00
Ilya Golshtein
d90302aa3b output_format_avro_rows_in_file 2021-10-18 19:01:06 +03:00
feng lv
ab0d2504e5 fix 2021-10-18 15:44:54 +00:00
kssenii
d5d4817350 Add disk proxies back, add setReadUntilPosition to ReadBuffer 2021-10-18 18:35:11 +03:00
Nikolay Degterinsky
d76976e272 Merge branch 'master' into query_parameters 2021-10-18 17:34:20 +03:00
Nikolai Kochetov
6e479b301a Update memory optimisation for MergingSorted. 2021-10-18 12:54:12 +03:00
Anton Popov
71b474793c
Merge pull request #30174 from CurtizJ/better-interfaces 
Better interfaces for `IDataType` and `ISerialization`
 2021-10-18 12:07:04 +03:00
kssenii
710ee96878 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-18 08:31:25 +00:00
feng lv
81fb4bcf76 fix style 2021-10-18 08:14:54 +00:00
kssenii
3d39ea8e37 Fix WHERE 2021-10-18 11:06:30 +03:00
feng lv
0c628726af fix build 2021-10-18 07:48:58 +00:00
feng lv
1e3df5780c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-18 07:33:11 +00:00
Azat Khuzhin
6d4af3bac1 Move SquashingTransform to Interpreters (to fix split build) 
clickhouse_common_io requires clickhouse_core:

    ld.lld: error: undefined symbol: DB::blocksHaveEqualStructure(DB::Block const&, DB::Block const&)
    >>> referenced by SquashingTransform.cpp:92 (/src/ch/clickhouse/src/Common/SquashingTransform.cpp:92)
    >>>               src/CMakeFiles/clickhouse_common_io.dir/Common/SquashingTransform.cpp.o:(void DB::SquashingTransform::append<DB::Block&&>(DB::Block&&))
    >>> referenced by SquashingTransform.cpp:92 (/src/ch/clickhouse/src/Common/SquashingTransform.cpp:92)
    >>>               src/CMakeFiles/clickhouse_common_io.dir/Common/SquashingTransform.cpp.o:(void DB::SquashingTransform::append<DB::Block const&>(DB::Block const&))

while clickhouse_core requires clickhouse_common_io:

    "clickhouse_core" of type SHARED_LIBRARY
      depends on "roaring" (weak)
      depends on "clickhouse_common_io" (weak)
      depends on "clickhouse_common_config" (weak)
      depends on "clickhouse_common_zookeeper" (weak)
      depends on "clickhouse_dictionaries_embedded" (weak)
      depends on "clickhouse_parsers" (weak)

Follow-up for: #30247 (cc @KochetovNicolai)
 2021-10-18 10:28:36 +03:00
feng lv
6cf82f859c fix 2021-10-18 07:27:37 +00:00
feng lv
a32abda322 fix 2021-10-18 07:08:37 +00:00
feng lv
074e02eb14 fix 2021-10-18 06:38:47 +00:00
alexey-milovidov
7beffb04ad
Merge pull request #29495 from ClickHouse/improvement_for_strange_mutations 
Minor improvement for strange mutations
 2021-10-17 20:57:06 +03:00
feng lv
9378b93e41 fix 2021-10-17 14:49:27 +00:00
Kseniia Sumarokova
5633865df1
Update src/Storages/StorageFile.cpp 
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
 2021-10-17 14:16:40 +03:00
Nikolai Kochetov
2a95d7fc87
Merge pull request #29918 from amosbird/projection-improve5 
Add primary key to minmax_count_projection
 2021-10-17 14:04:18 +03:00
kssenii
059fc1de69 Allow symlinks in file storage 2021-10-17 09:11:06 +00:00
Nikolai Kochetov
bfcbf5abe0 Merge branch 'master' into removing-data-streams-folder 2021-10-17 10:42:37 +03:00
alexey-milovidov
4f11cfa59d
Merge pull request #28760 from azat/mutator-forbid-cleaner 
Forbid cleaning of tmp directories that can be used by an active mutation/merge.
 2021-10-17 03:18:01 +03:00
kssenii
4a0bdd296d Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-16 18:35:48 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
alesapin
9418eda122
Merge pull request #30111 from ClickHouse/remove_trash 
Remove trash from MergeTreeReadPool
 2021-10-16 15:41:41 +03:00
Amos Bird
9ceb668a4d
Fix TSan 2021-10-16 18:13:41 +08:00
Amos Bird
77c4a5fa18
Better 2021-10-16 18:13:41 +08:00
Amos Bird
77a2022cf5
Add primary key to minmax_count_projection 2021-10-16 18:13:41 +08:00
Nikolai Kochetov
067eaadadd Merge branch 'master' into removing-data-streams-folder 2021-10-16 09:46:05 +03:00
Nikolai Kochetov
c668696047
Merge pull request #30171 from ClickHouse/remove-stream-interfaces 
Remove stream interfaces
 2021-10-16 09:34:01 +03:00
feng lv
8b7721ae6f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-16 05:14:11 +00:00
Anton Popov
d2ba3bcaa4
Merge pull request #28363 from amosbird/fastload2 
Speed up part loading for JBOD
 2021-10-16 02:37:32 +03:00
Azat Khuzhin
07e8b2b3c7 Do not try to remove temporary paths that is currently in written by merge/mutation 
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
 2021-10-16 00:43:52 +03:00
Azat Khuzhin
0b3bf43d61 Rewrite MergeTreeData::clearOldTemporaryDirectories() to use early continue 2021-10-16 00:43:51 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
mergify[bot]
477ea62257
Merge branch 'master' into remove-sample-by 2021-10-15 15:14:17 +00:00
kssenii
759f747080 Fix build 2021-10-15 17:12:34 +03:00
kssenii
7b81766168 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into remote-connection 2021-10-15 12:36:44 +00:00
kssenii
ccb02cac56 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-15 12:36:30 +00:00
kssenii
881ae8617e Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-15 15:09:56 +03:00
kssenii
edfb1a5521 Better performance for smaller number of threads 2021-10-15 15:07:39 +03:00
Nikita Mikhaylov
d1138a8a25
Merge pull request #29691 from azat/fix-merge-mutate-memory-tracking 
Fix memory tracking for merges and mutations
 2021-10-15 13:53:52 +03:00
Nikolai Kochetov
ad8a344b46 Move TTL streams and algo 2021-10-15 13:11:57 +03:00
Maksim Kita
7beaeb8b13
Merge branch 'master' into indexcache 2021-10-15 12:59:09 +03:00
Maksim Kita
27fdeef4b6
Merge pull request #29650 from kitaisreal/add-default-database-visitor-support-dict-get 
AddDefaultDatabaseVisitor support dictGet
 2021-10-15 12:28:58 +03:00
kssenii
3995506d37 Adjust range reader for remote fs reads 2021-10-15 11:36:26 +03:00
Azat Khuzhin
2a7a1d8df5 Avoid losing any allocations context from merges/mutations 2021-10-15 01:43:28 +03:00
Azat Khuzhin
8a209a78d7 Set query_id for mutations/merges 
This will allow to distinguish allocations in trace_log.
 2021-10-15 01:43:28 +03:00
Azat Khuzhin
8cc45bea7b Avoid accounting memory from another mutation/merge 
Before this path it was possible that one merge/mutation may account
memory from another, due to ThreadStatus::untracked_memory.

And this cause flakiness of 01200_mutations_memory_consumption.
 2021-10-15 01:43:28 +03:00
Azat Khuzhin
fd38cbb0df Fix memory tracking for merges and mutations (by destroying earlier) 
It fixes only some tiny allocations, and so it should not affect any
huge mutations/merges.

And plus, this should not be a real fix, since peak_memory_usage is
obtained before even destrying this objects, and destroing objects will
unlikely update peak memory usage (although it is possible).

v0: do this in dtors
v2: do this earlier
 2021-10-15 01:43:27 +03:00
Azat Khuzhin
e4b39ca99c Remove unused Block member from MutateTask 
v0: Fix excessive memory usage in MutateTask
    This is relevant only for 01200_mutations_memory_consumption test, since
    this is just a block.
v2: just remove unused Block member, since other part had been fixed in #29768
 2021-10-15 01:43:27 +03:00
alexey-milovidov
bc1662b9fe
Merge pull request #29946 from azat/log-fix-data-race 
Fix data-race between LogSink::writeMarks() and LogSource in StorageLog
 2021-10-15 01:30:58 +03:00
alexey-milovidov
8d694cbcdc
Merge pull request #29959 from azat/log-data-race-file-checker-fix 
Fix possible data-race between FileChecker and StorageLog/StorageStripeLog
 2021-10-15 01:30:23 +03:00
mergify[bot]
92dd528c78
Merge branch 'master' into remove_trash 2021-10-14 14:14:07 +00:00
Anton Popov
476d7a411f allow to remove SAMPLE BY expression 2021-10-14 17:05:20 +03:00
Maksim Kita
c1f04452de AddDefaultDatabaseVisitor support dictGet 2021-10-14 16:31:32 +03:00
feng lv
400a31db06 reduce poll timeout 
fix
 2021-10-14 11:45:18 +00:00
Anton Popov
c2d376d0d6
Merge branch 'master' into better-interfaces 2021-10-14 14:33:36 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Nikolai Kochetov
3d3e143c29
Merge pull request #30001 from ClickHouse/remove-streams-from-formats 
Remove streams from formats.
 2021-10-14 12:39:07 +03:00
Maksim Kita
f589084a94
Merge pull request #29899 from Enmk/Governance/dictionary_comment 
Governance/dictionary comment
 2021-10-14 12:35:10 +03:00
Anton Popov
92413aed68 better interfaces for IDataType and ISerialization 2021-10-14 05:36:49 +03:00
feng lv
21f7ecbd9f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-14 02:09:51 +00:00
feng lv
3116767834 fix 2021-10-14 02:09:43 +00:00
Maksim Kita
659a46a972 WriteBufferToRabbitMQProduced fixed queue usage 2021-10-14 00:33:18 +03:00
Maksim Kita
04047f76c7 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
d55561422f Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
c9b6c2661b Refactor ConcurrentBoundedQueue 2021-10-14 00:33:18 +03:00
Azat Khuzhin
eba6410fd5 Fix data-race between LogSink::writeMarks() and LogSource::readData() in StorageLog 
CI founds [1], TSan report:

    WARNING: ThreadSanitizer: data race (pid=497)
      Write of size 8 at 0x7b1c00e4e8d8 by thread T922:
        4 DB::LogSink::writeMarks() obj-x86_64-linux-gnu/../src/Storages/StorageLog.cpp:462:72 (clickhouse+0x1665cdaa)
        5 DB::LogSink::consume() obj-x86_64-linux-gnu/../src/Storages/StorageLog.cpp:324:5 (clickhouse+0x1665c216)

      Previous read of size 8 at 0x7b1c00e4e8d8 by thread T711:
        1 DB::LogSource::readData()::$_0::operator()(bool) const::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Storages/StorageLog.cpp:188:26 (clickhouse+0x16661baf)
        9 DB::LogSource::readData() obj-x86_64-linux-gnu/../src/Storages/StorageLog.cpp:204:20 (clickhouse+0x1665bbcb)

      [1]: https://clickhouse-test-reports.s3.yandex.net/29930/4bc90d1dd7dbd4b8a9b6920d00ca24e8b160358e/stress_test_(thread).html#fail1

Fixes: #29470
v2: rework locking, but this produce deadlock for 01499_log_deadlock test
v3: introduce per-file marks mutex
 2021-10-14 00:17:45 +03:00
Azat Khuzhin
e4db862d85 Fix possible data-race between FileChecker and StorageLog/StorageStripeLog 2021-10-14 00:16:58 +03:00
Nikolay Degterinsky
63faf0212d Merge branch 'master' into query_parameters 2021-10-13 23:55:09 +03:00
Nikolai Kochetov
2957971ee3 Remove some last streams. 2021-10-13 21:22:02 +03:00
alesapin
1801ea4955 Merge branch 'master' into remove_trash 2021-10-13 18:10:31 +03:00
feng lv
168020e0db fix 2021-10-13 14:02:32 +00:00
alesapin
d2dfbb5ab6 Remove trash from MergeTreeReadPool 2021-10-13 15:25:44 +03:00
feng lv
c82fd3491d fix 
fix
 2021-10-13 11:18:19 +00:00
feng lv
868dbc4bad use poll replace select 
fix

fix
 2021-10-13 09:13:35 +00:00
mergify[bot]
cfda5592fe
Merge branch 'master' into fix-writing-storage-log-marks 2021-10-13 07:18:33 +00:00
feng lv
74e1900034 fix 2021-10-13 05:23:00 +00:00
Amos Bird
23602f4607
Speed up part loading for JBOD 2021-10-13 12:33:54 +08:00
feng lv
0409b775bf fix 2021-10-13 02:11:15 +00:00
Nikolay Degterinsky
49c1a52f34 Merge branch 'master' into query_parameters 2021-10-13 03:04:40 +03:00
Azat Khuzhin
e9ce859b02 Fix race between MOVE PARTITION and merges/mutations for MergeTree 
From time to time 00975_move_partition_merge_tree test got failed, the
reason is that there is merge in progress that writes data to the source
table again:

<details>

    2021.10.12 13:20:40.243839 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> executeQuery: (from [::1]:38060) (comment: 00975_move_partition_merge_tree.sql) ALTER TABLE test_move_partition_src MOVE PARTITION 1 TO TABLE test_move_partition_dest;
    2021.10.12 13:20:40.244482 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_2_2_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_21_21_0
    ...
    2021.10.12 13:20:40.373487 [ 378 ] {} <Trace> test_t1tzb2.test_move_partition_src (467d7145-b47e-444e-867d-7145b47ea44e) (MergerMutator): Merged 6 parts: from 1_2_2_0 to 1_11_11_0
    ...
    2021.10.12 13:20:40.379750 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_15_15_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_28_28_0

</details>

And also remove cleaning of mutations since this will cause deadlock
after doing MOVE PARTITION under
currently_processing_in_background_mutex.

CI: https://clickhouse-test-reports.s3.yandex.net/0/a59c6b1c8eb47ebf77189a491ee0c3980b38e91c/functional_stateless_tests_
Fixes: 00975_move_partition_merge_tree
 2021-10-13 00:14:24 +03:00
feng lv
b2a17cb320 fix build and conflict 2021-10-12 16:50:50 +00:00
Kseniia Sumarokova
bb15b57753
Merge pull request #30016 from ucasfl/sample-by-tuple 
Fix  crash of sample by tuple()
 2021-10-12 19:46:21 +03:00
Nikolai Kochetov
f2d8cc547f Fix Kafka source. 2021-10-12 18:09:28 +03:00
Nikolai Kochetov
c36bb9206e
Merge pull request #29879 from amosbird/queryidholderfix 
Fix potential leak of query_id_holder
 2021-10-12 15:42:02 +03:00
Nikolai Kochetov
077aba4a97
Merge pull request #29520 from amosbird/projection-improve4 
Get rid of naming limitation of projections.
 2021-10-12 14:25:29 +03:00
Nikolai Kochetov
93b3fcc195 Fix rabbitmq 2021-10-12 13:05:20 +03:00
kssenii
9814cb1b45 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-12 09:49:08 +00:00
Vladimir C
969999ff10
Merge pull request #29544 from Algunenano/join_deadlock 2021-10-12 12:34:04 +03:00
Maksim Kita
91a48e9651
Merge pull request #29896 from kitaisreal/system-data-skipping-indices-added-size 
System table data skipping indices added size
 2021-10-12 12:16:57 +03:00
Maksim Kita
c15b67c182 Fix naming 2021-10-12 11:42:24 +03:00
alesapin
5ac0d36392
Merge pull request #30005 from ClickHouse/more_strict_check_for_intersecting_parts 
More strict check for intersecting parts
 2021-10-12 11:38:09 +03:00
kssenii
e7dbfff5d2 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-12 06:19:15 +00:00
feng lv
b5d69d599e fix sample by tuple() 
add tests

fix

fix

fix
 2021-10-12 05:19:32 +00:00
alexey-milovidov
f1b550cdaf
Merge pull request #29974 from ClickHouse/minimized-toolchain 
Experiment with minimized toolchain (2)
 2021-10-11 22:04:01 +03:00
Alexander Tokmakov
706e2b6b88 more strict check for intersecting parts 2021-10-11 21:42:46 +03:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
kssenii
6d93253a67 Fix tests 2021-10-11 11:34:08 +00:00
kssenii
1de55cfeb4 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into remote-connection 2021-10-11 11:33:35 +00:00
Alexey Milovidov
2370a8d6da Fix PVS-Studio 2021-10-11 14:28:46 +03:00
Amos Bird
72bccaa501
Fix path name 2021-10-11 19:12:08 +08:00
Maksim Kita
b0d887a0fe Added tests 2021-10-11 14:00:10 +03:00
Amos Bird
83717b7c3b
Get rid of naming limitation of projections. 2021-10-11 18:32:17 +08:00
Kseniia Sumarokova
20e8074b5e
Merge pull request #29912 from taiyang-li/remove_unused_var 
Remove unused variable in s3cluster
 2021-10-11 12:45:04 +03:00
Maksim Kita
ce0c41e1ad Fixed tests 2021-10-11 11:39:50 +03:00
Maksim Kita
2d069acc22 System table data skipping indices added size 2021-10-11 11:39:50 +03:00
Vasily Nemkov
2c55807b2a Create dictionary with comment 2021-10-11 11:26:59 +03:00
Amos Bird
ebfb013ea1
Fix potential leak of query_id_holder 2021-10-11 15:56:00 +08:00
Vladimir C
dc4bb920d3
Merge branch 'master' into join_deadlock 2021-10-11 10:18:17 +03:00
Vladimir C
6c7fbf0b88
Merge pull request #29956 from azat/fix-tuple-filter 2021-10-11 10:02:47 +03:00
feng lv
a57e97d2e0 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-11 06:09:43 +00:00
feng lv
d8bd1474bc fix 2021-10-11 04:18:21 +00:00
feng lv
4d9604e298 fix 2021-10-11 04:06:31 +00:00
kssenii
448d63a0fc Range read with upper bound offset 2021-10-11 01:53:51 +03:00
alexey-milovidov
0adbcd58c9
Merge pull request #29930 from azat/buffer-fix-data-race 
Fix data-race between flush() and startup() in StorageBuffer
 2021-10-11 01:11:39 +03:00
kssenii
aa4b797808 Turn off tasks stealing for remote disk 2021-10-10 23:22:58 +03:00
Azat Khuzhin
6c33eaee32 Fix filtering by tuple (some conditions was lost during analyzing) 
Fixes: #29281
Fixes: test_cluster_copier/test_three_nodes.py::test
 2021-10-10 22:37:52 +03:00
Nikolai Kochetov
a95c28ec4b
Merge pull request #29898 from ClickHouse/remove-native-stream 
Remove some more streams.
 2021-10-10 21:01:16 +03:00
feng lv
045704274c implement directory watcher 2021-10-10 16:31:32 +00:00
kssenii
eaebd84cd5 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-10 15:17:35 +00:00
Azat Khuzhin
4bc90d1dd7 Fix data-race between flush() and startup() in StorageBuffer 
Stress tests found [1], TSan report:

    ==================
    WARNING: ThreadSanitizer: data race (pid=485)
      Read of size 8 at 0x7b5001280bd8 by thread T567 (mutexes: write M612061890855345680):
        1 std::__1::shared_ptr<DB::BackgroundSchedulePoolTaskInfo>::operator bool() const obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2851:62 (clickhouse+0x159140a6)
        2 bool std::__1::operator!=<DB::BackgroundSchedulePoolTaskInfo>() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3447:30 (clickhouse+0x159140a6)
        3 DB::BackgroundSchedulePoolTaskHolder::operator bool() const obj-x86_64-linux-gnu/../src/Core/BackgroundSchedulePool.h:164:46 (clickhouse+0x159140a6)
        4 DB::StorageBuffer::flush() obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:675:10 (clickhouse+0x159140a6)

      Previous write of size 8 at 0x7b5001280bd8 by thread T586 (mutexes: write M191819750614415520):
        2 std::__1::shared_ptr<DB::BackgroundSchedulePoolTaskInfo>::operator=(std::__1::shared_ptr<DB::BackgroundSchedulePoolTaskInfo>&&) obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3243:34 (clickhouse+0x15913e22)
        3 DB::BackgroundSchedulePoolTaskHolder::operator=() obj-x86_64-linux-gnu/../src/Core/BackgroundSchedulePool.h:156:110 (clickhouse+0x15913e22)
        4 DB::StorageBuffer::startup() obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:668:18 (clickhouse+0x15913e22)
        5 DB::InterpreterCreateQuery::doCreateTable() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterCreateQuery.cpp:1092:10 (clickhouse+0x149bef7b)
        6 DB::InterpreterCreateQuery::createTable() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterCreateQuery.cpp:952:20 (clickhouse+0x149ba9f5)
        7 DB::InterpreterCreateQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterCreateQuery.cpp:1302:16 (clickhouse+0x149c1086)

  [1]: https://clickhouse-test-reports.s3.yandex.net/0/1c9778603ff49563d1d3d0d357de0608167e504d/stress_test_(thread).html

Fixes: #29416
 2021-10-10 04:03:36 +03:00
Maksim Kita
0358d664b9
Merge pull request #29882 from MaxWk/add-alias-for-name 
Add table alias to system.tables and database alias to system.databases
 2021-10-09 12:44:40 +03:00
Vasily Nemkov
67ff0f5dba Refactoring: moved comment out of ASTStorage to ASTCreateQuery 2021-10-09 12:25:27 +03:00
Nikolai Kochetov
939d38b13e Fix gtest build. 2021-10-09 10:01:58 +03:00
taiyang-li
860f74db62 remove unused var 2021-10-09 11:15:13 +08:00
tavplubix
fa1396a9dd
Merge pull request #29790 from ClickHouse/fix-bad-cast-in-parser-create 
Fix bad cast in ParserCreateQuery
 2021-10-08 23:53:38 +03:00
Nikolai Kochetov
c6bce1a4cf Update Native. 2021-10-08 20:21:19 +03:00
Nikolai Kochetov
340b53ef85 Remove some more streams. 2021-10-08 17:03:54 +03:00
kssenii
f3f820c6e0 HTTP read buffer retries 2021-10-08 15:41:55 +03:00
MaxWk
f0e193d5a5 add database alias 2021-10-08 20:25:37 +08:00
Nikolai Kochetov
9eddee5517
Merge pull request #29768 from ClickHouse/remove-merging-streams 
Remove some merging streams
 2021-10-08 13:15:10 +03:00
MaxWk
b081f58db5 add alias 2021-10-08 16:49:02 +08:00
Maksim Kita
717998a37f
Merge pull request #29825 from azat/memory_profiler_step-cleaner-api 
Make memory_profiler_step API cleaner
 2021-10-07 22:44:47 +03:00
tavplubix
de2fd9d762
Merge pull request #29812 from ClickHouse/better_diagnostic_for_optimize 
Better diagnostic for OPTIMIZE
 2021-10-07 22:22:25 +03:00
Kseniia Sumarokova
185d58a7cc
Merge pull request #29766 from ClickHouse/remove_redundant_seeks 
Less seeks in compressed buffers
 2021-10-07 17:04:50 +03:00
Alexander Tokmakov
7e2bc184ec fix another suspicious places, add test 2021-10-07 16:43:49 +03:00
feng lv
4233e2cca4 fix 2021-10-07 13:23:25 +00:00
feng lv
4841b31b56 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-07 13:20:08 +00:00
tavplubix
754e038eec
Merge pull request #29803 from ClickHouse/intersects-previous-part-usability 
Improve usability of error messages when error is caused by sophisticated interventions
 2021-10-07 15:24:17 +03:00
feng lv
bd1ea5cd16 try fix undefine in ifstream 2021-10-07 12:15:36 +00:00
Alexander Tokmakov
5365cc686f Merge branch 'master' into improvement_for_strange_mutations 2021-10-07 13:26:58 +03:00
Alexander Tokmakov
4119848a41 better diagnostic for OPTIMIZE 2021-10-07 13:08:10 +03:00
Vladimir C
0422939893
Merge pull request #29281 from lingtaolf/bugfix/tupleFilter 2021-10-07 12:24:26 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams (#29491) 
* Remove more streams.

* Fixing build.

* Fixing build.

* Rename files.

* Fix fast test.

* Fix StorageKafka.

* Try fix kafka test.

* Move createBuffer to KafkaSource ctor.

* Revert "Move createBuffer to KafkaSource ctor."

This reverts commit 81fa94d27e.

* Revert "Try fix kafka test."

This reverts commit 2107e54969.

* Comment some rows in test.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-10-07 11:26:08 +03:00
Azat Khuzhin
71cffbf521 Make memory_profiler_step API cleaner 
Right now to configure memory_profiler_step/total_memory_profiler_step
you need to call:

    MemoryTracker::setOrRaiseProfilerLimit()
    MemoryTracker::setProfilerStep()

But it is easy to forget about setOrRaiseProfilerLimit(), since there is
no even any comments about this.

So instead, make setOrRaiseProfilerLimit() private and call it from
setProfilerStep()
 2021-10-07 10:52:16 +03:00
feng lv
127a0f3591 fix 2021-10-07 07:31:41 +00:00
Maksim Kita
4ec7311d4d
Merge pull request #29738 from kitaisreal/added-function-ngram 
Added function ngram
 2021-10-07 10:21:00 +03:00
feng lv
d9959a4f22 fix 2021-10-07 05:56:09 +00:00
feng lv
28f8e1c297 fix 
update test
 2021-10-07 05:33:12 +00:00
Maksim Kita
b7b39da8cc Fixed build 2021-10-06 23:22:29 +03:00
Nikolai Kochetov
d0c6f11fcb More. 2021-10-06 20:59:27 +03:00
Alexey Milovidov
1455cfeddd Improve usability of error messages when error is caused by sophisticated interventions 2021-10-06 15:28:52 +03:00
Maksim Kita
33353be4ed Fixed tests 2021-10-06 15:12:07 +03:00
kssenii
1f91e2de46 Update install.sh 2021-10-06 12:08:37 +00:00
kssenii
c96b403809 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into remote-connection 2021-10-06 11:54:28 +00:00
feng lv
c801fa65df fix and add new test 
fix

fix
 2021-10-06 10:56:42 +00:00
Nikolai Kochetov
6b619512aa Fix style. 2021-10-06 12:37:56 +03:00
feng lv
acdb8233c3 fix 
fix

fix
 2021-10-06 09:06:33 +00:00
kssenii
5a87b6eb4f Predefined configuration for table function remote 2021-10-06 05:55:09 +00:00
Maksim Kita
90eb74c933 Fixed tests 2021-10-06 00:37:45 +03:00
feng lv
f2a62ef9ee try to fix data race found in DirectoryWatcher 2021-10-05 17:38:18 +00:00
mergify[bot]
ab55225c8f
Merge branch 'master' into remove-merging-streams 2021-10-05 14:16:34 +00:00
Nikolai Kochetov
96d070a5ba Fix tests. 2021-10-05 16:58:24 +03:00
feng lv
0709b2e690 fix 2021-10-05 13:28:45 +00:00
mergify[bot]
3b1f49aac7
Merge branch 'master' into nv/parts-uuid-move-shard-kill 2021-10-05 12:36:02 +00:00
alesapin
9aab0eb0a2 Fixup 2021-10-05 13:41:09 +03:00
alesapin
8186339307 Less seeks in compressed buffers 2021-10-05 12:11:25 +03:00
Vitaly Baranov
8a01b32cba
Merge pull request #28637 from vitlibar/fix-materialized-column-as-sharding-key 
Fix materialized column as sharding key
 2021-10-05 10:53:24 +03:00
feng lv
86d31445fd Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-05 03:58:27 +00:00
Alexander Tokmakov
bb32432943 more fixes 2021-10-05 00:13:18 +03:00
Nikolai Kochetov
2001ebbf9d Fix build. 2021-10-04 21:52:31 +03:00
Nikita Mikhaylov
b31d11478b
Merge pull request #29680 from azat/defines 
Cleanup common defines
 2021-10-04 20:13:29 +03:00
Nikita Mikhaylov
36516dd503
Merge pull request #29681 from azat/MemoryTracker-mutate-merge 
Apply max_untracked_memory/memory_profiler_step/memory_profiler_sample_probability during mutate/merge
 2021-10-04 20:04:57 +03:00
Maksim Kita
3fe9fdac96 Added function ngram 2021-10-04 18:53:17 +03:00
Vitaly Baranov
200f655a2f Fix writing marks in StorageLog. 2021-10-04 18:45:04 +03:00
Nikolay Degterinsky
e03bf0c526 Merge branch 'master' into query_parameters 2021-10-04 17:23:17 +03:00
feng lv
873805d4da update poco 
update

fix unbundled build
 2021-10-04 13:55:49 +00:00
Anton Popov
362e84a336
Merge pull request #29429 from Enmk/Governance/database_comment 
Implemented creating databases with comments
 2021-10-04 15:12:33 +03:00
feng lv
45a49b399a fix typo 2021-10-04 08:23:02 +00:00
Vitaly Baranov
1636ee24bb Fix using materialized column as sharding key. 2021-10-04 10:56:42 +03:00
feng lv
f40a17c300 fix bug in updateFileInfos 
update
 2021-10-04 07:45:37 +00:00
Kseniia Sumarokova
5ab17d795f
Merge pull request #29673 from ClickHouse/url-fix-initialization 
Fix initialization of `StorageURLSource`
 2021-10-04 10:23:52 +03:00
feng lv
8c82e8b9bc fix 
fix
 2021-10-03 17:49:00 +00:00
kssenii
dbc114e6fc Fix special build check 2021-10-03 16:49:02 +00:00
kssenii
7ea9037939 Fix 2021-10-03 16:41:12 +00:00
Azat Khuzhin
117e9e77c8 Apply max_untracked_memory/memory_profiler_{step,sample_probability} during mutate/merge 2021-10-03 17:39:07 +03:00
kssenii
0cb7193fde Fix url with failover case 2021-10-03 13:58:03 +00:00
Azat Khuzhin
ae5ee2dd28 Move macros for distributed engine into separate header 2021-10-03 14:34:03 +03:00
Azat Khuzhin
6a9dd9828d Move protocol macros into separate header 
Defines.h is a very common header, so lots of modules will be recompiled
on changes.
Move macros for protocol into separate header, this should significantly
decreases number of units to compile on it's changes.
 2021-10-03 14:34:03 +03:00
feng lv
9683513c52 fix build 2021-10-03 11:03:20 +00:00
feng lv
8a4092c55b Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-03 10:00:16 +00:00
feng lv
3a56cc5141 fix 
update test
 2021-10-03 09:49:11 +00:00
Alexey Milovidov
868215d0dc Fix initialization of StorageURLSource 2021-10-03 07:28:28 +03:00
Maksim Kita
395119339f Merge branch 'master' into rename-common 2021-10-02 17:15:10 +03:00
alexey-milovidov
86d1ce63cf
Merge pull request #29649 from CurtizJ/merging-29376 
Merging #29376
 2021-10-02 12:02:42 +03:00
Alexey Milovidov
2ad5dbdfe8 Rename "common" to "base" 2021-10-02 11:30:38 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Anton Popov
bb92ff52d8
Update IMergeTreeDataPart.cpp 2021-10-02 00:50:19 +03:00
Anton Popov
d404efbf7e add test for freeze in-memory parts 2021-10-01 22:05:36 +03:00
Anton Popov
f0363c524d Merge branch 'master' of git://github.com/mo-avatar/ClickHouse into merging-29376 2021-10-01 20:29:33 +03:00
Maksim Kita
57666725f9
Merge pull request #29623 from kitaisreal/bloom-filter-indexes-map-support-fix-unsafe-identifier-cast 
BloomFilter indexes map data type support fix unsafe identifier cast
 2021-10-01 19:28:04 +03:00
Nikita Mikhaylov
b2f7b6fae8
Fix race between MergeTask and storage destruction (#29614) 2021-10-01 18:55:04 +03:00
feng lv
9c1dcf41b8 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-10-01 14:47:21 +00:00
feng lv
5bb5e04fb3 fix assert 2021-10-01 14:47:05 +00:00
vdimir
7444d64156
Fix for recursive tuples in MergeTreeWhereOptimizer 2021-10-01 17:01:03 +03:00
vdimir
23ce41e3bb
MergeTreeWhereOptimizer supports constant tuple in both sides of equals 2021-10-01 17:01:03 +03:00
vdimir
64ffacd9e0
MergeTreeWhereOptimizer::tryAnalyzeTuple, fix fuzz 2021-10-01 17:01:02 +03:00
vdimir
e326b12a37
Fix style in MergeTreeWhereOptimizer.cpp 2021-10-01 17:01:02 +03:00
凌涛
f660224a5f
add tests 2021-10-01 17:01:01 +03:00
凌涛
ea68944055
fixed index didn't work in tuple where 2021-10-01 17:01:01 +03:00
Maksim Kita
c793c8e7fa BloomFilter indexes map data type support fix unsafe identifier cast 2021-10-01 14:40:00 +03:00
feng lv
6eeade23fa revert drop table immediately 
remove unused code

fix pvs check

fix
 2021-10-01 11:36:49 +00:00
Vasily Nemkov
2f16806cd0 Implemented creating databases with comments 2021-10-01 11:51:26 +03:00
feng lv
ac9af1ca38 fix wrong offset 2021-10-01 08:24:57 +00:00
feng lv
8c20a184ca optimize event handle 2021-10-01 05:06:19 +00:00
Alexey Milovidov
4271f8c738 Merge remote-tracking branch 'origin/master' into yandex-to-clickhouse-in-configs 2021-10-01 01:33:58 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
Kseniia Sumarokova
3e0d4faa1e
Merge pull request #29503 from azat/external-db-log-query 
Log queries to external databases (since they may be rewritten)
 2021-09-30 21:08:57 +03:00
feng lv
4390c8f029 fix 2021-09-30 17:13:45 +00:00
feng lv
ee02e52d3d Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into filelog-engine 2021-09-30 16:02:31 +00:00
feng lv
1758ff2d32 fix 2021-09-30 16:02:17 +00:00
Alexey Milovidov
551c52c236 Merge remote-tracking branch 'origin/master' into yandex-to-clickhouse-in-configs 2021-09-30 17:45:14 +03:00
Raúl Marín
6f2447c027 clang-tidy fix 2021-09-30 15:48:54 +02:00
Raúl Marín
f58742014c Consistent naming 2021-09-30 11:05:21 +02:00
Nikolay Degterinsky
7f3d395bf3 Merge branch 'master' into query_parameters 2021-09-30 02:01:33 +03:00
Azat Khuzhin
a4f2663b82 Log queries to external databases (since they may be rewriten) 
By rewritten means that some expression from WHERE can be removed, yes
there is external_table_strict_query, but sometimes it is too strict.
 2021-09-29 21:56:28 +03:00