alexey-milovidov
4df7dd8c2b
Merge pull request #10656 from azat/kafka-consumer-hang
...
Kafka cleanup
2020-05-07 00:20:31 +03:00
bobrovskij artemij
a2916c23a5
fixes/style
2020-05-06 02:42:44 +03:00
alexey-milovidov
114f09cfca
Merge pull request #10666 from NanoBjorn/refactor-volume
...
Volumes and storages refactoring
2020-05-05 16:25:26 +03:00
alexey-milovidov
c54f6d5a66
Merge pull request #10546 from kyprizel/master
...
Lexer, ast, checksum and column specs fuzzers added
2020-05-05 14:50:55 +03:00
alexey-milovidov
09756a036a
Merge pull request #10645 from ClickHouse/check-suspicious-codecs
...
Check for suspicious codecs #4966
2020-05-04 23:33:59 +03:00
Gleb Novikov
c637d99e07
Volumes and storages refactoring:
...
1. Moved Volume to separate file
2. Created IVolume interface and implemented current behaviour in implementation of new interface — VolumeJBOD
3. Replaced all old volume usages with new VolumeJBOD. Where it is unnecessary to have JBOD — left just IVolume.
4. Removed old Volume completely
5. Moved StoragePolicy to separated files
6. Moved DiskSelector to separated files
7. Removed DiskSpaceMonitor file
2020-05-04 23:15:38 +03:00
alesapin
d47d28d51f
Fix mutations assignment
2020-05-04 19:30:39 +03:00
Azat Khuzhin
61c5df5662
Drop claimed from the KafkaBlockInputStream
2020-05-04 17:23:09 +03:00
Azat Khuzhin
a10cfc3ae5
Pass stream_cancelled to the copyData as stub in StorageKafka
2020-05-04 17:23:09 +03:00
Azat Khuzhin
c69217e9f3
Force destrying of kafka buffers from the shutdown in StorageKafka
2020-05-04 17:23:09 +03:00
Azat Khuzhin
9e14db2091
Add missing kafka settings into docs
2020-05-04 17:23:05 +03:00
Alexey Milovidov
99c18c5a09
Check for suspicious codecs #4966
2020-05-04 03:11:49 +03:00
alexey-milovidov
884c2aa631
Merge pull request #10641 from ClickHouse/storage-buffer-nullptr-dereference
...
Fix nullptr dereference in StorageBuffer
2020-05-04 02:01:57 +03:00
Alexey Milovidov
a5bdc375ca
Fix Arcadia #10569
2020-05-04 00:36:11 +03:00
Alexey Milovidov
f8866af0c1
Fix nullptr dereference in StorageBuffer
2020-05-03 23:53:54 +03:00
Azat Khuzhin
6ffdd53b6a
Share auto-increment for first batch and tmp file in DistributedBlockOutputStream
2020-05-03 14:47:59 +03:00
Azat Khuzhin
53c470cab4
Fix directory monitor initialization from INSERT into Distributed
...
This also fixes hardlink code (when one file should be sent to multiple
servers, i.e. internal_replication == false) of writeToShard() with
distributed_storage_policy (i.e. when StorageDistributed::getPath() will
path to different filesystems).
Plus also cleanup DistributedBlockOutputStream::writeToShard() a little.
2020-05-03 14:47:51 +03:00
Azat Khuzhin
e97e1f06db
Do not schedule distributed send if there were no error
...
Since in this case it will be scheduled from the
DistributedBlockOutputStream with the
distributed_directory_monitor_max_sleep_time_ms, and this will overwrite
timer that was set by the DistributedBlockOutputStream, not good.
2020-05-03 14:46:44 +03:00
Azat Khuzhin
947b3942dd
Schedule distributed sends after the file has been written
2020-05-03 14:46:43 +03:00
Azat Khuzhin
0157fd5d93
Fix distributed send that are scheduled by INSERT query
...
Before this patch each INSERT query re-schedule distributed send, thus
each time it resets the timer, while this is not the expected behaviour,
since in on frequent INSERT distributed sends will not be triggered at
all.
Fix this by not resetting the timer.
2020-05-03 14:46:42 +03:00
Azat Khuzhin
6bb39dafc3
Drop decreated code (cond var and note for thread) in DirectoryMonitor
2020-05-03 14:46:41 +03:00
alexey-milovidov
db4c2352e1
Merge pull request #10569 from zhang2014/fix/ISSUES-10551
...
ISSUES-10551 add backward compatibility for create bloom filter index
2020-05-02 22:25:29 +03:00
alexey-milovidov
229f666dea
Merge pull request #10611 from azat/optimize_skip_unused_shards-LowCardinality
...
Fix optimize_skip_unused_shards with LowCardinality
2020-05-02 22:14:33 +03:00
alexey-milovidov
4f1b115e76
Merge pull request #10610 from azat/optimize_skip_unused_shards-fixes
...
Fix logging for optimize_skip_unused_shards
2020-05-02 22:13:01 +03:00
tavplubix
a073cd6c79
Fix segfault in StorageBuffer when exception on server startup ( #10609 )
...
* fix segfault in StorageBuffer when exception on server startup
* Trigger CI
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
2020-05-02 19:54:20 +03:00
Alexey Milovidov
81adcc9058
Remove obsolete records from system.build_options
2020-05-02 17:00:03 +03:00
alexey-milovidov
404452bad3
Update MergeTreeIndices.cpp
2020-05-01 21:40:56 +03:00
alesapin
4d8660f270
Remove obsolete comment
2020-05-01 13:05:37 +03:00
Azat Khuzhin
63d8ab8f03
Make createSelector() static (in storage) and const (in stream)
2020-05-01 11:31:05 +03:00
Azat Khuzhin
f22ba15b4a
Reduce copy-paste of DistributedBlockOutputStream::createSelector
...
This will make it less error prone.
2020-05-01 02:59:40 +03:00
Azat Khuzhin
cdd7013438
Drop superfluous "Skipping irrelevant shards" messages
...
Before this patch it printed 3 times:
- from StorageDistributed::getProcessingStageImpl()
- from StorageDistributed::read()
- from StorageDistributed::getProcessingStageImpl() (from StorageDistributed::read() -> getSampleBlock())
(But this should be optimized)
2020-05-01 02:56:13 +03:00
Azat Khuzhin
c648c300bf
Fix optimize_skip_unused_shards with LowCardinality
2020-05-01 02:39:58 +03:00
Azat Khuzhin
4cbe625567
Fix shard numbers output in logs (full cluster had been printed over optimized)
2020-05-01 02:13:07 +03:00
Nikolai Kochetov
19dadb8c2d
Add parallel final.
2020-04-30 12:59:08 +03:00
alesapin
15e8f37839
Merge pull request #10531 from CurtizJ/polymorphic-parts-2
...
Fix index corruption in merges with compact parts.
2020-04-30 12:15:25 +03:00
alesapin
5e2afad549
Merge pull request #10477 from ClickHouse/refactor_istorage
...
Remove getColumn and hasColumn from IStorage, better virtuals.
2020-04-30 10:42:52 +03:00
Anton Popov
6ed3092c8a
make fillIndexGranularity less complicated
2020-04-30 00:57:58 +03:00
alesapin
69fdf2a6a3
Review fixes and better test
2020-04-29 15:15:23 +03:00
zhang2014
8ac709b5d3
ISSUES-10551 add backward compatibility for create bloom filter index
2020-04-29 11:57:28 +08:00
Azat Khuzhin
038235684d
Add optimize_distributed_group_by_sharding_key and disable it by default
...
I know at least one way to fool that optimization, by using as sharding
key something like `if(col1>0, col1, col2)` (although this is not common
sharding key I would say, but can be useful if this will work
correctly), so let's disable it by default.
2020-04-29 00:09:25 +03:00
robot-clickhouse
de4e88ff97
Auto version update to [20.5.1.1] [54435]
2020-04-28 20:12:16 +03:00
bobrovskij artemij
0cc3d780b3
Support write for XDBCStorage
2020-04-28 14:03:24 +03:00
alesapin
f981649213
Fix pushing to views stream and refactor virtuals
2020-04-28 13:38:57 +03:00
bobrovskij artemij
1bcd077fbf
Support write for XDBCStorage
2020-04-28 03:56:44 +03:00
Eldar Zaitov
670ed4bf1b
lexer, ast, checksum and column specs fuzzers added
2020-04-28 02:12:18 +03:00
Anton Popov
032a0aa282
fix clang-tidy
2020-04-27 21:12:17 +03:00
alesapin
7c2f4d121a
Fix func
2020-04-27 20:48:53 +03:00
alesapin
4badd0fd28
Better code
2020-04-27 20:46:51 +03:00
alesapin
228dbf475e
Finalize mutations in background ( #10526 )
2020-04-27 19:19:04 +03:00
alesapin
01db4877f6
Fix style check
2020-04-27 18:44:33 +03:00
alesapin
c24ab563d7
Fix several wrong usages
2020-04-27 18:38:35 +03:00
alesapin
b0c5c1d685
Fix style
2020-04-27 18:21:37 +03:00
alesapin
6cfe0f7046
Fix after refactoring
2020-04-27 18:20:03 +03:00
alesapin
cefc58e77c
Fix style
2020-04-27 18:17:08 +03:00
alesapin
5c6316afbc
Fix bug
2020-04-27 18:10:50 +03:00
alesapin
4c1ac83112
Fix in IStorage
2020-04-27 17:44:32 +03:00
alesapin
e85b50c911
Revert "Strange fix, how it was possible earlier?"
...
This reverts commit 8d20644c99
.
2020-04-27 17:17:12 +03:00
alesapin
8d20644c99
Strange fix, how it was possible earlier?
2020-04-27 17:12:54 +03:00
alesapin
18c550df15
Better virtuals logic
2020-04-27 16:55:30 +03:00
Anton Popov
bee343dae1
fix writing of index in compact parts
2020-04-27 16:36:50 +03:00
alesapin
2829774105
Merge branch 'master' into refactor_istorage
2020-04-27 15:34:21 +03:00
Azat Khuzhin
20b4eed9a1
Disable GROUP BY sharding_key optimization for WITH ROLLUP/CUBE/TOTALS
2020-04-27 01:30:54 +03:00
alexey-milovidov
c9334d3fde
Merge pull request #10491 from azat/dist-shutdown
...
Proper Distributed shutdown (fixes UAF, avoid waiting for sending all batches)
2020-04-25 23:47:59 +03:00
Azat Khuzhin
747a74215f
Avoid processing all batches before Distributed shutdown
2020-04-25 02:03:27 +03:00
Azat Khuzhin
8ad6b37913
Proper StorageDistributed shutdown to avoid UAF in DistributedMonitor
...
StorageDistributed::shutdown() does not acquire the lock, that controls
access to the cluster_nodes_data, thus it does not synced with the
requireDirectoryMonitor(), hence some monitors can be untracked that
will trigger UAF (use-after-free) after DROP TABLE dist:
This is for the SIGSEGV from the DirectoryMonitor (with already destroyed storage):
0 0x0000000008e9f760 in std::__1::__cxx_atomic_load<int> (__order=std::__1::memory_order::seq_cst, __a=0x0)
1 std::__1::__atomic_base<int, false>::load (__m=std::__1::memory_order::seq_cst, this=0x0) <-- this is nullptr
2 std::__1::__atomic_base<int, false>::operator int (this=0x0)
3 DB::ActionBlocker::isCancelled (this=0x7f85e31c9bb8) at ../src/Common/ActionBlocker.h:18
4 DB::StorageDistributedDirectoryMonitor::run (this=0x7f85f93b2a00) at ../src/Storages/Distributed/DirectoryMonitor.cpp:140
2020-04-25 02:03:26 +03:00
tavplubix
7928f2eaa1
Merge pull request #7512 from ClickHouse/database_atomic
...
DatabaseAtomic
2020-04-24 22:28:18 +03:00
alesapin
793f4b734a
Remove obsolete comment
2020-04-24 13:31:03 +03:00
alesapin
b4425c2dc4
Remove getColumn and hasColumn methods from IStorage
2020-04-24 13:20:03 +03:00
alesapin
0c622370c5
Better
2020-04-24 12:21:38 +03:00
alesapin
dc2dd77d2e
Remove redundant overrides from IStorage
2020-04-24 12:20:09 +03:00
Alexander Tokmakov
f413061e0d
remove Context from some IDatabase methods
2020-04-23 19:51:48 +03:00
alesapin
5ae25b68e4
Remove redundant virtual from function
2020-04-23 18:18:33 +03:00
alesapin
b0a73589b0
Remove virtual columns as separate field from IStorage
2020-04-23 18:09:00 +03:00
Alexander Tokmakov
04d6b59ac0
Merge branch 'master' into database_atomic
2020-04-23 17:31:37 +03:00
Nikita Mikhaylov
3520755ab5
fix c++20 usage
2020-04-23 12:00:43 +03:00
alexey-milovidov
4f6bb00424
Merge pull request #10420 from ClickHouse/clang-tidy-8
...
Fixes for clang-tidy on clang-10
2020-04-23 00:33:41 +03:00
Nikita Mikhaylov
3cf2016867
Merge pull request #10188 from nikitamikhaylov/quorum_bug
...
insert_quorum vs drop partition or ttl
2020-04-23 00:11:57 +04:00
Alexey Milovidov
73dd78bac1
Merge branch 'master' into clang-tidy-8
2020-04-22 21:04:47 +03:00
Alexander Tokmakov
fefbbd37df
Merge branch 'master' into database_atomic
2020-04-22 17:02:30 +03:00
zhang2014
0a0555d4d7
ISSUES-9623 fix failure test case
2020-04-22 18:34:42 +08:00
zhang2014
bfb87f0eab
ISSUES-9623 check the number and type of arguments when creating BloomFilter index
2020-04-22 16:17:09 +08:00
Alexey Milovidov
03e17405cd
Checkpoint
2020-04-22 10:03:43 +03:00
Alexey Milovidov
603d132d45
Checkpoint
2020-04-22 09:34:20 +03:00
Alexey Milovidov
1e325a9fd9
Checkpoint
2020-04-22 09:22:14 +03:00
Alexey Milovidov
be22a4b94e
Checkpoint
2020-04-22 08:39:31 +03:00
Ivan
555488a458
Fix build in Arcadia ( #10413 )
2020-04-21 23:23:44 +03:00
Nikita Mikhaylov
2173bcf28e
Merge branch 'master' of github.com:ClickHouse/ClickHouse into quorum_bug
2020-04-21 14:35:06 +03:00
Alexey Milovidov
4cc0269de0
Fix flattening of Array(Tuple(...)) #10259
2020-04-21 02:44:51 +03:00
alexey-milovidov
8c2839d3c9
Merge pull request #10308 from ClickHouse/shared-context-lifetime
...
Shared context lifetime
2020-04-21 00:12:45 +03:00
alexey-milovidov
ab922d44d5
Merge pull request #10379 from ClickHouse/only_fetch_repliac
...
Add setting always_fetch_merged_part
2020-04-21 00:07:00 +03:00
alexey-milovidov
69b6f135e7
Merge pull request #10382 from excitoon-favorites/altercodec
...
Allowed to alter column in non-modifying data mode when the same type is specified
2020-04-20 22:54:53 +03:00
alexey-milovidov
f6de9c6ffd
Update AlterCommands.cpp
2020-04-20 22:54:08 +03:00
alexey-milovidov
1eca5088ec
Update AlterCommands.cpp
2020-04-20 22:53:19 +03:00
Vladimir Chebotarev
82985ff594
Allowed to alter column in non-modifying data mode when the same type is specified.
2020-04-20 14:15:06 +03:00
Alexander Tokmakov
b29bddac12
Merge branch 'master' into database_atomic
2020-04-20 14:09:09 +03:00
Nikita Mikhaylov
05206df88c
better
2020-04-20 13:56:59 +03:00
alesapin
1bc84ec8b4
Fix comments
2020-04-20 13:56:23 +03:00
alesapin
8767ea7efa
Add setting which restrict replica to merge parts by itself and always prefer dowloading from other replicas.
2020-04-20 13:53:29 +03:00
Nikolai Kochetov
052598534a
Try fix unit tests.
2020-04-20 12:42:56 +03:00
Alexey Milovidov
d7264b292d
Merge branch 'master' into sorting-processors
2020-04-20 09:29:41 +03:00
Alexey Milovidov
47e4d14ece
Whitespace
2020-04-20 09:13:04 +03:00
alexey-milovidov
1577d771df
Merge pull request #10341 from azat/auto_distributed_group_by_no_merge
...
Auto distributed_group_by_no_merge on GROUP BY sharding key
2020-04-20 08:30:27 +03:00
Alexey Milovidov
a3195d56ae
Fix translation
2020-04-20 07:19:50 +03:00
Alexey Milovidov
7023e096ec
Check for error code in part check #6269
2020-04-20 04:44:24 +03:00
Azat Khuzhin
e44d5c5749
Fix clang readability-container-size-empty warning in StorageDistributed::canForceGroupByNoMerge()
2020-04-20 01:12:22 +03:00
Alexey Milovidov
25f3184dbe
Merge branch 'master' into async-metric-memory-usage
2020-04-19 23:18:12 +03:00
Azat Khuzhin
be1dec9239
Fix distributed_group_by_no_merge optimization for Distributed-over-Distributed
2020-04-19 21:11:14 +03:00
Azat Khuzhin
93d049fe64
Allow auto distributed_group_by_no_merge for DISTINCT of sharding key
2020-04-19 18:53:37 +03:00
Azat Khuzhin
de4a723264
Auto distributed_group_by_no_merge on GROUP BY injective function of sharding key
2020-04-19 18:33:49 +03:00
Azat Khuzhin
5d11118cc9
Use thread pool (background_distributed_schedule_pool_size) for distributed sends
...
After #8756 the problem with 1 thread for each (distributed table, disk)
for distributed sends became even worse (since there can be multiple
disks), so use predefined thread pool for this tasks, that can be
controlled with background_distributed_schedule_pool_size knob.
2020-04-19 12:01:56 +03:00
Azat Khuzhin
673ddc9d77
Drop superfluous locking for atomic in DirectoryMonitor
2020-04-19 00:22:48 +03:00
alexey-milovidov
aac7cc7330
Merge pull request #10126 from Jokser/replicated-merge-tree-s3
...
Support ReplicatedMergeTree over S3
2020-04-18 16:13:04 +03:00
alexey-milovidov
46a5c75beb
Update MergeTreeData.h
2020-04-18 16:11:57 +03:00
alexey-milovidov
8cea45315f
Merge pull request #10133 from 4ertus2/joins
...
JOIN with Dictionary
2020-04-18 16:08:15 +03:00
alexey-milovidov
959e6cd032
Merge pull request #10221 from ClickHouse/remove-part-special-case
...
Support for one special case of removing incompletely written parts
2020-04-18 15:58:58 +03:00
alexey-milovidov
72dd453c3f
Merge pull request #10234 from vitlibar/show-columns-of-not-loaded-dictionary
...
Show information about not loaded dictionaries in system tables.
2020-04-18 15:06:08 +03:00
alexey-milovidov
17b6e0b3e5
Merge pull request #10315 from azat/buffer-flush-bg-pool
...
Use background thread pool for background buffer flushes
2020-04-18 15:04:35 +03:00
Vitaly Baranov
d62345bbaa
Show information about not loaded dictionaries in system.columns, system.tables
...
and in "SHOW TABLES" executed in any database with the "Dictionary" engine.
2020-04-18 12:50:54 +03:00
Nikolai Kochetov
84faa9af26
Merge branch 'master' into shared-context-lifetime
2020-04-17 19:13:13 +03:00
Anton Popov
f7b7b5eb74
remove unused flags
2020-04-17 14:59:10 +03:00
Nikolai Kochetov
d51a9c551b
Remove shared_ptr to SharedContext from Context.
2020-04-17 12:47:40 +03:00
Alexey Milovidov
fdc5688d2d
Added memory usage to AsynchronousMetrics
2020-04-17 07:09:41 +03:00
Alexey Milovidov
fb095ad787
Merge branch 'master' into remove-part-special-case
2020-04-17 06:14:15 +03:00
Alexey Milovidov
b16ee538ab
Merge branch 'master' into sorting-processors
2020-04-17 05:31:55 +03:00
alexey-milovidov
17e7d4d88a
Merge pull request #10307 from abyss7/arcadia-4
...
Changes for auto-sync with Arcadia
2020-04-17 05:08:34 +03:00
Nikita Mikhaylov
6b58998a36
better
2020-04-16 22:57:39 +03:00
Nikita Mikhaylov
c153cc8fe5
fixed :)
2020-04-16 22:54:11 +03:00
Nikita Mikhaylov
9a80eb6e47
Merge branch 'master' of github.com:ClickHouse/ClickHouse into quorum_bug
2020-04-16 21:47:34 +03:00
Nikita Mikhaylov
85fbf413c1
server crashes, just to save
2020-04-16 21:47:20 +03:00
Azat Khuzhin
a9b1b36907
Use separate pool for buffer flushes (background_buffer_flush_schedule_pool_size)
2020-04-16 20:14:28 +03:00
Artem Zuikov
47cb670213
Merge branch 'master' into joins
2020-04-16 19:06:13 +03:00
Alexander Tokmakov
605f3b2119
fix DROP vs OPTIMIZE race in ReplicatedMergeTree
2020-04-16 18:30:18 +03:00
Ivan Lezhankin
e230632645
Changes required for auto-sync with Arcadia
2020-04-16 15:31:57 +03:00
Alexey Milovidov
cdeda4ab91
Fix usage of max_parser_depth setting; remove harmful default function arguments
2020-04-16 04:06:10 +03:00
Alexey Milovidov
f704c465a7
Make aware of one special case of imcompletely written parts #9940
2020-04-16 00:52:21 +03:00
Nikolai Kochetov
153f795ebe
Merge branch 'master' into sorting-processors
2020-04-15 12:07:05 +03:00
Alexey Milovidov
8ad04d4fec
Remove useless code
2020-04-15 00:05:45 +03:00
Nikolai Kochetov
12f4cfb2c7
Use processors for final.
2020-04-14 13:04:49 +03:00
Nikolai Kochetov
434a13a8bd
Move algorithm to separate files.
2020-04-14 12:21:24 +03:00
alexey-milovidov
9de981d8d3
Merge pull request #10052 from Vxider/mv_parallel_insert
...
Enable parallel INSERT for materialized view
2020-04-14 01:30:44 +03:00
alexey-milovidov
2219f05139
Merge pull request #10235 from ClickHouse/fix_range_hashed_ddl_dictionaries
...
Fix 'Cannot add column' in DDL dictionary
2020-04-14 01:28:24 +03:00
Nikita Mikhaylov
89f80a6353
Merge branch 'master' into quorum_bug
2020-04-13 23:25:55 +04:00
Alexander Tokmakov
b7d4bccc77
fix build
2020-04-13 22:02:13 +03:00
Artem Zuikov
6106b2bfc7
Merge branch 'master' into joins
2020-04-13 20:37:27 +03:00
Nikita Mikhaylov
68e82c11e0
fix build
2020-04-13 18:58:29 +03:00
Alexander Tokmakov
6c0d9723e0
Merge branch 'master' into database_atomic
2020-04-13 18:47:47 +03:00
Nikita Mikhaylov
441bb13d4a
better
2020-04-13 18:21:05 +03:00
alesapin
9cc7d0f06a
Fix 'Cannot add column' error while creating range_hashed dictionary using DDL queries
2020-04-13 17:34:01 +03:00
alexey-milovidov
396360cd90
Merge pull request #10209 from ClickHouse/akz/fix_segfault_in_system_tables
...
Fix N.P. segfault in StorageSystemTables for database engine "Lazy"
2020-04-13 16:33:27 +03:00
alexey-milovidov
eedb5a3e3d
Merge pull request #10218 from ClickHouse/fix-generate-random-date
...
Fixed "generateRandom" function for Date type
2020-04-13 16:26:08 +03:00