thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-04 13:32:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
41,851 Commits 9,179 Branches 1,782 Tags 2.3 GiB
936cc9d573
Commit Graph

66 Commits

Author SHA1 Message Date
alexey-milovidov
82e849e6a1
Update StorageDistributed.cpp 2020-06-06 18:57:52 +03:00
Azat Khuzhin
ff85125326 Fix readability-qualified-auto 2020-06-04 20:23:46 +03:00
Azat Khuzhin
86c5465bf8 Rewrite StorageSystemDistributionQueue interfaces 2020-06-04 03:04:32 +03:00
Azat Khuzhin
389f78ceee Add system.distribution_queue 
system.distribution_queue contains the following columns:
- database
- table
- data_path
- is_blocked
- error_count
- data_files
- data_compressed_bytes
 2020-06-04 02:36:16 +03:00
Azat Khuzhin
60d10f1bac Fix typo in StorageDistributed 2020-06-04 02:36:16 +03:00
alesapin
3847ea892d Merge branch 'master' into consistent_metadata3 2020-06-01 13:17:59 +03:00
Alexey Milovidov
25f941020b Remove namespace pollution 2020-05-31 00:57:37 +03:00
alesapin
52ca6b2051 I'm able to build it 2020-05-28 15:37:05 +03:00
Alexey Milovidov
7e1813825b Return old names of macros 2020-05-24 01:24:01 +03:00
Alexey Milovidov
85f84550ba Progress on task 2020-05-23 23:37:37 +03:00
Alexey Milovidov
9d2a0d2dd7 Apply all transformations again 2020-05-23 21:59:49 +03:00
Alexey Milovidov
a2ad11897f Remove duplicate whitespaces (preparation) 2020-05-23 21:53:58 +03:00
Alexey Milovidov
1f13515a65 Make all LOG in single line (preparation) 2020-05-23 21:31:37 +03:00
Alexey Milovidov
e391b77d81 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/' 2020-05-23 19:56:05 +03:00
Alexey Milovidov
ee4ffbc332 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}", \4);/' 2020-05-23 19:47:56 +03:00
Alexey Milovidov
8d2e80a5e2 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+"\)' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+, "[^"]+")\)/\1_FORMATTED(\2)/' 2020-05-23 19:42:39 +03:00
Azat Khuzhin
bc4b75dead Add table name into logs for StorageDistributed 2020-05-23 11:57:14 +03:00
alesapin
59b3bc0c05
Merge branch 'master' into fix_deadlock_system_logs_startup 2020-05-21 22:57:52 +03:00
alesapin
c036af0261 Fix deadlock after clickhouse-server update (with changes in one of system log tables structure) during startup between concurrent merge and table rename. 2020-05-21 17:11:56 +03:00
Azat Khuzhin
d93b9a57f6 Forward declaration for Context as much as possible. 
Now after changing Context.h 488 modules will be recompiled instead of 582.
 2020-05-21 01:53:18 +03:00
Gleb Novikov
1a25ac6e1f Merge branch 'master' into refactor-reservations 2020-05-16 23:34:45 +03:00
alesapin
352c402cf2 Merge branch 'add_alter_rename_column_to_distributed' of https://github.com/vzakaznikov/ClickHouse into vzakaznikov-add_alter_rename_column_to_distributed 2020-05-13 14:14:23 +03:00
alexey-milovidov
33d491edf3
Merge pull request #10516 from azat/dist-GROUP_BY-sharding_key-fixes 
Disable GROUP BY sharding_key optimization by default (and fix for WITH ROLLUP/CUBE/TOTALS)
 2020-05-11 12:03:27 +03:00
Gleb Novikov
390b39b272 VolumePtr instead of DiskPtr in MergeTreeData* 2020-05-10 00:24:15 +03:00
Vitaliy Zakaznikov
90e52e7fea Adding support for ALTER RENAME COLUMN query to Distributed table engine. 2020-05-07 14:54:35 +02:00
Gleb Novikov
c637d99e07 Volumes and storages refactoring: 
1. Moved Volume to separate file
  2. Created IVolume interface and implemented current behaviour in implementation of new interface — VolumeJBOD
  3. Replaced all old volume usages with new VolumeJBOD. Where it is unnecessary to have JBOD — left just IVolume.
  4. Removed old Volume completely
  5. Moved StoragePolicy to separated files
  6. Moved DiskSelector to separated files
  7. Removed DiskSpaceMonitor file
 2020-05-04 23:15:38 +03:00
alexey-milovidov
229f666dea
Merge pull request #10611 from azat/optimize_skip_unused_shards-LowCardinality 
Fix optimize_skip_unused_shards with LowCardinality
 2020-05-02 22:14:33 +03:00
Azat Khuzhin
63d8ab8f03 Make createSelector() static (in storage) and const (in stream) 2020-05-01 11:31:05 +03:00
Azat Khuzhin
f22ba15b4a Reduce copy-paste of DistributedBlockOutputStream::createSelector 
This will make it less error prone.
 2020-05-01 02:59:40 +03:00
Azat Khuzhin
cdd7013438 Drop superfluous "Skipping irrelevant shards" messages 
Before this patch it printed 3 times:
- from StorageDistributed::getProcessingStageImpl()
- from StorageDistributed::read()
- from StorageDistributed::getProcessingStageImpl() (from StorageDistributed::read() -> getSampleBlock())
(But this should be optimized)
 2020-05-01 02:56:13 +03:00
Azat Khuzhin
c648c300bf Fix optimize_skip_unused_shards with LowCardinality 2020-05-01 02:39:58 +03:00
Azat Khuzhin
4cbe625567 Fix shard numbers output in logs (full cluster had been printed over optimized) 2020-05-01 02:13:07 +03:00
Azat Khuzhin
038235684d Add optimize_distributed_group_by_sharding_key and disable it by default 
I know at least one way to fool that optimization, by using as sharding
key something like `if(col1>0, col1, col2)` (although this is not common
sharding key I would say, but can be useful if this will work
correctly), so let's disable it by default.
 2020-04-29 00:09:25 +03:00
alesapin
f981649213 Fix pushing to views stream and refactor virtuals 2020-04-28 13:38:57 +03:00
alesapin
4badd0fd28 Better code 2020-04-27 20:46:51 +03:00
alesapin
01db4877f6 Fix style check 2020-04-27 18:44:33 +03:00
alesapin
18c550df15 Better virtuals logic 2020-04-27 16:55:30 +03:00
alesapin
2829774105 Merge branch 'master' into refactor_istorage 2020-04-27 15:34:21 +03:00
Azat Khuzhin
20b4eed9a1 Disable GROUP BY sharding_key optimization for WITH ROLLUP/CUBE/TOTALS 2020-04-27 01:30:54 +03:00
alexey-milovidov
c9334d3fde
Merge pull request #10491 from azat/dist-shutdown 
Proper Distributed shutdown (fixes UAF, avoid waiting for sending all batches)
 2020-04-25 23:47:59 +03:00
Azat Khuzhin
747a74215f Avoid processing all batches before Distributed shutdown 2020-04-25 02:03:27 +03:00
Azat Khuzhin
8ad6b37913 Proper StorageDistributed shutdown to avoid UAF in DistributedMonitor 
StorageDistributed::shutdown() does not acquire the lock, that controls
access to the cluster_nodes_data, thus it does not synced with the
requireDirectoryMonitor(), hence some monitors can be untracked that
will trigger UAF (use-after-free) after DROP TABLE dist:

This is for the SIGSEGV from the DirectoryMonitor (with already destroyed storage):
    0  0x0000000008e9f760 in std::__1::__cxx_atomic_load<int> (__order=std::__1::memory_order::seq_cst, __a=0x0)
    1  std::__1::__atomic_base<int, false>::load (__m=std::__1::memory_order::seq_cst, this=0x0) <-- this is nullptr
    2  std::__1::__atomic_base<int, false>::operator int (this=0x0)
    3  DB::ActionBlocker::isCancelled (this=0x7f85e31c9bb8) at ../src/Common/ActionBlocker.h:18
    4  DB::StorageDistributedDirectoryMonitor::run (this=0x7f85f93b2a00) at ../src/Storages/Distributed/DirectoryMonitor.cpp:140
 2020-04-25 02:03:26 +03:00
alesapin
793f4b734a Remove obsolete comment 2020-04-24 13:31:03 +03:00
alesapin
dc2dd77d2e Remove redundant overrides from IStorage 2020-04-24 12:20:09 +03:00
Alexander Tokmakov
04d6b59ac0 Merge branch 'master' into database_atomic 2020-04-23 17:31:37 +03:00
Alexey Milovidov
1e325a9fd9 Checkpoint 2020-04-22 09:22:14 +03:00
Alexander Tokmakov
b29bddac12 Merge branch 'master' into database_atomic 2020-04-20 14:09:09 +03:00
Alexey Milovidov
d7264b292d Merge branch 'master' into sorting-processors 2020-04-20 09:29:41 +03:00
alexey-milovidov
1577d771df
Merge pull request #10341 from azat/auto_distributed_group_by_no_merge 
Auto distributed_group_by_no_merge on GROUP BY sharding key
 2020-04-20 08:30:27 +03:00
Azat Khuzhin
e44d5c5749 Fix clang readability-container-size-empty warning in StorageDistributed::canForceGroupByNoMerge() 2020-04-20 01:12:22 +03:00