thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
32,897 Commits 9,338 Branches 1,783 Tags 2.1 GiB
c4e9784464
Commit Graph

5543 Commits

Author SHA1 Message Date
Azat Khuzhin
3a72e1c12a Add checksum for extra info/query in distributed sends 
This extras includes:
- server revision
- query settings
- query

Otherwise the code can try to interpret data, and got for instance
std::length_error exception, which is not catched (to mark the part as
broken). Also this will protect from the corruptions on disk.

And add a simple test, since dbms/tests/integration/test_insert_into_distributed too complex.

Also simplify the code by using readStringBinary() over
readVarUInt()+b.readStrict() (this also gains additional checks that
string is not bigger then 1GB).

Refs: #4852 (8ef7f3589a)
v2: avoid ABI breakage (suggested by @vitlibar)
v3: minor code fixes (suggested by @vitlibar) and as a consequence clang-8 build fix
v4: drop DBMS_MIN_REVISION_WITH_EXTRAS_CHECKSUM_IN_DIST_BATCH and also
    revert some renames to make the patch cleaner
 2019-12-04 22:54:38 +03:00
Mikhail Filimonov
34275626de Fix for https://github.com/ClickHouse/ClickHouse/issues/8003 : old producers do not write timestamp, but column still should have enough rows 2019-12-03 23:03:43 +01:00
Mikhail Filimonov
3de3ba336a Fix Native format in Kafka 2019-12-03 22:12:47 +01:00
Mikhail Filimonov
6e426592a7 Make the formats like RowBinaryWithNamesAndTypes work in Kafka & do not try to parse buffer when it at eof 2019-12-03 22:03:22 +01:00
alesapin
fa4728d44f Fix style 2019-12-03 20:57:00 +03:00
alesapin
2a43daec5c Buildable code 2019-12-03 20:15:16 +03:00
alesapin
20e4e92c49 Fix some conflicts 2019-12-03 19:32:41 +03:00
Pavel Kovalenko
a9e2327ec0 AWS S3 SDK integration. 2019-12-03 19:23:24 +03:00
Alexander Burmak
21623c3308 Merge remote-tracking branch 'refs/remotes/upstream/master' into vfs 2019-12-03 18:16:19 +03:00
alexey-milovidov
f09c29a2b6
Merge pull request #7992 from ClickHouse/excitoon-favorites-s3auth 
Merging #7623
 2019-12-03 15:21:12 +03:00
alexey-milovidov
87adf355e8
Merge pull request #7935 from filimonov/kafka_perf 
[wip] attempt to improve kafka parsing performance
 2019-12-03 03:52:40 +03:00
alexey-milovidov
b6edda194f
Update KafkaBlockInputStream.cpp 2019-12-03 03:52:02 +03:00
Alexey Milovidov
d70b73f033 Merge branch 's3auth' of https://github.com/excitoon-favorites/ClickHouse into excitoon-favorites-s3auth 2019-12-03 03:06:01 +03:00
alexey-milovidov
fe05565cec
Update StorageS3.cpp 2019-12-03 03:03:44 +03:00
Alexey Milovidov
fc174dce6e More simple 2019-12-03 02:50:53 +03:00
Alexey Milovidov
9d099bde8e Merge branch 'master' into fix_check_table_with_empty_pk 2019-12-03 02:36:21 +03:00
alexey-milovidov
e44dc0a6c8
Merge pull request #7981 from ClickHouse/fix_number_of_streams_for_storages_with_globs 
Limit number of streams for read from StorageFile and StorageHDFS
 2019-12-03 02:23:10 +03:00
chertus
a179c5025f Merge branch 'master' into joins 2019-12-02 21:08:39 +03:00
chertus
5ce65546e9 semi & anti join 2019-12-02 21:08:04 +03:00
Vitaly Baranov
b2685a3710
Merge pull request #7727 from kekekekule/master 
Added lifetime to system.dictionaries #6820
 2019-12-02 15:15:44 +03:00
alesapin
08dfd1d1c4 Limit number of streams for read from StorageFile and StorageHDFS 2019-12-02 14:25:41 +03:00
alesapin
a4ca30b5fd Fix bug in check table for tables without primary key 2019-12-02 13:30:04 +03:00
Vladimir Chebotarev
8dbfcdb3c3 Fixed failing tests. 2019-12-01 20:41:35 +03:00
Vladimir Chebotarev
ef5c4fb5a4 Remake of result_part_path. 2019-12-01 20:41:35 +03:00
Vladimir Chebotarev
b23d411f84 Filling result_part_path column of system.merges during the merge. 2019-12-01 20:41:35 +03:00
Vladimir Chebotarev
41c965e4ba Attempt to add is_move, source_part_paths, result_part_path columns to system.merges. 2019-12-01 20:41:35 +03:00
Vladimir Chebotarev
94684c07bf Added syntax for access_key_id and secret_access_key in S3 table function and storage. 2019-12-01 10:56:11 +03:00
Артем Стрельцов
25aa2114b2 Deleted redundant empty lines 2019-12-01 03:49:11 +03:00
Alexander Burmak
e5ffdc0224 Merge remote-tracking branch 'refs/remotes/upstream/master' into vfs 2019-11-29 18:27:56 +03:00
Nikolay Degterinsky
6b39253b71 Fixed naming 2019-11-29 17:15:12 +03:00
Nikolay Degterinsky
59002ddb91 Merge branch 'master' of https://github.com/kekekekule/ClickHouse into dict_lifetime 2019-11-29 17:06:38 +03:00
Mikhail Filimonov
ad136ec62d get rid of cloneWithColumns in internal loop and some other trash 2019-11-29 11:46:25 +01:00
Nikolai Kochetov
7a32ca057d
Merge pull request #7670 from excitoon-favorites/move_pool 
Separated pool for background moves
 2019-11-29 11:08:51 +03:00
hcz
dd7051bd98 Merge remote-tracking branch 'origin/master' into patch-2 2019-11-29 15:07:53 +08:00
Mikhail Filimonov
4cc0d0ca5d make virtual_columns local 2019-11-29 07:21:31 +01:00
Mikhail Filimonov
07722dfa93 more progress - cleanup internal cycles 2019-11-29 00:06:03 +01:00
alexey-milovidov
2e34f02a36
Merge pull request #7936 from ClickHouse/branch-for-slightly-improvement-of-usability-of-exception-messages 
The rest of PR #7897 about improvement of exception message
 2019-11-28 22:07:29 +03:00
Mikhail Filimonov
6217cb51f6 Get rid of InputStream layer for Kafka, use InputStream directly. Reopening port during reset. 2019-11-28 17:43:06 +01:00
stavrolia
295633b7ef Another try 2019-11-28 17:31:35 +03:00
Anton Popov
9fe987c8e1
Merge pull request #7920 from excitoon-favorites/datetimettl 
Removed check for using `Date` or `DateTime` column from TTL expressions
 2019-11-27 19:55:57 +03:00
stavrolia
f23273fa80 Fix 2019-11-27 17:29:00 +03:00
Nikolai Kochetov
bd40c9cd13
Merge pull request #7933 from azat/dist-mon-codec-errors-mark-broken 
DirectoryMonitor: mark batch as broken on UNKNOWN_CODEC/CANNOT_DECOMPRESS errors
 2019-11-27 16:03:44 +03:00
Alexander Burmak
3e5ef56644 Filesystem abstraction layer 2019-11-27 14:33:27 +03:00
Vladimir Chebotarev
ecaa98e4e4 Allowed to use constant TTL expressions, updated tests. 2019-11-27 13:11:17 +03:00
Vladimir Chebotarev
c396af7c40 Removed check for using Date or DateTime column from TTL expressions. 2019-11-27 13:11:17 +03:00
stavrolia
f1e5e58480 Some rest of PR about improvement of exception message 2019-11-27 04:18:26 +03:00
Mikhail Filimonov
deec48a3e0 attempt to fix kafka parsing performance regression https://github.com/ClickHouse/ClickHouse/issues/7261 2019-11-27 00:46:19 +01:00
Azat Khuzhin
362505b129 DirectoryMonitor: mark batch as broken on UNKNOWN_CODEC/CANNOT_DECOMPRESS errors 
There is no point in retrying batch after those errors (for the same
reasons as CHECKSUM_DOESNT_MATCH is there).
 2019-11-26 20:21:05 +03:00
Vladimir Chebotarev
37bc15779d Better logging of BackgroundMovePool. 2019-11-26 16:09:57 +03:00
Vladimir Chebotarev
37bebd1370 Improved BackgroundProcessingPool to use with background moves task. 2019-11-26 16:09:57 +03:00
Vladimir Chebotarev
e96c24786b Separated pool for background moves. 2019-11-26 16:08:59 +03:00
Olga Khvostikova
47e499d71b
Merge pull request #7833 from excitoon-favorites/detachedparts 
Added disk info to `system.detached_parts`
 2019-11-26 11:56:50 +03:00
Nikolai Kochetov
c97212d132 Remove unused variable. 2019-11-25 16:35:28 +03:00
Nikolai Kochetov
d3118f0a76 Support SAMPLE and FINAL together in MergeTree. 2019-11-25 15:57:14 +03:00
Nikolai Kochetov
9b42e9341d Support SAMPLE and FINAL together in MergeTree. 2019-11-25 15:49:05 +03:00
Nikolai Kochetov
f770934e60 Support SAMPLE and FINAL together in MergeTree. 2019-11-25 15:05:29 +03:00
Nikita Mikhaylov
728ede25f9
Merge pull request #7840 from apbodrov/read_write_gzip 
Gzip read/write to file/s3/url/hdfs
 2019-11-22 11:48:58 +03:00
tavplubix
b75b6c294d
Merge pull request #7762 from ClickHouse/possible_segfaults_in_table_functions 
Fix Storage lifetime in table functions
 2019-11-22 02:29:06 +03:00
Nikita Mikhaylov
762d295ab0
Merge pull request #7780 from ClickHouse/aku/parallel-parsing 
Merging parallel format parsing
 2019-11-22 01:54:47 +03:00
Vladimir Chebotarev
2b4f6af736 Fixed errors with space reservation introduced in #7558 and #7602. 2019-11-21 12:09:43 +03:00
Andrei Bodrov
decdd7d476 fix typos 2019-11-20 18:40:42 +03:00
Andrei Bodrov
c2849d0864 move to helpers 2019-11-20 17:48:01 +03:00
Alexander Kuzmenkov
7e5731d64e Merge remote-tracking branch 'origin/master' into HEAD 2019-11-19 16:15:47 +03:00
Andrei Bodrov
8cbc52e711 add gzip read/write to file/s3/url/hdfs 2019-11-19 15:46:07 +03:00
Vladimir Chebotarev
5cbbcd9cdb Added disk info to system.detached_parts. 2019-11-19 09:44:10 +03:00
Alexander Tokmakov
2173bf5f4a Merge branch 'master' into possible_segfaults_in_table_functions 2019-11-17 16:48:13 +03:00
Vitaly Baranov
e40c140982 Add the ignorable flag for the settings. 2019-11-17 03:56:18 +03:00
Vitaly Baranov
f3d72b9ec1 Skip unknown settings with warnings. 2019-11-17 03:51:58 +03:00
Vitaly Baranov
ff6cba192b Replace pointer with reference in SettingsCollection<T>::MemberInfos::findStrict(). 
Rename SettingsCommon.h -> SettingsCollection.h for consistency.
 2019-11-17 03:51:51 +03:00
Alexander Tokmakov
55f60a6bad hold shared pointers to table function Storage and Context from InterpreterSelectQuery 2019-11-15 21:42:32 +03:00
Alexander Kuzmenkov
723e02fef7
Merge pull request #7759 from CurtizJ/fix-order-by 
Fix order by in case of sorting by primary key prefix and non primary key suffix.
 2019-11-15 20:52:31 +03:00
CurtizJ
475356f2b7 more clarifying names to avoid mistakes 2019-11-15 17:03:42 +03:00
Артем Стрельцов
2a65b4818a Fixed bugs of last non-fictive commit 2019-11-15 11:33:27 +03:00
Артем Стрельцов
5c46a8a7a8 Empty commit to restart tests 2019-11-14 22:36:31 +03:00
Alexander Kuzmenkov
7f34c0089c Merge remote-tracking branch 'origin/master' into HEAD 2019-11-14 18:53:36 +03:00
chertus
8286bbf913 Merge branch 'master' into joins 2019-11-14 16:16:52 +03:00
Nikolai Kochetov
caad7e636e Use shared_ptr for storage in StorageFileBlockInputStream. 2019-11-13 15:28:14 +03:00
Nikolai Kochetov
943dd12a96 Use shared_ptr for storage in StorageFileBlockInputStream. 2019-11-13 15:17:31 +03:00
Артем Стрельцов
ec636e95cd Added lifetime to system.dictionaries 2019-11-12 00:34:28 +03:00
chertus
8321deefac Merge branch 'master' into joins 2019-11-11 20:54:45 +03:00
alesapin
c74538153f Fix use of destroyed variable captured in lambda 2019-11-11 11:40:00 +03:00
Mikhail Korotov
37a6ccfa49 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into url_engine_problem 2019-11-08 18:29:06 +03:00
robot-clickhouse
94c7719c73 Auto version update to [19.18.1.1] [54429] 2019-11-08 11:36:41 +03:00
millb
7a2dcf451a Merge branch 'master' into url_engine_problem 2019-11-08 02:19:09 +03:00
alexey-milovidov
7f01fd13b3
Merge pull request #7624 from azat/_shard_num 
[RFC] Add _shard_num virtual column for the Distributed engine
 2019-11-08 00:55:41 +03:00
alexey-milovidov
a2e79cc427
Merge pull request #7574 from ClickHouse/processors-4 
Fix tests for processors
 2019-11-08 00:46:30 +03:00
chertus
b8d464aa03 new consistent ANY JOIN (LEFT, INNER, RIGHT) 2019-11-08 00:32:44 +03:00
alesapin
18e59f2771 Remove some functions from mutations interpreter 2019-11-07 11:40:36 +03:00
Azat Khuzhin
0677ae048c Wrap _shard_num with toUInt32 for simpler JOIN with system.clusters.shard_num 2019-11-06 21:02:42 +03:00
Azat Khuzhin
92bd96fc0d Simplify buildWhereExpression() by using makeASTFunction() helper 2019-11-06 20:37:30 +03:00
alesapin
8c7deaa70d More safier, but still ugly mutations interpreter 2019-11-06 20:28:03 +03:00
Azat Khuzhin
81aeff2d2a Add _shard_num virtual column for the Distributed engine 
With JOIN from system.clusters one can figure out from which server data
came.

TODO:
- optimization to avoid communicating with unrelated shards (for queries
  like "AND _shard_num = n")
- fix aliases (see tests with serverError expected)

v0: AddingConstColumnBlockInputStream
v2: VirtualColumnUtils::rewriteEntityInAst
v3: fix remote(Distributed) by appending _shard_num only if has been requested
 2019-11-06 15:26:12 +03:00
Alexander Kuzmenkov
233212ac68
Merge pull request #7531 from amosbird/getkeymapped 
Unify hash tables interface (separated for review)
 2019-11-06 13:49:39 +03:00
alexey-milovidov
687be5f7af
Merge pull request #7618 from excitoon-favorites/looks-like-wrong-copy-paste 
Fixed exception in case of using 1 argument while defining S3, URL and HDFS storages
 2019-11-06 02:11:38 +03:00
alexey-milovidov
56c89349d3
Merge pull request #7639 from CurtizJ/final_mark_without_primary_key 
Fix final mark with empty primary key.
 2019-11-05 22:53:52 +03:00
Alexey Milovidov
9ac4e4c7fb Little better #7600 2019-11-05 22:31:07 +03:00
alexey-milovidov
1bfade54c2
Merge pull request #7600 from azat/DirectoryMonitor-current_batch.txt-corruption 
Write current batch for distributed send atomically (using .tmp + rename)
 2019-11-05 22:24:15 +03:00
CurtizJ
3d968b0259 fix final mark with empty primary key 2019-11-05 20:42:35 +03:00
Nikolai Kochetov
055f21c9b7 Less wrappers for procesors pipeline. 2019-11-05 20:33:03 +03:00
Amos Bird
fbfbe161ec
Unify hash tables interface. 2019-11-06 00:08:04 +08:00
millb
a03f8145bb Created check for HDFS and S3 storages 
:
 2019-11-05 15:40:49 +03:00
millb
d62cd6efa8 Merge branch 'master' into url_engine_problem 2019-11-05 14:42:53 +03:00
hcz
e0e5e812a7
Allow relative user_files_path 2019-11-05 17:48:49 +08:00
Nikita Mikhaylov
817b52314e Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel_parsing 2019-11-05 12:32:26 +03:00
Azat Khuzhin
4dfffdd24a Write current batch for distributed send atomically (using .tmp + rename) 
Otherwise the following can happen after reboot:

    2019.11.01 11:46:12.217143 [ 187 ] {} <Error> dist.Distributed.DirectoryMonitor: Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected \n before: S\'^A\0^]\0\0<BE>4^A\0r<87>\0\0<A2><D7>^D^Y\0<F2>{^E<CD>\0\0Hy\0\0<F2>^_^C\0^_&\0\0<FF><D3>\0\0
    <8D><91>\0\0<C0>9\0\0<C0><B0>^A\0^G<AA>\0\0<B5><FE>^A\0<BF><A7>^A\0<9B><CB>^A\0I^R^A\0<B7><AB>^A\0<BC><8F>\0\0˲^B\0Zy\0\0<94><AA>\0\0<98>
    <8F>\0\0\f<A5>\0\0^QN\0\0<E3><C6>\0\0<B1>6^B\0ɳ\0\0W<99>\0\0<B9><A2>\0\0:<BB>\0\0)<B1>\0\0#<8B>\0\0aW\0\0<ED>#\0\0<F1>@\0\0ˀ^B\0<D7><FC>\0\0<DF>, Stack trace:

    0. 0x559e27222e60 StackTrace::StackTrace() /usr/bin/clickhouse
    1. 0x559e27222c45 DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) /usr/bin/clickhouse
    2. 0x559e26de4473 ? /usr/bin/clickhouse
    3. 0x559e272494b5 DB::assertString(char const*, DB::ReadBuffer&) /usr/bin/clickhouse
    4. 0x559e2a5dab45 DB::StorageDistributedDirectoryMonitor::processFilesWithBatching(std::map<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) /usr/bin/clickhouse
    5. 0x559e2a5db5fa DB::StorageDistributedDirectoryMonitor::processFiles() /usr/bin/clickhouse
    6. 0x559e2a5dba78 DB::StorageDistributedDirectoryMonitor::run() /usr/bin/clickhouse
    7. 0x559e2a5ddbbc ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::StorageDistributedDirectoryMonitor::*)(), DB::StorageDistributedDirectoryMonitor*>(void (DB::StorageDistributedDirectoryMonitor::*&&)(), DB::StorageDistributedDirectoryMonitor*&&)::{lambda()#1}::operator()() const /usr/bin/clickhouse
    8. 0x559e2726b07c ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) /usr/bin/clickhouse
    9. 0x559e2bbc3640 ? /usr/bin/clickhouse
    10. 0x7fbd62b3cfb7 start_thread /lib/x86_64-linux-gnu/libpthread-2.29.so
    11. 0x7fbd62a692ef __clone /lib/x86_64-linux-gnu/libc-2.29.so
     (version 19.17.1.1)

v2: remove fsync, to avoid possible stalls (https://github.com/ClickHouse/ClickHouse/pull/7600#discussion_r342010874)
 2019-11-05 00:24:38 +03:00
alexey-milovidov
5aceca9e17
Merge pull request #7615 from amosbird/fix21 
Global context should be unique.
 2019-11-05 00:16:37 +03:00
Vladimir Chebotarev
9869ee141c Fixed exception in case of using 1 argument while defining S3, URL and HDFS storages. 2019-11-04 20:11:50 +03:00
Amos Bird
e2cfc3ecab
Global context should be unique. 2019-11-04 23:43:13 +08:00
alexey-milovidov
b261e6033a
Merge pull request #7602 from excitoon-favorites/cloneAndLoadDataPartOnSameDisk 
Made `MergeTreeData::cloneAndLoadDataPart` only work for the same disk.
 2019-11-04 14:42:30 +03:00
zhang2014
08894484ee fix cluster system is_local 2019-11-04 10:56:12 +08:00
Vladimir Chebotarev
e76bbfc5ad Made MergeTreeData::cloneAndLoadDataPart only work for the same disk. 2019-11-04 01:27:05 +03:00
alexey-milovidov
15c4a19bb1
Merge pull request #7585 from ernestp/macos_catalina_2019_11_01 
Fixed compile errors on macOS Catalina
 2019-11-02 15:46:17 +03:00
alexey-milovidov
6e4d63e394
Merge pull request #7575 from Alex-Burmak/escaping_fix 
Fixed missing file name escaping in FREEZE command
 2019-11-02 13:20:28 +03:00
alexey-milovidov
ad01f7b1ac
Merge pull request #7552 from azat/do-not-account-memory-for-Buffer-in-max_memory_usage 
[RFC] Do not account memory for Buffer engine in max_memory_usage limit
 2019-11-02 00:57:24 +03:00
alexey-milovidov
dda93e2275
Merge pull request #7561 from ClickHouse/achimbab-master 
Merge for #7363
 2019-11-02 00:57:13 +03:00
Ernest Poletaev
de1e5d0e53 Fixed compile errors on macOS Catalina 2019-11-01 23:33:43 +07:00
Nikolai Kochetov
239d92a94c Code cleanup. 2019-11-01 18:31:02 +03:00
alexey-milovidov
4081b0be90
Merge pull request #7497 from amosbird/countopt 
Early constant folding.
 2019-11-01 16:54:02 +03:00
Alexander Burmak
9b8fc9a899 Fixed missing file name escaping in FREEZE command 2019-11-01 16:17:08 +03:00
Ivan
21db9157f7
Don't initialize some constant blocks in a loop (#7475) 2019-11-01 14:34:29 +03:00
Nikolai Kochetov
388bbb950b Add flag to set that set is created. 2019-11-01 14:03:35 +03:00
Nikolai Kochetov
1f7a798c97 Add flag to set that set is created. 2019-11-01 13:58:29 +03:00
alesapin
6ef0a92535
Merge pull request #7558 from excitoon/patch-1 
Made mutation to choose the same disk in `ReplicatedMergeTree`.
 2019-11-01 13:52:45 +03:00
Nikolai Kochetov
d31cac33b2 Small fixes. 2019-11-01 12:41:20 +03:00
Nikolai Kochetov
5bb47e2484
Merge pull request #7181 from ClickHouse/processors-and-storages 
Processors and storages
 2019-11-01 12:04:49 +03:00
Azat Khuzhin
dbe45800d9 Do not account memory for Buffer engine in max_memory_usage limit 
Since background flush will not be accounted there, and it can be too
tricky to calclulate this limit, in case you have multiple materialized
views with Buffer engine.

v2: test adjustment
 2019-10-31 22:26:50 +03:00
Nikolai Kochetov
9a5a0a2a3b Fix types for set elements. 2019-10-31 19:14:06 +03:00
Vitaly Baranov
56011d4649
Merge pull request #7553 from vitlibar/add-constraints-columns-to-table-system-settings 
Add constraints columns to table system.settings
 2019-10-31 19:06:46 +03:00
Amos Bird
e04cdd4ebb
Early constant folding. 2019-10-31 23:54:32 +08:00
Vladimir Chebotarev
faf1b070ac
Typo was fixed. 2019-10-31 16:26:11 +03:00
Vladimir Chebotarev
0dea6f5458
Added comment for selecting disk for mutation. 2019-10-31 16:19:38 +03:00
Nikolai Kochetov
be1ccaa890 Fix clang build. 2019-10-31 16:18:21 +03:00
Vladimir Chebotarev
f44cee623a
Made mutation to choose the same disk in ReplicatedMergeTree. 2019-10-31 15:22:25 +03:00
Nikolai Kochetov
18378417df Fix build. 2019-10-31 14:48:35 +03:00
Nikolai Kochetov
a80338e066 Merged with master 2019-10-31 14:41:29 +03:00
Nikolai Kochetov
a38124c68e Review fixes. 2019-10-31 14:32:24 +03:00
Vitaly Baranov
5601d6ae66 Add constraints columns to table system.settings 2019-10-31 01:45:57 +03:00
alexey-milovidov
88d43488e1
Merge pull request #7412 from 4ertus2/strange 
merge_max_block_size in settings
 2019-10-31 00:24:43 +03:00
alexey-milovidov
bb75c7469a
Merge pull request #7545 from ClickHouse/fix_only_tuple_columns 
Fix only tuple columns in table.
 2019-10-30 23:34:23 +03:00
alesapin
e2424bda4c Fix style 2019-10-30 19:00:40 +03:00
alesapin
2b5ef8bdef Fix bug when table contains only tuple columns 2019-10-30 17:50:59 +03:00
Amos Bird
2c75a51d4f
Instant count() for MergeTree 
Use (Replicated)MergeTree's metadata to do trivial count()
 2019-10-29 17:45:47 +08:00
Nikita Mikhaylov
756a10707d Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel_parsing 2019-10-28 13:51:52 +03:00
alesapin
04ca163518 Merge branch 'master' into dictionaries_ddl_loader 2019-10-25 10:58:04 +03:00
alesapin
5d8d1df802
Merge pull request #7434 from excitoon-favorites/fix-7424 
Allowed to have some parts on destination disk or volume in MOVE PARTITION
 2019-10-24 19:45:22 +03:00
Artem Zuikov
39b64dff87
Merge pull request #7454 from 4ertus2/refactoring 
Refactoring: extract non aliases logic out of QueryNormalizer
 2019-10-24 14:34:53 +03:00
Vladimir Chebotarev
64f158ff28 Fixed message for ALTER MOVE PART. 2019-10-24 13:56:32 +03:00
Alexander Kuzmenkov
29052b6a37
Merge pull request #7377 from azat/INSERT-Distributed-MATERIALIZED-cols 
* Fix INSERT into Distributed non local node with MATERIALIZED columns

Previous patch e527def18a ("Fix INSERT
into Distributed() table with MATERIALIZED column") fixes it only for
cases when the node is local, i.e. direct insert.

This patch address the problem when the node is not local
(`is_local == false`), by erasing materialized columns on INSERT into
Distributed.

And this patch fixes two cases, depends on `insert_distributed_sync`
setting:

- `insert_distributed_sync=0`

    ```
    Not found column value in block. There are only columns: date. Stack trace:

    2. 0x7ffff7be92e0 DB::Exception::Exception() dbms/src/Common/Exception.h:27
    3. 0x7fffec5d6cf6 DB::Block::getByName(...) dbms/src/Core/Block.cpp:187
    4. 0x7fffec2fe067 DB::NativeBlockInputStream::readImpl() dbms/src/DataStreams/NativeBlockInputStream.cpp:159
    5. 0x7fffec2d223f DB::IBlockInputStream::read() dbms/src/DataStreams/IBlockInputStream.cpp:61
    6. 0x7ffff7c6d40d DB::TCPHandler::receiveData() dbms/programs/server/TCPHandler.cpp:971
    7. 0x7ffff7c6cc1d DB::TCPHandler::receivePacket() dbms/programs/server/TCPHandler.cpp:855
    8. 0x7ffff7c6a1ef DB::TCPHandler::readDataNext(unsigned long const&, int const&) dbms/programs/server/TCPHandler.cpp:406
    9. 0x7ffff7c6a41b DB::TCPHandler::readData(DB::Settings const&) dbms/programs/server/TCPHandler.cpp:437
    10. 0x7ffff7c6a5d9 DB::TCPHandler::processInsertQuery(DB::Settings const&) dbms/programs/server/TCPHandler.cpp:464
    11. 0x7ffff7c687b5 DB::TCPHandler::runImpl() dbms/programs/server/TCPHandler.cpp:257
    ```

- `insert_distributed_sync=1`

    ```
    2019.10.18 13:23:22.114578 [ 44 ] {a78f669f-0b08-4337-abf8-d31e958f6d12} <Error> executeQuery: Code: 171, e.displayText() = DB::Exception: Block structure mismatch in RemoteBlockOutputStream stream: different number of columns:
    date Date UInt16(size = 1), value Date UInt16(size = 1)
    date Date UInt16(size = 0): Insertion status:
    Wrote 1 blocks and 0 rows on shard 0 replica 0, 127.0.0.1:59000 (average 0 ms per block)
    Wrote 0 blocks and 0 rows on shard 1 replica 0, 127.0.0.2:59000 (average 2 ms per block)
     (version 19.16.1.1) (from [::1]:3624) (in query: INSERT INTO distributed_00952 VALUES ), Stack trace:

    2. 0x7ffff7be92e0 DB::Exception::Exception() dbms/src/Common/Exception.h:27
    3. 0x7fffec5da4e9 DB::checkBlockStructure<void>(...)::{...}::operator()(...) const dbms/src/Core/Block.cpp:460
    4. 0x7fffec5da671 void DB::checkBlockStructure<void>(...) dbms/src/Core/Block.cpp:467
    5. 0x7fffec5d8d58 DB::assertBlocksHaveEqualStructure(...) dbms/src/Core/Block.cpp:515
    6. 0x7fffec326630 DB::RemoteBlockOutputStream::write(DB::Block const&) dbms/src/DataStreams/RemoteBlockOutputStream.cpp:68
    7. 0x7fffe98bd154 DB::DistributedBlockOutputStream::runWritingJob(DB::DistributedBlockOutputStream::JobReplica&, DB::Block const&)::{lambda()#1}::operator()() const dbms/src/Storages/Distributed/DistributedBlockOutputStream.cpp:280
    <snip>
    ````

Fixes: #7365
Fixes: #5429
Refs: #6891

* Cover INSERT into Distributed with MATERIALIZED columns and !is_local node

I guess that adding new cluster into server-test.xml is not required,
but it won't harm.

* Update DistributedBlockOutputStream.cpp
 2019-10-24 12:35:09 +03:00
Alexander Kuzmenkov
050de71ef4
Update DistributedBlockOutputStream.cpp 2019-10-24 12:33:45 +03:00
alesapin
a6199b7e69 Merge with master 2019-10-24 12:33:40 +03:00
Vladimir Chebotarev
255da8f5e0 Fixed style. 2019-10-24 12:11:06 +03:00
Vladimir Chebotarev
3debdc2119 Added integration tests for ALTER MOVE PARTITION and fixed minor things. 2019-10-24 11:52:33 +03:00
chertus
9818eada69 rename: merge_max_block_size 2019-10-24 02:18:21 +03:00
Azat Khuzhin
ab9d9f8997 Fix INSERT into Distributed non local node with MATERIALIZED columns 
Previous patch e527def18a ("Fix INSERT
into Distributed() table with MATERIALIZED column") fixes it only for
cases when the node is local, i.e. direct insert.

This patch address the problem when the node is not local
(`is_local == false`), by erasing materialized columns on INSERT into
Distributed.

And this patch fixes two cases, depends on `insert_distributed_sync`
setting:

- `insert_distributed_sync=0`

    ```
    Not found column value in block. There are only columns: date. Stack trace:

    2. 0x7ffff7be92e0 DB::Exception::Exception() dbms/src/Common/Exception.h:27
    3. 0x7fffec5d6cf6 DB::Block::getByName(...) dbms/src/Core/Block.cpp:187
    4. 0x7fffec2fe067 DB::NativeBlockInputStream::readImpl() dbms/src/DataStreams/NativeBlockInputStream.cpp:159
    5. 0x7fffec2d223f DB::IBlockInputStream::read() dbms/src/DataStreams/IBlockInputStream.cpp:61
    6. 0x7ffff7c6d40d DB::TCPHandler::receiveData() dbms/programs/server/TCPHandler.cpp:971
    7. 0x7ffff7c6cc1d DB::TCPHandler::receivePacket() dbms/programs/server/TCPHandler.cpp:855
    8. 0x7ffff7c6a1ef DB::TCPHandler::readDataNext(unsigned long const&, int const&) dbms/programs/server/TCPHandler.cpp:406
    9. 0x7ffff7c6a41b DB::TCPHandler::readData(DB::Settings const&) dbms/programs/server/TCPHandler.cpp:437
    10. 0x7ffff7c6a5d9 DB::TCPHandler::processInsertQuery(DB::Settings const&) dbms/programs/server/TCPHandler.cpp:464
    11. 0x7ffff7c687b5 DB::TCPHandler::runImpl() dbms/programs/server/TCPHandler.cpp:257
    ```

- `insert_distributed_sync=1`

    ```
    2019.10.18 13:23:22.114578 [ 44 ] {a78f669f-0b08-4337-abf8-d31e958f6d12} <Error> executeQuery: Code: 171, e.displayText() = DB::Exception: Block structure mismatch in RemoteBlockOutputStream stream: different number of columns:
    date Date UInt16(size = 1), value Date UInt16(size = 1)
    date Date UInt16(size = 0): Insertion status:
    Wrote 1 blocks and 0 rows on shard 0 replica 0, 127.0.0.1:59000 (average 0 ms per block)
    Wrote 0 blocks and 0 rows on shard 1 replica 0, 127.0.0.2:59000 (average 2 ms per block)
     (version 19.16.1.1) (from [::1]:3624) (in query: INSERT INTO distributed_00952 VALUES ), Stack trace:

    2. 0x7ffff7be92e0 DB::Exception::Exception() dbms/src/Common/Exception.h:27
    3. 0x7fffec5da4e9 DB::checkBlockStructure<void>(...)::{...}::operator()(...) const dbms/src/Core/Block.cpp:460
    4. 0x7fffec5da671 void DB::checkBlockStructure<void>(...) dbms/src/Core/Block.cpp:467
    5. 0x7fffec5d8d58 DB::assertBlocksHaveEqualStructure(...) dbms/src/Core/Block.cpp:515
    6. 0x7fffec326630 DB::RemoteBlockOutputStream::write(DB::Block const&) dbms/src/DataStreams/RemoteBlockOutputStream.cpp:68
    7. 0x7fffe98bd154 DB::DistributedBlockOutputStream::runWritingJob(DB::DistributedBlockOutputStream::JobReplica&, DB::Block const&)::{lambda()#1}::operator()() const dbms/src/Storages/Distributed/DistributedBlockOutputStream.cpp:280
    <snip>
    ````

Fixes: #7365
Fixes: #5429
Refs: #6891
 2019-10-23 21:54:27 +03:00
chertus
20093fa065 extract more logic out of QueryNormalizer 2019-10-23 16:59:03 +03:00
Amos Bird
295864e6e0
better scalar query 2019-10-23 21:37:54 +08:00
millb
eca81dc32e Fixed bugs 2019-10-23 14:58:35 +03:00
Colum
a413770a97 Fixed spelling error in error message 2019-10-22 10:02:51 -07:00
Vladimir Chebotarev
9446fd2c4d Allowed to have some parts on destination disk or volume in MOVE PARTITION. #7424 2019-10-22 17:45:01 +03:00
Nikita Mikhaylov
34d8ade4a0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel_parsing 2019-10-22 15:56:51 +03:00
Ivan
7ba4e36e73
Don't push to MVs when inserting into Kafka table (#7265) 
* Do not insert values into MV when inserting directly to Kafka
* Add method noPushingToViews() to IStorage interface
  To separate InterpreterInsertQuery and StorageKafka
 2019-10-22 13:31:28 +03:00
chertus
f024b007a2 default_merge_block_size setting 2019-10-21 20:57:26 +03:00
Nikolai Kochetov
2b334a4adb Review fixes. 2019-10-21 19:26:29 +03:00
Nikolai Kochetov
2893c35930 Merged with master. 2019-10-21 18:33:44 +03:00
Nikolai Kochetov
f7d2e1b758 Added Pipe class. Updated MergeTreeDataSelectExecutor. 2019-10-21 18:16:08 +03:00
millb
8b8ca9a141 Support redirect improvement 2019-10-21 17:45:57 +03:00
robot-clickhouse
7381489d5d Auto version update to [19.17.1.1] [54428] 2019-10-21 15:48:00 +03:00
millb
e6d0c2a66c Redirect support added 2019-10-21 12:42:28 +03:00
alesapin
26c87e46c9 Merge branch 'master' into dictionaries_ddl_loader 2019-10-21 11:46:49 +03:00
alesapin
35e66d6696 Merge with master 2019-10-21 11:46:11 +03:00
millb
f64f6658ce Merge branch 'master' into url_engine_problem 2019-10-21 11:44:31 +03:00
Aleksey Akulovich
a019e45093 fix typos in docs and comment (#7385) 2019-10-20 12:43:54 +08:00
alexey-milovidov
52bfa9927c
Merge pull request #7384 from ClickHouse/fix_ub_in_system_parts 
Fix UB in StoragesInfoStream
 2019-10-19 09:59:39 +03:00
Azat Khuzhin
260a9fba2b StorageDistributed: Drop extra inclusion of materializeBlock.h 2019-10-18 23:38:33 +03:00
Alexander Kuzmenkov
5e2cc37381
Distinguish Array and Tuple field types when creating from array of Fields. (#7179) 2019-10-18 18:57:05 +03:00
Alexander Tokmakov
06b28f9fe3 fix uninitialized offset for Lazy db 2019-10-18 16:09:24 +03:00
vzakaznikov
66675edf2d Fixing data race with no_users_thread. (#7353) 2019-10-17 17:47:36 +03:00
vzakaznikov
ee1a2ce027 Fixing AddressSanitizer error in the LIVE VIEW getHeader() method. (#7271) 2019-10-17 17:43:11 +03:00
tavplubix
0046b9f137 Wait for jobs to finish on exception (fixes rare segfaults) (#7350) 2019-10-17 17:41:27 +03:00
alesapin
6dfe5c7142 Better introspection 2019-10-17 16:05:12 +03:00
Ivan
e595df05d9
Fix CREATE TABLE AS from table with virtual columns (#7183) 
* Disable undefined instrumentation for rdkafka
 2019-10-17 14:12:35 +03:00
philip.han
8aac4d04de Fixed weird type-casting for IN-operator in bloom_filter 2019-10-17 19:53:41 +09:00
Vladimir Chebotarev
512e97ddb3
Improved readability a little bit (MergeTreeData::getActiveContainingPart). 2019-10-16 22:01:50 +03:00
philip.han
a2bf848e12 Made bloom_filter type of index supporting LowCardinality and Nullable 2019-10-16 15:22:56 +09:00
alexey-milovidov
6a44336b86
Merge pull request #7304 from sundy-li/ttl_check 
Check TTL in StorageFactory
 2019-10-14 20:11:37 +03:00
sundy-li
b2109e4ece Check TTL in StorageFactory 2019-10-14 17:03:39 +08:00
alesapin
155731a676 Merge branch 'master' into dictionaries_ddl_interpreter 2019-10-14 11:49:18 +03:00
Nikolai Kochetov
757488369f Merge branch 'master' into processors-3 2019-10-12 19:24:02 +03:00
Alexander Kuzmenkov
31bd359100
Merge pull request #7231 from ClickHouse/aku/podarray-includes 
Include PODArray.h into fewer files.
 2019-10-12 15:08:41 +03:00
alesapin
653aa7c005 Fix unit tests and segfault 2019-10-11 18:41:52 +03:00
alesapin
967fe64793 Add simpliest tests 2019-10-11 18:22:30 +03:00
alesapin
ae42dc0d4c Attach detach and introspection queries for dictionaries 2019-10-11 16:21:52 +03:00
Alexander Kuzmenkov
8a1d57bddb Include PODArray.h into fewer files. 2019-10-11 14:28:16 +03:00
alesapin
3dfe961291 Merge branch 'master' into dictionaries_ddl_interpreter 2019-10-11 12:14:07 +03:00
alesapin
7159cde160 Remove accident change 2019-10-11 12:07:44 +03:00
Nikolai Kochetov
e48755d045 Merge branch 'master' into processors-and-storages 2019-10-11 00:46:37 +03:00
Vitaly Baranov
18ccb4d64d Move backQuote() and quoteString() to a separate file, use StringRefs. 2019-10-11 00:18:52 +03:00
alesapin
2a5713e6e4 Compilable code 2019-10-10 23:47:47 +03:00
alesapin
67fc3347c2 Preparing databases for dictionaries 2019-10-10 20:33:01 +03:00
Nikolai Kochetov
ef14df4632 Added more comments. 2019-10-10 17:16:15 +03:00