Commit Graph

22061 Commits

Author SHA1 Message Date
Nikolai Kochetov
a15d8f36d5
Merge pull request #7270 from azat/uniqCombined-memory-usage-tests-fixes
uniqCombined memory usage tests fixes
2019-10-10 23:53:55 +03:00
Nikolai Kochetov
0ce86e1f8f Disable memory tracker for exception stack. 2019-10-10 21:55:16 +03:00
Azat Khuzhin
e25bdc8d25 tests/00212_shard_aggregate_function_uniq: adjust values for uniqCombined(12)(String)
In e373862c83 the maxium size for HashTable in
CombinedCardinalityEstimator had been reduced for power of 2 for String
(since otherwise the size of the hashtable becames bigger then the
sizeof HLL).

Refs: https://github.com/ClickHouse/ClickHouse/pull/7236#issuecomment-540496270
Fixes: e373862c83 ("Do not use more then 98K of memory for uniqCombined*")
2019-10-10 21:47:51 +03:00
Azat Khuzhin
eb7d4797e3 tests/00212_shard_aggregate_function_uniq: break into sections 2019-10-10 21:47:51 +03:00
Azat Khuzhin
495d1de46f clickhouse-test: introduce -U/--unified option (pass to diff(1)) 2019-10-10 21:47:51 +03:00
alexey-milovidov
9c1c596146
Merge pull request #7252 from ClickHouse/fix-bad-code-in-transform-query-for-external-database
Fix bad code in transforming query for external database.
2019-10-10 21:10:36 +03:00
Nikolai Kochetov
f011c1020e Disable memory tracker for exception stack. 2019-10-10 20:37:34 +03:00
Nikolai Kochetov
742b948156 Disable memory tracker for exception stack. 2019-10-10 19:30:33 +03:00
Alexander Kazakov
03c7b7ff8d
Prevent from using non-deterministic functions in mutations of replicated tables (#7247)
* Added isDeterministic() to IFunctionBuilder i-face
* A test for non-deterministic mutations
* In MutationsInterpreter::validate() ensure deterministic functions
* Adjusted nondeterministic mutation tests
* Moved around some code in MutationsInterpreter::validate()
* Dropped unnecessary check in findFirstNonDeterministicFuncName()
* Corrections to IFunction{Base,Builder} interface
2019-10-10 17:38:08 +03:00
alesapin
51c8188903
Merge pull request #7253 from ClickHouse/dictionaries_ddl_parser
Refactor parser create query
2019-10-10 11:34:21 +03:00
alesapin
777ba2e47e Remove some code 2019-10-10 00:31:29 +03:00
Alexey Milovidov
3fbac60328 Added a test 2019-10-09 23:16:17 +03:00
Alexey Milovidov
e65b57369e transformQueryForExternalDatabase: fixed constant folding for Date and DateTime 2019-10-09 23:13:34 +03:00
alesapin
fae3f85a0f Refactor parser create query 2019-10-09 23:09:10 +03:00
Alexey Milovidov
b302fec362 Fixed bad code in transformQueryForExternalDatabase 2019-10-09 22:58:50 +03:00
alexey-milovidov
689411abe0
Merge pull request #7222 from azat/uniqCombined64
[RFC] Introduce uniqCombined64() to get sane results for cardinality > UINT_MAX
2019-10-09 20:29:56 +03:00
alexey-milovidov
6cd3e9a458
Update AggregateFunctionUniqCombined.h 2019-10-09 20:26:01 +03:00
alexey-milovidov
81a9c81ba7
Merge pull request #7236 from azat/uniqCombined-memory-usage
Do not use more then 98K of memory for uniqCombined*
2019-10-09 20:19:30 +03:00
alexey-milovidov
e3861e0e88
Merge pull request #7242 from azat/sparse_hashed-name
Return SparseHashed name (system.dictionaries:type) for the sparse_hashed layout
2019-10-09 20:12:24 +03:00
alexey-milovidov
01e08a23c2
Merge pull request #7240 from ClickHouse/function-get-macro
Added function `getMacro`
2019-10-09 15:28:54 +03:00
Alexander Kuzmenkov
1965c612ab Fix sorting in clickhouse-test. 2019-10-09 14:56:51 +03:00
Alexander Kuzmenkov
89ebd4885f Fix test filtering in clickhouse-test. 2019-10-09 13:51:59 +03:00
alesapin
ef0b2f5936
Merge pull request #7209 from ClickHouse/dictionaries_ddl_parser
Dictionaries ddl parser
2019-10-09 12:51:08 +03:00
Azat Khuzhin
0ff823b574 Return SparseHashed name (system.dictionaries:type) for the sparse_hashed layout
Due to tons of rebasing this bit had been forgotten.

Refs: 420089c301 ("Add new dictionary layout (sparse_hashed) that is more memory efficient")
2019-10-09 11:11:41 +03:00
alexey-milovidov
9b2e025918
Merge pull request #7234 from ClickHouse/aku/null-field-serialization
Serialize Null Fields correctly in DataTypeNullable.
2019-10-09 04:30:37 +03:00
Alexey Milovidov
6d2a79e832 Added a test 2019-10-09 04:18:12 +03:00
Alexey Milovidov
552e4b4be2 Added example config with macros for tests 2019-10-09 04:16:34 +03:00
Alexey Milovidov
a19f6513ce Added example config with macros 2019-10-09 04:15:23 +03:00
Alexey Milovidov
9f8d562543 Add function "getMacro" #7239 2019-10-09 04:14:57 +03:00
Azat Khuzhin
e373862c83 Do not use more then 98K of memory for uniqCombined*
uniqCombined() uses hashtable for medium cardinality, and since
HashTable resize by the power of 2 (well actually HashTableGrower grows
double by the power of 2, hence HashTableGrower::increaseSize() should
be overwritten to change this), with 1<<13 (default for uniqCombined)
and UInt64 HashValueType, the HashTable will takes:

  getBufferSizeInBytes() == 131072

While it should be not greater then sizeof(HLL) ~= 98K, so reduce the
maximum cardinality for hashtable to 1<<12 with UInt64 HashValueType and
to 1<13 with UInt32, overwrite HashTableGrower::increaseSize() and cover
this using max_memory_usage.

Refs: https://github.com/ClickHouse/ClickHouse/pull/7221#issuecomment-539672742

v2: cover uniqCombined() with non-default K
2019-10-09 02:39:23 +03:00
alexey-milovidov
4c1f0177b6
Merge pull request #7109 from infinivision/aggBitmapAnd
added groupBitmapAnd, groupBitmapOr, groupBitmapXor
2019-10-08 22:51:46 +03:00
alexey-milovidov
5c2d478543
Merge pull request #7212 from ClickHouse/aku/field-includes
Include Field.h and FieldVisitor.h into fewer files.
2019-10-08 22:40:30 +03:00
alesapin
5296be18f3 Fix desc table query 2019-10-08 22:39:20 +03:00
alexey-milovidov
b9aa5d812d
Merge pull request #7171 from nikvas0/nikvas0/lazy_db
Lazy Database
2019-10-08 22:15:55 +03:00
Azat Khuzhin
f46c5a47c0 Introduce uniqCombined64() to get sane results for cardinality > UINT_MAX
By default uniqCombined() uses 32-bit hash for all types except String,
so this means that you cannot use uniqCombined() with i.e UInt64 and
cardinality > UINT_MAX, although you can use uniqCombined(toString())
and this will lead to 64-bit hash, but will not have good performance.

So uniqCombined64() had been introduced to fix this.

Requires: #7213
2019-10-08 21:59:35 +03:00
alexey-milovidov
ddc2299901
Update 01016_null_part_minmax.sql 2019-10-08 21:52:48 +03:00
alexey-milovidov
d8b3db65f7
Merge pull request #7213 from azat/fix-uniqCombined-approximation
Fix uniqCombined() result for >UINT_MAX values (return UInt64 to avoid overflow)
2019-10-08 21:45:23 +03:00
alexey-milovidov
e619304b2c
Merge pull request #7216 from ClickHouse/nicelulu-issue-6615
Merging "Optimize empty IN subquery and empty INNER/RIGHT JOIN"
2019-10-08 21:36:57 +03:00
Alexander Kuzmenkov
a5b1496fe9 Serialize Null Fields correctly in DataTypeNullable. 2019-10-08 21:21:24 +03:00
alesapin
5ea1a12c05 Refactoring and comments 2019-10-08 16:26:15 +03:00
Alexander Kuzmenkov
2b30f2696c Include Field.h in less files. 2019-10-08 16:23:24 +03:00
alesapin
7b545f9b18 Fix style 2019-10-08 15:35:17 +03:00
alesapin
cd05564b17 Add show create dictionary 2019-10-08 15:34:04 +03:00
Alexander Kuzmenkov
46f5115601 Accept many test name regexes in clickhouse-test.
Sometimes it is more convenient to supply space-separated test names
without having to concatenate them into one regex with pipes.
2019-10-08 14:32:03 +03:00
alesapin
0883b3f8a9 Parser for drop dictionary query 2019-10-08 14:10:29 +03:00
alesapin
db20681207 Better declaration and lifetime parser 2019-10-08 13:50:57 +03:00
alesapin
5f81f47637 Add recursive parsing of pairs 2019-10-08 12:47:17 +03:00
Vasilyev Nikita
21b2b20729 Merge branch 'nikvas0/lazy_db' of https://github.com/nikvas0/ClickHouse into nikvas0/lazy_db 2019-10-08 11:15:12 +03:00
Vasilyev Nikita
030aa1cf10 fix 2019-10-08 11:10:55 +03:00
alexey-milovidov
3bd59edc3c
Update StorageSystemTables.cpp 2019-10-08 03:31:50 +03:00
alexey-milovidov
c92fc70f66
Update StorageSystemTables.cpp 2019-10-08 03:29:41 +03:00
alexey-milovidov
704529801e
Update StorageSystemColumns.cpp 2019-10-08 03:27:57 +03:00
alexey-milovidov
20f5b27b8b
Update DatabaseOnDisk.h 2019-10-08 03:24:26 +03:00
Alexey Milovidov
15bba8b498 Updated test 2019-10-07 22:16:45 +03:00
alexey-milovidov
bcc4e3f0ba
Merge branch 'master' into issue-6615 2019-10-07 22:12:25 +03:00
Alexander Tokmakov
1c5ee018df update docs 2019-10-07 21:55:12 +03:00
Azat Khuzhin
92c88248a1 Fix uniqCombined() result for >UINT_MAX values (return UInt64 to avoid overflow)
uniqCombined() return type is UInt64, but uniqCombined() uses
CombinedCardinalityEstimator, and CombinedCardinalityEstimator::size()
return type is UInt32, while the underlying HyperLogLog::size() is
UInt64.

So after this patch uniqCombined() can be used for >UINT_MAX values, the
outcome is not ideal (#2073) but at least sane.
2019-10-07 21:38:58 +03:00
Alexander Kuzmenkov
56b065881a Include FieldVisitors.h in less files. 2019-10-07 20:23:05 +03:00
Alexander Kuzmenkov
367c537106 Do not include Field.h everywhere through SettingsCommon.h.
Move the methods of SettingsCollection<> that are dependent on Field to
a separate file, and include it once for each instantiation. This allows
to work on Field without always recompiling the entire project.
2019-10-07 20:23:05 +03:00
Alexander Tokmakov
73aa1d1518 diagnostic info for bad csv delimiters 2019-10-07 19:47:25 +03:00
alesapin
0e664807e9 Additional test 2019-10-07 19:46:35 +03:00
Alexander Tokmakov
a2c83ac82e add tests 2019-10-07 19:31:30 +03:00
alesapin
cec3af45e5 First working version of parser 2019-10-07 19:23:16 +03:00
Alexander Tokmakov
32b0b8272c setup csv delimiter in Template format 2019-10-07 19:03:01 +03:00
Alexander Tokmakov
b4d9dc2a90 query parameters in evaluateConstantExpression 2019-10-07 18:46:50 +03:00
alesapin
369a7d262c Merge branch 'master' into dictionaries_ddl_parser 2019-10-07 12:46:08 +03:00
alexey-milovidov
d096119529
Merge pull request #7156 from ClickHouse/fix-header-for-now
Fix header for function 'now'
2019-10-05 22:35:23 +03:00
alexey-milovidov
ce12fb6b3c
Update ExpressionAnalyzer.cpp 2019-10-05 22:34:57 +03:00
alexey-milovidov
40b22061f0
Update InterpreterSelectQuery.cpp 2019-10-05 22:34:25 +03:00
proller
594c535bd4 Build fixes (#7063)
* Build fixes

*            fix

* Fix

* fix

* Fix install(export..

* Freebsd fixes

* Freebsd fixes

* Fix warning

* fix

* More PRIVATE linking

* Fix review

* Timmy

* fix

* fix
2019-10-05 22:25:31 +03:00
alexey-milovidov
acbadfefbf
Merge pull request #7200 from azat-archive/unbundled-build-fix
Add missing linking with PocoXML for clickhouse_common_io
2019-10-05 21:54:24 +03:00
alexey-milovidov
bb8414bdeb
Merge pull request #7201 from vzakaznikov/enabling_again_live_view_tests
A quick fix to resolve crash in LIVE VIEW table and re-enabling all LIVE VIEW tests.
2019-10-05 20:47:34 +03:00
Vitaliy Zakaznikov
6b8e61737a * Fixing crash in LIVE VIEW
* Re-enabling all LIVE VIEW tests
2019-10-05 08:05:24 -04:00
Vasilyev Nikita
5fb84b55a6 fix nl 2019-10-05 13:21:47 +03:00
Vasilyev Nikita
b8647f49b5 fix 2019-10-05 13:21:04 +03:00
Alexander Tokmakov
a90d126b93 support defaults_for_omitted_fields in TSV 2019-10-04 22:44:48 +03:00
Alexander Tokmakov
d936cee836 support null_as_default in most text formats 2019-10-04 22:42:35 +03:00
Alexander Tokmakov
ec79e5aa5b refactor null_as_default 2019-10-04 20:19:49 +03:00
Azat Khuzhin
4b9d711f07 Add missing linking with PocoXML for clickhouse_common_io
Otherwise unbundled build fail with:
  FAILED: dbms/libclickhouse_common_iod.so
  ...
  ../dbms/src/IO/WriteBufferFromS3.cpp:143: error: undefined reference to 'Poco::XML::InputSource::InputSource(std::istream&)'
  ../dbms/src/IO/WriteBufferFromS3.cpp:144: error: undefined reference to 'Poco::XML::DOMParser::DOMParser(Poco::XML::NamePool*)'
  ../dbms/src/IO/WriteBufferFromS3.cpp:145: error: undefined reference to 'Poco::XML::DOMParser::parse(Poco::XML::InputSource*)'
  ../dbms/src/IO/WriteBufferFromS3.cpp:146: error: undefined reference to 'Poco::XML::Document::getElementsByTagName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
  ../dbms/src/IO/WriteBufferFromS3.cpp:144: error: undefined reference to 'Poco::XML::DOMParser::~DOMParser()'
  ../dbms/src/IO/WriteBufferFromS3.cpp:143: error: undefined reference to 'Poco::XML::InputSource::~InputSource()'
  ../dbms/src/IO/WriteBufferFromS3.cpp:144: error: undefined reference to 'Poco::XML::DOMParser::~DOMParser()'
  ../dbms/src/IO/WriteBufferFromS3.cpp:143: error: undefined reference to 'Poco::XML::InputSource::~InputSource()'
  collect2: error: ld returned 1 exit status
2019-10-04 19:48:10 +03:00
alexey-milovidov
9c2f3ba41a
Merge pull request #6984 from achimbab/pr_bf
Made bloom_filter type of index supporting arrays
2019-10-04 18:08:42 +03:00
tavplubix
d73db7f779
Merge pull request #6781 from ClickHouse/merging_values_with_expressions
Merging Values with expressions
2019-10-04 12:51:22 +03:00
Nikolai Kochetov
90472293fd Added test with function randConstant. 2019-10-04 12:22:07 +03:00
Guillaume Tassery
0f66da0e91 Resolve lock issue after review 2019-10-04 06:37:19 +02:00
alexey-milovidov
fadd003f5b
Merge pull request #7185 from ClickHouse/fix_bug_with_alter_attach_part
Fix bug with segmentation fault in ATTACH PART query
2019-10-04 01:18:42 +03:00
alesapin
4e83f65089 Fix bug with segmentation fault in ATTACH PART query 2019-10-03 21:07:47 +03:00
alesapin
cf094cd62b
Merge pull request #7155 from ClickHouse/dictionaries_ddl
Preparation of ExternalLoader for Dictionaries DDL
2019-10-03 20:12:19 +03:00
Nikolai Kochetov
76d1412953 Fix actions for distributed. 2019-10-03 18:47:42 +03:00
Vasilyev Nikita
d6490892a7 fix metadata time 2019-10-03 17:18:17 +03:00
Vasilyev Nikita
66506e26d0 fix 2019-10-03 16:40:53 +03:00
Vasilyev Nikita
6b1a1b5b5f fix 2019-10-03 15:49:22 +03:00
Nikolai Kochetov
e5c1df010f Remove allow_different_constant_values from InterpreterSelectQuery. 2019-10-03 15:02:30 +03:00
Nikolai Kochetov
a706cc47cf Remove setting allow_different_constant_values from ConvertingTransform and ConvertingBlockInputStream. 2019-10-03 14:59:45 +03:00
Nikolai Kochetov
3142955e6c InterpreterSelectQuery: reuse actions from getSampleBlock in readImpl. 2019-10-03 14:58:52 +03:00
Guillaume Tassery
4f26f3fcfa Fix style 2019-10-03 12:52:32 +02:00
Vasilyev Nikita
f0f5134759 fixed includes 2019-10-03 12:31:59 +03:00
Vasilyev Nikita
c27ce44d19 set -> list 2019-10-03 12:20:21 +03:00
Vasilyev Nikita
2f4e7bbe7f get rid of copy-paste 2019-10-03 11:35:58 +03:00
Vasilyev Nikita
7ac9fadc0d reduce copy-paste 2019-10-03 11:27:43 +03:00
Guillaume Tassery
f54c90ccce Merge branch 'master' of github.com:PerformanceVision/ClickHouse into move_partition 2019-10-03 10:24:39 +02:00
Vasilyev Nikita
080815833f refactor 2019-10-03 10:59:48 +03:00
Alexander Kuzmenkov
8e245d4e8c
Merge pull request #7135 from ClickHouse/aku/msan-field
Fix some MemorySanitizer warnings for Fields.
2019-10-03 10:33:44 +03:00
Ivan
05b17976d1
Commit to Kafka explicitly after the writing is finalized (#7175) 2019-10-03 00:17:19 +03:00
alexey-milovidov
255a57d02e
Merge pull request #7172 from excitoon/patch-1
Fixed time calculation in `MergeTreeData`.
2019-10-03 00:15:16 +03:00
Alexander Tokmakov
a58996817f allow query parameters in Values 2019-10-02 22:54:40 +03:00
Alexander Tokmakov
565c273eaa Merge branch 'master' into merging_values_with_expressions 2019-10-02 20:51:44 +03:00
Alexander Tokmakov
80e5976d10 fixes after review 2019-10-02 20:51:00 +03:00
Artem Zuikov
d23b7ae8ef
Merge pull request #7166 from 4ertus2/some
Refactoring: remove copy-paste
2019-10-02 17:28:38 +03:00
Alexander Kuzmenkov
4af8e76bdd Use proper types for Field.
When a Field is inserted into a column, its type should be the same as
the column data type to avoid implicit reinterpret_cast. Change a
couple of usages where these types were different.
2019-10-02 16:48:26 +03:00
Vasilyev Nikita
136b1665a3 test rename 2019-10-02 16:06:57 +03:00
Vasilyev Nikita
de3deb9e85 fix 2019-10-02 15:58:13 +03:00
chertus
3b0d870804 rename function 2019-10-02 15:06:34 +03:00
Vitaliy Zakaznikov
ffe8931cda Revert "Merge pull request #6770 from yandex/temporary-remove-live-view-tests"
This reverts commit 6cf5327269, reversing
changes made to 4155771106.
2019-10-02 06:52:40 -04:00
Vasilyev Nikita
ff405865f7 db on disk 2019-10-02 13:10:45 +03:00
Nikita Mikhaylov
13e2034084
fix possible dataraces in ReplicatedMergeTree 2019-10-02 12:59:44 +03:00
Vasilyev Nikita
54237e3f27 name, data, metadata -> functions 2019-10-02 11:58:28 +03:00
Vladimir Chebotarev
e79a15c39c
Fixed time calculation in MergeTreeData. 2019-10-02 10:46:53 +03:00
Vasilyev Nikita
dab68e796e removed unused consts 2019-10-02 09:47:12 +03:00
Vasilyev Nikita
f515e9465b fix 2019-10-02 09:45:27 +03:00
Vasilyev Nikita
657d200d42 better logs 2019-10-02 09:40:51 +03:00
Vasilyev Nikita
45ecff99a1 upd test 2019-10-02 09:31:23 +03:00
alexey-milovidov
96ca241fba
Merge pull request #7158 from azat-archive/MetricsTransmitter-use-of-uninitialized-value
Avoid use of uninitialized values in MetricsTransmitter
2019-10-02 01:01:00 +03:00
chertus
c46cc3ff1a refactoring: remove copy-paste 2019-10-01 21:51:33 +03:00
Nikita Mikhaylov
0e9cad2276
fix datarace in stress test in storagemergetree 2019-10-01 21:04:42 +03:00
Nikita Mikhaylov
fe5df2c1ed
Update ASTAlterQuery.cpp 2019-10-01 20:48:22 +03:00
Nikita Mikhaylov
f6985028db
Update ASTAlterQuery.cpp 2019-10-01 20:47:25 +03:00
alesapin
3fe7d84233 Add ASTs for dictionaries 2019-10-01 17:54:28 +03:00
Artem Zuikov
48ab9afce6
Merge pull request #7122 from 4ertus2/mjoin
Squash left-hand blocks in partial merge join (optimization)
2019-10-01 17:23:10 +03:00
Vasilyev Nikita
71a2002417 test with sleep 2019-10-01 16:57:58 +03:00
Vasilyev Nikita
c5087219c4 test 2019-10-01 16:53:55 +03:00
Vasilyev Nikita
c630d9457b opt system tables 2019-10-01 16:07:00 +03:00
chertus
038428bccf do not affect other SquashingBlockInputStream instances 2019-10-01 16:01:08 +03:00
Vasilyev Nikita
5793a02ef9 impr Lazy database 2019-10-01 15:44:17 +03:00
Nikolai Kochetov
9e74b8a6fe Disable check for constants in Converting stream while converting to header. 2019-10-01 13:58:39 +03:00
Vasilyev Nikita
69677d19e4 fix iterations 2019-10-01 13:48:19 +03:00
Alexander Tokmakov
605046761e fix build 2019-10-01 13:44:28 +03:00
Vasilyev Nikita
5ef295eeff fixes 2019-10-01 13:24:09 +03:00
Nikolai Kochetov
58a432be62 Updated functions today, yestarday, randConstant. 2019-10-01 13:15:20 +03:00
Nikolai Kochetov
b9b2ca342d Updated functions today, yestarday, randConstant. 2019-10-01 13:11:16 +03:00
Nikolai Kochetov
92ba6081a4 Updated functions today, yestarday, randConstant. 2019-10-01 13:10:07 +03:00
Nikolai Kochetov
0d94bdb6a1 Updated functions today, yestarday, randConstant. 2019-10-01 13:03:55 +03:00
alesapin
8429f46f3c Return getUpdateTime method to interface 2019-10-01 11:58:47 +03:00
Azat Khuzhin
740e4c5851
Avoid use of uninitialized values in MetricsTransmitter
Defer thread creation after values had been initialized correctly.
2019-10-01 10:16:39 +03:00
alesapin
709783a1bc Add missed header 2019-09-30 22:42:31 +03:00
Alexander Tokmakov
e267081389 better PeekableReadBuffer interface 2019-09-30 21:21:58 +03:00
Nikolai Kochetov
486deee1f1 Use new iface for now function. Use current time value from function builder. 2019-09-30 20:36:03 +03:00
Nikolai Kochetov
393bfdcd9f Use new iface for now function. Use current time value from function builder. 2019-09-30 20:31:36 +03:00
alesapin
0e6331e892 Add comments 2019-09-30 19:54:50 +03:00
alesapin
08681ac210 Preparation for AST configurations for external dictionaries 2019-09-30 19:12:08 +03:00
chertus
5df8f870c4 try remove some PVS errors 2019-09-30 18:45:53 +03:00
Alexander Tokmakov
979f235e38 Merge branch 'master' into merging_values_with_expressions 2019-09-30 18:06:58 +03:00