Commit Graph

9010 Commits

Author SHA1 Message Date
proller
fa2484ea90 Make gperftools (libtcmalloc) as submodule and update. (#1381)
* Update tcmalloc and make submodule

* wip

* wip

* wip

* wip

* clean

* fix

* fix

* fix

* Fix tests
2017-10-24 22:11:42 +03:00
alexey-milovidov
6d62a117a1 Merge pull request #1395 from proller/fix4
Fix stripping debug info from huge .o (-g0)
2017-10-24 22:04:21 +03:00
Alexey Zatelepin
6bb34a4494 Merge branch 'master' into arbitrary-partitioning
Conflicts:
	dbms/src/Storages/StorageFactory.cpp
2017-10-24 17:23:18 +03:00
Alexey Zatelepin
b6b5d1cf25 store rows count for part on disk to allow tables without fixed size columns [#CLICKHOUSE-3000] 2017-10-24 17:11:53 +03:00
proller
2d2a4afa3f Fix stripping debug info from huge .o (-g0) 2017-10-24 16:30:44 +03:00
proller
82134e44cb Split ComplexKeyCacheDictionary to faster compile (part2) 2017-10-24 00:26:42 +03:00
proller
86a587e482 Fix test (supress stderr) 2017-10-23 21:40:07 +03:00
alexey-milovidov
3c2155df0b Update StorageKafka.cpp 2017-10-23 20:47:33 +03:00
alexey-milovidov
139e2080ae Update StorageKafka.cpp 2017-10-23 20:47:33 +03:00
alexey-milovidov
5bfb4c213c Update JSONEachRowRowInputStream.cpp 2017-10-23 20:47:33 +03:00
Marek Vavruša
3926851343 StorageKafka: use config string instead of just server name 2017-10-23 20:47:33 +03:00
Marek Vavruša
a7c391fbf7 StorageKafka: use std::vector instead of stack array for error string buffers 2017-10-23 20:47:33 +03:00
Marek Vavruša
f4c1149606 Settings: renamed schema to format_schema 2017-10-23 20:47:33 +03:00
Marek Vavruša
6be0ffd943 DataStreams/JSONEachRow: read exact number of rows
When using a LIMIT to reduce the number of rows
read, JSONEachRow format would always read an
extra row as it triggered a scan to seek the
EOF or beginning of the next row. This is not
an issue with physical tables, but it is
for streaming tables as an extra row is
unintentionally consumed on each read.
2017-10-23 20:47:33 +03:00
Alexey Milovidov
c57a62611b Removed StorageTrivialBuffer because it is underdeveloped [#CLICKHOUSE-2]. 2017-10-23 20:38:56 +03:00
proller
c66228a746 Log query id in executeQuery; Better type mismatch error; change format in report tool (#1372)
* Log query id in executeQuery; Better type mismatch error; change format in report tool

* Better log query_id

* fix message

* Use c++11 thread_local instaed of gcc's __thread

* lock mutex before notifying waiting thread in sync insertion into distributed [#CLICKHOUSE-3379]

* Cmake: fix build without downloaded submodules (#1379)

* fix
2017-10-23 20:32:39 +03:00
Alexey Zatelepin
8ccebefae6 add tests for extended storage definition syntax [#CLICKHOUSE-3000] 2017-10-23 18:21:44 +03:00
alexey-milovidov
6555d08782 Merge pull request #1390 from proller/fix6
Fix: An Error SQL will make server down (#1388)
2017-10-23 18:10:42 +03:00
Alexey Milovidov
d77686685a Using PODArray instead of std::vector in ZooKeeper library [#CLICKHOUSE-2]. 2017-10-23 18:08:31 +03:00
alexey-milovidov
91aa0713e5 Merge pull request #1371 from vavrusa/fix-zookeeper-stack-smashing
ZooKeeper: fixed stack smashing with tryGet()
2017-10-23 17:40:12 +03:00
alexey-milovidov
4f86ec9d58 Update ZooKeeper.cpp 2017-10-23 17:39:29 +03:00
proller
d8038cf746 Fix: An Error SQL will make server down (#1388) 2017-10-23 17:37:46 +03:00
Nikolai Kochetov
adae8205a7 increased timeout for test_insertion_sync_fails_with_timeout 2017-10-23 16:56:42 +03:00
Nikolai Kochetov
5966574405 fix timeout lock in DistributedBlockOutputStream 2017-10-23 16:47:00 +03:00
Nikolai Kochetov
3317251e2f fix jobs flags in DistributedBlockOutputStream 2017-10-23 15:00:56 +03:00
alexey-milovidov
a7189820a1 Merge pull request #1375 from yandex/fix-hangup-in-sync-writing
lock mutex before notifying waiting thread in sync insertion
2017-10-22 12:45:54 +03:00
alexey-milovidov
c5b5f2b4f8 Merge pull request #1374 from proller/fix4
Use c++11 thread_local instead of gcc's __thread
2017-10-22 12:43:11 +03:00
Marek Vavruša
5f53df7dbe ParserCreateQuery: allow ATTACH TABLE x shorthand statement
Allow `ATTACH TABLE [db.]name` if the table was previously detached,
an the table structure can be read from disk. This makes reattaching
tables less cumbersome:

```
CREATE TABLE test.t (x UInt8) ENGINE = Null;
DETACH TABLE test.t;
ATTACH TABLE test.t;
```
2017-10-21 13:38:39 -07:00
Marek Vavruša
975a7ada42 StorageMaterializedView: allow CREATE MATERIALIZED VIEW x TO y
This allows creation of materialized views without inner tables,
using an existing table for materialized rows instead.
This is useful for cases when you want to detach the materializing
view, but keep the already materialized data readable, especially
when the inner table is replicated.
2017-10-21 13:34:59 -07:00
Alexey Zatelepin
dd42d53856 additional help messages for extended syntax [#CLICKHOUSE-3000] 2017-10-20 23:02:55 +03:00
Alexey Zatelepin
c61d4106e8 allow loading all MergeTree settings from a query [#CLICKHOUSE-3000] 2017-10-20 21:28:45 +03:00
Alexey Zatelepin
a7c85e0c89 save index_granularity to ATTACH query if it wasn't set explicitly [#CLICKHOUSE-3000] 2017-10-20 21:28:44 +03:00
Alexey Zatelepin
1f76900500 move CREATE query rewriting for views to Interpreter [#CLICKHOUSE-2] 2017-10-20 21:28:43 +03:00
Alexey Zatelepin
75c65c7b59 use extended storage definition syntax for MergeTree engines [#CLICKHOUSE-3000] 2017-10-20 21:28:42 +03:00
Alexey Zatelepin
6b9ead4bd8 parse PARTITION BY, ORDER BY, SAMPLE BY, SETTINGS clauses in a storage definition [#CLICKHOUSE-3000] 2017-10-20 21:28:42 +03:00
Alexey Zatelepin
48dea685d0 use more typesafe AST objects in ASTCreateQuery [#CLICKHOUSE-3000] 2017-10-20 21:28:41 +03:00
proller
8dd5c9dac2 Cmake: fix build without downloaded submodules (#1379) 2017-10-20 20:59:44 +03:00
Nikolai Kochetov
4262c00122 lock mutex before notifying waiting thread in sync insertion into distributed [#CLICKHOUSE-3379] 2017-10-19 22:52:45 +03:00
proller
443088ce31 Use c++11 thread_local instaed of gcc's __thread 2017-10-19 22:36:37 +03:00
Alexey Zatelepin
e51f95872f fix backward compatibility for new part format: [#CLICKHOUSE-3000]
* do not write out partition and minmax index files for old MergeTreeData format
* download the part from another replica if there are missing or extra files
2017-10-19 21:24:30 +03:00
Marek Vavruša
e2a8526990 ZooKeeper: fixed stack smashing with tryGet()
The tryGet() operation creates a 1MB buffer on stack. This may or
may not work depending on the default stack size for threads,
whether the stack protector is enabled or not, recursion depth,
and the actual value size.

This is probably going to slow down some ZK operations, but I don't
see how else this could work reliably with the existing API.
2017-10-18 15:13:42 -07:00
robot-metrika-test
8a2b7cf0b6 Auto version update to [54300] 2017-10-16 13:49:50 +03:00
Marek Vavruša
927c0f7b74 dbms: Partially revert previous change
Instead of trying to convert long/unsigned long to Poco types, just
don't use them.
2017-10-13 23:46:10 -07:00
Marek Vavruša
8a0571af24 dbms: Better fix for missing unsigned long to Poco::UInt64 alias 2017-10-13 21:39:14 -07:00
Marek Vavruša
70543200be dbms: Fix build with clang after Poco update
In the current version of Poco, unsigned long no longer aliases to
UInt64 with LP64. The size_t aliases to unsigned long with clang,
so all the uses of size_t instead of UInt64 when interacting with
Poco interfaces are gone. I replaced uses with UInt64 where it makes
sense, and added an overloaded function for readVarUInt() to support size_t.
2017-10-13 18:26:24 -07:00
Marek Vavruša
2ac5a743cc dbms: fix build with OS X built-in clang
The built-in clang doesn’t support value() for
`std::experimental::optional`. It however supports
dereference operator, which is basically the
same thing:

```
/clickhouse/dbms/src/DataStreams/NullableAdapterBlockInputStream.cpp:83:67: error: call to unavailable member function 'value':
                    res.insert({elem.column, elem.type, rename[i].value()});
                                                        ~~~~~~~~~~^~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/experimental/optional:547:17: note: candidate function has been explicitly made unavailable
    value_type& value()
                ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/experimental/optional:539:33: note: candidate function has been explicitly made unavailable
    constexpr value_type const& value() const
```
2017-10-13 18:12:29 -07:00
proller
4c7850fdc2 Optional kafka (#1355)
* Zookeeper include fix

* Update submodule contrib/poco

* Fix zookeeper submodule ho-to-make doc

* Update zookeeper submodule

* Update submodule zookeeper

* Fix compile with external zookeeper

* Optional kafka

* fix

* Fix

* fix
2017-10-14 03:53:01 +03:00
Alexey Milovidov
f42c6014ce Merge branch 'master' of github.com:yandex/ClickHouse 2017-10-14 02:06:11 +03:00
alexey-milovidov
9ccdb5420b Merge branch 'master' into upstream-add-kafka 2017-10-13 23:52:42 +03:00
proller
20329ce794 Zookeeper include fix (#1353) 2017-10-13 23:30:55 +03:00
alexey-milovidov
adedf47715 Update StorageKafka.cpp 2017-10-13 23:18:43 +03:00
proller
fd072b58bb zstd, lz4 as submodule (#1214)
* Contrib: use zstd, lz4 as submodule

* fix shared build

* re-test me.

* wip

* Update CHANGELOG_RU.md

* Update CHANGELOG.md

* Update CHANGELOG_RU.md

* Whitespaces [#CLICKHOUSE-2].

* Implemented TODO [#CLICKHOUSE-2].

* Whitespaces [#CLICKHOUSE-2].

* Remove wrong test

* Implemented TODO [#CLICKHOUSE-2].

* Moved chown to correct place [#CLICKHOUSE-2].

* Resolves #1273. Exception safe users update. [#CLICKHOUSE-3]

* Less noisy logging. [#CLICKHOSUE-2]

* Add metrics for RWLockFIFO. [#CLICKHOUSE-3246]

* Executable dictionaries: fail if program returns non zero exit code (#CLICKHOUSE-3171)

* Add better logging if OPTIMIZE cannot be executed. [#CLICKHOUSE-2]

* test me

* Fixing documentation fragment about "default" user. It is not mandatory anymore.

* append yurial/clickhouse-client to docs/interfaces

* Fixed FREEZE PARTITION: using only active data parts; acquire snapshot of parts [#CLICKHOUSE-3369].

* Removed tcp_ssl_port by default [#CLICKHOUSE-2].

* Update MergeTreeDataMerger.cpp

* Update ShellCommand.cpp

* Disable part sendings and fetches before ALTER. [#CLICKHOUSE-3343]

* Update ExecutableDictionarySource.cpp

* Update ExecutableDictionarySource.cpp

* Miscellaneous changes after merge [#CLICKHOUSE-2].

* Improve tests: allow redefine some values (clickhouse path, ports, ...)

* Received signal Segmentation fault (#1300) (#1302)

* Received signal Segmentation fault (#1300)

* Add test

* Tests: Use new possibly redefined values from env (in 2 tests)

* Proper fix for the issue: better exception message [#CLICKHOUSE-2].

* Split GatherUtils.cpp for faster compile (#1312)

* Split GatherUtils.cpp for faster compile

* remove GatherUtils.cpp

* Fix array writing (#1314)

* changed MergedBlockOutputStream [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* disabled checkNoMultidimensionalArrays [#CLICKHOUSE-3341]

* fix IMergedBlockOutputStream::writeDataImpl [#CLICKHOUSE-3341]

* fix IMergedBlockOutputStream::writeDataImpl [#CLICKHOUSE-3341]

* fix IMergedBlockOutputStream::writeDataImpl [#CLICKHOUSE-3341]

* fix IMergedBlockOutputStream::writeDataImpl [#CLICKHOUSE-3341]

* added test [#CLICKHOUSE-3341]

* fixed test [#CLICKHOUSE-3341]

* refactoring and comments [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* Update ColumnArray.h

* Update ActionBlocker.h

* Fix section tabulation

* Fixed infinite recursion in expression analyzer. [#CLICKHOUSE-3125]

* Update ActionBlocker.h

* Improvement [#CLICKHOUSE-2].

* Try fix strange terminate (#1329)

* Tests: External: rename --use_http  => --no_http and fix

* Try fix strange terminate

* Misc [#CLICKHOUSE-2].

*  Fix compile CallPointInPolygon on clang4 (Thanks to @vavrusa) (#1333)

* Fix compile with boost 1.65.1+ and clang 3.8 ( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222439 )

* Fix compile CallPointInPolygon on clang4 (Thanks to @vavrusa)

* Fix complex queries with GLOBAL IN and UNION ALL  (#CLICKHOUSE-3356) (#1339)

* TEST only: why initQueryAnalyzer ?

* Better tests

* missing file

* Missing file

* Add test

* Test fixes

* Fixed FREEZE PARTITION: using only active data parts; acquire snapshot of parts [#CLICKHOUSE-3369].

* Removed tcp_ssl_port by default [#CLICKHOUSE-2].

* Better tests

* comment

* clean

* REmove wrong code

* clean

* dbms: Added compression level for ZSTD. [#METR-26742]

* dbms: CompressionSettingsSelector. [#METR-21516]

* dbms: Minor fix. [#METR-21516]

* Fix SummingMergeTree argument checking logic.

This patch fixes the Nested Column Name checking logic, which allows nested
columns be explicitly specified in SummingMergeTree engine.
2017-10-13 21:52:23 +03:00
Marek Vavruša
41b0bea84c SummingSortedBlockInputStream: use a separate structure for MapDescription 2017-10-13 21:51:03 +03:00
Marek Vavruša
39a5ae97c9 SummingSortedBlockInputStream: added comments, destructor for state 2017-10-13 21:51:03 +03:00
alexey-milovidov
d561d8d701 Update SummingSortedBlockInputStream.cpp 2017-10-13 21:51:03 +03:00
alexey-milovidov
a2fa58a2e4 Update SummingSortedBlockInputStream.cpp 2017-10-13 21:51:03 +03:00
alexey-milovidov
fee883d7c5 Update SummingSortedBlockInputStream.cpp 2017-10-13 21:51:03 +03:00
alexey-milovidov
6405620dfb Update AggregateFunctionSumMap.h 2017-10-13 21:51:03 +03:00
alexey-milovidov
a5955d527d Update AggregateFunctionSumMap.h 2017-10-13 21:51:03 +03:00
alexey-milovidov
4124465d0b Update AggregateFunctionSumMap.h 2017-10-13 21:51:03 +03:00
alexey-milovidov
725379e23d Update AggregateFunctionSumMap.h 2017-10-13 21:51:03 +03:00
alexey-milovidov
9eca78273f Update AggregateFunctionSumMap.h 2017-10-13 21:51:03 +03:00
alexey-milovidov
1984782321 Update AggregateFunctionSumMap.h 2017-10-13 21:51:03 +03:00
Marek Vavruša
2edf5492d0 AggregateFunctionSumMap: support variardic arguments, specialized keys
The function is rewritten to avoid allocations on every insert with
Field deserialising each array. The key type is now specialized,
so it can be accessed directly. The value type is variant type,
but only individual values are deserialised (which is cheap, since they're PODs).
The function also support summing of multiple columns by the same key.

The SummingSortedBlockInputStream uses the function in case of
Nested structure with one numeric key and many values to sum.
2017-10-13 21:51:03 +03:00
Marek Vavruša
be2faa82ff DataStreams/test: test case for nested field with two arrays for summing 2017-10-13 21:51:03 +03:00
Marek Vavruša
c09a43a7b2 DataStreams/SummingSortedBlockInputStream: use builtin aggregated functions
This replaces custom summation function implementations with an implementation
using built-in aggregation functions (sum and sumMap). The goal is to be able to
use specialized variants of aggregation functions, and to have a single
efficient implementation.
2017-10-13 21:51:03 +03:00
Marek Vavruša
c9be3719bc DataStreams/tests: add SummingSortedBlockInputStream test variant
This is basically useful for testing SummingSortedBlockInputStream
against the AggregatingBlockInputStream, which are used in the
{Summing,Aggregating}MergeTree table engines respectively.
2017-10-13 21:51:03 +03:00
alexey-milovidov
3a1ce26764 Update Context.cpp 2017-10-13 21:46:32 +03:00
proller
c0e9813b90 Fix access for inserting into temporary table (#CLICKHOUSE-3360) 2017-10-13 21:46:32 +03:00
proller
49f8080e52 Small debud improvement 2017-10-13 21:44:26 +03:00
Amos Bird
f30841d5ab Fix SummingMergeTree argument checking logic.
This patch fixes the Nested Column Name checking logic, which allows nested
columns be explicitly specified in SummingMergeTree engine.
2017-10-13 15:42:32 +03:00
Alexey Milovidov
03dd341804 Merge branch 'master' of github.com:yandex/ClickHouse 2017-10-13 04:24:14 +03:00
Vladimir Chebotarev
c05dfc201f dbms: Minor fix. [#METR-21516] 2017-10-13 04:14:57 +03:00
Alexey Milovidov
0ec218a853 Merge branch 'master' of github.com:yandex/ClickHouse 2017-10-13 04:09:51 +03:00
Vladimir Chebotarev
1e0241e262 dbms: CompressionSettingsSelector. [#METR-21516] 2017-10-13 04:08:23 +03:00
Vladimir Chebotarev
4e313f3732 dbms: Added compression level for ZSTD. [#METR-26742] 2017-10-13 04:08:23 +03:00
proller
1e961cc69d Fix complex queries with GLOBAL IN and UNION ALL (#CLICKHOUSE-3356) (#1339)
* TEST only: why initQueryAnalyzer ?

* Better tests

* missing file

* Missing file

* Add test

* Test fixes

* Fixed FREEZE PARTITION: using only active data parts; acquire snapshot of parts [#CLICKHOUSE-3369].

* Removed tcp_ssl_port by default [#CLICKHOUSE-2].

* Better tests

* comment

* clean

* REmove wrong code

* clean
2017-10-13 00:28:24 +03:00
Vitaliy Lyudvichenko
7bc58340b0 Fixed infinite recursion in expression analyzer. [#CLICKHOUSE-3125] 2017-10-13 00:27:25 +03:00
proller
fd85fffa0d Fix compile CallPointInPolygon on clang4 (Thanks to @vavrusa) (#1333)
* Fix compile with boost 1.65.1+ and clang 3.8 ( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222439 )

* Fix compile CallPointInPolygon on clang4 (Thanks to @vavrusa)
2017-10-13 00:22:04 +03:00
Alexey Milovidov
d3d976a6fb Misc [#CLICKHOUSE-2]. 2017-10-12 23:45:55 +03:00
proller
b993667156 Try fix strange terminate (#1329)
* Tests: External: rename --use_http  => --no_http and fix

* Try fix strange terminate
2017-10-12 23:42:58 +03:00
Alexey Milovidov
3a6705af0f Improvement [#CLICKHOUSE-2]. 2017-10-12 23:34:01 +03:00
alexey-milovidov
b7c1076df2 Update ActionBlocker.h 2017-10-12 23:26:54 +03:00
alexey-milovidov
96b603a023 Update ActionBlocker.h 2017-10-12 23:26:54 +03:00
Vitaliy Lyudvichenko
62ea1133dd Disable part sendings and fetches before ALTER. [#CLICKHOUSE-3343] 2017-10-12 23:26:54 +03:00
KochetovNicolai
83925f1d5e Fix array writing (#1314)
* changed MergedBlockOutputStream [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* disabled checkNoMultidimensionalArrays [#CLICKHOUSE-3341]

* fix IMergedBlockOutputStream::writeDataImpl [#CLICKHOUSE-3341]

* fix IMergedBlockOutputStream::writeDataImpl [#CLICKHOUSE-3341]

* fix IMergedBlockOutputStream::writeDataImpl [#CLICKHOUSE-3341]

* fix IMergedBlockOutputStream::writeDataImpl [#CLICKHOUSE-3341]

* added test [#CLICKHOUSE-3341]

* fixed test [#CLICKHOUSE-3341]

* refactoring and comments [#CLICKHOUSE-3341]

* fix build [#CLICKHOUSE-3341]

* Update ColumnArray.h
2017-10-12 23:13:26 +03:00
proller
3de5e465d0 Split GatherUtils.cpp for faster compile (#1312)
* Split GatherUtils.cpp for faster compile

* remove GatherUtils.cpp
2017-10-12 22:58:39 +03:00
Alexey Milovidov
b93f9ba686 Merge branch 'master' of github.com:yandex/ClickHouse 2017-10-12 22:56:55 +03:00
Alexey Milovidov
e57e79a5ff Proper fix for the issue: better exception message [#CLICKHOUSE-2]. 2017-10-12 22:56:47 +03:00
proller
26c44d5993 Tests: Use new possibly redefined values from env (in 2 tests) 2017-10-12 22:54:56 +03:00
proller
f8548156a0 Improve tests: allow redefine some values (clickhouse path, ports, ...) 2017-10-12 22:48:39 +03:00
proller
cbc81a6a9f Received signal Segmentation fault (#1300) (#1302)
* Received signal Segmentation fault (#1300)

* Add test
2017-10-12 22:46:40 +03:00
Alexey Milovidov
54eb2bbb0a Miscellaneous changes after merge [#CLICKHOUSE-2]. 2017-10-12 22:40:42 +03:00
alexey-milovidov
c2647e0c60 Update ExecutableDictionarySource.cpp 2017-10-12 22:36:27 +03:00
alexey-milovidov
beb93bb9d6 Update ExecutableDictionarySource.cpp 2017-10-12 22:36:27 +03:00
alexey-milovidov
44e6ecfe78 Update ShellCommand.cpp 2017-10-12 22:36:27 +03:00
proller
4969038865 test me 2017-10-12 22:36:27 +03:00
proller
1505ad6bac Executable dictionaries: fail if program returns non zero exit code (#CLICKHOUSE-3171) 2017-10-12 22:36:27 +03:00
Vitaliy Lyudvichenko
18af75a4f1 Add metrics for RWLockFIFO. [#CLICKHOUSE-3246] 2017-10-12 22:32:31 +03:00
alexey-milovidov
c1285c7d6b Update MergeTreeDataMerger.cpp 2017-10-12 22:28:24 +03:00