This contains many fixes and corrections for the Kafka engine.
Most notably it now supports extended configuration similarly to GraphiteMergeTree.
Now it also allows specification of consumer count to parallelize consumption of
multiple partitions both in materialized views and in SELECT queries.
The offsets are now committed in the insertSuffix() method after all rows
are successfully read. If an exception is thrown during reading, affected consumer
unsubscribes from all assignments and rejoins the consumer group to rewind offsets.
This means that the consumer won't lose messages in case of write failures.
This addresses one of the remarks in the PR.
All format schemas are required to be in the <format_schema_path> directory.
This makes loading schema files less tedious, as the path can be omitted.
Cap'n Proto is a binary message format.
Like Protocol Buffers and Thrift (but unlike JSON or MessagePack), Cap'n Proto messages are strongly-typed and not self-describing. Due to this, it requires a schema setting to specify schema file and the root object. The schema is parsed on runtime and cached for each SQL statement.
By default sum(x) promotes the result type to
largest possible integral type to avoid
arithmetic overflow when summing values from
smaller data types.
This is generally desirable behaviour, but it doesn’t
work with summing merge tree, as the result is
expected to be of same type as the input.
Allow `ATTACH TABLE [db.]name` if the table was previously detached,
an the table structure can be read from disk. This makes reattaching
tables less cumbersome:
```
CREATE TABLE test.t (x UInt8) ENGINE = Null;
DETACH TABLE test.t;
ATTACH TABLE test.t;
```
This allows creation of materialized views without inner tables,
using an existing table for materialized rows instead.
This is useful for cases when you want to detach the materializing
view, but keep the already materialized data readable, especially
when the inner table is replicated.
* update presentations
* CLICKHOUSE-2936: redirect from clickhouse.yandex.ru and clickhouse.yandex.com
* update submodule
* lost files
* CLICKHOUSE-2981: prefer sphinx docs over original reference
* CLICKHOUSE-2981: docs styles more similar to main website + add flags to switch language links
* update presentations
* Less confusing directory structure (docs -> doc/reference/)
* Minify sphinx docs too
* Website release script: fail fast + pass docker hash on deploy
* Do not underline links in docs
* shorter
* cleanup docker images
* tune nginx config
* CLICKHOUSE-3043: get rid of habrastorage links
* Lost translation
* CLICKHOUSE-2936: temporary client-side redirect
* behaves weird in test
* put redirect back
* CLICKHOUSE-3047: copy docs txts to public too
* move to proper file
* remove old pages to avoid confusion
* Remove reference redirect warning for now
* Refresh README.md
* Yellow buttons in docs
* Use svg flags instead of unicode ones in docs
* fix test website instance
* Put flags to separate files
* wrong flag
* Copy Yandex.Metrica introduction from main page to docs
* Yet another home page structure change, couple new blocks (CLICKHOUSE-3045)
* Update Contacts section
* CLICKHOUSE-2849: more detailed legal information
* CLICKHOUSE-2978 preparation - split by files
* More changes in Contacts block
* Tune texts on index page
* update presentations
* One more benchmark
* Add usage sections to index page, adapted from slides
* Get the roadmap started, based on slides from last ClickHouse Meetup
* CLICKHOUSE-2977: some rendering tuning
* Get rid of excessive section in the end of getting started
* Make headers linkable
* CLICKHOUSE-2981: links to editing reference - https://github.com/yandex/ClickHouse/issues/849
* CLICKHOUSE-2981: fix mobile styles in docs
* Ban crawling of duplicating docs
* Open some external links in new tab
* Ban old docs too
* Lots of trivial fixes in english docs
* Lots of trivial fixes in russian docs
* Remove getting started copies in markdown
* Add Yandex.Webmaster
* Fix some sphinx warnings
* More warnings fixed in english docs
* More sphinx warnings fixed
* Add code-block:: text
* More code-block:: text
* These headers look not that well
* Better switch between documentation languages
* merge use_case.rst into ya_metrika_task.rst
* Edit the agg_functions.rst texts
* Add lost empty lines
* Lost blank lines
* Add new logo sizes
* update presentations
* Next step in migrating to new documentation
* Fix all warnings in en reference
* Fix all warnings in ru reference
* Re-arrange existing reference
* Move operation tips to main reference
* Fix typos noticed by milovidov@
* Get rid of zookeeper.md
* Looks like duplicate of tutorial.html
* Fix some mess with html tags in tutorial
* No idea why nobody noticed this before, but it was completely not clear whet to get the data
* Match code block styling between main and tutorial pages (in favor of the latter)
* Get rid of some copypaste in tutorial
* Normalize header styles
* Move example_datasets to sphinx
* Move presentations submodule to website
* Move and update README.md
* No point in duplicating articles from habrahabr here
* Move development-related docs as is for now
* doc/reference/ -> docs/ (to match the URL on website)
* Adapt links to match the previous commit
* Adapt development docs to rst (still lacks translation and strikethrough support)
* clean on release
* blacklist presentations in gulp
* strikethrough support in sphinx
* just copy development folder for now
* fix weird introduction in style article
* Style guide translation (WIP)
* Finish style guide translation to English
* gulp clean separately
* Update year in LICENSE
* Initial CONTRIBUTING.md
* Fix remaining links to old docs in tutorial
* Some tutorial fixes
* Typo
* Another typo
* Update list of authors from yandex-team accoding to git log
* Fix diff with master
* couple fixes in en what_is_clickhouse.rst
* Try different link to blog in Russian
* Swap words
* Slightly larger line height
* CLICKHOUSE-3089: disable hyphenation in docs
* update presentations
* Fix copying of txt files
* update submodule
* CLICKHOUSE-3108: fix overflow issues in mobile version
* Less weird tutorial header in mobile version
* CLICKHOUSE-3073: skip sourcemaps by default
* CLICKHOUSE-3067: rename item in docs navigation
* fix list markup
* CLICKHOUSE-3067: some documentation style tuning
* CLICKHOUSE-3067: less laggy single page documentation
* update presentations
* YQL-3278: add some links to ClickHouse Meetup in Berlin on October 5, 2017
* Add "time series" keyword
* Switch link to next event
* Switch link to next event #2
* smaller font
Added a new table engine backed by Apache Kafka,
it allows near real-time ingestion or ETL with
exactly-once semantics.
It doesn’t store any data on disk, so data must
be materialized into a physical table to allow
random access.
Currently it only supports reading, no insertion.
* update presentations
* CLICKHOUSE-2936: redirect from clickhouse.yandex.ru and clickhouse.yandex.com
* update submodule
* lost files
* CLICKHOUSE-2981: prefer sphinx docs over original reference
* CLICKHOUSE-2981: docs styles more similar to main website + add flags to switch language links
* update presentations
* Less confusing directory structure (docs -> doc/reference/)
* Minify sphinx docs too
* Website release script: fail fast + pass docker hash on deploy
* Do not underline links in docs
* shorter
* cleanup docker images
* tune nginx config
* CLICKHOUSE-3043: get rid of habrastorage links
* Lost translation
* CLICKHOUSE-2936: temporary client-side redirect
* behaves weird in test
* put redirect back
* CLICKHOUSE-3047: copy docs txts to public too
* move to proper file
* remove old pages to avoid confusion
* Remove reference redirect warning for now
* Refresh README.md
* Yellow buttons in docs
* Use svg flags instead of unicode ones in docs
* fix test website instance
* Put flags to separate files
* wrong flag
* Copy Yandex.Metrica introduction from main page to docs
* Yet another home page structure change, couple new blocks (CLICKHOUSE-3045)
* Update Contacts section
* CLICKHOUSE-2849: more detailed legal information
* CLICKHOUSE-2978 preparation - split by files
* More changes in Contacts block
* Tune texts on index page
* update presentations
* One more benchmark
* Add usage sections to index page, adapted from slides
* Get the roadmap started, based on slides from last ClickHouse Meetup
* CLICKHOUSE-2977: some rendering tuning
* Get rid of excessive section in the end of getting started
* Make headers linkable
* CLICKHOUSE-2981: links to editing reference - https://github.com/yandex/ClickHouse/issues/849
* CLICKHOUSE-2981: fix mobile styles in docs
* Ban crawling of duplicating docs
* Open some external links in new tab
* Ban old docs too
* Lots of trivial fixes in english docs
* Lots of trivial fixes in russian docs
* Remove getting started copies in markdown
* Add Yandex.Webmaster
* Fix some sphinx warnings
* More warnings fixed in english docs
* More sphinx warnings fixed
* Add code-block:: text
* More code-block:: text
* These headers look not that well
* Better switch between documentation languages
* merge use_case.rst into ya_metrika_task.rst
* Edit the agg_functions.rst texts
* Add lost empty lines
* Lost blank lines
* Add new logo sizes
* update presentations
* Next step in migrating to new documentation
* Fix all warnings in en reference
* Fix all warnings in ru reference
* Re-arrange existing reference
* Move operation tips to main reference
* Fix typos noticed by milovidov@
* Get rid of zookeeper.md
* Looks like duplicate of tutorial.html
* Fix some mess with html tags in tutorial
* No idea why nobody noticed this before, but it was completely not clear whet to get the data
* Match code block styling between main and tutorial pages (in favor of the latter)
* Get rid of some copypaste in tutorial
* Normalize header styles
* Move example_datasets to sphinx
* Move presentations submodule to website
* Move and update README.md
* No point in duplicating articles from habrahabr here
* Move development-related docs as is for now
* doc/reference/ -> docs/ (to match the URL on website)
* Adapt links to match the previous commit
* Adapt development docs to rst (still lacks translation and strikethrough support)
* clean on release
* blacklist presentations in gulp
* strikethrough support in sphinx
* just copy development folder for now
* fix weird introduction in style article
* Style guide translation (WIP)
* Finish style guide translation to English
* gulp clean separately
* Update year in LICENSE
* Initial CONTRIBUTING.md
* Fix remaining links to old docs in tutorial
* Some tutorial fixes
* Typo
* Another typo
* Update list of authors from yandex-team accoding to git log
* Fix diff with master
* couple fixes in en what_is_clickhouse.rst
* Try different link to blog in Russian
* Swap words
* Slightly larger line height
* CLICKHOUSE-3089: disable hyphenation in docs
* update presentations
* Fix copying of txt files
* update submodule
* CLICKHOUSE-3108: fix overflow issues in mobile version
* Less weird tutorial header in mobile version
* CLICKHOUSE-3073: skip sourcemaps by default
* CLICKHOUSE-3067: rename item in docs navigation
* fix list markup
* CLICKHOUSE-3067: some documentation style tuning
* CLICKHOUSE-3067: less laggy single page documentation
* update presentations
* YQL-3278: add some links to ClickHouse Meetup in Berlin on October 5, 2017
* update presentations
* CLICKHOUSE-2936: redirect from clickhouse.yandex.ru and clickhouse.yandex.com
* update submodule
* lost files
* CLICKHOUSE-2981: prefer sphinx docs over original reference
* CLICKHOUSE-2981: docs styles more similar to main website + add flags to switch language links
* update presentations
* Less confusing directory structure (docs -> doc/reference/)
* Minify sphinx docs too
* Website release script: fail fast + pass docker hash on deploy
* Do not underline links in docs
* shorter
* cleanup docker images
* tune nginx config
* CLICKHOUSE-3043: get rid of habrastorage links
* Lost translation
* CLICKHOUSE-2936: temporary client-side redirect
* behaves weird in test
* put redirect back
* CLICKHOUSE-3047: copy docs txts to public too
* move to proper file
* remove old pages to avoid confusion
* Remove reference redirect warning for now
* Refresh README.md
* Yellow buttons in docs
* Use svg flags instead of unicode ones in docs
* fix test website instance
* Put flags to separate files
* wrong flag
* Copy Yandex.Metrica introduction from main page to docs
* Yet another home page structure change, couple new blocks (CLICKHOUSE-3045)
* Update Contacts section
* CLICKHOUSE-2849: more detailed legal information
* CLICKHOUSE-2978 preparation - split by files
* More changes in Contacts block
* Tune texts on index page
* update presentations
* One more benchmark
* Add usage sections to index page, adapted from slides
* Get the roadmap started, based on slides from last ClickHouse Meetup
* CLICKHOUSE-2977: some rendering tuning
* Get rid of excessive section in the end of getting started
* Make headers linkable
* CLICKHOUSE-2981: links to editing reference - https://github.com/yandex/ClickHouse/issues/849
* CLICKHOUSE-2981: fix mobile styles in docs
* Ban crawling of duplicating docs
* Open some external links in new tab
* Ban old docs too
* Lots of trivial fixes in english docs
* Lots of trivial fixes in russian docs
* Remove getting started copies in markdown
* Add Yandex.Webmaster
* Fix some sphinx warnings
* More warnings fixed in english docs
* More sphinx warnings fixed
* Add code-block:: text
* More code-block:: text
* These headers look not that well
* Better switch between documentation languages
* merge use_case.rst into ya_metrika_task.rst
* Edit the agg_functions.rst texts
* Add lost empty lines
* Lost blank lines
* Add new logo sizes
* update presentations
* Next step in migrating to new documentation
* Fix all warnings in en reference
* Fix all warnings in ru reference
* Re-arrange existing reference
* Move operation tips to main reference
* Fix typos noticed by milovidov@
* Get rid of zookeeper.md
* Looks like duplicate of tutorial.html
* Fix some mess with html tags in tutorial
* No idea why nobody noticed this before, but it was completely not clear whet to get the data
* Match code block styling between main and tutorial pages (in favor of the latter)
* Get rid of some copypaste in tutorial
* Normalize header styles
* Move example_datasets to sphinx
* Move presentations submodule to website
* Move and update README.md
* No point in duplicating articles from habrahabr here
* Move development-related docs as is for now
* doc/reference/ -> docs/ (to match the URL on website)
* Adapt links to match the previous commit
* Adapt development docs to rst (still lacks translation and strikethrough support)
* clean on release
* blacklist presentations in gulp
* strikethrough support in sphinx
* just copy development folder for now
* fix weird introduction in style article
* Style guide translation (WIP)
* Finish style guide translation to English
* gulp clean separately
* Update year in LICENSE
* Initial CONTRIBUTING.md
* Fix remaining links to old docs in tutorial
* Some tutorial fixes
* Typo
* Another typo
* Update list of authors from yandex-team accoding to git log
* Fix diff with master
* couple fixes in en what_is_clickhouse.rst
* Try different link to blog in Russian
* Swap words
* Slightly larger line height
* CLICKHOUSE-3089: disable hyphenation in docs
* update presentations
* Fix copying of txt files
* update submodule
* CLICKHOUSE-3108: fix overflow issues in mobile version
* Less weird tutorial header in mobile version
* CLICKHOUSE-3073: skip sourcemaps by default
* CLICKHOUSE-3067: rename item in docs navigation
* fix list markup
* CLICKHOUSE-3067: some documentation style tuning
* CLICKHOUSE-3067: less laggy single page documentation
* update presentations
* CLICKHOUSE-2936: redirect from clickhouse.yandex.ru and clickhouse.yandex.com
* update submodule
* lost files
* CLICKHOUSE-2981: prefer sphinx docs over original reference
* CLICKHOUSE-2981: docs styles more similar to main website + add flags to switch language links
* update presentations
* Less confusing directory structure (docs -> doc/reference/)
* Minify sphinx docs too
* Website release script: fail fast + pass docker hash on deploy
* Do not underline links in docs
* shorter
* cleanup docker images
* tune nginx config
* CLICKHOUSE-3043: get rid of habrastorage links
* Lost translation
* CLICKHOUSE-2936: temporary client-side redirect
* behaves weird in test
* put redirect back
* CLICKHOUSE-3047: copy docs txts to public too
* move to proper file
* remove old pages to avoid confusion
* Remove reference redirect warning for now
* Refresh README.md
* Yellow buttons in docs
* Use svg flags instead of unicode ones in docs
* fix test website instance
* Put flags to separate files
* wrong flag
* Copy Yandex.Metrica introduction from main page to docs
* Yet another home page structure change, couple new blocks (CLICKHOUSE-3045)
* Update Contacts section
* CLICKHOUSE-2849: more detailed legal information
* CLICKHOUSE-2978 preparation - split by files
* More changes in Contacts block
* Tune texts on index page
* update presentations
* One more benchmark
* Add usage sections to index page, adapted from slides
* Get the roadmap started, based on slides from last ClickHouse Meetup
* CLICKHOUSE-2977: some rendering tuning
* Get rid of excessive section in the end of getting started
* Make headers linkable
* CLICKHOUSE-2981: links to editing reference - https://github.com/yandex/ClickHouse/issues/849
* CLICKHOUSE-2981: fix mobile styles in docs
* Ban crawling of duplicating docs
* Open some external links in new tab
* Ban old docs too
* Lots of trivial fixes in english docs
* Lots of trivial fixes in russian docs
* Remove getting started copies in markdown
* Add Yandex.Webmaster
* Fix some sphinx warnings
* More warnings fixed in english docs
* More sphinx warnings fixed
* Add code-block:: text
* More code-block:: text
* These headers look not that well
* Better switch between documentation languages
* merge use_case.rst into ya_metrika_task.rst
* Edit the agg_functions.rst texts
* Add lost empty lines
* Lost blank lines
* Add new logo sizes
* update presentations
* Next step in migrating to new documentation
* Fix all warnings in en reference
* Fix all warnings in ru reference
* Re-arrange existing reference
* Move operation tips to main reference
* Fix typos noticed by milovidov@
* Get rid of zookeeper.md
* Looks like duplicate of tutorial.html
* Fix some mess with html tags in tutorial
* No idea why nobody noticed this before, but it was completely not clear whet to get the data
* Match code block styling between main and tutorial pages (in favor of the latter)
* Get rid of some copypaste in tutorial
* Normalize header styles
* Move example_datasets to sphinx
* Move presentations submodule to website
* Move and update README.md
* No point in duplicating articles from habrahabr here
* Move development-related docs as is for now
* doc/reference/ -> docs/ (to match the URL on website)
* Adapt links to match the previous commit
* Adapt development docs to rst (still lacks translation and strikethrough support)
* clean on release
* blacklist presentations in gulp
* strikethrough support in sphinx
* just copy development folder for now
* fix weird introduction in style article
* Style guide translation (WIP)
* Finish style guide translation to English
* gulp clean separately
* Update year in LICENSE
* Initial CONTRIBUTING.md
* Fix remaining links to old docs in tutorial
* Some tutorial fixes
* Typo
* Another typo
* Update list of authors from yandex-team accoding to git log
* Fix diff with master
* couple fixes in en what_is_clickhouse.rst
* Try different link to blog in Russian
* Swap words
* Slightly larger line height
* CLICKHOUSE-3089: disable hyphenation in docs
* update presentations
* CLICKHOUSE-2936: redirect from clickhouse.yandex.ru and clickhouse.yandex.com
* update submodule
* lost files
* CLICKHOUSE-2981: prefer sphinx docs over original reference
* CLICKHOUSE-2981: docs styles more similar to main website + add flags to switch language links
* update presentations
* Less confusing directory structure (docs -> doc/reference/)
* Minify sphinx docs too
* Website release script: fail fast + pass docker hash on deploy
* Do not underline links in docs
* shorter
* cleanup docker images
* tune nginx config
* CLICKHOUSE-3043: get rid of habrastorage links
* Lost translation
* CLICKHOUSE-2936: temporary client-side redirect
* behaves weird in test
* put redirect back
* CLICKHOUSE-3047: copy docs txts to public too
* move to proper file
* remove old pages to avoid confusion
* Remove reference redirect warning for now
* Refresh README.md
* Yellow buttons in docs
* Use svg flags instead of unicode ones in docs
* fix test website instance
* Put flags to separate files
* wrong flag
* Copy Yandex.Metrica introduction from main page to docs
* Yet another home page structure change, couple new blocks (CLICKHOUSE-3045)
* Update Contacts section
* CLICKHOUSE-2849: more detailed legal information
* CLICKHOUSE-2978 preparation - split by files
* More changes in Contacts block
* Tune texts on index page
* update presentations
* One more benchmark
* Add usage sections to index page, adapted from slides
* Get the roadmap started, based on slides from last ClickHouse Meetup
* CLICKHOUSE-2977: some rendering tuning
* Get rid of excessive section in the end of getting started
* Make headers linkable
* CLICKHOUSE-2981: links to editing reference - https://github.com/yandex/ClickHouse/issues/849
* CLICKHOUSE-2981: fix mobile styles in docs
* Ban crawling of duplicating docs
* Open some external links in new tab
* Ban old docs too
* Lots of trivial fixes in english docs
* Lots of trivial fixes in russian docs
* Remove getting started copies in markdown
* Add Yandex.Webmaster
* Fix some sphinx warnings
* More warnings fixed in english docs
* More sphinx warnings fixed
* Add code-block:: text
* More code-block:: text
* These headers look not that well
* Better switch between documentation languages
* merge use_case.rst into ya_metrika_task.rst
* Edit the agg_functions.rst texts
* Add lost empty lines
* Lost blank lines
* Add new logo sizes
* update presentations
* Next step in migrating to new documentation
* Fix all warnings in en reference
* Fix all warnings in ru reference
* Re-arrange existing reference
* Move operation tips to main reference
* Fix typos noticed by milovidov@
* Get rid of zookeeper.md
* Looks like duplicate of tutorial.html
* Fix some mess with html tags in tutorial
* No idea why nobody noticed this before, but it was completely not clear whet to get the data
* Match code block styling between main and tutorial pages (in favor of the latter)
* Get rid of some copypaste in tutorial
* Normalize header styles
* Move example_datasets to sphinx
* Move presentations submodule to website
* Move and update README.md
* No point in duplicating articles from habrahabr here
* Move development-related docs as is for now
* doc/reference/ -> docs/ (to match the URL on website)
* Adapt links to match the previous commit
* Adapt development docs to rst (still lacks translation and strikethrough support)
* clean on release
* blacklist presentations in gulp
* strikethrough support in sphinx
* just copy development folder for now
* fix weird introduction in style article
* Style guide translation (WIP)
* Finish style guide translation to English
* gulp clean separately
* Update year in LICENSE
* Initial CONTRIBUTING.md
* Fix remaining links to old docs in tutorial
* Some tutorial fixes
* Typo
* Another typo
* Update list of authors from yandex-team accoding to git log
This commit implements a basic IP prefix dictionary
that allows storing IPv4/IPv6 prefixes and
matching them against a single IP address on query
time. This allows for doing IP -> ASN matching and
other similar things on query time.
The implementation is basic for start, using a
simple bitwise trie and reusing interface for
complex key dictionaries (so using tuple instead
of UInt32/FixedString(16) as the key). A faster
bitwise trie implementation (like poptrie) is
desired to improve lookup performance and
memory consumption with large prefix tables.
