ClickHouse® is a real-time analytics DBMS
Go to file
Azat Khuzhin e7592c140e Create consumers for Kafka tables on fly (but keep them for 1min since last used)
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-27 09:49:07 +01:00
.github Run build report check job on build failures, fix (#58135) 2023-12-21 21:30:40 +01:00
base Adjust all std::ios implementations in poco to set failbit/badbit by default 2023-12-22 18:34:34 +01:00
benchmark
cmake
contrib Maybe fix the build, maybe not fix it 2023-12-20 15:32:33 +00:00
docker Merge pull request #58168 from ClickHouse/udpate-statefull-run-sh 2023-12-23 01:40:53 +01:00
docs fix doc 2023-12-26 16:57:18 +11:00
packages
programs Merge pull request #58210 from chhetripradeep/pchhetri/param-query-clickhouse-local 2023-12-25 13:34:04 +01:00
rust Replace rust's BLAKE3 with llvm's implementation 2023-12-18 15:29:01 +01:00
src Create consumers for Kafka tables on fly (but keep them for 1min since last used) 2023-12-27 09:49:07 +01:00
tests Merge pull request #58224 from amosbird/part_offset_pk 2023-12-26 14:51:57 +01:00
utils Merge pull request #58095 from azat/metrics/max-rss 2023-12-23 13:13:44 +01:00
.clang-format
.clang-tidy
.clangd
.editorconfig
.exrc
.git-blame-ignore-revs
.gitattributes
.gitignore
.gitmessage Add GH status for PR formatting (#58050) 2023-12-21 11:53:01 +01:00
.gitmodules Implement punycodeEncode()/Decode() 2023-12-17 17:04:28 +00:00
.pylintrc
.snyk
.yamllint
AUTHORS
CHANGELOG.md
CMakeLists.txt
CODE_OF_CONDUCT.md
CONTRIBUTING.md
format_sources
LICENSE
PreLoad.cmake
README.md
SECURITY.md

Website Apache 2.0 License

The ClickHouse company logo.

ClickHouse® is an open-source column-oriented database management system that allows generating analytical data reports in real-time.

How To Install (Linux, macOS, FreeBSD)

curl https://clickhouse.com/ | sh

Upcoming Events

Also, keep an eye out for upcoming meetups around the world. Somewhere else you want us to be? Please feel free to reach out to tyler clickhouse com.

Recent Recordings

  • Recent Meetup Videos: Meetup Playlist Whenever possible recordings of the ClickHouse Community Meetups are edited and presented as individual talks. Current featuring "Modern SQL in 2023", "Fast, Concurrent, and Consistent Asynchronous INSERTS in ClickHouse", and "Full-Text Indices: Design and Experiments"
  • Recording available: v23.10 Release Webinar All the features of 23.10, one convenient video! Watch it now!
  • All release webinar recordings: YouTube playlist

Interested in joining ClickHouse and making it your full-time job?

We are a globally diverse and distributed team, united behind a common goal of creating industry-leading, real-time analytics. Here, you will have an opportunity to solve some of the most cutting-edge technical challenges and have direct ownership of your work and vision. If you are a contributor by nature, a thinker and a doer - well definitely click!

Check out our current openings here: https://clickhouse.com/company/careers

Can't find what you are looking for, but want to let us know you are interested in joining ClickHouse? Email careers@clickhouse.com!