thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,193 Commits 9,326 Branches 1,783 Tags 2.1 GiB
296a0bf981
Commit Graph

15521 Commits

Author SHA1 Message Date
Kseniia Sumarokova
06438cc17b
Merge pull request #57578 from ClickHouse/allow-to-change-some-cache-settings-without-restart 
Allow to apply some fs cache config settings changes without server restart
 2023-12-15 11:18:19 +01:00
flynn
2d45213f70 Fix transform query for external database 2023-12-15 08:34:58 +00:00
skyoct
3319934055 reafactor function 2023-12-15 04:49:38 +00:00
Alexey Milovidov
23cee7ef37
Merge pull request #57851 from azat/revert-55271-local-lazy-database 
Revert "Replace --no-system-tables with loading virtual tables of system database lazily"
 2023-12-15 03:25:59 +01:00
Andrey Zvonov
0174b94469
Merge branch 'ClickHouse:master' into zvonand-fix-57819 2023-12-14 20:14:54 +01:00
Raúl Marín
546484d46b Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 19:03:59 +01:00
Duc Canh Le
93dd6b83e7
Do not allow creating replicated table with inconsistent merge params (#56833) 
* save all merge params to zookeeper

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* calculate hash for graphite merge params

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* add graphite params hash to zookeeper + fix tests

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* install new graphite for testing

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* fix backward incompatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* minor fix test

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* Update src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* remove peekString and add more comments

- peekString doesn't always work even for ReadBufferFromString
- more comment re. backward compatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

---------

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-12-14 17:26:35 +01:00
joelynch
b4bef78e63
bugfix: addresses_expr ignored for psql named collections 2023-12-14 17:17:14 +01:00
Andrey Zvonov
f9579580e9 Retrigger CI + make docs a bit better 2023-12-14 16:34:10 +01:00
kssenii
3605caa8d9 Fxi 2023-12-14 16:05:40 +01:00
kssenii
ea7e55b929 Simplify 2023-12-14 15:58:06 +01:00
skyoct
0a4882d781 feat: update style 2023-12-14 14:26:47 +00:00
skyoct
d700eb2501 feat: support trivial count optimization for Merge 2023-12-14 14:08:24 +00:00
kssenii
a4a264c785 Fix 2023-12-14 14:04:17 +01:00
Andrey Zvonov
7d7aea9826 try fix memory leak in StorageHDFS 2023-12-14 13:01:47 +01:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
Igor Nikonov
42d0afef1f
Merge pull request #57749 from ClickHouse/pr-perf-improve-hope 
Parallel replicas: announcement response handling improvement
 2023-12-14 11:48:51 +01:00
Raúl Marín
efcacd3c34 Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 11:08:50 +01:00
Azat Khuzhin
0e86e16a01
Revert "Replace --no-system-tables with loading virtual tables of system database lazily" 2023-12-14 09:07:47 +01:00
Alexey Milovidov
7a2edb4804
Merge pull request #57822 from azat/kafka-disable-stat 
Disable system.kafka_consumers by default (due to possible live memory leak)
 2023-12-14 04:37:50 +01:00
Yakov Olkhovskiy
9e1d583f54 fix style 2023-12-13 23:36:54 +00:00
Kseniia Sumarokova
79db3c66df
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-13 23:33:59 +01:00
Yakov Olkhovskiy
441050adce fix style, cleanup 2023-12-13 22:26:38 +00:00
Julia Kartseva
b8d274d070
Add malformed output generation to JSON fuzzer (#57646) 
Randomly modify structural characters of a valid JSON ('{', '}', '[', ']',
':', '"', ',') to generate output that cannot be parsed as JSON.

Follow-up to https://github.com/ClickHouse/ClickHouse/pull/56490
 2023-12-13 19:59:31 +01:00
Alexey Milovidov
bf4964b177
Merge pull request #57712 from ClickHouse/clickbench-ci 
Add ClickBench to CI
 2023-12-13 19:52:44 +01:00
Nikita Taranov
560e66f3ca
More respect to min_number_of_marks in ParallelReplicasReadingCoordinator (#57763) 2023-12-13 19:51:49 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Yakov Olkhovskiy
f4e2933f5d use storage uuid instead of table name as a key in storage info because there are can be multiple dropped table with the same name 2023-12-13 18:08:41 +00:00
Azat Khuzhin
055c231438 Disable system.kafka_consumers by default (due to possible live memory leak) 
It is not safe to use statistics because of how KafkaEngine works - it
pre-creates consumers, and this leads to the situation when this
statistics entries generated (RD_KAFKA_OP_STATS), but never consumed.

Which creates a live memory leak for a server with Kafka tables, but
without materialized view attached to it (and no SELECT).

Another problem is that this makes shutdown very slow, because of how
pending queue entries are handled in librdkafka, it uses
TAILQ_INSERT_SORTED, which is sorted insert into linked list, which
works incredibly slow (likely you will never wait till it ends and kill
the server)

For instance in my production setup the server was running for ~67 days
with such table, and it got 1'942'233 `TAILQ_INSERT_SORTED` entries
(which perfectly matches by the way - `67*86400/3` = 1'929'600), and it
moved only 289'806 entries for a few hours, though I'm not sure how much
time the process was in the running state, since most of the time it was
with debugger attached.

So for now let's disable it, to make this patch easy for backporting,
and I will think about long term fix - do not pre-create consumers in
Kafka engine.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-13 18:13:13 +01:00
Yakov Olkhovskiy
0ea4b16530
Merge branch 'master' into ft-dropped_tables_parts 2023-12-13 09:01:59 -05:00
Igor Nikonov
c165be76ab
Parallel replicas: friendly settings (#57542) 2023-12-13 14:42:06 +01:00
Raúl Marín
dd405a655e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-13 13:39:58 +01:00
Kseniia Sumarokova
b1325450e9
Merge pull request #57754 from ClickHouse/fix-loading-dependent-table-materialized-postgresql 
Table engine MaterializedPostgreSQL fix dependency loading
 2023-12-13 12:17:01 +01:00
Yakov Olkhovskiy
5121bfcd78 major refactoring 2023-12-13 07:16:30 +00:00
Alexey Milovidov
62b6d1ef5e iMerge branch 'master' of github.com:ClickHouse/ClickHouse into clickbench-ci 2023-12-13 01:41:27 +01:00
Alexey Milovidov
f11b90e7bb Allow buckets without List access 2023-12-13 00:10:58 +01:00
Igor Nikonov
616d054dee Merge remote-tracking branch 'origin/master' into pr-perf-improve-hope 2023-12-12 21:27:31 +00:00
Igor Nikonov
7f4a028196 Test and fix 2023-12-12 21:26:58 +00:00
serxa
57d7a58dbe Rename system.async_loader into system.asynchronous_loader 2023-12-12 16:55:27 +00:00
kssenii
cc71b40002 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 16:40:04 +01:00
Alexander Gololobov
09d6e6b945
Merge pull request #57743 from ClickHouse/other_timeout_for_wait 
Increase async block cache deduplication timeout
 2023-12-12 15:40:04 +01:00
Raúl Marín
049fb60c73 Fix error on retries due to TABLE_IS_READ_ONLY 2023-12-12 15:37:28 +01:00
kssenii
fa0b2b9e50 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 15:31:01 +01:00
Chen Lixiang
68af81196d
Merge branch 'ClickHouse:master' into bytes_uncompressed 2023-12-12 22:06:45 +08:00
Kruglov Pavel
8a447bf57c
Merge pull request #55892 from Avogar/schema-inference-union 
Add 'union' mode for schema inference
 2023-12-12 15:02:06 +01:00
Nikolai Kochetov
696a2d9f4f
Merge pull request #57614 from ClickHouse/vdimir/join_array_lowcard_totals_crash 
Fix type correction in HashJoin for nested low cardinality
 2023-12-12 14:56:41 +01:00
Raúl Marín
923c3b7b32 Implement retries when ZK connection fails without committing the transaction 2023-12-12 12:00:46 +01:00
Raúl Marín
9d8d5dfdf6 Partially revert "make stages commit" 
This reverts commit e4becc01ba.
 2023-12-12 11:26:48 +01:00
Chen Lixiang
0b89fbbdd9 fix style issue 2023-12-12 16:49:58 +08:00
Zhuo Qiu
e34c13b9d9 Consider lightweight deleted rows when selecting parts to merge 2023-12-12 11:40:19 +08:00
alesapin
7cdc3d29fb Get rid of tiny function 2023-12-11 22:29:51 +01:00
alesapin
bfc2a7bb75 Update cache only after conflicts 2023-12-11 21:27:48 +01:00
alesapin
4b3e845dab Merge branch 'master' into other_timeout_for_wait 2023-12-11 20:43:21 +01:00
Igor Nikonov
30dabd42d3 One lookup for 2 checks 
(1) if it's the same part
(2) if annouced part has intersections with parts in working set
 2023-12-11 19:37:42 +00:00
kssenii
05e56f76ed Merge remote-tracking branch 'origin/master' into fix-loading-dependent-table-materialized-postgresql 2023-12-11 20:17:26 +01:00
Kseniia Sumarokova
907d0a79eb
Merge pull request #57568 from ClickHouse/fix-materialized-pg-issue-with-generated-columns 
MaterializedPostgreSQL: Support "generated columns" and default values
 2023-12-11 19:57:41 +01:00
Kruglov Pavel
3b1ffbd77c
Merge branch 'master' into schema-inference-union 2023-12-11 18:59:15 +01:00
kssenii
48ce040621 Fix 2023-12-11 17:30:23 +01:00
Raúl Marín
e1965bb6b5 WIP: Remove UNCERTAIN_COMMIT in INSERT 2023-12-11 17:27:56 +01:00
vdimir
ea09080c73
fix 2023-12-11 15:50:27 +00:00
Raúl Marín
e6be38be4d Adapt from HEAD 2023-12-11 16:28:00 +01:00
Raúl Marín
a55a0c068e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-11 15:27:21 +01:00
Igor Nikonov
4ccf4e11a8 Parallel replicas (perf): announcement response handling improvement 2023-12-11 13:00:08 +00:00
kssenii
1e4d61d55a Change error code 2023-12-11 13:44:12 +01:00
kssenii
b8caa00d24 Commit forgotten file 2023-12-11 13:28:36 +01:00
alesapin
a62a0b92de Increase async block cache deduplication timeout 2023-12-11 12:46:22 +01:00
kssenii
966a09fb70 Review fixes 2023-12-11 12:42:23 +01:00
kssenii
a06179da00 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-11 11:15:00 +01:00
Vitaly Baranov
f1ccf4e29f Show owner query ids in the message for the DEADLOCK_AVOIDED error. 2023-12-11 00:56:17 +01:00
Alexey Milovidov
644ef7b63f
Revert "Fix RWLock inconsistency after write lock timeout" 2023-12-10 23:51:59 +03:00
Alexey Milovidov
0a20ce5d32
Merge pull request #55271 from azat/local-lazy-database 
Replace --no-system-tables with loading virtual tables of system database lazily
 2023-12-10 19:51:17 +01:00
Alexey Milovidov
d119052895
Merge pull request #51858 from ClickHouse/CurtizJ-fix-ttl-non-deterministic 
Merging #37286
 2023-12-10 17:50:36 +01:00
Vitaly Baranov
a058a26b7e
Merge pull request #57454 from vitlibar/fix-rwlock 
Fix RWLock inconsistency after write lock timeout
 2023-12-10 14:09:29 +01:00
Alexey Milovidov
100b36e8a9 Do not check in replication 2023-12-10 01:32:42 +01:00
Alexey Milovidov
33b0ec7d55 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-10 01:21:43 +01:00
Azat Khuzhin
4f9d90761b Attach tables lazily only for clickhouse-local 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
c14721bb03 Move lazy table initialization interface into IDatabase 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
cc9afe6f6e Replace --no-system-tables with loading virtual tables of system database lazily 
First of all the problem is that that simple 'SELECT 1' cannot be runned
without system.one, which makes --no-system-tables almost useless:

    $ ./clickhouse-debug local --no-system-tables -q "select 1"
    Code: 81. DB::Exception: Database system does not exist. (UNKNOWN_DATABASE)

Secondly there are just too much flags, and this one
(--no-system-tables) is too damn specific.

This patch should improve startup time of the clickhouse-local almost
3x in debug builds.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Alexey Milovidov
c77183a597
Merge pull request #57480 from azat/dist/async-INSERT-fixes 
Fix possible distributed sends stuck due to "No such file or directory" (during recovering batch from disk)
 2023-12-09 17:11:35 +01:00
Alexey Milovidov
56cd9196b0
Merge pull request #57641 from azat/system.stack_trace-fix 
Avoid hangs of system.stack_trace by correctly prohibit parallel read from it
 2023-12-09 17:07:48 +01:00
Chen Lixiang
7cd770e9a7 fix doc and comments 2023-12-09 20:09:41 +08:00
Nikolay Degterinsky
c5e52d260f
Merge pull request #57553 from evillique/fix-local-crash 
Fix crash in clickhouse-local
 2023-12-08 22:57:52 +01:00
Gagan Goel
e547db0a8c Issue 43666: Add skip_unavailable_shards as a setting for Distributed table. 
This setting, when enabled (disabled by default), allows ClickHouse to
silently skip unavailable shards of a Distributed table during a query
execution, instead of throwing an exception to the client.
 2023-12-08 15:43:59 -05:00
avogar
ee7af95bc0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-12-08 20:29:28 +00:00
Nikita Mikhaylov
c979124cfe Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-08 16:25:02 +00:00
Azat Khuzhin
b1043b7ad3 Increase log verbosity of some messages in system.stack_trace 
This can help to debug 01051_system_stack_trace test failures.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 16:18:49 +01:00
Kseniia Sumarokova
928993f2bd
Merge branch 'master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 15:37:29 +01:00
Kseniia Sumarokova
bd3665863c
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-08 15:36:40 +01:00
zhongyuankai
7b0f8d44e8
Make DirectoryMonitor handle cluster node list change (#42826) 2023-12-08 14:41:51 +01:00
kssenii
d77938c3ff Add validation of columns 2023-12-08 12:42:38 +01:00
kssenii
ffd0d16f79 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 12:42:21 +01:00
Azat Khuzhin
16cbcf700e Prohibit parallel read from system.stack_trace 
Before rewriting system.stack_trace to handle max_block_size (in #54946)
parallel reading from system.stack_trace was prohibited, because this
could lead to hang of system.stack_trace table.

But that rewrite broke this guarantee, so let's fix it to avoid possible
hung.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 11:05:25 +01:00
Alexey Milovidov
ce9e134043 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-08 03:10:09 +01:00
alesapin
46f04d6d0e Throw more clear exception 2023-12-07 17:15:39 +01:00
kssenii
8be3c9d218 Merge remote-tracking branch 'origin/master' into allow-to-change-some-cache-settings-without-restart 2023-12-07 12:14:24 +01:00
kssenii
614da21144 Better 2023-12-07 12:12:10 +01:00
Chen Lixiang
9e95fe5d29 conflict fixed. 2023-12-07 18:10:45 +08:00
Kseniia Sumarokova
b3f22449f6
Merge pull request #57515 from ClickHouse/fix-materialized-pg-issue-with-incorrect-connection-options 
MaterializedPostgreSQL: fix issue #41922, add test for #41923
 2023-12-07 10:42:14 +01:00
Alexey Milovidov
b85214ca1a
Merge pull request #57546 from rschu1ze/revert-revert-support_alias_column_in_indices 
Revert "Revert "Support use alias column in indices""
 2023-12-07 01:22:03 +01:00
Alexey Milovidov
5fe1b8926c
Merge pull request #57595 from ClickHouse/23.12-prepare 
Update version after release
 2023-12-06 23:53:53 +01:00