thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,599 Commits 9,328 Branches 1,783 Tags 2.1 GiB
852f397a97
Commit Graph

15616 Commits

Author SHA1 Message Date
skyoct
0a4882d781 feat: update style 2023-12-14 14:26:47 +00:00
skyoct
d700eb2501 feat: support trivial count optimization for Merge 2023-12-14 14:08:24 +00:00
kssenii
a4a264c785 Fix 2023-12-14 14:04:17 +01:00
Andrey Zvonov
7d7aea9826 try fix memory leak in StorageHDFS 2023-12-14 13:01:47 +01:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
Igor Nikonov
42d0afef1f
Merge pull request #57749 from ClickHouse/pr-perf-improve-hope 
Parallel replicas: announcement response handling improvement
 2023-12-14 11:48:51 +01:00
Raúl Marín
efcacd3c34 Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 11:08:50 +01:00
Azat Khuzhin
0e86e16a01
Revert "Replace --no-system-tables with loading virtual tables of system database lazily" 2023-12-14 09:07:47 +01:00
Alexey Milovidov
7a2edb4804
Merge pull request #57822 from azat/kafka-disable-stat 
Disable system.kafka_consumers by default (due to possible live memory leak)
 2023-12-14 04:37:50 +01:00
Yakov Olkhovskiy
9e1d583f54 fix style 2023-12-13 23:36:54 +00:00
Kseniia Sumarokova
79db3c66df
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-13 23:33:59 +01:00
Yakov Olkhovskiy
441050adce fix style, cleanup 2023-12-13 22:26:38 +00:00
Julia Kartseva
b8d274d070
Add malformed output generation to JSON fuzzer (#57646) 
Randomly modify structural characters of a valid JSON ('{', '}', '[', ']',
':', '"', ',') to generate output that cannot be parsed as JSON.

Follow-up to https://github.com/ClickHouse/ClickHouse/pull/56490
 2023-12-13 19:59:31 +01:00
Alexey Milovidov
bf4964b177
Merge pull request #57712 from ClickHouse/clickbench-ci 
Add ClickBench to CI
 2023-12-13 19:52:44 +01:00
Nikita Taranov
560e66f3ca
More respect to min_number_of_marks in ParallelReplicasReadingCoordinator (#57763) 2023-12-13 19:51:49 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Yakov Olkhovskiy
f4e2933f5d use storage uuid instead of table name as a key in storage info because there are can be multiple dropped table with the same name 2023-12-13 18:08:41 +00:00
Azat Khuzhin
055c231438 Disable system.kafka_consumers by default (due to possible live memory leak) 
It is not safe to use statistics because of how KafkaEngine works - it
pre-creates consumers, and this leads to the situation when this
statistics entries generated (RD_KAFKA_OP_STATS), but never consumed.

Which creates a live memory leak for a server with Kafka tables, but
without materialized view attached to it (and no SELECT).

Another problem is that this makes shutdown very slow, because of how
pending queue entries are handled in librdkafka, it uses
TAILQ_INSERT_SORTED, which is sorted insert into linked list, which
works incredibly slow (likely you will never wait till it ends and kill
the server)

For instance in my production setup the server was running for ~67 days
with such table, and it got 1'942'233 `TAILQ_INSERT_SORTED` entries
(which perfectly matches by the way - `67*86400/3` = 1'929'600), and it
moved only 289'806 entries for a few hours, though I'm not sure how much
time the process was in the running state, since most of the time it was
with debugger attached.

So for now let's disable it, to make this patch easy for backporting,
and I will think about long term fix - do not pre-create consumers in
Kafka engine.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-13 18:13:13 +01:00
Anton Popov
7598403f78 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-13 16:55:17 +00:00
Yakov Olkhovskiy
0ea4b16530
Merge branch 'master' into ft-dropped_tables_parts 2023-12-13 09:01:59 -05:00
Igor Nikonov
c165be76ab
Parallel replicas: friendly settings (#57542) 2023-12-13 14:42:06 +01:00
Raúl Marín
dd405a655e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-13 13:39:58 +01:00
Kseniia Sumarokova
b1325450e9
Merge pull request #57754 from ClickHouse/fix-loading-dependent-table-materialized-postgresql 
Table engine MaterializedPostgreSQL fix dependency loading
 2023-12-13 12:17:01 +01:00
Yakov Olkhovskiy
5121bfcd78 major refactoring 2023-12-13 07:16:30 +00:00
Alexey Milovidov
62b6d1ef5e iMerge branch 'master' of github.com:ClickHouse/ClickHouse into clickbench-ci 2023-12-13 01:41:27 +01:00
Alexey Milovidov
f11b90e7bb Allow buckets without List access 2023-12-13 00:10:58 +01:00
Igor Nikonov
616d054dee Merge remote-tracking branch 'origin/master' into pr-perf-improve-hope 2023-12-12 21:27:31 +00:00
Igor Nikonov
7f4a028196 Test and fix 2023-12-12 21:26:58 +00:00
serxa
57d7a58dbe Rename system.async_loader into system.asynchronous_loader 2023-12-12 16:55:27 +00:00
kssenii
cc71b40002 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 16:40:04 +01:00
Alexander Gololobov
09d6e6b945
Merge pull request #57743 from ClickHouse/other_timeout_for_wait 
Increase async block cache deduplication timeout
 2023-12-12 15:40:04 +01:00
Raúl Marín
049fb60c73 Fix error on retries due to TABLE_IS_READ_ONLY 2023-12-12 15:37:28 +01:00
kssenii
fa0b2b9e50 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 15:31:01 +01:00
Chen Lixiang
68af81196d
Merge branch 'ClickHouse:master' into bytes_uncompressed 2023-12-12 22:06:45 +08:00
Kruglov Pavel
8a447bf57c
Merge pull request #55892 from Avogar/schema-inference-union 
Add 'union' mode for schema inference
 2023-12-12 15:02:06 +01:00
Nikolai Kochetov
696a2d9f4f
Merge pull request #57614 from ClickHouse/vdimir/join_array_lowcard_totals_crash 
Fix type correction in HashJoin for nested low cardinality
 2023-12-12 14:56:41 +01:00
Raúl Marín
923c3b7b32 Implement retries when ZK connection fails without committing the transaction 2023-12-12 12:00:46 +01:00
Raúl Marín
9d8d5dfdf6 Partially revert "make stages commit" 
This reverts commit e4becc01ba.
 2023-12-12 11:26:48 +01:00
Chen Lixiang
0b89fbbdd9 fix style issue 2023-12-12 16:49:58 +08:00
Zhuo Qiu
e34c13b9d9 Consider lightweight deleted rows when selecting parts to merge 2023-12-12 11:40:19 +08:00
alesapin
7cdc3d29fb Get rid of tiny function 2023-12-11 22:29:51 +01:00
alesapin
bfc2a7bb75 Update cache only after conflicts 2023-12-11 21:27:48 +01:00
alesapin
4b3e845dab Merge branch 'master' into other_timeout_for_wait 2023-12-11 20:43:21 +01:00
Igor Nikonov
30dabd42d3 One lookup for 2 checks 
(1) if it's the same part
(2) if annouced part has intersections with parts in working set
 2023-12-11 19:37:42 +00:00
kssenii
05e56f76ed Merge remote-tracking branch 'origin/master' into fix-loading-dependent-table-materialized-postgresql 2023-12-11 20:17:26 +01:00
Kseniia Sumarokova
907d0a79eb
Merge pull request #57568 from ClickHouse/fix-materialized-pg-issue-with-generated-columns 
MaterializedPostgreSQL: Support "generated columns" and default values
 2023-12-11 19:57:41 +01:00
Kruglov Pavel
3b1ffbd77c
Merge branch 'master' into schema-inference-union 2023-12-11 18:59:15 +01:00
kssenii
48ce040621 Fix 2023-12-11 17:30:23 +01:00
Raúl Marín
e1965bb6b5 WIP: Remove UNCERTAIN_COMMIT in INSERT 2023-12-11 17:27:56 +01:00
vdimir
ea09080c73
fix 2023-12-11 15:50:27 +00:00
Raúl Marín
e6be38be4d Adapt from HEAD 2023-12-11 16:28:00 +01:00
Raúl Marín
a55a0c068e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-11 15:27:21 +01:00
Igor Nikonov
4ccf4e11a8 Parallel replicas (perf): announcement response handling improvement 2023-12-11 13:00:08 +00:00
kssenii
1e4d61d55a Change error code 2023-12-11 13:44:12 +01:00
kssenii
b8caa00d24 Commit forgotten file 2023-12-11 13:28:36 +01:00
alesapin
a62a0b92de Increase async block cache deduplication timeout 2023-12-11 12:46:22 +01:00
kssenii
966a09fb70 Review fixes 2023-12-11 12:42:23 +01:00
kssenii
a06179da00 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-11 11:15:00 +01:00
Vitaly Baranov
f1ccf4e29f Show owner query ids in the message for the DEADLOCK_AVOIDED error. 2023-12-11 00:56:17 +01:00
Alexey Milovidov
644ef7b63f
Revert "Fix RWLock inconsistency after write lock timeout" 2023-12-10 23:51:59 +03:00
Alexey Milovidov
0a20ce5d32
Merge pull request #55271 from azat/local-lazy-database 
Replace --no-system-tables with loading virtual tables of system database lazily
 2023-12-10 19:51:17 +01:00
Alexey Milovidov
d119052895
Merge pull request #51858 from ClickHouse/CurtizJ-fix-ttl-non-deterministic 
Merging #37286
 2023-12-10 17:50:36 +01:00
Vitaly Baranov
a058a26b7e
Merge pull request #57454 from vitlibar/fix-rwlock 
Fix RWLock inconsistency after write lock timeout
 2023-12-10 14:09:29 +01:00
Alexey Milovidov
100b36e8a9 Do not check in replication 2023-12-10 01:32:42 +01:00
Alexey Milovidov
33b0ec7d55 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-10 01:21:43 +01:00
Azat Khuzhin
4f9d90761b Attach tables lazily only for clickhouse-local 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
c14721bb03 Move lazy table initialization interface into IDatabase 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
cc9afe6f6e Replace --no-system-tables with loading virtual tables of system database lazily 
First of all the problem is that that simple 'SELECT 1' cannot be runned
without system.one, which makes --no-system-tables almost useless:

    $ ./clickhouse-debug local --no-system-tables -q "select 1"
    Code: 81. DB::Exception: Database system does not exist. (UNKNOWN_DATABASE)

Secondly there are just too much flags, and this one
(--no-system-tables) is too damn specific.

This patch should improve startup time of the clickhouse-local almost
3x in debug builds.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Alexey Milovidov
c77183a597
Merge pull request #57480 from azat/dist/async-INSERT-fixes 
Fix possible distributed sends stuck due to "No such file or directory" (during recovering batch from disk)
 2023-12-09 17:11:35 +01:00
Alexey Milovidov
56cd9196b0
Merge pull request #57641 from azat/system.stack_trace-fix 
Avoid hangs of system.stack_trace by correctly prohibit parallel read from it
 2023-12-09 17:07:48 +01:00
Chen Lixiang
7cd770e9a7 fix doc and comments 2023-12-09 20:09:41 +08:00
Nikolay Degterinsky
c5e52d260f
Merge pull request #57553 from evillique/fix-local-crash 
Fix crash in clickhouse-local
 2023-12-08 22:57:52 +01:00
Gagan Goel
e547db0a8c Issue 43666: Add skip_unavailable_shards as a setting for Distributed table. 
This setting, when enabled (disabled by default), allows ClickHouse to
silently skip unavailable shards of a Distributed table during a query
execution, instead of throwing an exception to the client.
 2023-12-08 15:43:59 -05:00
avogar
ee7af95bc0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-12-08 20:29:28 +00:00
Azat Khuzhin
f876bea050 Add support for system.stack_trace filtering optimizations for analyzer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 18:13:36 +01:00
Nikita Mikhaylov
c979124cfe Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-08 16:25:02 +00:00
Azat Khuzhin
b1043b7ad3 Increase log verbosity of some messages in system.stack_trace 
This can help to debug 01051_system_stack_trace test failures.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 16:18:49 +01:00
Kseniia Sumarokova
928993f2bd
Merge branch 'master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 15:37:29 +01:00
Kseniia Sumarokova
bd3665863c
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-08 15:36:40 +01:00
zhongyuankai
7b0f8d44e8
Make DirectoryMonitor handle cluster node list change (#42826) 2023-12-08 14:41:51 +01:00
kssenii
d77938c3ff Add validation of columns 2023-12-08 12:42:38 +01:00
kssenii
ffd0d16f79 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 12:42:21 +01:00
Azat Khuzhin
16cbcf700e Prohibit parallel read from system.stack_trace 
Before rewriting system.stack_trace to handle max_block_size (in #54946)
parallel reading from system.stack_trace was prohibited, because this
could lead to hang of system.stack_trace table.

But that rewrite broke this guarantee, so let's fix it to avoid possible
hung.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 11:05:25 +01:00
Alexey Milovidov
ce9e134043 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-08 03:10:09 +01:00
alesapin
46f04d6d0e Throw more clear exception 2023-12-07 17:15:39 +01:00
kssenii
8be3c9d218 Merge remote-tracking branch 'origin/master' into allow-to-change-some-cache-settings-without-restart 2023-12-07 12:14:24 +01:00
kssenii
614da21144 Better 2023-12-07 12:12:10 +01:00
Chen Lixiang
9e95fe5d29 conflict fixed. 2023-12-07 18:10:45 +08:00
Kseniia Sumarokova
b3f22449f6
Merge pull request #57515 from ClickHouse/fix-materialized-pg-issue-with-incorrect-connection-options 
MaterializedPostgreSQL: fix issue #41922, add test for #41923
 2023-12-07 10:42:14 +01:00
Alexey Milovidov
b85214ca1a
Merge pull request #57546 from rschu1ze/revert-revert-support_alias_column_in_indices 
Revert "Revert "Support use alias column in indices""
 2023-12-07 01:22:03 +01:00
Alexey Milovidov
5fe1b8926c
Merge pull request #57595 from ClickHouse/23.12-prepare 
Update version after release
 2023-12-06 23:53:53 +01:00
Alexey Milovidov
c8fb574fc8 Update autogenerated version to 23.12.1.1 and contributors 2023-12-06 23:53:06 +01:00
Yakov Olkhovskiy
ac7210b9b3
Merge pull request #57461 from ClickHouse/fix-ephemeral-matview 
Fix: don't exclude ephemeral column when building pushing to view chain
 2023-12-06 16:46:54 -05:00
Alexey Milovidov
05bc8ef1e0
Merge pull request #57548 from ClickHouse/s3queue_production_ready 
`S3Queue` is production ready
 2023-12-06 22:21:08 +01:00
Alexey Milovidov
3723573f81 Revert "Merge pull request #56724 from canhld94/ch_replicated_column_mismatch" 
This reverts commit a9bb76378f, reversing
changes made to 588fd16518.
 2023-12-06 19:09:07 +01:00
kssenii
806061642a Support GENERATED and DEFAULT columns 2023-12-06 15:01:43 +01:00
vdimir
8f32293104
Merge pull request #57416 from ClickHouse/vdimir/fix_test_encrypted_disk 
Change type of s3_cache in test_encrypted_disk
 2023-12-06 14:37:40 +01:00
Yakov Olkhovskiy
8f00678abe fix style 2023-12-06 13:24:25 +00:00
Alexey Milovidov
f8de7f6bc2 Merge branch 'master' into s3queue_production_ready 2023-12-06 12:25:29 +01:00
Kseniia Sumarokova
e02883cd4a
Merge branch 'master' into fix-materialized-pg-issue-with-incorrect-connection-options 2023-12-06 12:16:04 +01:00
alesapin
e8b017d323
Merge pull request #57513 from ClickHouse/enable_async_block_ids_cache_by_default 
Enable async block ids cache by default
 2023-12-06 11:35:25 +01:00
Nikolai Kochetov
718b542f40
Merge pull request #57518 from ClickHouse/cleanup-around-distributed 
Simple cleanup in distributed (while dealing with parallel replicas)
 2023-12-06 10:22:17 +01:00
Yakov Olkhovskiy
f5fb169c65 add system.dropped_tables_parts table 2023-12-06 06:00:00 +00:00
Nikolay Degterinsky
ed618d3be1 Fix crash in clickhouse-local 2023-12-06 04:24:43 +00:00
Alexey Milovidov
76bbeaa676 S3Queue is production ready 2023-12-05 21:10:19 +01:00
Robert Schulze
8e4e37d8ed
Revert "Revert "Support use alias column in indices"" 
This reverts commit d0b591be2a.
 2023-12-05 18:22:32 +00:00
Alexey Milovidov
d0b591be2a
Revert "Support use alias column in indices" 2023-12-05 18:21:45 +03:00
Anton Popov
eb990d863d fix tests 2023-12-05 15:15:46 +00:00
Anton Popov
913ae1d267 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-05 14:47:32 +00:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Igor Nikonov
87a4c0bde4 Simple cleanup in distributed (while dealing with parallel replicas) 2023-12-05 12:25:02 +00:00
kssenii
0d313acd03 Fix 2023-12-05 13:22:35 +01:00
Alexander Sapin
93a16b7cd6 Fix comments 2023-12-05 12:39:00 +01:00
Alexander Sapin
2587f368a6 Enable async block ids cache by default 2023-12-05 12:37:22 +01:00
Anton Popov
5770def9af
Merge pull request #57491 from jkartseva/limit-table-projections 
limit merge tree projections
 2023-12-05 12:13:41 +01:00
Julia Kartseva
2ac104240d limit merge tree projections 
Fixes: https://github.com/ClickHouse/ClickHouse/issues/56427
 2023-12-05 06:44:39 +00:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
kssenii
4a28f10c3d Minor cache changes 2023-12-04 19:02:37 +01:00
Alexey Milovidov
02439eee58
Merge pull request #49351 from ClickHouse/async-loader-integration 
Asynchronous loading of tables
 2023-12-04 17:16:12 +01:00
Azat Khuzhin
7986fe619a Introduce DistributedAsyncInsertionFailures - event for async INSERT failures 
Useful for alerts

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
Azat Khuzhin
604cec475a Fix possible distributed sends stuck due to "No such file or directory" 
In case of restoring from current_batch.txt it is possible that the some
file from the batch will not be exist, and the fix submitted in #49884
was not complete, since it will fail later in markAsSend() (due to it
tries to obtain file size there):

    2023.12.04 05:43:12.676658 [ 5006 ] {} <Error> dist.DirectoryMonitor.work4: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in file_size: No such file or directory ["/work4/clickhouse/data/dist/shard8_all_replicas//150426396.bin"], Stack trace (when copying this message, always include the lines below):

    0. ./.build/./contrib/llvm-project/libcxx/include/exception:134: std::runtime_error::runtime_error(String const&) @ 0x00000000177e83f4 in /usr/lib/debug/usr/bin/clickhouse.debug
    1. ./.build/./contrib/llvm-project/libcxx/include/string:1499: std::system_error::system_error(std::error_code, String const&) @ 0x00000000177f0fd5 in /usr/lib/debug/usr/bin/clickhouse.debug
    2. ./.build/./contrib/llvm-project/libcxx/include/__filesystem/filesystem_error.h:42: std::__fs::filesystem::filesystem_error::filesystem_error[abi:v15000](String const&, std::__fs::filesystem::path const&, std::error_code) @ 0x000000000b844ca1 in /usr/lib/debug/usr/bin/clickhouse.debug
    3. ./.build/./contrib/llvm-project/libcxx/include/__filesystem/filesystem_error.h:90: void std::__fs::filesystem::__throw_filesystem_error[abi:v15000]<String&, std::__fs::filesystem::path const&, std::error_code const&>(String&, std::__fs::filesystem::path const&, std::error_code const&) @ 0x000000001778f953 in /usr/lib/debug/usr/bin/clickhouse.debug
    4. ./.build/./contrib/llvm-project/libcxx/src/filesystem/filesystem_common.h:0: std::__fs::filesystem::detail::(anonymous namespace)::ErrorHandler<unsigned long>::report(std::error_code const&) const @ 0x0000000017793ef7 in /usr/lib/debug/usr/bin/clickhouse.debug
    5. ./.build/./contrib/llvm-project/libcxx/src/filesystem/operations.cpp:0: std::__fs::filesystem::__file_size(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000017793e26 in /usr/lib/debug/usr/bin/clickhouse.debug
    6. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:707: DB::DistributedAsyncInsertDirectoryQueue::markAsSend(String const&) @ 0x0000000011cd92c5 in /usr/lib/debug/usr/bin/clickhouse.debug
    7. ./.build/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h💯 DB::DistributedAsyncInsertBatch::send() @ 0x0000000011cdd81c in /usr/lib/debug/usr/bin/clickhouse.debug
    8. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:0: DB::DistributedAsyncInsertDirectoryQueue::processFilesWithBatching() @ 0x0000000011cd5054 in /usr/lib/debug/usr/bin/clickhouse.debug
    9. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:417: DB::DistributedAsyncInsertDirectoryQueue::processFiles() @ 0x0000000011cd3440 in /usr/lib/debug/usr/bin/clickhouse.debug
    10. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:0: DB::DistributedAsyncInsertDirectoryQueue::run() @ 0x0000000011cd3878 in /usr/lib/debug/usr/bin/clickhouse.debug
    11. ./.build/./contrib/llvm-project/libcxx/include/__functional/function.h:0: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x00000000103dbc34 in /usr/lib/debug/usr/bin/clickhouse.debug
    12. ./.build/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x00000000103de1b6 in /usr/lib/debug/usr/bin/clickhouse.debug
    13. ./.build/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000103de7d1 in /usr/lib/debug/usr/bin/clickhouse.debug
    14. ./.build/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000b8c5502 in /usr/lib/debug/usr/bin/clickhouse.debug
    15. ./.build/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000b8c936e in /usr/lib/debug/usr/bin/clickhouse.debug
    16. ? @ 0x00007f1be8b30fd4 in ?
    17. ? @ 0x00007f1be8bb15bc in ?

And instead of ignoring errors, DistributedAsyncInsertBatch::valid() had
been added, that should be called when the files had been read from the
current_batch.txt, if it is not valid (some files from the batch did not
exist), then there is no sense in trying to send the same batch, so just
this file will be ignored, and files will be processed in a regular
order.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
Azat Khuzhin
638d0102f8 Fix error_count in case of distributed_directory_monitor_max_sleep_time_ms>5min 
In this case the error counter will be decremented everytime.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
vdimir
7e624ae911
Review fixes 2023-12-04 13:56:51 +00:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data 
Background merges correctly use temporary data storage in the cache
 2023-12-04 14:52:20 +01:00
Anton Popov
8d30e22a09 fix lighweight delete with heavy delete 2023-12-04 13:00:50 +00:00
vdimir
b2e29d1416
Change type of s3_cache in test_encrypted_disk 2023-12-04 10:34:13 +00:00
robot-ch-test-poll1
d63b652dae
Merge pull request #57385 from vitlibar/fix-inconsistent-metadata-for-backup-2 
Stop using INCONSISTENT_METADATA_FOR_BACKUP that much
 2023-12-04 10:29:57 +01:00
Yakov Olkhovskiy
b601b66146 don't exclude ephemeral column when pushing to view 2023-12-04 06:56:33 +00:00
Vitaly Baranov
0457aba4a9 Show owner query ids in the message for the DEADLOCK_AVOIDED error. 2023-12-04 00:15:31 +01:00
Nikolay Degterinsky
2150308c23
Merge pull request #56948 from amosbird/fix-56932 
Prevent incompatible ALTER of projection columns
 2023-12-03 04:24:51 +01:00
Sema Checherinda
c61cd6aab2
Merge pull request #57335 from sichenzhao/increase_default_replicated_deduplication_window 
increase default replicated_deduplication_window from 100 to 1k
 2023-12-02 21:23:39 +01:00
Nikolay Degterinsky
a53e604429
Merge branch 'master' into fix-56932 2023-12-02 21:19:45 +01:00
Sema Checherinda
32e3b2657b
Merge pull request #57205 from CheSema/zero-locks-leak 
fix zero-copy locks leaking
 2023-12-02 15:54:12 +01:00
Sema Checherinda
a1c9501c07 fix 01158_zookeeper_log_long and 02442_auxiliary_zookeeper_endpoint 2023-12-02 04:57:33 +01:00
Anton Popov
7fbe7462b6 add mutation command to apply deleted mask 2023-12-01 19:12:05 +00:00
JackyWoo
8a68d51261
Add index to table system.numbers (#50909) 2023-12-01 19:59:25 +01:00
robot-ch-test-poll
1b49463bd2
Merge pull request #55841 from nickitat/optimize_reading3 
Optimize reading from cache
 2023-12-01 17:36:57 +01:00
Nikolay Degterinsky
ceed0f73d3
Merge pull request #57391 from evillique/better-partitioned-write-to-file 
Allow wildcards in directories for partitioned write with File storage
 2023-12-01 17:23:54 +01:00
Nikolai Kochetov
5430ac2c02
Merge pull request #57099 from ClickHouse/analyzer-fuzz-3 
Analyzer fuzzer 3 (aggregate_functions_null_for_empty for projections)
 2023-12-01 16:57:34 +01:00
Anton Popov
61c517bc02
Merge pull request #57352 from kitaisreal/mutations-reuse-source-part-granularity 
MergeTree mutations reuse source part index granularity
 2023-12-01 16:45:41 +01:00
Nikolai Kochetov
0b4131546a
Merge pull request #56813 from jsc0218/SystemTablesFilterEngine 
Able to Filter Engine When Scanning System Tables
 2023-12-01 16:02:27 +01:00
alesapin
fb98b212c5
Merge pull request #57311 from ClickHouse/make_alter_mat_view_non_experimental 
Make alter materialized view non experimental
 2023-12-01 12:18:43 +01:00
vdimir
fd373024f5
Update buffer re-reading in MergeTask.cpp 2023-12-01 10:11:08 +00:00
Sema Checherinda
82270627ed fix the relock 2023-12-01 02:12:35 +01:00
Vitaly Baranov
1bc1563e0e Stop using INCONSISTENT_METADATA_FOR_BACKUP that much. If possible prefer to continue scanning instead of stopping and starting the scanning for backup again. 2023-11-30 21:37:42 +01:00
Sema Checherinda
47daa4f235 remarks 2023-11-30 21:13:56 +01:00
Sema Checherinda
42f648f0d2 relock block when sessin expired 2023-11-30 21:13:56 +01:00
Sema Checherinda
e4becc01ba make stages commit 2023-11-30 21:13:56 +01:00
Sema Checherinda
7efa76f08d remove rename_part_to_temporary 2023-11-30 21:13:56 +01:00
Sema Checherinda
2cc31752b1 unlockSharedData in 1 more place, and do it in a best effort way 2023-11-30 21:13:56 +01:00
Sema Checherinda
ad6897ddbd make shellcheck happy 2023-11-30 21:13:55 +01:00
Sema Checherinda
ad7b51c3c8 fix zero locks leak 2023-11-30 21:13:55 +01:00
Sergey Kviatkevich
376d297da2 fix alter table tests 2023-11-30 21:42:13 +03:00
Sergey Kviatkevich
afae432e99 allow ALTER for TEMPORARY table 2023-11-30 21:42:12 +03:00
Nikolay Degterinsky
95b5b9eb35 Fix partitioned write for File storage 2023-11-30 16:18:12 +00:00
Kruglov Pavel
575ee1e665
Merge pull request #56868 from zvonand/zvonand-fileCluster 
Introduce `fileCluster` table function
 2023-11-30 13:49:53 +01:00
Antonio Andelic
9955f571ef
Merge pull request #57318 from mkmkme/mkmkme/remove-unused-vars 
Remove unused Strings from MergeTreeData
 2023-11-30 13:48:54 +01:00
vdimir
54fe2e1410
Fix clang tidy 2023-11-30 12:40:34 +00:00
alesapin
d8bdbf29c4 Merge branch 'master' into make_alter_mat_view_non_experimental 2023-11-30 11:14:25 +01:00
robot-ch-test-poll
07ca6236f6
Merge pull request #57372 from jkartseva/fuzz-json-verbose-exception 
verbose exception messages for StorageFuzzJSON
 2023-11-30 08:20:54 +01:00
zvonand
6730a29a0b upd 2023-11-30 01:28:25 +01:00
Julia Kartseva
ed65c26faf verbose exception messages for StorageFuzzJSON 
Follow-up to https://github.com/ClickHouse/ClickHouse/pull/56490#discussion_r1409556111
 2023-11-29 20:42:21 +00:00
Kruglov Pavel
c6fecfb1af
Merge pull request #56901 from KevinyhZou/Fix_allow_cr_end_of_csv_line 
Fix allow cr end of line for csv
 2023-11-29 20:57:58 +01:00
Nikolai Kochetov
a36c11b21e Disable transform_null_in as well just in case. 2023-11-29 17:30:22 +00:00
Nikolai Kochetov
fc02b88596 Merge branch 'master' into analyzer-fuzz-3 2023-11-29 17:19:56 +00:00
Nikolai Kochetov
dc43a36ee2 Fixing style 2023-11-29 16:40:18 +00:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
Nikolai Kochetov
390c27f7d8 Ignore aggregate_functions_null_for_empty for projections at insert. 2023-11-29 16:16:22 +00:00
zvonand
c329e382ab resolve conflicts 2023-11-29 16:03:07 +01:00
zvonand
a4d9fbdfc2 Fix indents, fix style, fix test 2023-11-29 15:55:24 +01:00
Kruglov Pavel
30148972ed
Merge pull request #57328 from Avogar/size-virtual-column-table-functions 
Add information about new _size virtual column in file/s3/url/hdfs/azure table functions
 2023-11-29 15:05:28 +01:00
Maksim Kita
6578fe5baa Updated implementation 2023-11-29 16:21:07 +03:00
Maksim Kita
f0e53fde64 MergeTree mutations reuse source part index granularity 2023-11-29 14:36:57 +03:00
Han Fei
4c0efb0262
Merge pull request #53240 from hanfei1991/hanfei/statistic 
use statistic to order prewhere conditions better
 2023-11-29 12:26:44 +01:00
Nikolai Kochetov
731a1bfcf8 Cleanup 2023-11-29 11:11:58 +00:00
vdimir
470a092e3d
Merge pull request #51745 from ClickHouse/vdimir/setting_join_algorithm_order 
Setting JoinAlgorithm respect specified order
 2023-11-29 11:24:39 +01:00
Raúl Marín
b102145a4f
Merge pull request #57301 from Algunenano/fix_reports 
Fix code reports
 2023-11-29 11:11:33 +01:00
Chen Lixiang
88920f08f1 add tests and fix whitespace issue 2023-11-29 17:26:11 +08:00
Robert Schulze
bc3da5032e
Merge pull request #57330 from rschu1ze/fix-no-lib-build 
Follow-up to #56490: Fix build with `cmake -DENABLE_LIBRARIES=0`
 2023-11-29 09:42:12 +01:00
Antonio Andelic
f8811351fa
Merge pull request #57267 from ClickHouse/fix-readonly-replica 
Fix `ReadonlyReplica` metric for all cases
 2023-11-29 08:32:25 +01:00
Sichen Zhao
1b8b1c0a63 increase default replicated_deduplication_window from 100 to 1k 2023-11-28 14:24:06 -08:00
Robert Schulze
5f5e8633c7
Merge pull request #57220 from ucasfl/alias_column_in_indices 
Support use alias column in indices
 2023-11-28 21:31:34 +01:00
Robert Schulze
7d26990752
Follow-up to #56490: Fix build with -DENABLE_LIBRARIES=0 2023-11-28 19:59:43 +00:00
vdimir
b748405976
Intorduce *List definition for muli enum settings 2023-11-28 19:09:02 +00:00
vdimir
a6eb2ad930
Fix JoinStep::describeJoinActions 2023-11-28 19:09:01 +00:00
Nikolai Kochetov
4d906d6e3c Cleanup 2023-11-28 18:35:19 +00:00
avogar
4d9a1b50f9 Add information about new _size virtual column in file/s3/url/hdfs/azure table functions 2023-11-28 18:15:07 +00:00
Nikolai Kochetov
84194f10f2 Fixing tests. 2023-11-28 18:04:30 +00:00
Nikolai Kochetov
08a7575984 Re-implement filtering a bit. 2023-11-28 16:17:35 +00:00
Han Fei
e4421e2808 remove wrong code 2023-11-28 16:46:55 +01:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Han Fei
6733438adc Merge branch 'master' into hanfei/statistic 2023-11-28 16:35:00 +01:00
Han Fei
8e84c734bf rename some code 2023-11-28 16:32:47 +01:00
zvonand
c306d21b54 merge master + resolve conflicts 2023-11-28 15:51:21 +01:00
Mikhail Koviazin
daaa575106
Remove unused Strings from MergeTreeData 
Compiler does not emit a warning when there is an unused variable of
a type with a non-trivial constructor. std::string is one of them.
Hence, these two variables were overlooked. This commit removes them
 2023-11-28 16:33:44 +02:00
Alexander Tokmakov
e40c71a74c
Merge pull request #56533 from ClickHouse/rmt_check_shutdown_flags_in_retry_loops 
ReplicatedMergeTree: check shutdown flags in retry loops
 2023-11-28 15:24:40 +01:00
Alexander Sapin
2cd9f5f3cc Fix known issue 2023-11-28 15:13:12 +01:00
Alexander Sapin
e6855cef4d Followup 2023-11-28 14:01:31 +01:00
Alexander Sapin
b5404f4a9f Followup 2023-11-28 13:34:05 +01:00
Alexander Sapin
e1aa62d9cd Make alter materialized view non expiremental and deprecate allow_experimental_alter_materialized_view_structure obsolete 2023-11-28 13:17:55 +01:00