thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-13 09:52:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
107,643 Commits 9,324 Branches 1,783 Tags 2.1 GiB
1b207d9fa0
Commit Graph

11394 Commits

Author SHA1 Message Date
Anton Popov
fb3f7409fa allow vertical merges from compact to wide parts 2023-01-26 22:18:04 +00:00
Alexander Tokmakov
061026c307 fix 2023-01-26 19:01:29 +01:00
Alexey Milovidov
d942a4ddef Allow LowCardinality in GenerateRandom 2023-01-26 15:35:20 +01:00
Alexander Tokmakov
067b1f5f13 Merge branch 'master' into exception_message_patterns4 2023-01-26 15:20:58 +01:00
Alexander Tokmakov
f676e9eaff
Merge pull request #45613 from azat/replicated-improve-log 
Improve "at least part X is missing" error message
 2023-01-26 16:48:18 +03:00
Anton Popov
92fa39798b Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-26 13:29:23 +00:00
Anton Popov
b58b73b0e7
Merge pull request #45529 from CurtizJ/fix-storage-s3-race 
Try to fix test `test_storage_s3/test.py::test_wrong_s3_syntax` (race in `StorageS3`)
 2023-01-26 14:21:32 +01:00
Anton Popov
87af86e9a3 fix projections 2023-01-26 13:11:31 +00:00
Alexander Tokmakov
14db798191 fix 2023-01-26 13:56:16 +01:00
Anton Popov
b52cf17fa6
Merge pull request #45612 from ClickHouse/CurtizJ-patch-7 
Remove useless code
 2023-01-26 12:38:52 +01:00
alesapin
84af5eb58b Fxi 2023-01-26 11:42:53 +01:00
alesapin
d5036f2ebc Merge branch 'fix_hang_during_drop_in_zero_copy_replication' of github.com:ClickHouse/ClickHouse into fix_hang_during_drop_in_zero_copy_replication 2023-01-26 11:42:30 +01:00
alesapin
1f61abce46 One useful logging line 2023-01-26 11:41:43 +01:00
Smita Kulkarni
8c885ffeae Updated checking of SYSTEM SYNC REPLICA 
Implementation:
* Updated to wait for current last entry to be processed (after pulling shared log) instead of queue size becoming 0.
* Updated Subscriber to notify both queue size and removed log_entry_id.
 2023-01-26 11:40:09 +01:00
Alexander Tokmakov
a584ad0eb1 forbid runtime strings 2023-01-26 10:52:47 +01:00
Antonio Andelic
6fe9e9a67f
Merge pull request #45617 from ClickHouse/fix-bfngram-index 
Correctly check types when using N-gram bloom filter index
 2023-01-26 09:05:56 +01:00
Alexander Tokmakov
9b670946db Merge branch 'master' into exception_message_patterns5 2023-01-26 00:41:32 +01:00
Mikhail f. Shiryaev
120c31d57f
Merge pull request #45634 from ClickHouse/23.2-prepare 
Update version after release
 2023-01-26 00:10:53 +01:00
Mikhail f. Shiryaev
53d89844be
Update version to 23.2.1.1 2023-01-25 23:57:29 +01:00
Anton Popov
79ea6a1c9c fix performance of short queries 2023-01-25 22:20:01 +00:00
Alexander Tokmakov
92c62e2de2
Update src/Storages/StorageReplicatedMergeTree.cpp 2023-01-26 00:28:46 +03:00
Han Fei
0221b50028
Merge pull request #45601 from hanfei1991/hanfei/async-cache-conflict 
fix race of async-insert-blockids-cache
 2023-01-25 22:28:42 +01:00
Alexander Tokmakov
3744fa2c63 format more messages 2023-01-25 21:16:42 +01:00
Anton Popov
0f46c1ecc6 fix build 2023-01-25 20:13:54 +00:00
alesapin
d0428a9b42 Fxi 2023-01-25 20:30:19 +01:00
Kruglov Pavel
cd3d3fdd68
Merge pull request #45483 from Avogar/fix-hdfs-cluster 
Fix schema inference from insertion table in hdfsCluster
 2023-01-25 20:24:51 +01:00
Azat Khuzhin
00c341850f Improve "at least part X is missing" error message 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-25 19:53:45 +01:00
Anton Popov
a455083b87 Merge remote-tracking branch 'upstream/master' into refactoring-merge-tree-parts 2023-01-25 18:01:12 +00:00
Kruglov Pavel
83a212a128
Merge branch 'master' into more-fsync-fixes 2023-01-25 18:40:41 +01:00
Anton Popov
8e3698c91f refactoring of code near merge tree parts 2023-01-25 17:34:09 +00:00
Antonio Andelic
7fa906487c Fix ngram index with null 2023-01-25 16:40:04 +00:00
Anton Popov
d9aabb48c3 remove useless code 2023-01-25 16:11:15 +00:00
alesapin
55af44ca2f Better fix 2023-01-25 14:40:39 +01:00
Alexander Tokmakov
b9e5c586e6
Merge pull request #45566 from ClickHouse/resubmit_45493 
Resubmit "Fix possible in-use table after DETACH"
 2023-01-25 16:39:55 +03:00
Alexander Tokmakov
6eb557b2ba Merge branch 'master' into exception_message_patterns4 2023-01-25 13:49:17 +01:00
Robert Schulze
59528cfca0
Merge pull request #45460 from ClickHouse/inv-index-cleanup 
Cleanup of inverted index
 2023-01-25 13:23:38 +01:00
alesapin
96d8546b1c Merge branch 'master' into fix_hang_during_drop_in_zero_copy_replication 2023-01-25 13:07:14 +01:00
Han Fei
8c4bbcd0fc fix race of async-insert-blockids-cache 2023-01-25 12:41:01 +01:00
Sergei Trifonov
0d1ea05ff6
Merge pull request #45007 from ClickHouse/cancellable-mutex-integration 
Fast shared mutex integration
 2023-01-25 11:15:46 +01:00
Kseniia Sumarokova
2b8b1ad5d4
Additional check in MergeTreeReadPool (#45515) 
* Check ranges

* Check equality just in case

* Check under ndebug
 2023-01-24 23:56:26 +01:00
Alexander Tokmakov
6621834e37 fix 2023-01-24 17:10:54 +01:00
avogar
07ad750366 Fix tests 2023-01-24 15:08:19 +00:00
Alexander Tokmakov
6ecae8388e Merge branch 'master' into exception_message_patterns4 2023-01-24 14:42:36 +01:00
Alexander Tokmakov
42a976fe3d Revert "Revert "Merge pull request #45493 from azat/fix-detach"" 
This reverts commit 9dc4f02d15.
 2023-01-24 14:15:51 +01:00
Azat Khuzhin
54ef885a61 Fsync all small files at once after mutation 
Everything else is handled in MergedBlockOutputStream::finalizePart()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Azat Khuzhin
4b775ccdd5 Do fsync all files at once for fetched parts to decrease latency 
For filessystems like ext4, fsync of one file will handle all operations
before, so this can be pretty time consuming.

And in case of you write multiple files in a loop, and at the end of
each iteration sync each file, then during writing of this file there
can be other operations in journal, and hence more work for fsync.

Let's call fsync for all files at once instead, like
MergedBlockOutputStream does.

Hope that keeping all file buffers till the end, will not cause troubles
(buffering and so forth).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Azat Khuzhin
b370e3faa5 Move fsync inside transaction callback in DataPartStorageOnDisk::rename() 
Otherwise, it is useless.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-24 14:08:37 +01:00
Anton Popov
5c0307bc6a fix race in StorageURL and StorageHDFS 2023-01-24 12:34:43 +00:00
Alexander Tokmakov
9dc4f02d15 Revert "Merge pull request #45493 from azat/fix-detach" 
This reverts commit a182a6b851, reversing
changes made to c47a29a089.
 2023-01-24 12:20:37 +01:00
Robert Schulze
9ff2bfcbf5
Merge remote-tracking branch 'origin/master' into inv-index-cleanup 
src/Interpreters/GinFilter.cpp
	src/Interpreters/InterpreterCreateQuery.cpp
	src/Storages/MergeTree/MergeTreeData.cpp
	src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp
	src/Storages/MergeTree/MergeTreeIndexInverted.cpp
 2023-01-24 10:09:42 +00:00
Alexey Milovidov
dda6a69f9a
Merge branch 'master' into tavplubix-patch-1 2023-01-24 13:04:25 +03:00
Alexey Milovidov
54bc501d39
Update StorageReplicatedMergeTree.cpp 2023-01-24 13:03:28 +03:00
Anton Popov
8a1ea62aec fix race on cancelation of query in StorageS3 2023-01-24 01:12:01 +00:00
Alexander Tokmakov
414693feb2 fixes 2023-01-24 00:46:03 +01:00
Alexander Tokmakov
bb4c8e169f check number of parameters in format string 2023-01-23 23:16:16 +01:00
Anton Popov
b987ed5037
Merge pull request #45499 from CurtizJ/add-fields-system-formats 
Add fields to table `system.formats`
 2023-01-23 23:08:35 +01:00
Alexander Tokmakov
3f6594f4c6 forbid old ctor of Exception 2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Robert Schulze
7adec3d8a4
Update src/Storages/MergeTree/MergeTreeIndexInverted.cpp 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-01-23 20:53:25 +01:00
Robert Schulze
d8af5bc2ef
Update src/Storages/MergeTree/MergeTreeIndexInverted.cpp 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2023-01-23 20:53:17 +01:00
Robert Schulze
0ab7ae6c17
Merge pull request #43797 from ClickHouse/query-result-cache 
Query result cache [experimental]
 2023-01-23 19:54:35 +01:00
Maksim Kita
7b48c75e82
Merge pull request #45485 from kitaisreal/remove-unnecessary-get-total-row-count-function-calls 
Remove unnecessary getTotalRowCount function calls
 2023-01-23 21:02:51 +03:00
Anton Popov
08b2e524c7 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-23 15:34:53 +00:00
Kseniia Sumarokova
8ec39ab6f3
Merge pull request #45442 from kssenii/minor-changes 
Minor improvements around reading from remote
 2023-01-23 15:46:30 +01:00
Alexander Tokmakov
a182a6b851
Merge pull request #45493 from azat/fix-detach 
Fix possible in-use table after DETACH
 2023-01-23 17:36:45 +03:00
Aleksandr
2caeed901b
Merge branch 'master' into fix_backup_restore_num_files 2023-01-23 13:43:20 +01:00
Sergei Trifonov
0fbfa17863
Merge branch 'master' into cancellable-mutex-integration 2023-01-23 12:44:09 +01:00
Azat Khuzhin
51019bc9f3 Fix a race between Distributed table creation and INSERT into it 
Initializing queues for pending on-disk files for async INSERT cannot be
done after table had been attached and visible to user, since it
initializes the per-table counter, that is used during INSERT.

Now there is a window, when this counter is not initialized and it will
start from the beginning, and this could lead to CANNOT_LINK error:

    Destination file /data/clickhouse/data/urls_v1/urls_in/shard6_replica1/13129817.bin is already exist and have different inode

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-23 09:55:43 +01:00
Anton Popov
8a39337c7c add fields to table system.formats 2023-01-23 02:04:31 +00:00
Robert Schulze
340f406553
Merge branch 'master' into query-result-cache 2023-01-22 13:21:36 +01:00
Robert Schulze
531e64dc39
Merge branch 'master' into inv-index-cleanup 2023-01-22 13:20:38 +01:00
Azat Khuzhin
ca8af3d18c Fix possible in-use table after DETACH 
Right now in case of DETACH/ATTACH there can be a window when after the
table had been DETACH'ed someone will still use it, the common example
here is MVs handling.

It happens because TableExclusiveLockHolder does not guards the
shard_ptr of the IStorage, and so if someone holds it, then it can use
it. So if ATTACH will be done for this table then, you can have multiple
instances of it.

It is not possible for DROP, because before using a table, you should
lock it and after table had been DROP'ed you cannot lock it anymore.

So let's do the same for DETACH.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-22 13:18:56 +01:00
Alexander Gololobov
91fb1eab8c
Merge pull request #45373 from ClickHouse/throttler-metrics 
Add detailed profile events for throttling
 2023-01-22 11:20:45 +01:00
Alexander Tokmakov
1174eaa132
Merge pull request #45492 from azat/revert/dist/async-INSERT-metrics 
Revert "Merge pull request #44922 from azat/dist/async-INSERT-metrics"
 2023-01-22 00:45:10 +03:00
Azat Khuzhin
a55798626a Revert "Merge pull request #44922 from azat/dist/async-INSERT-metrics" 
There are the following problems with this patch:
- Looses files on exception
- Existing current_batch.txt on startup leads to ENOENT error and hung
  of distributed sends without ATTACH/DETACH
- Race between creating the queue for sending at table startup and
  INSERT, if it had been created from INSERT, then it will not be
  initialized from disk

They were addressed in #45491, but it makes code more cmoplex and plus
since, likely, the release is comming, it is better to revert the
change.

This reverts commit 94604f71b7, reversing
changes made to 80f6a45376.
 2023-01-21 22:42:00 +01:00
Maksim Kita
19f1bae5ed
Merge pull request #45254 from kitaisreal/planner-small-fixes 
Planner small fixes
 2023-01-21 19:54:17 +03:00
Nikolai Kochetov
b877c484d2
Merge pull request #45481 from ClickHouse/fix-deadlock-with-allow_asynchronous_read_from_io_pool_for_merge_tree 
Fix possible deadlock with allow_asynchronous_read_from_io_pool_for_merge_tree in case of exception from ThreadPool::schedule
 2023-01-21 12:05:34 +01:00
Sema Checherinda
962894afc8
Merge pull request #44909 from CheSema/intersect-prev-part 
Do not merge over a gap with outdated undeleted parts
 2023-01-21 11:51:21 +01:00
Maksim Kita
47385a19e7 Remove unnecessary getTotalRowCount function calls 2023-01-21 11:27:07 +01:00
avogar
eed1db7e07 Fix schema inference in hdfsCluster 2023-01-20 21:17:35 +00:00
Anton Popov
41a199e175
Fix crash when ListObjects request fails (#45371) 2023-01-20 20:10:23 +01:00
Nikolai Kochetov
dcd84c152a Fix possible deadlock with allow_asynchronous_read_from_io_pool_for_merge_tree in case of exception from ThreadPool::schedule 2023-01-20 18:57:47 +00:00
Alexander Tokmakov
910d6dc0ce
Merge pull request #45342 from ClickHouse/exception_message_patterns 
Save message format strings for DB::Exception
 2023-01-20 18:46:52 +03:00
Sema Checherinda
b76b612d23
fix typo 2023-01-20 14:55:58 +01:00
Robert Schulze
1a966a9590
Fix bad comparison 2023-01-20 13:05:06 +00:00
Sema Checherinda
02f22f04e8
fix typos 2023-01-20 13:35:23 +01:00
kssenii
8d20af8127 Fix 2023-01-20 13:34:23 +01:00
Robert Schulze
bfc3b4f5ca
Suffix "GinFilter" --> "Inverted" 2023-01-20 12:02:35 +00:00
Robert Schulze
0738b2499c
Use GinFilters typedef where possible 2023-01-20 11:52:04 +00:00
Robert Schulze
0b77f07f67
Remove superfluous check (the same is checked in MergeTreeIndices.cpp) 2023-01-20 11:50:35 +00:00
Robert Schulze
d2c830ec39
Cosmetics 2023-01-20 11:49:08 +00:00
Robert Schulze
72973076c9
Rename MergeTreeIndexGin.h/cpp to MergeTreeIndexInverted.h/cpp 2023-01-20 11:42:36 +00:00
Robert Schulze
1ef2704539
Cosmetics 2023-01-20 11:39:23 +00:00
Robert Schulze
463cc843de
"segment file" --> "segment metadata file" 2023-01-20 11:26:22 +00:00
Robert Schulze
58df3953bb
Move some code around (no other changes) 2023-01-20 11:24:23 +00:00
Robert Schulze
3267ac2787
Prefix more typedefs in DB namespace with "Gin" 2023-01-20 11:19:07 +00:00
Robert Schulze
919b67f117
Cosmetics 2023-01-20 11:15:28 +00:00
Sema Checherinda
09f3a5c599 add a comment, add a check, fix test 2023-01-20 12:10:31 +01:00
Robert Schulze
98e117dca6
SegmentDictionary --> GinSegmentDictionary, also move typedef 2023-01-20 11:09:49 +00:00
Robert Schulze
908fa83f72
Move some typedefs around 2023-01-20 11:08:19 +00:00
Robert Schulze
44618927f9
Inline two short methods + uppercase 2023-01-20 11:04:35 +00:00
Robert Schulze
5c3cc5283f
"term dictionary" --> "dictionary" 2023-01-20 10:53:41 +00:00
Robert Schulze
be936b257c
Make version enum private 2023-01-20 10:48:43 +00:00
Robert Schulze
0653f86de9
Various cosmetic cleanups 2023-01-20 10:45:35 +00:00
Maksim Kita
23e26032ca
Merge pull request #45399 from aalexfvk/alexfvk/mdb-21326_fix_system_dictionaries_when_dictionary_with_bad_structure 
Fix select from system.dictionaries when there is dictionary with bad structure
 2023-01-20 13:36:32 +03:00
Aleksandr Musorin
838acb22b7
added num_processed_files and processed_files_size 2023-01-20 10:20:41 +01:00
Frank Chen
f867c4c206 Add an overriden fromCurrentException to simply the code 2023-01-20 16:43:10 +08:00
Frank Chen
38799eb852 Merge two methods as one 2023-01-20 16:35:28 +08:00
Frank Chen
be8ceeb4bb Save stack trace in part_log 2023-01-20 16:22:40 +08:00
Frank Chen
1fe9bde594 Set replica name to zero copy exclusive lock 2023-01-20 15:48:33 +08:00
Frank Chen
fb8452aa49 Add last_exception_time to replication_queue 2023-01-20 15:20:57 +08:00
Maksim Kita
2c56b0b2b9 Planner small fixes 2023-01-19 19:05:49 +01:00
kssenii
4ce8950712 Minor changes 2023-01-19 17:53:10 +01:00
larryluogit
52ae33dba7
Merge branch 'master' into ftsearch 2023-01-19 11:34:11 -05:00
Aleksei Filatov
afada0ecb3 Fix review notes 2023-01-19 17:02:57 +03:00
Alexander Tokmakov
7bb65cc002
Update StorageReplicatedMergeTree.cpp 2023-01-19 16:45:41 +03:00
Han Fei
f661dad0e9
Merge pull request #45106 from hanfei1991/hanfei/async-cache 
support cache for async inserts block ids
 2023-01-19 10:59:25 +01:00
Ilya Yatsishin
d16b59b662
Merge pull request #45422 from Avogar/fix-s3-cluser-si 2023-01-19 10:36:54 +01:00
avogar
117ec13c9e Fix s3Cluster schema inference when structure from insertion table is used 2023-01-18 20:33:50 +00:00
Aleksei Filatov
5e9340f682 Add integration test 2023-01-18 17:50:38 +03:00
Aleksei Filatov
7f4a01b903 Add handling of bad dictionary structure 2023-01-18 17:27:03 +03:00
Sema Checherinda
ae1dfb9ce5
Update src/Storages/MergeTree/MergeTreeData.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-01-18 15:21:11 +01:00
Sema Checherinda
a344b526a6
Update src/Storages/StorageMergeTree.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-01-18 15:16:18 +01:00
Antonio Andelic
8f8b14148a
Merge pull request #45215 from ClickHouse/fix-crash-kv-store 
Fix crash when prepared set with different type used in KV stores
 2023-01-18 13:27:40 +01:00
Vitaly Baranov
7cdb2c4c7f
Merge pull request #45351 from vitlibar/fix-backup-with-killed-mutations 
Fix backup with killed mutations
 2023-01-18 13:14:27 +01:00
Han Fei
e51123c9b0 fix data race 2023-01-18 13:11:07 +01:00
Antonio Andelic
cfba9b19eb
Merge pull request #45360 from azat/dist/fix-startup-race 
Fix race in Distributed table startup
 2023-01-18 10:09:54 +01:00
Antonio Andelic
f57ee043ae
Merge pull request #45319 from ClickHouse/disable-prewhere-in-merge-different-types 
Disable PREWHERE in storage Merge if types don't match
 2023-01-18 10:02:06 +01:00
Antonio Andelic
f3469ee077
Merge branch 'master' into dist/fix-startup-race 2023-01-18 09:44:52 +01:00
Sergei Trifonov
c443c1ece0
Merge branch 'master' into hanfei/async-cache 2023-01-18 00:19:49 +01:00
Robert Schulze
4f90824347
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-17 22:49:53 +00:00
Anton Popov
f40fd7a151
Add checks for compilation of regexps (#45356) 2023-01-17 23:46:04 +01:00
Alexander Tokmakov
5cd90c1a3e Merge branch 'master' into exception_message_patterns 2023-01-17 20:04:04 +01:00
Alexander Tokmakov
72e8615bec formatting of some exception messages 2023-01-17 20:03:56 +01:00
serxa
ce7e22b87b add detailed profile events for throttling 2023-01-17 18:29:24 +00:00
alesapin
e732f510f0
Merge branch 'master' into fix_hang_during_drop_in_zero_copy_replication 2023-01-17 19:24:36 +01:00
Alexander Tokmakov
8b13b85ea0
Merge pull request #44543 from ClickHouse/text_log_add_pattern 
Add a column with a message pattern to text_log
 2023-01-17 20:19:32 +03:00
alesapin
69925647eb Fix style 2023-01-17 15:59:55 +01:00
alesapin
f6131101bb Fix no shared id during drop for the fourth time 2023-01-17 15:51:49 +01:00
Han Fei
8a74238fe0 improve 2023-01-17 15:47:52 +01:00
Kruglov Pavel
582aa8b770
Merge pull request #45253 from Avogar/fix-s3-heap-use-after-free 
Fix heap-use-after-free in reading from s3
 2023-01-17 15:32:26 +01:00
HarryLeeIBM
e7add8218f Addressed more review comments and ClangTidy errors 2023-01-17 06:29:13 -08:00
Azat Khuzhin
54fc6859ae Fix race in Distributed table startup 
Before this patch it was possible to have multiple directory monitors
for the same directory, one from the INSERT context, another one on
storage startup().

Here are an example of logs for this scenario:

    2022.12.07 12:12:27.552485 [ 39925 ] {a47fcb32-4f44-4dbd-94fe-0070d4ea0f6b} <Debug> DDLWorker: Executed query: DETACH TABLE inc.dist_urls_in
    ...
    2022.12.07 12:12:33.228449 [ 4408 ] {20c761d3-a46d-417b-9fcd-89a8919dd1fe} <Debug> executeQuery: (from 0.0.0.0:0, user: ) /* ddl_entry=query-0000089229 */ ATTACH TABLE inc.dist_urls_in (stage: Complete)
    ... this is the DirectoryMonitor created from the context of INSERT for the old StoragePtr that had not been destroyed yet (becase of "was 1" this can be done only from the context of INSERT) ...
    2022.12.07 12:12:35.556048 [ 39536 ] {} <Trace> inc.dist_urls_in.DirectoryMonitor: Files set to 173 (was 1)
    2022.12.07 12:12:35.556078 [ 39536 ] {} <Trace> inc.dist_urls_in.DirectoryMonitor: Bytes set to 29750181 (was 71004)
    2022.12.07 12:12:35.562716 [ 39536 ] {} <Trace> Connection (i13.ch:9000): Connected to ClickHouse server version 22.10.1.
    2022.12.07 12:12:35.562750 [ 39536 ] {} <Debug> inc.dist_urls_in.DirectoryMonitor: Sending a batch of 10 files to i13.ch:9000 (0.00 rows, 0.00 B bytes).
    ... this is the DirectoryMonitor that created during ATTACH ...
    2022.12.07 12:12:35.802080 [ 39265 ] {} <Trace> inc.dist_urls_in.DirectoryMonitor: Files set to 173 (was 0)
    2022.12.07 12:12:35.802107 [ 39265 ] {} <Trace> inc.dist_urls_in.DirectoryMonitor: Bytes set to 29750181 (was 0)
    2022.12.07 12:12:35.834216 [ 39265 ] {} <Debug> inc.dist_urls_in.DirectoryMonitor: Sending a batch of 10 files to i13.ch:9000 (0.00 rows, 0.00 B bytes).
    ...
    2022.12.07 12:12:38.532627 [ 39536 ] {} <Trace> inc.dist_urls_in.DirectoryMonitor: Sent a batch of 10 files (took 2976 ms).
    ...
    2022.12.07 12:12:38.601051 [ 39265 ] {} <Error> inc.dist_urls_in.DirectoryMonitor: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in file_size: No such file or directory ["/data6/clickhouse/data/inc/dist_urls_in/shard13_replica1/66827403.bin"], Stack trace (when copying this message, always include the lines below):
    ...
    2022.12.07 12:12:54.132837 [ 4408 ] {20c761d3-a46d-417b-9fcd-89a8919dd1fe} <Debug> DDLWorker: Executed query: ATTACH TABLE inc.dist_urls_in

And eventually both monitors (for a short period of time, one replaces
another) are trying to process the same batch (current_batch.txt), and
one of them fails because such file had been already removed.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-17 14:51:00 +01:00
Vitaly Baranov
692065e5fe Fix backup if mutations got killed during the backup process. 2023-01-17 11:05:34 +01:00
Vitaly Baranov
b13498d9ba
Merge pull request #45288 from vitlibar/fix-s3-requests-without-region 
Fix s3 requests without region
 2023-01-17 09:24:59 +01:00
Antonio Andelic
76eb3e3b3c Fix test 2023-01-17 07:34:39 +00:00
Alexander Tokmakov
522686f78b less empty patterns 2023-01-17 01:19:44 +01:00
Alexander Tokmakov
870cfcc36a less fmt::runtime usages 2023-01-17 00:11:59 +01:00
Alexander Tokmakov
e7899825e6 save format strings for DB::Exceptions 2023-01-16 23:20:33 +01:00
Sema Checherinda
dbe89cd5d8 fix that optimize final waits for currently running merges 2023-01-16 16:47:12 +01:00
Sema Checherinda
90fa1ecd49 make that old_parts_lifetime=0 deletes files instantly at drop/truncate 2023-01-16 16:47:12 +01:00
Sema Checherinda
8f660afab3 style fix 2023-01-16 16:47:12 +01:00
Sema Checherinda
c51f4d7be1 do not merge over a gap with uotdate parts, delete empty parts with respect to old_parts_lifetime 2023-01-16 16:47:11 +01:00
Sema Checherinda
25e16388d7 better message in MergeTreeDataMergerMutator when parts intersect 2023-01-16 16:47:11 +01:00
Kruglov Pavel
04d95f4877
Fix 2023-01-16 16:47:04 +01:00
Antonio Andelic
108b2384e7 Disable prewhere in storage merge if types don't match 2023-01-16 13:39:46 +00:00
Anton Popov
6863cd152f
Merge pull request #42181 from CurtizJ/optimize-loading-parts 
Do not load inactive parts at startup
 2023-01-16 14:38:50 +01:00
Han Fei
30a798182a Merge branch 'master' into hanfei/async-cache 2023-01-16 14:07:36 +01:00
alesapin
190c9b3156
Merge pull request #44682 from hanfei1991/hanfei/support-advance-dedup 
deduplicate async inserts in the same block earlier
 2023-01-16 12:19:30 +01:00
Alexander Tokmakov
94604f71b7
Merge pull request #44922 from azat/dist/async-INSERT-metrics 
Optimize and fix metrics for Distributed async INSERT
 2023-01-16 14:12:56 +03:00
Han Fei
3481b4d50a fix style 2023-01-16 10:41:35 +01:00
Robert Schulze
099e30ef2a
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-16 08:04:49 +00:00
Han Fei
5617f7f616 address comments 2023-01-15 22:51:10 +01:00
Vitaly Baranov
a955504043 Move throw_on_error parameter to the end. 2023-01-15 20:28:16 +01:00
Han Fei
701dc88d6f
Merge branch 'master' into hanfei/support-advance-dedup 2023-01-15 19:46:28 +01:00
Han Fei
c859f8dbe5
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2023-01-15 19:46:16 +01:00
Han Fei
bb2c0914e9
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2023-01-15 19:46:09 +01:00
Alexander Tokmakov
d857d62a03 remove another set of macros 2023-01-13 20:34:31 +01:00
Alexander Tokmakov
2d7773fccc Merge branch 'master' into text_log_add_pattern 2023-01-13 20:33:46 +01:00
Han Fei
ed49ebf01a update setting explain 2023-01-13 20:26:08 +01:00
Han Fei
2fb2f503e3 Update src/Storages/MergeTree/MergeTreeSettings.h 
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
 2023-01-13 20:20:08 +01:00
Han Fei
bcf813fedc Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
 2023-01-13 20:19:30 +01:00
Han Fei
9e99c7e116 Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
 2023-01-13 20:19:13 +01:00
Han Fei
a258a39eb1 Merge branch 'master' into hanfei/async-cache 2023-01-13 20:17:58 +01:00
Anton Popov
487de70d01 fix locking at loading outdated data parts 2023-01-13 17:05:32 +00:00
avogar
6cb7c4d175 Better commit, mark noexcept 2023-01-13 16:33:11 +00:00
avogar
76c89c6d20 Fix heap-use-after-free in reading from s3 2023-01-13 16:31:30 +00:00
Alexander Tokmakov
9d5ec474a3
Merge pull request #43998 from evillique/make_system_replicas_parallel 
Make `system.replicas` parallel
 2023-01-13 16:33:36 +03:00
Anton Popov
71188c22ee fix race on 'relative_data_path' 2023-01-13 12:19:41 +00:00
Alexander Tokmakov
51d94314d6
Merge pull request #45235 from ClickHouse/more_verbose_logs_about_replication_log_entries 
More verbose logs about replication log entries
 2023-01-13 15:05:21 +03:00
Maksim Kita
44f4184e11
Merge pull request #44540 from kitaisreal/analyzer-support-distributed 
Analyzer support distributed queries processing
 2023-01-13 14:45:36 +03:00
Antonio Andelic
99548c8c15 Merge branch 'master' into fix-crash-kv-store 2023-01-13 08:42:08 +00:00
Robert Schulze
0c528fca29
Introduce SourceFromChunks to avoid unnecessary partial result chunk concatenation 2023-01-12 23:01:00 +00:00
Raúl Marín
9f121d5d9c
Fix background_fetches_pool_size config reload (#45189) 2023-01-12 23:41:25 +01:00
Alexander Tokmakov
d9b8cf9c04 more verbose logs about replication log entries 2023-01-12 20:32:39 +01:00
serxa
693489a8ad review fixes 2023-01-12 15:51:04 +00:00
Sergei Trifonov
12d8543578
Merge branch 'master' into cancellable-mutex-integration 2023-01-12 16:03:49 +01:00
Kseniia Sumarokova
db3e0219fc
Merge pull request #41687 from ClickHouse/40907_Parameterized_views_as_table_functions 
40907 Parameterized views as table functions
 2023-01-12 14:24:32 +01:00
Ilya Yatsishin
e2a42f0079
Merge pull request #45212 from ucasfl/minor 
remove unused function
 2023-01-12 14:12:06 +01:00
Anton Popov
ca80240d84 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-12 12:13:45 +00:00
Kseniia Sumarokova
d6289c3a0d
Merge pull request #45181 from kssenii/fix-heavy-calculation-in-system-detached-parts 
Calculate only required columns in system.detached_parts
 2023-01-12 13:02:56 +01:00
alesapin
df8ceeefdc
Merge pull request #45134 from ClickHouse/fix_flaky_azure_test 
Fix flaky azure test
 2023-01-12 13:02:34 +01:00
Anton Popov
8bdf63f8e5
Merge pull request #45191 from ClickHouse/fix-ip-part-name 
Fix part ID generation for IP types for backward compatibility
 2023-01-12 12:15:06 +01:00
Maksim Kita
43a0996356 Fixed tests 2023-01-12 12:07:58 +01:00
Maksim Kita
a140d6c5b1 Fixed code review issues 2023-01-12 12:07:58 +01:00
Maksim Kita
47f4159909 Analyzer support distributed queries processing 2023-01-12 12:07:58 +01:00
Vladimir C
ac5ee471c7
Merge pull request #45184 from ClickHouse/vdimir/storage_join_null_44940 2023-01-12 11:51:11 +01:00
Antonio Andelic
60896170bc Check type of prepared set 2023-01-12 10:44:55 +00:00
Igor Nikonov
edbcb24da0
Merge pull request #44954 from ClickHouse/fix-insert-delay 
Fix: insert delay calculation
 2023-01-12 09:43:00 +01:00
flynn
68f747ed32 remove unused function 2023-01-12 06:48:24 +00:00
Alexey Milovidov
7f642f7e10
Merge pull request #45057 from kssenii/filelog-fix-possible-cannot-read-all-data 
Fix possible cannot-read-all-data in storage FileLog
 2023-01-12 05:50:56 +03:00
HarryLeeIBM
6d576cf157 Added more documentation 2023-01-11 13:40:20 -08:00
Kruglov Pavel
29240ef380
Merge pull request #43927 from pufit/mmap-for-storage-file 
Added mmap for StorageFile
 2023-01-11 21:25:02 +01:00
Yakov Olkhovskiy
9c4ea5a16b fix part ID generation for IP types for backward compatibility 2023-01-11 18:51:37 +00:00
vdimir
3e9d142066
Fix wrong column nullability in StoreageJoin 2023-01-11 16:17:21 +00:00
kssenii
6499e8e687 Calculate only required column in system.detached_parts 2023-01-11 16:45:38 +01:00
alesapin
fe8f373aa6 Less stupid tests 2023-01-11 15:20:18 +01:00
Alexander Tokmakov
0fba3d6d81
Merge pull request #44468 from ClickHouse/fix_get_part_name 
Fix incorrect usages of `getPartName()`
 2023-01-11 16:24:02 +03:00
Igor Nikonov
1d002e45a9 Fix test & review comments 2023-01-11 12:49:01 +00:00
Nikolai Kochetov
5e7a6ac619
Merge pull request #45122 from ClickHouse/revert-45121-revert-44653-custom-reading-for-mutation 
Revert "Revert "Custom reading for mutation""
 2023-01-11 12:37:32 +01:00
Nikolai Kochetov
d0c1079242
Merge pull request #45133 from ClickHouse/fix-additinal-tables-filter-with-count-projection 
Fix additional_table_filters with minmax/count projection.
 2023-01-11 12:25:40 +01:00
Maksim Kita
f0567f7a25
Merge pull request #45055 from kitaisreal/analyzer-aggregation-without-column-fix 
Analyzer aggregation without column fix
 2023-01-11 12:03:32 +03:00
Anton Popov
85734bb321 fix build 2023-01-11 08:32:33 +00:00
Anton Popov
6c4f53bdca Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-11 01:00:57 +00:00
HarryLeeIBM
150f9b48a4 Fix functional test for full text search 2023-01-10 14:35:06 -08:00
Igor Nikonov
543221b8b9 Merge remote-tracking branch 'origin/master' into fix-insert-delay 2023-01-10 22:15:41 +00:00
Igor Nikonov
e75df5e76b Test 
+ fix some review comments
 2023-01-10 22:14:54 +00:00
Alexander Tokmakov
881b17492f Merge branch 'master' into fix_get_part_name 2023-01-10 21:39:35 +01:00
Alexander Tokmakov
255881a2ff
Merge pull request #44466 from ClickHouse/parallel_parts_cleanup_with_zero_copy 
Parallel parts cleanup with zero copy replication
 2023-01-10 22:44:39 +03:00
Ilya Yatsishin
0efceae6ce
Merge pull request #44917 from azat/mt-fix-log-message 
Fix logging message in MergeTreeDataMergerMutator (about merged parts)
 2023-01-10 19:49:31 +01:00
Nikolai Kochetov
9a81f27fb2 Fix additional_table_filters with minmax/count projection. 2023-01-10 17:32:26 +00:00
Alexander Tokmakov
5ff4c6228a
Merge pull request #43840 from Algunenano/system_tables_nolock 
Do not acquire read locks in system.tables if possible
 2023-01-10 20:26:21 +03:00
Kruglov Pavel
ce6962614d
Merge branch 'master' into mmap-for-storage-file 2023-01-10 17:34:01 +01:00
HarryLeeIBM
bc48b17817 Addressed more review comments. 2023-01-10 08:26:27 -08:00
Maksim Kita
4571c74fdd Fixed build 2023-01-10 16:49:55 +01:00
Maksim Kita
fbba28b31e Analyzer aggregation without column fix 2023-01-10 16:49:55 +01:00
Nikolai Kochetov
4673b3fe1d
Revert "Revert "Custom reading for mutation"" 2023-01-10 16:31:01 +01:00
Alexander Tokmakov
c8ec130be4
Revert "Custom reading for mutation" 2023-01-10 17:51:30 +03:00
Han Fei
bef6168c05 support cache for async inserts block ids 2023-01-10 13:19:12 +01:00
Anton Popov
b9bf92fa5d check for network errors while loading parts 2023-01-10 11:55:08 +00:00
Nikolai Kochetov
11418963c0
Merge pull request #44653 from ClickHouse/custom-reading-for-mutation 
Custom reading for mutation
 2023-01-10 12:16:24 +01:00
alesapin
616e37ba43
Merge pull request #44847 from ClickHouse/fix_rare_race 
Fix rare race which can lead to queue hang
 2023-01-10 12:09:10 +01:00
Robert Schulze
923fa2c15a
Fix review comments, pt. II 2023-01-10 10:21:08 +00:00
kssenii
613ff3387d Better 2023-01-10 11:16:20 +01:00
Anton Popov
79e89cf69c
Merge pull request #44939 from ClickHouse/revert-44493-s3_optimize 
Revert "If user only need virtual columns, we don't need to initialize ReadBufferFromS3"
 2023-01-10 10:42:18 +01:00
Robert Schulze
0c3b034887
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-10 09:02:41 +00:00
Robert Schulze
ad4884b394
Fix review comments, pt. I 2023-01-09 21:42:51 +00:00
Nikolai Kochetov
3e430b1b6e Add comments. Remove unused flag from Context. 2023-01-09 19:52:11 +00:00
Alexander Gololobov
742887ce53
Merge pull request #44718 from ClickHouse/lwd_mutation_always_sync 
Make lightweight deletes always synchronous
 2023-01-09 15:58:02 +01:00
Anton Popov
1e5d6e44a5 fix deadlock at shutdown 2023-01-09 13:07:46 +00:00
Kseniia Sumarokova
b22b51213f
Merge pull request #44868 from kssenii/improve-async-buffer-from-remote 
Small perf improvement for async buffer from remote
 2023-01-09 13:31:15 +01:00
Nikita Mikhaylov
857799fbca
Parallel distributed insert select with s3Cluster [3] (#44955) 
* Revert "Revert "Resurrect parallel distributed insert select with s3Cluster (#41535)""

This reverts commit b8d9066004.

* Fix build

* Better

* Fix test

* Automatic style fix

Co-authored-by: robot-clickhouse <robot-clickhouse@users.noreply.github.com>
 2023-01-09 13:30:32 +01:00
Kseniia Sumarokova
9ae5ad202a
Merge pull request #44707 from xiedeyantu/s3-add-config 
S3 support upload object using storage class INTELLIGENT_TIERING
 2023-01-09 13:28:53 +01:00
kssenii
7764fd9ac9 Fix possible cannot-read-all-data 2023-01-09 12:47:55 +01:00
Vladimir C
75e9a45341
Merge pull request #44716 from ClickHouse/vdimir/merge_engine_nullable 2023-01-09 11:38:04 +01:00
Sergei Trifonov
81d2ea30ba
Merge branch 'master' into cancellable-mutex-integration 2023-01-07 19:37:46 +01:00
Anton Popov
1f32ffedf8
Merge pull request #43221 from ClickHouse/refactoring-ip-types 
Replace domain IP types (IPv4, IPv6) with native
 2023-01-07 12:01:21 +01:00
Alexey Milovidov
9fcd111724
Merge pull request #45003 from amosbird/issue_44979 
Fix minmax_count_projection with _partition_value
 2023-01-07 03:36:56 +03:00
serxa
15bb127b01 replace every std::shared_mutex with DB::FastSharedMutex 2023-01-06 23:35:38 +00:00
Amos Bird
200d40c53d
Fix minmax_count_projection with _partition_value 
Check if virtual column `_partition_value` can be used when building
minmax_count_projection block.
 2023-01-07 05:38:28 +08:00