thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 19:32:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
89,665 Commits 9,319 Branches 1,783 Tags 2.1 GiB
ec4e8d71b2
Commit Graph

8085 Commits

Author SHA1 Message Date
alesapin
a97754f462 Fix 2022-04-18 13:39:09 +02:00
alesapin
c4b70f3ed3 Merge branch 'master' into fix_trash 2022-04-18 12:20:09 +02:00
zhanglistar
5670dcf750 Fix crash in debug mode of ReadBufferFromHDFS. 2022-04-18 17:52:59 +08:00
zhanglistar
355333b361
Merge branch 'ClickHouse:master' into fixhdfsreadbuffer 2022-04-18 16:42:03 +08:00
Memo
b3adf150b5 add_query_level_settings 2022-04-18 12:15:41 +08:00
Alexey Milovidov
f6ab2bd523
Merge pull request #36312 from ClickHouse/remove-arcadia 
Remove remaining parts of Arcadia
 2022-04-18 07:02:54 +03:00
Alexey Milovidov
36595e4206
Merge pull request #36320 from ClickHouse/fix-clang-tidy-14 
Fix clang-tidy-14 (part 1)
 2022-04-18 07:02:10 +03:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
Robert Schulze
1e1df8e101
Replace remove-erase idiom by C++20 erase()/erase_if() 
- makes the code less verbose while being as efficient
 2022-04-17 12:04:47 +02:00
Alexey Milovidov
294efeccfe Fix clang-tidy-14 (part 1) 2022-04-16 04:54:04 +02:00
Anton Popov
2de6668b3f show names of erroneous files 2022-04-16 00:10:47 +00:00
Alexey Milovidov
cbeeb7ec4f Remove Arcadia 2022-04-16 00:20:47 +02:00
alesapin
1706ae9e15 Some trash implementation 2022-04-15 18:36:23 +02:00
alesapin
5a8419a48e Remove more trash 2022-04-15 17:05:17 +02:00
alesapin
eb7593f786 Remove more trash 2022-04-15 16:24:38 +02:00
zhanglistar
fe93fd83f5
Merge branch 'ClickHouse:master' into fixhdfsreadbuffer 2022-04-15 16:27:20 +08:00
zhanglistar
e22a175afb Fix ReadBufferFromHDFS crash in debug mode 2022-04-15 14:04:30 +08:00
Anton Popov
305dd57262
Merge branch 'master' into fix_storage_distributed_ttl 2022-04-14 14:51:15 +02:00
tavplubix
3af2384066
Merge pull request #36190 from azat/revert-27002-mutation-stuck 
Revert "Fix possible mutation stuck due to race with DROP_RANGE"
 2022-04-14 15:03:35 +03:00
alesapin
bd629f4cd0
Merge pull request #36163 from ClickHouse/maybe_fix_36098 
Some fixes for ReplicatedMergeTree
 2022-04-14 12:12:55 +02:00
tavplubix
fd23c42ea7
Update StorageReplicatedMergeTree.cpp 2022-04-13 22:33:55 +03:00
avogar
42726639f3 Check ORC/Parquet/Arrow format magic bytes before loading file in memory 2022-04-13 19:27:38 +00:00
avogar
1c065f8c7a Some refactoring around schema inference with globs 2022-04-13 17:02:48 +00:00
Alexander Tokmakov
66fdf35dfd remove outdated parts immediately on drop partition 2022-04-13 18:01:22 +02:00
Alexander Gololobov
fcb83a12ff
Merge pull request #35953 from azat/ttl-mutations 
Require mutations for per-table TTL only when it had been changed
 2022-04-13 17:14:04 +02:00
Alexander Tokmakov
40fea2b833 do not unfold {uuid} macro 2022-04-13 16:51:59 +02:00
kssenii
0912b038fa Merge master 2022-04-13 15:30:01 +02:00
Azat Khuzhin
6dbc138e76 Remove duplicated parts removal from mutation. 
This commit reverts 701e2ffd79 ("Fix
possible mutation stuck due to race with DROP_RANGE"), and also adds
some clarification comments.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-13 16:09:17 +03:00
Sergei Trifonov
9d1a2da685 merge master and fix conflicts 2022-04-13 14:53:36 +02:00
Mikhail Artemenko
2fd86cc564
Merge branch 'master' into MeiliSearch 2022-04-13 12:05:46 +03:00
Mikhail Artemenko
4ed6ec6780 change fast_test references 2022-04-13 10:35:37 +03:00
Julian Gilyadov
a4f56f3330
Throw exception when file cant be executed instead of displaying success 2022-04-12 17:52:44 +02:00
mergify[bot]
0b3c15c07a
Merge branch 'master' into projection-fix1 2022-04-12 13:49:28 +00:00
Alexander Tokmakov
457a9e9691 fixes for ReplicatedMergeTree 2022-04-12 14:14:26 +02:00
Vladimir C
8873d7b0e7
Merge pull request #36082 from bigo-sg/cache_hive_files 2022-04-12 11:42:08 +02:00
Kseniia Sumarokova
b29202ac88
Merge pull request #36150 from kssenii/fix-cache-test 
Fix test 02241_remote_filesystem_cache_on_insert for database Ordinary
 2022-04-12 11:08:44 +02:00
taiyang-li
9d31c44d39 fix bug of read buffer from hdfs 2022-04-12 12:01:51 +08:00
Kseniia Sumarokova
51dd39b42f
Update StorageSystemRemoteDataPaths.cpp 2022-04-11 22:46:51 +02:00
kssenii
fc62e95d76 Fix 2022-04-11 22:24:45 +02:00
Kruglov Pavel
972357ebf6
Merge pull request #35748 from bigo-sg/s3_cluster_join 
Fixed : elimate AMBIGUOUS_COLUMN_NAME exception in hdfs/s3 cluster join query
 2022-04-11 16:22:53 +02:00
Nikolai Kochetov
ac74757f92
Merge pull request #35973 from filimonov/kafka_read_consumers_in_parallel 
Make parallel reading from several kafka consumers work again
 2022-04-11 15:30:52 +02:00
Kseniia Sumarokova
beea4b103f
Merge pull request #36109 from ucasfl/window-view-fix 
Fix window view when is proc time and window kind larger than day
 2022-04-11 09:54:14 +02:00
taiyang-li
090fd72884 fix bug 2022-04-11 11:19:31 +08:00
Denny Crane
383076ca0e
fix typo 2022-04-10 16:42:43 -03:00
fenglv
cac2df914b fix 2022-04-10 17:28:17 +00:00
fenglv
24e86f81cb Fix window view when is proc time and window kind larger than day 
fix
 2022-04-10 17:26:09 +00:00
taiyang-li
7e89f760f3 remove useless code 2022-04-09 10:43:58 +08:00
tavplubix
faae540a85
Merge pull request #24258 from ClickHouse/mvcc_prototype 
MVCC prototype for MergeTree
 2022-04-08 22:02:13 +03:00
Kseniia Sumarokova
1e54c5c57b
Merge pull request #35885 from kssenii/fix-postgres-test-2 
fix postgres test
 2022-04-08 19:55:05 +02:00
taiyang-li
70f4503ba5 use global context for cache 2022-04-09 00:28:07 +08:00
taiyang-li
cd807da838 finish test 2022-04-09 00:15:33 +08:00
taiyang-li
e319df1799 finish dev 2022-04-08 23:58:56 +08:00
taiyang-li
2c99ef0ecc refactor HiveTableMetadata 2022-04-08 23:04:24 +08:00
Alexander Tokmakov
141fbc6980 Merge branch 'master' into mvcc_prototype 2022-04-08 13:38:11 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
Anton Popov
a16ac67c0c
Merge pull request #36034 from amosbird/better-projection2 
Use storage_snapshot for projection analysis
 2022-04-08 13:22:53 +02:00
Vitaly Baranov
45dea9933b
Merge pull request #35883 from azat/format-settings 
Do not allow SETTINGS after FORMAT for INSERT queries
 2022-04-08 12:38:14 +02:00
kssenii
87855c4ae7 Merge master 2022-04-08 12:32:48 +02:00
kssenii
ef2b238444 Fix 2022-04-08 12:30:24 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements 
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
 2022-04-08 12:06:26 +02:00
taiyang-li
2e6f0db825 first commit 2022-04-08 15:12:24 +08:00
taiyang-li
dc4c638af8 Merge branch 'master' into optimize_trivial_hive_query 2022-04-08 11:36:55 +08:00
lgbo-ustc
759e4a9a05 Merge remote-tracking branch 'ck/master' into s3_cluster_join 2022-04-08 09:25:30 +08:00
Alexander Tokmakov
6a46da93ae Merge branch 'master' into mvcc_prototype 2022-04-07 23:22:19 +02:00
kssenii
5dce2f18b5 Better 2022-04-07 18:48:35 +02:00
Kseniia Sumarokova
5f9bc1c68e
Merge pull request #35987 from awakeljw/fork_chmaster 
[Improvement] improvement in PARTITION ALL
 2022-04-07 17:36:35 +02:00
Sergei Trifonov
01425df758 fix build 2022-04-07 16:07:12 +02:00
Azat Khuzhin
33d99c8ffb Introduce compatiblity mode for SETTINGS after FORMAT 
Add allow_settings_after_format_in_insert setting, OFF by default.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
v2: s/parser_settings_after_format_compact/allow_settings_after_format_in_insert/ (suggested by vitlibar)
v3: replace ParserSettings with a flag (requested by vitlibar)
 2022-04-07 16:36:34 +03:00
Alexander Tokmakov
7f54e7b422 Merge branch 'master' into mvcc_prototype 2022-04-07 15:14:06 +02:00
taiyang-li
87507ec9e8 fix conflicts 2022-04-07 20:52:54 +08:00
taiyang-li
d7c79c3a54 merge master and solve conflicts 2022-04-07 20:48:16 +08:00
taiyang-li
e9de38c52b fix bug 2022-04-07 20:45:07 +08:00
Amos Bird
a071bde59e
Use storage_snapshot for projection analysis 2022-04-07 19:56:31 +08:00
Alexander Tokmakov
8290ffa88d Merge branch 'master' into mvcc_prototype 2022-04-07 13:50:42 +02:00
Nikolai Kochetov
a998d73ee2
Update StorageKafka.cpp 2022-04-07 12:37:11 +02:00
Kseniia Sumarokova
5909b3c04d
Merge pull request #36015 from ucasfl/filelog-fix 
Fix data race in StorgeFileLog
 2022-04-07 12:23:12 +02:00
Sergei Trifonov
755d5c55f9 resolve conflict 2022-04-07 11:57:45 +02:00
Amos Bird
1238babb6f
Make SelectQueryInfo pseudo-copyable 2022-04-07 17:46:50 +08:00
Amos Bird
9cf5935604
Fix broken SET reuse during projection analysis. 2022-04-07 17:46:49 +08:00
jewisliu
77edd41b2e [Improvement] improvement in PARTITION ALL 
1. ASTPartition::formatImpl should output ALL while executing ALTER TABLE t DETACH PARTITION ALL
2. prohibit PARTITION ALL excepte DETACH PARTITION ALL
 2022-04-07 17:37:01 +08:00
Sergei Trifonov
9c2804b924 merge from master + resolve conflicts 2022-04-07 11:22:01 +02:00
Mikhail Artemenko
0f1119dd2a fix impl after merge 2022-04-07 11:56:30 +03:00
taiyang-li
2dc420c66b rename some symbols in hivefile 2022-04-07 15:48:42 +08:00
taiyang-li
4763a39802 merge bigo-sg/use_minmax_index and solve conflict 2022-04-07 15:45:28 +08:00
taiyang-li
046a2ba51c rename some symboles 2022-04-07 15:35:08 +08:00
taiyang-li
ad074fee91 merge use_minmax_index and solve conflict 2022-04-07 15:19:45 +08:00
taiyang-li
f02d769343 fix build error 2022-04-07 14:29:35 +08:00
taiyang-li
acc7046d54 remove some useless virtual and rename some functions in HiveFile 2022-04-07 11:46:57 +08:00
taiyang-li
df00bd214d merge bigo-sg/use_minmax_index and solve conflict 2022-04-07 11:18:24 +08:00
taiyang-li
2ef316801c Merge branch 'master' into use_minmax_index 2022-04-07 10:53:25 +08:00
fenglv
517c2ae8df Fix data race in StorgeFileLog 2022-04-07 02:20:20 +00:00
tavplubix
58db8cd9b3
Merge pull request #35946 from ClickHouse/faster_replicated_ddl 
Make some replicated DDL queries faster
 2022-04-06 19:25:25 +03:00
Mikhail Artemenko
151eeb1a27
Merge branch 'master' into MeiliSearch 2022-04-06 17:07:55 +03:00
Kseniia Sumarokova
2aa3d32887
Merge pull request #35980 from ucasfl/thread_num 
parallel reading files for FileLog Engine
 2022-04-06 15:05:41 +02:00
Antonio Andelic
cda8a18fe3
Merge pull request #35984 from ClickHouse/fix-url-data-race 
Fix data race in StorageURL
 2022-04-06 14:36:19 +02:00
kssenii
846faa51d8 Fix bug 2022-04-06 14:35:19 +02:00
Alexander Tokmakov
1309e781b6 apply suggestion 2022-04-06 13:56:26 +02:00
taiyang-li
0b0c8ef09e add integration tests 2022-04-06 18:47:34 +08:00
Kseniia Sumarokova
a383d9ee6a
Merge pull request #35916 from filimonov/kafka_metrics 
Add some metrics to engine Kafka
 2022-04-06 11:58:17 +02:00
Kseniia Sumarokova
bb26903675
Merge pull request #35771 from bigo-sg/fix-filebuffer-pos 
fix filebuffer pos in RemoteReadBuffer
 2022-04-06 11:57:05 +02:00
taiyang-li
acb9f1632e suppoort skip splits in orc and parquet 2022-04-06 16:40:22 +08:00
Antonio Andelic
84eef61d17 Pull under reader mutex 2022-04-06 06:39:56 +00:00
fenglv
a2ce366c34 parallel reading files for FileLog Engine 2022-04-06 04:49:43 +00:00
taiyang-li
43e8af697a fix code style 2022-04-06 11:41:16 +08:00
Mikhail Filimonov
f9a4cd12d4
Make parallel reading of kafka consumers work again (issue #35153) 2022-04-05 21:46:56 +02:00
Mikhail Filimonov
3412be9d4d
Change KafkaDirectReads to KafkaConsumersInUse 2022-04-05 20:25:05 +02:00
Nikolai Kochetov
def33323a0
Merge pull request #35788 from amosbird/better-projection1 
Improve minmax_count_projection
 2022-04-05 19:59:45 +02:00
kssenii
cbd7f4acfb Fix 2022-04-05 19:24:10 +02:00
Alexander Tokmakov
37a06eec1a fixes 2022-04-05 17:36:53 +02:00
Alexander Tokmakov
d9e3e9b69f Merge branch 'master' into faster_replicated_ddl 2022-04-05 14:38:38 +02:00
Alexander Tokmakov
1fe50ad201 Merge branch 'master' into mvcc_prototype 2022-04-05 14:38:02 +02:00
Azat Khuzhin
8ebaf84989 Fix clang-tidy readability-simplify-boolean-expr warning in AlterCommands.cpp 
I want to make the code more readable before, but clang-tidy is too
smart...

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-05 13:39:39 +03:00
alesapin
eaa72ef002
Merge pull request #35904 from ClickHouse/more_logs_on_unsuccessful_part_removal 
More logs on unsuccessful part removal
 2022-04-05 11:29:08 +02:00
Alexander Tokmakov
da00beaf7f Merge branch 'master' into mvcc_prototype 2022-04-05 11:14:42 +02:00
Azat Khuzhin
c6f182163a Require mutations for per-table TTL only when it had been changed 
Before this patch only per-column TTL did not requires mutation if it
had not been changed, after per-table TTL will also check whether it had
been changed or not.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-05 11:08:35 +03:00
Amos Bird
5bc09550d8
Fix tests 2022-04-05 15:56:39 +08:00
Amos Bird
35a8bb2a9b
add comment 2022-04-05 15:56:38 +08:00
Amos Bird
163664fad7
Improve minmax_count_projection 2022-04-05 15:56:37 +08:00
Vladimir C
54b4049ea5
Merge pull request #35794 from awakeljw/fork_chmaster 2022-04-05 09:44:45 +02:00
Alexander Tokmakov
4e9ec5dc2f make some replicated DDL faster 2022-04-05 00:51:48 +02:00
Alexey Milovidov
1b562923f9
Merge pull request #35941 from ClickHouse/revert-35184-clang-tidy-issues 
Revert "clang-tidy report issues with Medium priority"
 2022-04-04 23:42:21 +03:00
alesapin
69fceb7e6d
Merge pull request #35926 from ClickHouse/better_consumers 
Better check for kafka_num_consumers
 2022-04-04 22:42:05 +02:00
Alexey Milovidov
4d6c030d23
Revert "clang-tidy report issues with Medium priority" 2022-04-04 23:41:42 +03:00
Mikhail Artemenko
49236c691e replace curl with Poco::Net 2022-04-04 23:27:39 +03:00
Mikhail Filimonov
53c7376e37
Add some metrics to engine Kafka 2022-04-04 20:25:53 +02:00
alesapin
d04c48e67a Fix build 2022-04-04 20:14:09 +02:00
alesapin
1e4fe5e3b5
Merge pull request #35882 from ClickHouse/more_metadata_alters 
Make more alters of nested types metadata-only
 2022-04-04 20:11:31 +02:00
alesapin
0477e74f42 Get rid of caps 2022-04-04 19:41:54 +02:00
alesapin
72331856eb fix message 2022-04-04 18:28:26 +02:00
alesapin
86f42e7a3a Better check for kafka_num_consumers 2022-04-04 18:07:36 +02:00
Anton Popov
bd93be65aa
Merge pull request #35820 from azat/columns-ttl-fix 
Avoid processing per-column TTL multiple times
 2022-04-04 16:42:50 +02:00
Anton Popov
d08d4a2437
Merge pull request #35806 from CurtizJ/dynamic-columns-8 
Fix inserts to columns of type `Object` in partitioned tables
 2022-04-04 16:16:54 +02:00
kssenii
4e50da722b Better s3 settings 2022-04-04 16:14:56 +02:00
Antonio Andelic
db75bf6f5d
Merge pull request #35852 from ClickHouse/bigo-sg-add_hints 
Refactoring of hints for column descriptor
 2022-04-04 15:37:08 +02:00
alesapin
3ccf99c3d7
Merge pull request #35884 from ClickHouse/don_t_allow_to_reset_settings 
Forbid to reset non existing settings
 2022-04-04 15:27:34 +02:00
Alexander Tokmakov
a2167f12b8 Merge branch 'master' into mvcc_prototype 2022-04-04 14:24:23 +02:00
Nikita Taranov
bd89fcafdb
Make SortDescription::column_name always non-empty (#35805) 2022-04-04 14:17:15 +02:00
alesapin
9b75ef6ce9 Fix build 2022-04-04 14:03:16 +02:00
alesapin
885447342c More logs on unsuccessful part removal 2022-04-04 13:17:33 +02:00
alesapin
803a1a2a9c Fix tests and check 2022-04-04 11:19:02 +02:00
Anton Popov
ae53aae106 fix clang-tidy 2022-04-04 08:48:31 +00:00
taiyang-li
38f149b533 optimize trivial count hive query 2022-04-04 15:28:26 +08:00
Antonio Andelic
a926bc19ea Address PR comments 2022-04-04 07:24:42 +00:00
Antonio Andelic
d96b682a55 Refactor 2022-04-04 07:24:42 +00:00
taiyang-li
6bc1786047 fix style 2022-04-04 07:24:42 +00:00
taiyang-li
d6247338de fix failed stateless tests 2022-04-04 07:24:42 +00:00
taiyang-li
fd9a10ef53 add hints for projections 2022-04-04 07:24:42 +00:00
taiyang-li
4547ed370a add hints for column description 2022-04-04 07:24:42 +00:00
taiyang-li
4e2d5f1841 Merge remote-tracking branch 'bigo-sg/use_minmax_index' into optimize_trivial_hive_query 2022-04-04 10:42:28 +08:00
taiyang-li
cbfc0f6bac fix typo 2022-04-04 10:42:22 +08:00
alesapin
a8c1ccdb20 Forbit to reset non existing settings 2022-04-03 18:29:24 +02:00
Kseniia Sumarokova
30c1afe29d
Merge pull request #35646 from bigo-sg/reduce-mutex-scope 
narrow mutex scope when setenv LIBHDFS3_CONF
 2022-04-03 17:59:24 +02:00
alesapin
7b35920d4c Make more alters of nested types metadata-only 2022-04-03 14:03:34 +02:00
alesapin
91453fe4d6
Merge pull request #35759 from chen9t/fix_dirty_metacache 
Clean dirty meta cache when part is broken during part loading phase
 2022-04-03 13:33:46 +02:00
Azat Khuzhin
195b4c47ea Avoid processing per-column TTL multiple times 
Before this patch ttl.txt will not be written for per-column TTLs, and
hence it will be calculated every time after server restart of
DETACH/ATTACH cycle (note, that it will work w/o restart since in-memory
representation will avoid this).

v2: convert test to .sh to get correct current database over default for MV
v3: extract UUID to avoid error like in [1]:
    [ 490 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) []: Code: 349. DB::Exception: Cannot convert NULL value to non-Nullable type: While processing query_id LIKE concat('%', CAST(_CAST(NULL, 'Nullable(UUID)') AS uuid, 'String'), '%'): while pushing to view test_0hc2ro.this_text_log (c64e5af4-059e-4330-a728-354ecf83c031). (CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN)
    [1]: https://s3.amazonaws.com/clickhouse-test-reports/35820/a512d322b024d37d2f1082c4833f59f86057555f/stateless_tests_flaky_check__address__actions_.html
v4: add no-parallel to avoid issues with disappeared underlying table while pushing to text_log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-02 15:19:39 +03:00
shuchaome
a565a93740 reduce mutex scope when setenv LIBHDFS3_CONF 2022-04-02 06:52:39 +08:00
kssenii
6c8e073a61 Merge master 2022-04-01 16:48:29 +02:00
Alexander Tokmakov
6591d1ceb7 Merge branch 'master' into mvcc_prototype 2022-04-01 15:38:46 +02:00
fenglv
97523a3329 Try remove unneed variable 2022-04-01 09:58:23 +00:00
Kseniia Sumarokova
d3b3294872
Merge pull request #35365 from bigo-sg/improve_access_type 
Improve check access in table functions
 2022-04-01 10:47:02 +02:00
Kseniia Sumarokova
9c39320e75
Merge pull request #35800 from ucasfl/mutex 
remove unused variable
 2022-04-01 10:46:23 +02:00
Kseniia Sumarokova
04729ecc70
Merge pull request #35798 from xiedeyantu/patch-1 
Delete duplicate code
 2022-04-01 10:43:31 +02:00
jewisliu
9a4cc78dfb support ALTER TABLE t DETACH PARTITION ALL syntax 2022-04-01 13:13:07 +08:00
Alexey Milovidov
5c6b84c398
Merge pull request #35586 from kssenii/fix-virtual-columns-s3 
Fix s3 engine getting virtual columns
 2022-04-01 04:43:32 +03:00
Alexander Tokmakov
63e8d73ba4 better message for incorrect *MergeTree arguments 2022-03-31 17:08:30 +02:00
Anton Popov
42acb1dc29 fix inserts to columns of type Object in partitioned tables 2022-03-31 13:30:01 +00:00
fenglv
e2e4b02d13 remove unused variable 2022-03-31 12:01:10 +00:00
xiedeyantu
d141dbc388
Delete duplicate code 
Delete duplicate code
 2022-03-31 19:34:30 +08:00
Nikita Mikhaylov
f6bfdcc0c9
Added support for schema inference for hdfsCluster (#35602) 2022-03-31 12:47:36 +02:00
Alexander Tokmakov
5a50ad9de3 Merge branch 'master' into mvcc_prototype 2022-03-31 11:35:04 +02:00
Kseniia Sumarokova
a711fc0402
Merge pull request #35572 from bigo-sg/hdfs_config_prefix 
Deduce absolute hdfs config path
 2022-03-31 10:08:32 +02:00
Antonio Andelic
13b4fabee4
Merge pull request #35743 from bigo-sg/remote_filter_for_hive 
Respect remote_url_allow_hosts for hive
 2022-03-31 08:54:37 +02:00
taiyang-li
539ce5f1b8 Merge branch 'master' into use_minmax_index 2022-03-31 12:27:32 +08:00
taiyang-li
70087bc959 Merge branch 'master' into improve_access_type 2022-03-31 12:19:17 +08:00
taiyang-li
f6ccda8097 Merge branch 'master' into hdfs_config_prefix 2022-03-31 12:16:05 +08:00
chen9t
b40a69f59a Code style 2022-03-31 10:10:05 +08:00
kssenii
98e85dd68e Fix build 2022-03-30 20:27:04 +02:00
shuchaome
738966b6b6 fix filebuffer pos in RemoteReadBuffer 2022-03-31 00:03:45 +08:00
kssenii
099fa0e668 Merge master 2022-03-30 17:33:32 +02:00
kssenii
0fc92fe2aa Get rid of all "remote" mentions 2022-03-30 17:15:29 +02:00
chen9t
5cb2301e39 Clean dirty meta cache when part is broken during part loading phase 2022-03-30 20:40:20 +08:00
Antonio Andelic
a3ad99be03 Create CallbackRunner in Storage 2022-03-30 11:24:50 +00:00
kssenii
98ad3f4a91 Fix tests, rename some remote mentions 2022-03-30 12:21:59 +02:00
lgbo-ustc
08e1fdc6b8 Fixed :a elimate AMBIGUOUS_COLUMN_NAME exception 
elimate AMBIGUOUS_COLUMN_NAME exception in hdfs/s3 cluster when run a query with join
 2022-03-30 17:06:38 +08:00
Antonio Andelic
1173eab15e Use same IOThreadPool for URL and S3 2022-03-30 08:15:20 +00:00
taiyang-li
16bb4c4ad0 respect remote_url_allow_hosts for hive 2022-03-30 15:33:59 +08:00
Antonio Andelic
03a09dc914
Merge pull request #35571 from ClickHouse/parallel-s3-downloading 
Parallel S3 downloading
 2022-03-30 08:14:49 +02:00
Antonio Andelic
d85ed8f2a9
Merge pull request #35655 from ClickHouse/exception-compile-time-message-check 
Use compile-time check for `Exception` messages
 2022-03-30 08:11:32 +02:00
taiyang-li
0af6fdb576 fix building 2022-03-30 11:28:21 +08:00
taiyang-li
fcfabf4a0c Merge remote-tracking branch 'ck/master' into use_minmax_index 2022-03-30 11:23:24 +08:00
taiyang-li
e76ed3acde Merge branch 'master' into improve_access_type 2022-03-30 11:04:57 +08:00
taiyang-li
9ffb42d507 Merge branch 'master' into hdfs_config_prefix 2022-03-30 11:02:31 +08:00
Anton Popov
1cba31c305
Merge pull request #35687 from CurtizJ/dynamic-columns-7 
Fix reading from type object
 2022-03-30 00:21:32 +02:00
Alexander Tokmakov
287d858fda Merge branch 'master' into mvcc_prototype 2022-03-29 16:24:12 +02:00
Dmitry Novik
8f935a72d6
Merge pull request #33230 from CurtizJ/read_in_order_max_rows_to_read 
Proper handle of 'max_rows_to_read' in case of reading in order of sorting key
 2022-03-29 15:16:34 +02:00
Antonio Andelic
ef484547bf Fix GCC build 2022-03-29 13:16:11 +00:00
taiyang-li
67c3c0be3d Merge branch 'master' into improve_access_type 2022-03-29 20:42:57 +08:00
taiyang-li
631352ba4b Merge branch 'master' into hdfs_config_prefix 2022-03-29 20:40:32 +08:00
kssenii
4374415bd7 Fix clang-tidy, style check 2022-03-29 14:20:21 +02:00
Kruglov Pavel
fe60560c03
Merge pull request #35619 from Avogar/check-remote-url-allow-hosts 
Check remote_url_allow_hosts before schema inference in URL engine
 2022-03-29 14:08:02 +02:00
Anton Popov
d42632dc13
Merge pull request #32928 from bigo-sg/rocksdb_metacache 
Speed up parts loading process of MergeTree by metadata cache
 2022-03-29 14:05:22 +02:00
taiyang-li
59727a423e Merge remote-tracking branch 'ck/master' into use_minmax_index 2022-03-29 19:39:09 +08:00
Maksim Kita
a1a4552740
Merge pull request #35184 from DevTeamBK/clang-tidy-issues 
clang-tidy report issues with Medium priority
 2022-03-29 13:19:54 +02:00
taiyang-li
77314246f3 Merge branch 'master' into hdfs_config_prefix 2022-03-29 18:32:59 +08:00
taiyang-li
834ac3a1b9 Merge remote-tracking branch 'ck/master' into rocksdb_metacache 2022-03-29 18:32:06 +08:00
Nikolai Kochetov
86690f18ee
Merge pull request #35667 from amosbird/projection-fix2 
Another fix of projection part check
 2022-03-29 10:22:01 +02:00
Antonio Andelic
362e877529 Merge branch 'master' into parallel-s3-downloading 2022-03-29 07:05:26 +00:00
Amos Bird
3ab8fc1e7b
Fix projection part check 
Fix projection part check when projection part type is different from
parent part type.
 2022-03-29 10:39:07 +08:00
kssenii
44157e5c80 Merge master 2022-03-29 02:19:32 +02:00
kssenii
c6b1f86306 Merge with master 2022-03-29 02:03:28 +02:00
Nikita Mikhaylov
85282f1b4d
Made schema inference work with s3Cluster table function (#35544) 2022-03-29 00:46:35 +02:00
kssenii
1d49a85963 Merge master 2022-03-28 22:51:56 +02:00
kssenii
a2013ebe2b Better 2022-03-28 22:40:27 +02:00
Kseniia Sumarokova
9a8e9198e5
Merge pull request #35593 from kssenii/named-collections-fix 
Fix headers with named collections, add compression_method
 2022-03-28 22:10:32 +02:00
Alexander Tokmakov
208b242188 Merge branch 'master' into mvcc_prototype 2022-03-28 19:58:06 +02:00
Anton Popov
0e4af89f69 fix reading from type object 2022-03-28 17:23:34 +00:00
mergify[bot]
f09ebea2d8
Merge branch 'master' into read_in_order_max_rows_to_read 2022-03-28 13:30:34 +00:00
taiyang-li
38702be40c Merge remote-tracking branch 'ck/master' into rocksdb_metacache 2022-03-28 21:03:22 +08:00
mergify[bot]
70b339ea4e
Merge branch 'master' into check-remote-url-allow-hosts 2022-03-28 11:55:48 +00:00
mergify[bot]
a022de4a6e
Merge branch 'master' into fix-materialized-postgresql 2022-03-28 10:55:41 +00:00
Antonio Andelic
d9d826c813 Address PR review 2022-03-28 08:19:23 +00:00
kssenii
c4832f7633 Fix 2022-03-28 09:28:02 +02:00
Mikhail Artemenko
1886d9c2ea refactor parseTypeOfField & fix tests 2022-03-27 23:00:28 +03:00
kssenii
c6706b8bfa Better 2022-03-25 19:30:13 +01:00
kssenii
141c290344 Fix build 2022-03-25 18:31:15 +01:00
taiyang-li
b79cec6806 Merge branch 'use_minmax_index' of https://github.com/bigo-sg/ClickHouse into use_minmax_index 2022-03-25 23:33:49 +08:00
taiyang-li
eee8949150 fix code 2022-03-25 23:33:46 +08:00
Alexander Tokmakov
9ec0951de5 update comment for mismatching checksums 2022-03-25 15:54:05 +01:00
taiyang-li
4aaa361f2e Merge remote-tracking branch 'ck/master' into use_minmax_index 2022-03-25 22:48:03 +08:00
taiyang-li
fbadd8fc0e only deduce absolute path of hdfs config when it's relative path do not exists 2022-03-25 22:28:47 +08:00
taiyang-li
c013cd0b52 Merge branch 'master' into hdfs_config_prefix 2022-03-25 22:13:45 +08:00
taiyang-li
3ca487030d Merge branch 'master' into improve_access_type 2022-03-25 22:11:08 +08:00
avogar
ecc57b436c Check remote_url_allow_hosts before schema inference in URL engine 2022-03-25 13:30:45 +00:00
Antonio Andelic
394682b983 Merge branch 'master' into parallel-s3-downloading 2022-03-25 13:00:44 +00:00
kssenii
6769b84638 Merge master 2022-03-24 20:47:43 +01:00
kssenii
d9d9f3bc6d Fix 2022-03-24 19:10:53 +01:00
kssenii
e91e30aaa4 Fix 2022-03-24 17:10:27 +01:00
Sergei Trifonov
a90e83665d
Merge pull request #35343 from ClickHouse/parallel-multipart-upload-for-s3storage 
allow parallelization of multipart S3 storage upload; fixes #34350
 2022-03-24 15:58:29 +01:00
kssenii
d6ab6de30a Minor changes 2022-03-24 15:32:08 +01:00
taiyang-li
c0eb9f73a9 remote std::cout 2022-03-24 21:17:43 +08:00
taiyang-li
6eccd09e2b deduce hdfs config path 2022-03-24 18:04:29 +08:00
Antonio Andelic
62a495a5fc Small refactor 2022-03-24 09:45:30 +00:00
Nikolai Kochetov
283e20a9a5
Merge pull request #35395 from amosbird/distributedmultiplejoin 
Validate some thoughts over making sets
 2022-03-24 10:30:26 +01:00
Antonio Andelic
c52cee2219 Merge branch 'master' into parallel-s3-downloading 2022-03-24 08:11:52 +00:00
Maksim Kita
1df1721648
Merge pull request #35546 from kitaisreal/refactor-external-models-loader 
ExternalModelsLoader refactoring
 2022-03-24 01:24:45 +01:00
kssenii
d4161b5925 Add optin read_from_cache_if_exists_otherwise_bypass_cache (for merges) 2022-03-23 20:24:00 +01:00
Maksim Kita
3df6a86387 ExternalModelsLoader refactoring 2022-03-23 20:05:17 +01:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
Yakov Olkhovskiy
1631656803
Merge pull request #34353 from zvonand/issue_33147 
Extended usage of Milliseconds, Microseconds, Nanoseconds
 2022-03-23 13:33:07 -04:00
kssenii
e39aba37a2 Minor changes 2022-03-23 18:11:52 +01:00
Alexander Tokmakov
9aed0507b7 Merge branch 'master' into mvcc_prototype 2022-03-23 18:07:22 +01:00
Sergei Trifonov
bf9c6be6c6 move threadPoolCallbackRunner from Common into Interpreters folder to fix split build issue 2022-03-23 17:57:19 +01:00
Antonio Andelic
cbd20af706 Merge branch 'master' into parallel-s3-downloading 2022-03-23 16:35:05 +00:00
kssenii
c2c7f365ce Merge master 2022-03-23 16:58:35 +01:00
kssenii
8c1be8950e Improve test 2022-03-23 16:55:18 +01:00
kssenii
34c0690a69 Better 2022-03-23 15:43:14 +01:00
Antonio Andelic
1a5f5c32bb Remove commented out code 2022-03-23 14:36:25 +00:00
李扬
9cc528b01f
Update HiveFile.h 2022-03-23 21:57:58 +08:00
ianton-ru
32c6390400
Merge branch 'master' into MDB-16962 2022-03-23 16:53:30 +03:00
Anton Ivashkin
66c026d0ac Fix collision of S3 operation log revision 2022-03-23 16:48:05 +03:00
xiedeyantu
9441111833
fix INSERT INTO table FROM INFILE does not display progress bar (#35429) 
* fix INSERT INTO table FROM INFILE does not display progress bar

* Update StorageFile.cpp

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-03-23 14:24:59 +01:00
kssenii
f3e1ca44a9 Add system.remote_data_paths table 2022-03-23 13:01:18 +01:00
Antonio Andelic
93ad209910 Attach memory tracker 2022-03-23 11:52:31 +00:00
Alexander Tokmakov
bcec53aadc Merge branch 'master' into mvcc_prototype 2022-03-23 11:28:43 +01:00
Antonio Andelic
131b3a091c Refactor StorageS3 2022-03-23 08:57:01 +00:00
Antonio Andelic
bb6b9f3799 Merge branch 'parallel-downloading-url-engine' into parallel-s3-downloading 2022-03-23 07:46:14 +00:00
Antonio Andelic
9829f2d051 Merge branch 'master' into parallel-downloading-url-engine 2022-03-23 07:41:14 +00:00
taiyang-li
ae3d55c6a2 merge master and fix conflict 2022-03-23 14:31:12 +08:00
taiyang-li
6fe4c3d17e some minor changes 2022-03-23 12:13:42 +08:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00
taiyang-li
68d5b538aa fix build error 2022-03-23 11:15:42 +08:00
Mikhail f. Shiryaev
e6f5a3f98b
Apply black formatter to all *.py files in the repo 2022-03-22 17:39:58 +01:00
Anton Popov
93dc5c0b81
Merge pull request #35488 from CurtizJ/add-subcolumns-sizes 
Add sizes of subcolumns to `system.parts_columns` table
 2022-03-22 16:53:23 +01:00
Anton Popov
f693eba568 fix tests with approx rows 2022-03-22 14:30:40 +00:00
Antonio Andelic
6785ad165a Fix issue for mutliple download threads 2022-03-22 14:06:30 +00:00
Antonio Andelic
df99a9cc3c Merge branch 'parallel-downloading-url-engine' into parallel-s3-downloading 2022-03-22 12:49:35 +00:00
Antonio Andelic
5da3058368 Merge branch 'master' into parallel-downloading-url-engine 2022-03-22 12:20:36 +00:00
Antonio Andelic
246c66bf3e
Merge pull request #35393 from bigo-sg/hive_column_pruning_bug 
Fix erasing nonexistent column
 2022-03-22 08:59:33 +01:00
Anton Popov
0c210a831c add sizes of subcolumns to system.parts_columns table 2022-03-21 21:12:26 +00:00
kssenii
a9ba14e386 Fix checks 2022-03-21 21:53:46 +01:00
mergify[bot]
9a505476a5
Merge branch 'master' into mvcc_prototype 2022-03-21 20:33:07 +00:00
tavplubix
2fdc937ae1
Merge pull request #35480 from ClickHouse/fix_34586 
Fix show create for information_schema
 2022-03-21 22:20:34 +03:00
Alexander Tokmakov
3c762f566d Merge branch 'master' into mvcc_prototype 2022-03-21 20:16:29 +01:00
mergify[bot]
9b9959aae0
Merge branch 'master' into read_in_order_max_rows_to_read 2022-03-21 17:33:37 +00:00
Vitaly Baranov
72a2f31984
Merge pull request #33985 from vitlibar/improve-backup-and-restore 
Improve backup and restore (experimental)
 2022-03-21 17:59:51 +01:00
tavplubix
9226ea7b70
Merge pull request #35467 from ClickHouse/resubmit_21474 
Resubmit #21474
 2022-03-21 19:15:28 +03:00
Nikolai Kochetov
5e239762c6
Merge pull request #35334 from amosbird/fixpartitionpruneroverflow 
Fix incorrect partition pruning when constant in predicate has no supertype of related key columns
 2022-03-21 17:05:19 +01:00
Alexander Tokmakov
dba82a7732 fix show create for information_schema 2022-03-21 17:01:39 +01:00
Antonio Andelic
04e6e6ab5e Add ParallelReadBuffer for S3 2022-03-21 14:52:26 +00:00
kssenii
afd0c64a1a Better 2022-03-21 15:04:16 +01:00
mergify[bot]
e11ef05c2b
Merge branch 'master' into issue_33147 2022-03-21 13:40:17 +00:00
lgbo-ustc
967d5a8055 Merge remote-tracking branch 'ck/master' into hive_column_pruning_bug 2022-03-21 19:52:06 +08:00
kssenii
eabbce69a7 Add system drop remote filesystem cache query, allow to write cache on insert 2022-03-21 12:32:44 +01:00
Alexander Tokmakov
7cba97aeab Merge branch 'master' into resubmit_21474 2022-03-21 12:09:00 +01:00
Vitaly Baranov
7a46e7a0d1
Merge branch 'master' into improve-backup-and-restore 2022-03-21 11:49:26 +01:00
Vitaly Baranov
51fc556fcd Implement BACKUP and RESTORE for MatViews. 2022-03-21 11:45:21 +01:00
mergify[bot]
e5a5ab2a40
Merge branch 'master' into distributedmultiplejoin 2022-03-21 10:00:51 +00:00
Antonio Andelic
b8c43ff2f2 Merge branch 'master' into parallel-downloading-url-engine 2022-03-21 09:11:01 +00:00
kssenii
59b7394caf WriteSettings 2022-03-21 09:52:48 +01:00
kssenii
1d80262a2a Add write settings, file cache settings 2022-03-21 09:50:50 +01:00
taiyang-li
49b6f3dfc5 merge master and fix conflict 2022-03-21 15:05:43 +08:00
taiyang-li
bf05b94940 fix build 2022-03-21 15:03:28 +08:00
taiyang-li
d2bea0a72c Merge branch 'master' into improve_access_type 2022-03-21 14:25:19 +08:00
Alexey Milovidov
0ce4696c49
Revert "[WIP] New row policies" 2022-03-21 08:41:33 +03:00
taiyang-li
7d50bd1eb3 add access type hive 2022-03-21 11:19:45 +08:00
Amos Bird
243de091bb
Validate some thoughts over making sets 2022-03-21 10:58:44 +08:00
lgbo-ustc
f7aa40af5b update codes 2022-03-21 09:25:20 +08:00
Alexander Tokmakov
9e05b12d2c Merge branch 'master' into mvcc_prototype 2022-03-20 22:42:26 +01:00
Vitaly Baranov
ce25afb2e9 Storages and databases are hollow by default now. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
c884cf901f Use BackupEntryFromAppendOnlyFile for Log engine, 
and use a BackupEntriesBatch for Memort engine to improve performance.
A lot of minor corrections.
 2022-03-20 20:02:15 +01:00
Vitaly Baranov
e72a343994 Implement BACKUP & RESTORE for the Memory table engine. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7f89b98308 Rework BackupSettings and RestoreSettings a little, pass StorageRestoreSettings to storages. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
258a472001 Shorter names: rename IRestoreFromBackupTask -> IRestoreTask. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
5a883c6b5a Fixed checking access rights while producing backup, add IStorage::hasHollowBackup(). 2022-03-20 20:01:54 +01:00
Vitaly Baranov
7a63feb3f7 Make restore tasks explicit. 2022-03-20 20:01:31 +01:00
Vitaly Baranov
9fb50422ce
Merge pull request #35345 from vitlibar/new-row-policies 
[WIP] New row policies
 2022-03-20 20:00:25 +01:00
Alexey Milovidov
9b05801e0f
Merge pull request #35284 from CurtizJ/fix-mutations-sparse-columns 
Fix mutations in tables with enabled sparse columns
 2022-03-20 21:55:59 +03:00
mergify[bot]
7ac606fa65
Merge branch 'master' into parallel-multipart-upload-for-s3storage 2022-03-20 18:25:28 +00:00
alesapin
1bd5736e34 Fix build one more time 2022-03-19 20:00:08 +01:00
alesapin
9e24677a30 Fix build 2022-03-19 18:09:01 +01:00
alesapin
97e84e6dc2 fix build 2022-03-19 17:47:35 +01:00
alesapin
f2c5e2d3a0 Don't spam logs in zero copy replication 2022-03-19 17:31:33 +01:00
Kseniia Sumarokova
a918e68d9a
Merge pull request #35355 from kssenii/add-hosts-filter 
Add more checks with remoteHostsFilter
 2022-03-19 09:41:23 +01:00
Alexander Tokmakov
418d52044a try fix merge assignment 2022-03-18 20:31:44 +01:00
Alexander Tokmakov
c28008a2df more comments, minor fixes 2022-03-18 12:01:26 +01:00
Amos Bird
4d85c2fc8b
Use castColumnAccurateOrNull 2022-03-18 17:47:04 +08:00
Amos Bird
10996b1434
Fix mixed constant type during partition pruning 2022-03-18 17:47:03 +08:00
lgbo-ustc
e78cfe3b26 update codes 2022-03-18 15:07:52 +08:00
Alexander Tokmakov
07d952b728 use snapshots for semistructured data, durability fixes 2022-03-17 18:26:18 +01:00
Antonio Andelic
b789a41662 Fix ASAN problems 2022-03-17 15:18:20 +00:00
Anton Popov
11ab24acb1
Merge pull request #35356 from CurtizJ/fix-clang-tidy 
Fix clang-tidy
 2022-03-17 16:17:23 +01:00
Vitaly Baranov
470582e262 Change type of RowPolicyKind: bool -> enum. 2022-03-17 15:28:05 +01:00
Alexander Tokmakov
d04dc03fa4 Merge branch 'master' into mvcc_prototype 2022-03-17 15:24:32 +01:00
Nikolai Kochetov
ee9c2ec735
Merge pull request #34780 from azat/mt-delayed-part-flush 
Do not delay final part writing by default (fixes possible Memory limit exceeded during INSERT)
 2022-03-17 12:30:51 +01:00
Anton Popov
fe2dc85788 fix clang-tidy 2022-03-17 11:16:32 +00:00
kssenii
a2cd165d38 Add remote host filter 2022-03-17 11:48:42 +01:00
alesapin
92ab33f560
Merge pull request #35348 from ClickHouse/fix_yet_another_zero_copy_bug 
Fix race in zero copy replication
 2022-03-17 11:34:38 +01:00
alesapin
bb251938dc
Merge pull request #35344 from ClickHouse/changelog-22.3 
Changelog 22.3
 2022-03-17 11:25:36 +01:00
Antonio Andelic
103a3fa140 Merge branch 'master' into parallel-downloading-url-engine 2022-03-17 10:11:58 +00:00
Antonio Andelic
44de5a581c Fix split build 2022-03-17 10:09:16 +00:00
Antonio Andelic
cc623bbf5e Fix redirect_url tests 2022-03-17 08:03:22 +00:00
Alexey Milovidov
68ef49ea51 Fix something stupid 2022-03-17 05:57:13 +01:00
alesapin
03ff6d52f6 Fix race in zero copy replication 2022-03-16 21:51:33 +01:00
Anton Popov
de2cc23e15 fix race 2022-03-16 20:16:59 +00:00
Alexander Tokmakov
4b3e13a4fe Merge branch 'master' into mvcc_prototype 2022-03-16 21:06:19 +01:00
Alexander Tokmakov
c2ac8d4a5c review fixes 2022-03-16 21:05:34 +01:00
Sergei Trifonov
bbe657ba24 allow parallelization of multipart S3 storage upload fixes #34350 2022-03-16 19:41:41 +01:00
Antonio Andelic
0979155f2f Address PR comments 2022-03-16 15:45:17 +00:00
Anton Popov
0ba78c3c3a Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-16 15:28:09 +00:00
Antonio Andelic
84f86bf3da Limit number of bytes read per worker 2022-03-16 12:34:14 +00:00
Antonio Andelic
2291b7c1e1 Merge branch 'master' into parallel-downloading-url-engine 2022-03-16 09:59:52 +00:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts 
Reload remote_url_allow_hosts after config update.
 2022-03-15 22:07:16 +01:00
Alexander Tokmakov
1f571b7734 Merge branch 'master' into mvcc_prototype 2022-03-15 14:45:06 +01:00
Nikolai Kochetov
97aa6c82ce Reload remote_url_allow_hosts after config update. 2022-03-15 13:00:31 +00:00
Anton Popov
b5a04ff938
Merge pull request #35207 from bigo-sg/add_distributed_depth_in_querylog 
Add column  distributed_depth for system.query_log and system.processes
 2022-03-15 13:38:21 +01:00
Kseniia Sumarokova
427ae808ba
Merge pull request #35191 from DevTeamBK/RemoteHostFilter_improvement 
Added RemoteHostFilter check for MYSQL and postgresSQL
 2022-03-15 09:58:08 +01:00
Antonio Andelic
067b79b00b Merge branch 'master' into parallel-downloading-url-engine 2022-03-15 07:55:41 +00:00
roverxu
29a842bf22 feat(...): [LWD] support getting _part_offset of a row 2022-03-15 15:40:10 +08:00
lgbo-ustc
abfaa82bca fixed hive query bugs 2022-03-15 12:01:34 +08:00
Saad Ur Rahman
679abaafdc
[Storages] bugfix 
Extracting Version Id for S3::URI.
 2022-03-14 22:25:40 -04:00
Saad Ur Rahman
fb450b7068
[Storages] bugfix 
Corrected order of parameters by swapping bucket and versionId.
 2022-03-14 22:20:02 -04:00
Saad Ur Rahman
171260f40b
[Storages] Added Version ID parameter for S3. 2022-03-14 22:03:58 -04:00
Anton Popov
ccbddd53a3 fix mutations in tables with enabled sparse columns 2022-03-15 01:48:21 +00:00
alesapin
fbb1ebd9b8
Merge pull request #35274 from CurtizJ/fix-check-table-sparse-columns 
Fix check table in case when there exist sparse columns
 2022-03-14 21:56:04 +01:00
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
Alexander Tokmakov
278d779a01 log cleanup, more comments 2022-03-14 21:43:34 +01:00
Maksim Kita
2fdcf53a76 Fix clang-tidy warnings in Server, Storages folders 2022-03-14 18:17:35 +00:00
Anton Popov
063917786e minor fixes 2022-03-14 17:29:18 +00:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Anton Popov
428bbd6377 fix check table in case when there exist sparse columns 2022-03-14 15:22:23 +00:00
Anton Popov
630182b2b1 minor renames 2022-03-14 14:42:09 +00:00
Heena Bansal
c774458f96
Update src/Storages/StoragePostgreSQL.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-14 10:38:12 -04:00
Heena Bansal
01c8b2f71e
Update src/Storages/StorageMySQL.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-14 10:37:59 -04:00
Antonio Andelic
d3353f3f0c Merge branch 'master' into parallel-downloading-url-engine 2022-03-14 12:47:23 +00:00
Antonio Andelic
556fe2bcc5 Don't send RANGE with HEAD call 2022-03-14 12:46:22 +00:00
Antonio Andelic
ed3d71d83f
Merge pull request #35072 from azat/buffer-memory-tracker-leak 
Avoid MEMORY_LIMIT_EXCEEDED during INSERT into Buffer with AggregateFunction
 2022-03-14 12:31:04 +01:00
Maksim Kita
ad6b3693e1
Merge pull request #35123 from zhanghuajieHIT/fix_build_fail_with_gcc 
fix build fail with gcc
 2022-03-14 10:36:15 +01:00
Antonio Andelic
9dda2863d3 Split download threads when multiple URLs are used 2022-03-14 09:27:09 +00:00
HeenaBansal2009
09be30ac26 Added suggestion from review comments 2022-03-11 07:33:23 -08:00
zhanghuajie
53a8987b3b fix build fail with gcc --fix warnings without disabling some parameters 2022-03-11 21:59:19 +08:00
Antonio Andelic
f5d3a8a31d Polishing 2022-03-11 13:38:19 +00:00
taiyang-li
1f9e050152 add column distributed_depth for system.query_log and system.processes 2022-03-11 17:57:34 +08:00
Antonio Andelic
28e9508c4e Improve shared pool and add settings 2022-03-11 08:33:34 +00:00
Antonio Andelic
cce318273b Use a shared IO thread pool 2022-03-11 08:33:34 +00:00
Antonio Andelic
df0f5e20d0 Refactor 2022-03-11 08:33:34 +00:00
Antonio Andelic
e051587fc2 Format code 2022-03-11 08:33:34 +00:00
Antonio Andelic
29c32ed831 Refactor code 2022-03-11 08:33:34 +00:00
Antonio Andelic
0a1a3a230e Add support for parallel http range requests 2022-03-11 08:33:34 +00:00
Antonio Andelic
58557b9bec Test check for ranges support 2022-03-11 08:33:34 +00:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs 
Local cache for remote filesystem
 2022-03-11 07:23:10 +01:00
Alexander Tokmakov
7f47f20aba add kill transaction query 2022-03-10 22:29:58 +01:00
Anton Popov
37efe2ddb5
Apply suggestions from code review 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-10 22:24:19 +01:00
alesapin
c0d8ccc91b
Merge pull request #35178 from Varinara/master 
Added disk_name to system.part_log
 2022-03-10 22:22:37 +01:00
HeenaBansal2009
d981463b05 Added RemoteHostFilter check for MYSQL and postgresSQL 2022-03-10 08:58:48 -08:00
Maksim Kita
493169910b
Merge pull request #35174 from zhangyifan27/fix_typo 
fix typos
 2022-03-10 17:10:44 +01:00
Kseniia Sumarokova
3fc399b6e9
Merge pull request #35158 from kssenii/fix-materialized-postgresql 
Fix materialised postrgesql adding new table after manually removing it
 2022-03-10 17:02:32 +01:00
Kseniia Sumarokova
e30b0c5d57
Merge pull request #35162 from kssenii/fix-materialized-postgresql-table-override 
Fix materialised postgres `table overrides` for partition by, etc
 2022-03-10 17:01:24 +01:00
Varinara
f5523f7ff0 added disk_name to system.part_log 2022-03-10 18:44:19 +03:00
Rajkumar
3d3b6d1956 clang-tidy report issues with Medium priority 2022-03-10 07:23:49 -08:00
mergify[bot]
df01290e73
Merge branch 'master' into fix_typo 2022-03-10 13:35:04 +00:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
kssenii
1dc3f36a11 Better 2022-03-10 12:19:20 +01:00
kssenii
787a0805a5 Merge master 2022-03-10 11:42:19 +01:00
Kseniia Sumarokova
e6ee891c9c
Merge pull request #34957 from bigo-sg/hive_random_access_file_cache 
Optimization for first time to read a random access readbuffer in hive
 2022-03-10 11:36:22 +01:00
zhangyifan27
e6fa9f699a fix typo 2022-03-10 18:29:42 +08:00
kssenii
3cd1da1e11 Fix 2022-03-10 11:11:59 +01:00
Alexander Tokmakov
0906b59fba fixes 2022-03-09 21:38:18 +01:00
kssenii
af9d8d278e Fix 2022-03-09 19:25:43 +01:00
Vladimir C
ce266b5a3e
Merge pull request #35146 from amosbird/fixpartitionprunerin 2022-03-09 13:23:45 +01:00
Nikolai Kochetov
6bfee7aca2
Merge pull request #35132 from azat/parallel_distributed_insert_select-view 
Support view() for parallel_distributed_insert_select
 2022-03-09 09:10:34 +01:00
Azat Khuzhin
e2960e1a52 Avoid MEMORY_LIMIT_EXCEEDED during INSERT into Buffer with AggregateFunction 
In case of Buffer table has columns of AggregateFunction type,
aggregate states for such columns will be allocated from the query
context but those states can be destroyed from the server context (in
case of background flush), and thus memory will be leaked from the query
since aggregate states can be shared, and eventually this will lead to
MEMORY_LIMIT_EXCEEDED error.

To avoid this, prohibit sharing the aggregate states.

But note, that this problem only about memory accounting, not memory
usage itself.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-09 10:57:49 +03:00
Amos Bird
a19224bc9b
Fix partition pruner: non-monotonic function IN 2022-03-09 15:48:42 +08:00
Azat Khuzhin
3a5a39a9df Do not delay final part writing by default 
For async s3 writes final part flushing was defered until all the INSERT
block was processed, however in case of too many partitions/columns you
may exceed max_memory_usage limit (since each stream has overhead).

Introduce max_insert_delayed_streams_for_parallel_writes (with default
to 1000 for S3, 0 otherwise), to avoid this.

This should "Memory limit exceeded" errors in performance tests.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:17:36 +03:00
Alexander Tokmakov
d2f838bd91 fix another race condition 2022-03-08 20:11:47 +01:00
Azat Khuzhin
4843e210c3 Support view() for parallel_distributed_insert_select 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:05:57 +03:00
Azat Khuzhin
ced34dea84 Take flush_time into account for scheduling background flush of the Buffer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 21:58:10 +03:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0 Fix split build 2022-03-08 18:05:55 +01:00
Azat Khuzhin
c4b6342853
Improvements for parallel_distributed_insert_select (and related) (#34728) 
* Add a warning if parallel_distributed_insert_select was ignored

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Respect max_distributed_depth for parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Print warning for non applied parallel_distributed_insert_select only for initial query

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Remove Cluster::getHashOfAddresses()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Forbid parallel_distributed_insert_select for remote()/cluster() with different addresses

Before it uses empty cluster name (getClusterName()) which is not
correct, compare all addresses instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix max_distributed_depth check

max_distributed_depth=1 must mean not more then one distributed query,
not two, since max_distributed_depth=0 means no limit, and
distribute_depth is 0 for the first query.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix INSERT INTO remote()/cluster() with parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add a test for parallel_distributed_insert_select with cluster()/remote()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Return <remote> instead of empty cluster name in Distributed engine

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Make user with sharding_key and w/o in remote()/cluster() identical

Before with sharding_key the user was "default", while w/o it it was
empty.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 15:24:39 +01:00
Antonio Andelic
bc5d7aea57
Merge pull request #34876 from azat/long-INSERT-fix 
Fix possible "Part directory doesn't exist" during INSERT
 2022-03-08 12:44:53 +01:00
Kseniia Sumarokova
1eb2bae792
Merge pull request #34954 from bigo-sg/hive_read_columns_pruning 
read columns pruning for hive
 2022-03-08 10:17:24 +01:00
lgbo-ustc
256e92ffee Merge remote-tracking branch 'ck/master' into hive_random_access_file_cache 2022-03-08 14:14:40 +08:00
Azat Khuzhin
caffc144b5 Fix possible "Part directory doesn't exist" during INSERT 
In #33291 final part commit had been defered, and now it can take
significantly more time, that may lead to "Part directory doesn't exist"
error during INSERT:

    2022.02.21 18:18:06.979881 [ 11329 ] {insert} <Debug> executeQuery: (from 127.1:24572, user: default) INSERT INTO db.table (...) VALUES
    2022.02.21 20:58:03.933593 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18044_18044_0 to 20220214_270654_270654_0.
    2022.02.21 21:16:50.961917 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18197_18197_0 to 20220214_270689_270689_0.
    ...
    2022.02.22 21:16:57.632221 [ 64878 ] {} <Warning> db.table: Removing temporary directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/
    ...
    2022.02.23 12:23:56.277480 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18232_18232_0 to 20220214_273459_273459_0.
    2022.02.23 12:23:56.299218 [ 11329 ] {insert} <Error> executeQuery: Code: 107. DB::Exception: Part directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/ doesn't exist. Most likely it is a logical error. (FILE_DOESNT_EXIST) (version 22.2.1.1) (from 127.1:24572) (in query: INSERT INTO db.table (...) VALUES), Stack trace (when copying this message, always include the lines below):

Follow-up for: #28760
Refs: #33291

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 07:44:11 +03:00
lgbo-ustc
a8cfc2458a update codes 2022-03-08 11:55:15 +08:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
Maksim Kita
2f9361008b
Merge pull request #35089 from 1lann/1lann/fix-update_lag-typo 
Fix typo of update_lag
 2022-03-07 23:12:35 +01:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
Alexander Tokmakov
ea2f65fef6 fix tests with DiskS3 2022-03-07 17:35:47 +01:00
Kseniia Sumarokova
5511f2f6e6
Merge pull request #34940 from bigo-sg/hive_client_connection_pool 
Use connection pool in HiveMetastoreClient
 2022-03-07 17:14:56 +01:00
Kseniia Sumarokova
28b9ec01c0
Merge pull request #34945 from bigo-sg/hive_bug_fixed 
unexpected result when use `in` in hive query
 2022-03-07 17:13:11 +01:00
Anton Popov
0bc57da238 Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-07 14:46:08 +00:00
alesapin
d90bee1df9
Merge pull request #35080 from azat/mutate-exceptions 
Do not hide exceptions during mutations
 2022-03-07 12:22:49 +01:00
Vladimir C
678f05ca1e
Merge pull request #34912 from kssenii/fix-filelog-metadata-path 2022-03-07 11:45:29 +01:00
1lann
5423c5a45c Fix typo of update_lag 
In external dictionary providers, the allowed keys for configuration seemed to have a typo
of "update_lag" as "update_tag", preventing the use of "update_lag". This change fixes that.
 2022-03-07 18:31:20 +08:00
lgbo-ustc
8ae5296ee8 fixed compile errors 2022-03-07 17:26:48 +08:00
lgbo-ustc
eab925554d fixed code styles 2022-03-07 14:01:03 +08:00
lgbo-ustc
0c83b96d8c fixed code style 2022-03-07 13:13:33 +08:00
lgbo-ustc
cfeedd2cb5 fixed code style 2022-03-07 12:28:31 +08:00
lgbo-ustc
4507cc58aa update codes 2022-03-07 12:05:07 +08:00
lgbo-ustc
c37eedd887 update codes 2022-03-07 10:30:54 +08:00
lgbo-ustc
75a50a30c4 update codes 2022-03-07 09:43:53 +08:00
lgbo-ustc
d907b70cc4 update codes: get actual read block 2022-03-07 09:26:05 +08:00
lgbo-ustc
f4d8fb46c5 update codes 2022-03-07 09:26:05 +08:00
lgbo-ustc
62c1bd5ae9 hive read columns pruning 2022-03-07 09:26:05 +08:00
Azat Khuzhin
bc224dee36 Do not hide exceptions during mutations 
system.mutations includes only the message, but not stacktrace, and it
is not always obvious to understand the culprit w/o stacktrace.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-06 13:39:49 +03:00
Kseniia Sumarokova
3ec6cd3128
Update StorageFileLog.cpp 2022-03-06 11:03:22 +01:00
alexey-milovidov
f9b7df6ba1
Merge pull request #35050 from CurtizJ/fix-async-inserts-system-table 
Fix reading from `system.asynchronous_inserts` table
 2022-03-06 02:25:53 +03:00
Maksim Kita
7ae1f0fa3b
Merge pull request #34911 from larspars/master 
Allow LowCardinality strings for ngrambf_v1/tokenbf_v1 indexes. Fixes #21865
 2022-03-04 19:17:48 +01:00
Azat Khuzhin
2ef9d32448 Revert "Remove VERSION_DATE from system.build_options" 
As requested by @kitaisreal

This reverts commit 4a404532fb.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:32:54 +03:00
Azat Khuzhin
4a404532fb Remove VERSION_DATE from system.build_options 
It was set only bu utils/release/release_lib.sh, and seems that this
script is not used anymore, at least that part of it.

Also note, that GIT_DATE is the same, and it is date time, not only
date.

Plus VERSION_DATE is not installed for releases anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:37 +03:00