thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-19 04:42:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
120,708 Commits 9,326 Branches 1,783 Tags 2.2 GiB
9448d42aea
Commit Graph

1032 Commits

Author SHA1 Message Date
Sergei Trifonov
f61a98b5dc
Merge branch 'master' into io-scheduler 2022-12-16 15:55:08 +01:00
Kruglov Pavel
e3aa2d9d56
Merge pull request #43695 from Avogar/better-using-structure-in-table-functions 
Improve automatic usage of structure from insertion table in table functions file/hdfs/s3
 2022-12-16 15:45:30 +01:00
Smita Kulkarni
16d8fd02fb Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-12-16 08:52:48 +01:00
Robert Schulze
4db33f16e3
Make DoS attack on cache less likely 2022-12-15 21:42:51 +00:00
kssenii
dfefd8dfcd Fix write-throw-cache possible write finish 2022-12-15 19:58:54 +01:00
Robert Schulze
537c64aad2
Merge remote-tracking branch 'origin/master' into query-result-cache 
Conflicts:
- tests/queries/0_stateless/02117_show_create_table_system.reference
 2022-12-15 12:17:58 +00:00
kssenii
ea5b06023a Merge remote-tracking branch 'refs/remotes/upstream/master' into allow-single-disk-instead-of-storage-policy 2022-12-14 13:59:51 +01:00
Nikolay Degterinsky
809f15ac81
Merge branch 'master' into fix_settings_constraints 2022-12-13 20:33:16 +01:00
Robert Schulze
b78652e138
Query result cache 2022-12-12 10:37:10 +00:00
Sergei Trifonov
bae1519970
Merge branch 'master' into merge_tree_settings_constraints 2022-12-08 22:25:16 +01:00
Nikolay Degterinsky
bb9fdc0839 Fix CREATE query constraints 2022-12-07 00:53:27 +00:00
kssenii
d91f3808bd Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2022-12-05 12:54:32 +01:00
Alexander Tokmakov
17a9b24688
Merge branch 'master' into fix_some_flaky_tests2 2022-12-05 14:09:19 +03:00
serxa
006af1b6a4 constraints on merge tree settings 2022-12-02 19:30:14 +00:00
Vladimir C
7d6950d397
Revert "Temporary files evict fs cache" 2022-12-02 14:50:56 +01:00
Alexander Tokmakov
dea3420f20
Update Context.cpp 2022-12-02 15:00:30 +03:00
Alexander Tokmakov
9df3709c12
Merge branch 'master' into fix_some_flaky_tests2 2022-12-01 21:05:02 +03:00
Alexander Tokmakov
6adbc2164a add an assertion 2022-12-01 17:33:15 +01:00
vdimir
98fe3c6c02
Temporary files evict fs cache 2022-12-01 11:49:17 +00:00
Smita Kulkarni
953e1068b5 Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-11-28 19:10:32 +01:00
avogar
4aa3e23270 Improve automatic usage of structure from insertion table in table functions file/hdfs/s3 2022-11-25 19:33:47 +00:00
Sergei Trifonov
8186a06c28
Merge branch 'master' into io-scheduler 2022-11-24 12:02:52 +01:00
Sergei Trifonov
77100896e3
Merge branch 'master' into storage-system-moves 2022-11-18 16:33:43 +01:00
kssenii
41c1a2aad4 Revert "Revert "Better code around SettingsChanges"" 
This reverts commit 5e61022bac.
 2022-11-18 13:16:46 +01:00
kssenii
5e61022bac Revert "Better code around SettingsChanges" 
This reverts commit bed9f7594a.
 2022-11-18 12:37:34 +01:00
kssenii
5cdf216c8f Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2022-11-17 16:39:36 +01:00
Smita Kulkarni
c84a50e77c Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-11-16 12:40:04 +01:00
Smita Kulkarni
95abe5d027 Updated to support query parameters in HAVING clause and addressed review comments - 40907 Parameterized views as table functions 2022-11-16 12:27:08 +01:00
李扬
fc7fcafe40
Merge branch 'master' into alternative_keeper 2022-11-15 10:34:07 +08:00
taiyang-li
64d7087084 merge master and fix conflict 2022-11-15 10:33:10 +08:00
Kruglov Pavel
58b7f6ca04
Merge pull request #42320 from Avogar/improve-schema-inference-insert-select 
Improve using structure from insertion table in table functions
 2022-11-14 14:51:27 +01:00
Smita Kulkarni
4dec8337f0 Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-11-11 19:04:35 +01:00
kssenii
85e448f048 Fix 2022-11-08 13:32:49 +01:00
kssenii
b5b25c2676 Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2022-11-07 17:02:41 +01:00
kssenii
bed9f7594a Better code around SettingsChanges 2022-11-07 16:58:03 +01:00
Smita Kulkarni
3e5f1a9178 Added is_parameterized_view in ASTFunction and fixed review comments - 40907 Parameterized views as table functions 2022-11-07 15:22:45 +01:00
Kruglov Pavel
83c5c44b11
Merge branch 'master' into improve-schema-inference-insert-select 2022-11-03 18:41:51 +01:00
Smita Kulkarni
9aa06e48f6 Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-10-31 15:46:52 +01:00
Vladimir Chebotaryov
d17b7387f9 Reworked changes to std::shared_ptr<const RowPolicyFilter>. 2022-10-27 10:42:55 +03:00
Vladimir Chebotaryov
4cabe1f57c Added applied row-level policies to system.query_log. 2022-10-27 10:41:47 +03:00
Sergei Trifonov
34b43f8b4b
Merge branch 'master' into storage-system-moves 2022-10-25 22:56:45 +02:00
serxa
55a86bc2de Add active moves system table 2022-10-25 13:56:46 +00:00
Nikolai Kochetov
c4685f6994
Merge branch 'master' into fix-42575 2022-10-25 15:31:32 +02:00
Nikolai Kochetov
5c4444237e
Merge pull request #42617 from ClickHouse/revert-revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3 
Revert revert 41268 disable s3 parallel write for part moves to disk s3
 2022-10-25 14:29:27 +02:00
Raúl Marín
0016bc2b73
Improve the time to recover keeper connections (#42541) 
Co-authored-by: Nikita Mikhaylov <nikitamikhaylov@clickhouse.com>
Co-authored-by: robot-clickhouse <robot-clickhouse@clickhouse.com>
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2022-10-25 13:26:29 +02:00
Nikolai Kochetov
93310fe8b1 Fix typo. 2022-10-24 15:51:21 +00:00
Nikolai Kochetov
d5bd905a41 Try to fix data race on zookeeper vs DDLWorker at server shutdown. 2022-10-24 15:09:35 +00:00
Nikolai Kochetov
57ffcb7ed0 Revert "Merge pull request #41681 from ClickHouse/revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3" 
This reverts commit 7956c2becf, reversing
changes made to 57be648984.
 2022-10-20 19:51:27 +02:00
Kruglov Pavel
9588653a6e
Merge branch 'master' into improve-schema-inference-insert-select 2022-10-18 15:17:27 +02:00
Alexey Milovidov
e014293fe8 Fix tidy 2022-10-17 19:29:14 +02:00
Smita Kulkarni
3034229c37 Merge branch 'master' of github.com:ClickHouse/ClickHouse into 40907_Parameterized_views_as_table_functions 2022-10-17 18:37:31 +02:00
Smita Kulkarni
614fd4cf42 Added is_parameterized_view to Context and removed flag from ASTs, updated to use tryGetTable, addressed review comments - 40907 Parameterized views as table functions 2022-10-17 18:11:22 +02:00
Alexey Milovidov
f88ed8195b Fix trash 2022-10-17 04:21:08 +02:00
Vitaly Baranov
91c438eebe
Merge pull request #42263 from vitlibar/refactor-udf 
Refactor the implementation of user-defined functions
 2022-10-15 12:33:10 +02:00
Kruglov Pavel
11a9b5977a
Fix typos 2022-10-14 19:22:05 +02:00
nvartolomei
7d9f097d3b
Fix bad copy & paste when loading background_merges_mutations_concurrency_ratio setting (#42315) 2022-10-14 19:19:14 +02:00
Kruglov Pavel
f1dd247181
Merge branch 'master' into improve-schema-inference-insert-select 2022-10-14 17:16:58 +02:00
avogar
10335aec13 Improve using structure from insertion table in table functions 2022-10-14 15:09:35 +00:00
Vitaly Baranov
2e59d671fc Split UserDefinedSQLObjectsLoader to interface and implementation. 2022-10-13 10:32:32 +02:00
FArthur-cmd
b6304c46d4 move default settings to different part of ast 2022-10-11 18:25:28 +00:00
Smita Kulkarni
ec95294621 Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-10-05 11:48:14 +02:00
Smita Kulkarni
a0b1085f3b Fixed issues with using database name in SELECT and added tests - 40907 Parameterized views as table functions 2022-10-04 17:32:48 +02:00
Anton Popov
148995894f add system table asynchronous_inserts_log 2022-10-03 21:35:55 +00:00
kssenii
229414ed09 Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2022-09-30 17:22:23 +02:00
kssenii
d538b5f3b8 Allow custom disk configuration 2022-09-30 17:20:10 +02:00
vdimir
f495361e28
fixes for TemporaryDataOnDisk 2022-09-29 10:09:29 +00:00
vdimir
14cd2b632c
fix Context::shutdown 2022-09-29 09:51:49 +00:00
vdimir
1234b70118
Handle decorator checking tmp disk type 2022-09-29 09:51:49 +00:00
vdimir
0f1a7c252d
better TemporaryDataOnDisk 2022-09-29 09:51:46 +00:00
Vladimir C
dd20fa37f5
Fix typos 2022-09-29 09:51:45 +00:00
vdimir
9f3f34548c
Allow to create temporaty streams on leaf TemporaryDataOnDisk 2022-09-29 09:51:45 +00:00
vdimir
15c7a3be34
Temp data on disk: build 2022-09-29 09:51:41 +00:00
vdimir
c0898ce289
Use abstraction for temporary data on disk in Sort and Aggregation 2022-09-29 09:51:41 +00:00
vdimir
367c77f0fb
Check if tmp volume contains only local disks 2022-09-29 09:51:39 +00:00
Sergei Trifonov
8edac7eba1
Merge branch 'master' into io-scheduler 2022-09-28 22:17:17 +02:00
Smita Kulkarni
f78f846503 Fixed issues executeTableFunctions in Context.cpp to fall back on TableFunction if not parameterized view & updated test - 40907 Parameterized views as table functions 2022-09-28 13:23:11 +02:00
Smita Kulkarni
e6672832b9 Removed unused errorcodes in Context.cpp - 40907 Parameterized views as table functions 2022-09-28 10:17:04 +02:00
Smita Kulkarni
618f63d6c7 Updated executeTableFunctions in Context.cpp to check for table/view & fallback to function to fix test fails - 40907 Parameterized views as table functions 2022-09-28 10:00:12 +02:00
kssenii
01c4299481 Support single disk instead of storage policy 2022-09-27 22:34:31 +02:00
Smita Kulkarni
bbc33a54b2 Addressed review comments - 40907 Parameterized views as table functions 
Implementation
* Fix for clang-today build fails - updated to use const in Context.cpp & const function in ActionsVisitior.cpp
* Updated to use QueryParameterVisitor to check if query has query parameters
* Updated executeTableFunction to check if table/table exists instead of try-catch approach
* Fixed small review comments and style comments.
Documentation:
* Addressed review comments and added the LIVE view part which was removed by mistake in the previous commits.
 2022-09-27 16:30:59 +02:00
serxa
b057d07977 IO scheduling subsystem 2022-09-27 13:26:41 +00:00
kssenii
0a801dad2a Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-23 19:39:07 +02:00
kssenii
30726721ad Fix threadpool reader (for local fs) 2022-09-23 19:35:16 +02:00
Smita Kulkarni
bb451b2b27 40907 Parameterized views as table functions 
Implementation:
* Updated parsers by adding a bool allow_query_parameters while creating ordinary view, which is used in interpreters to allow query parameters in SELECT.
* Added a check in ActionsVisitor if multiple parameters have same names while creating parameterised view.
* Added bool in StorageView to represent parameterized view.
* Updated processing of SELECT with parameter values to check for views and added substitution of values in the query parameters.

Testing:
* Added a test tests/queries/0_stateless/02428_parameterized_view.sql

Documentation:
* Updated the english documentation for VIEW.
 2022-09-23 16:46:57 +02:00
kssenii
ab702e43fc Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-23 15:21:33 +02:00
Nikita Mikhaylov
dbcba1490c
Try to make MergeTreeRestartingThread more reactive in case of expired sessions (#41092) 2022-09-23 14:40:04 +02:00
Alexander Tokmakov
021b882455 Revert "Disable parallel s3 multipart upload for part moves. (#41268)" 
This reverts commit 0414d95878.
 2022-09-22 13:59:25 +03:00
Alexey Milovidov
45afacdae4
Merge pull request #41186 from ClickHouse/fix-three-fourth-of-trash 
Fix more than half of the trash
 2022-09-22 07:28:26 +03:00
Nikolai Kochetov
0414d95878
Disable parallel s3 multipart upload for part moves. (#41268) 
* Disable parallel s3 multipart upload for part moves.

* Add setting s3_allow_parallel_part_upload
 2022-09-21 19:10:32 +02:00
Alexey Milovidov
ab4db2d0c4 Fix 5/6 of trash 2022-09-19 08:50:53 +02:00
Robert Schulze
b32b02d844
Merge pull request #40897 from ClickHouse/catboost-bridge-resurrected 
Move CatBoost evaluation into clickhouse-library-bridge
 2022-09-16 13:12:09 +02:00
kssenii
52ef3758c4 Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-13 16:34:31 +02:00
Kseniia Sumarokova
b2c9c04c7b
Merge pull request #40821 from kssenii/improve-marks-cache-loading 
Allow to load marks in threadpool in advance
 2022-09-13 12:31:58 +02:00
kssenii
b7d751b782 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-11 13:23:30 +02:00
Alexander Tokmakov
9bcec2d49a stop background ops before converting db engine 2022-09-09 18:27:19 +02:00
kssenii
6a885430d7 Add assertion 2022-09-09 13:36:23 +02:00
Robert Schulze
c16707ff00
chore: delete obsolete modelEvaluate() function + SYSTEM.MODELS view 
- The deleted function modelEvaluate() was superseded by
  catboostEvaluate().

- Also delete the external model repository, as modelEvaluate() was it's
  last user. Additionally remove the system view SYSTEM.MODELS for
  inspecting the repository.

- SYSTEM RELOAD MODELS is also obsolete. HOWEVER, it was retained and
  made a no-op instead of deleted.

  Why?
  The reason is that RBAC in distributed setups works by storing
  privileges (granted and revoked) as plain SQL statements in Keeper.
  Nodes read these statements at startup and parse them. If a privilege
  for SYSTEM RELOAD MODELS exists but parser doesn't recognize it
  nodes would fail to come up.

  Considered but rejected alternatives:
  - Ignore SYSTEM RELOAD MODELS during parsing RBAC privileges and
    return an error for regular SYSTEM RELOAD MODELS SQL. Special-case
    of no-op behavior, too brittle.
  - Remove SYSTEM RELOAD MODELS manually from Keeper via command-line
    manipulation of Keeper nodes or via SQL by dropping the privileges.
    Needs user intervention during upgrade.
 2022-09-08 09:10:11 +00:00
kssenii
e51313b6b3 Get rid of static threadpools 2022-09-07 17:48:11 +02:00
kssenii
487bc0fba3 Fix heap use after free 2022-09-06 16:50:58 +02:00
kssenii
4c8aa04b73 Revert "Fix" 
This reverts commit ec55c90f36.
 2022-09-05 20:12:40 +02:00
kssenii
ec55c90f36 Fix 2022-09-01 17:04:34 +02:00
kssenii
a5f69517ae Allow to load marks in threadpool 2022-08-30 20:20:05 +02:00
Frank Chen
f17d56b528 Merge branch 'master' into tracing_context_propagation 2022-08-30 14:24:36 +08:00
Vladimir C
5cbe7e0846
Merge pull request #40548 from ClickHouse/vdimir/warn-suppress-40330 
Add config option warning_supress_regexp
 2022-08-29 14:02:00 +02:00
Azat Khuzhin
c801469e9e Fix LOGICAL_ERROR with max_read_buffer_size=0 during reading marks 
The problem is that the buffer size validated after marks reading in
MergeTreeReaderStream::init(), since it requires to read marks first.

And later it is passed to AsynchronousReadBufferFromFileDescriptor,
which throws LOGICAL_ERROR because buffer_size < alignment.

Fix this my simply disallow such values for max_read_buffer_size (I
thougt about modifying createReadBufferFromFileBase(), but it is not
used for remote reads -- remote_fs_buffer_size).

Fixes: #40669
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-27 14:12:07 +02:00
vdimir
414872b1b9 Rename logger.warning_supress_regex -> warning_supress_regexp 2022-08-25 11:00:01 +02:00
vdimir
19cb068f64 Add config option warning_supress_regex 2022-08-25 11:00:01 +02:00
Frank Chen
99c37ce6c6
Merge branch 'master' into tracing_context_propagation 2022-08-25 10:07:16 +08:00
vdimir
1321ac87b5
Minor fixes 2022-08-24 16:14:07 +00:00
kssenii
26384b2543 Fix 2022-08-22 00:02:05 +02:00
Frank Chen
a3b6ad2a65
Merge branch 'master' into tracing_context_propagation 2022-08-18 20:59:07 +08:00
Kseniia Sumarokova
d8fd1f9c37
Merge pull request #36171 from kssenii/make-cache-composable 
Make cache composable, allow not to evict certain files (regarding .idx, .mrk, ..), delete old cache version
 2022-08-18 12:37:34 +02:00
Vladimir C
7ac0eb1af1
Merge pull request #40035 from ClickHouse/vdimir/cluster-discovery-observer 
Add observer mode to (zoo)keeper cluster discovery feature
 2022-08-18 10:53:49 +02:00
kssenii
eb26b219b9 Merge master 2022-08-16 00:56:27 +02:00
Kseniia Sumarokova
a6cfc7bc3b
Merge pull request #34651 from alexX512/master 
New caching strategies
 2022-08-12 17:23:37 +02:00
vdimir
cef6c4f8ce
Add observer mode to (zoo)keeper cluster discovery feature 2022-08-12 13:26:12 +00:00
Nikita Taranov
17956cb668
Extend protocol with query parameters (#39906) 2022-08-12 14:28:35 +02:00
alexX512
62f84b88eb Review fixes 2022-08-07 20:02:21 +00:00
alexX512
2be72560f4 Add loading of mark cache policy 2022-08-07 19:59:39 +00:00
alexX512
1133e42367 Add CacheBase clas instead of LRUCache and SLRUCache fo simpler configuration oache policiesf 2022-08-07 19:16:40 +00:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
kssenii
21ebf8874e Merge remote-tracking branch 'upstream/master' into make-cache-composable 2022-07-19 13:36:42 +02:00
kssenii
0aed5a602a Merge master 2022-07-18 16:16:28 +02:00
Robert Schulze
13482af4ee
First try at reducing the use of StringRef 
- to be replaced by std::string_view
- suggested in #39262
 2022-07-17 17:26:02 +00:00
kssenii
8ad26b3111 Merge master 2022-07-16 22:50:03 +02:00
Robert Schulze
deda29b46b
Pass const StringRef by value, not by reference 
See #39224
 2022-07-15 11:34:56 +00:00
Kruglov Pavel
a944a92d4c
Merge pull request #39224 from Avogar/string-view-by-value 
Pass const std::string_view by value, not by reference
 2022-07-15 13:05:56 +02:00
avogar
9291d33080 Pass const std::string_view & by value, not by reference 2022-07-14 16:11:57 +00:00
Sergei Trifonov
b2317ff710 renaming to be consistent 2022-07-14 17:33:22 +02:00
Sergei Trifonov
15ab3bc99f use context->getWriteSettings() 2022-07-13 19:48:57 +02:00
Sergei Trifonov
f854507729 merge master and resolve conflicts 2022-07-13 19:07:28 +02:00
kssenii
0d2d3ac2b4 Merge master 2022-07-13 13:26:15 +02:00
Sergei Trifonov
43779ec280 add max_remote_{read,write}_network_bandwidth_for_server settings 2022-07-11 14:59:39 +02:00
Frank Chen
da57a993e4 Fix CI 2022-07-09 13:43:10 +08:00
Frank Chen
9711559391 Remove extra code 2022-07-08 18:56:07 +08:00
Frank Chen
2e8c530bed Move thread trace context out of ThreadStatus 2022-07-07 17:41:10 +08:00
kssenii
e55169ebad Fix 2022-07-05 20:15:35 +02:00
lingpeng0314
cca94f01ae add function tryGetKeeperDispatcher for context 
- to make sure other case will get error if dispatcher not initialized
 2022-07-04 13:35:08 +08:00
lingpeng0314
03a5ce542d Remove Exception in getKeeperDispatcher 
- to handle the nil case by caller
 2022-07-01 11:27:40 +08:00
lingpeng0314
1d463718ca Change getKeeperDispatcher error type 2022-06-20 17:11:15 +08:00
kssenii
4813b90281 Merge master 2022-06-13 21:32:00 +02:00
kssenii
aaf9a46f07 Merge master 2022-06-11 15:56:05 +02:00
Kseniia Sumarokova
580a30c6ae
Merge pull request #37859 from KinderRiven/support_max_request_cache_size 
Support to set max cache size for per query in local cache
 2022-06-10 16:17:18 +02:00
KinderRiven
6883ecda96 fix 2022-06-08 14:19:50 +08:00
KinderRiven
3b2a427002 fix 2022-06-07 11:50:56 +08:00
KinderRiven
13e74bef5e support skip_download_if_exceeds_query_cache 2022-06-06 02:36:23 +08:00
KinderRiven
3ffb0b3549 support_max_query_cache_size 2022-06-05 16:21:36 +08:00
Alexander Tokmakov
3d346c766a better code 2022-06-01 16:49:26 +02:00
Alexander Tokmakov
4e52f45695 Merge branch 'master' into fix_trash 2022-05-28 19:43:19 +02:00
kssenii
25b1ed526d Merge master 2022-05-27 16:50:45 +02:00
alesapin
6d6779f17a
Merge pull request #37139 from ClickHouse/i_object_storage 
Separate object storage operations from disks
 2022-05-27 13:59:50 +02:00
Nikolai Kochetov
fea2401f1f
Merge pull request #37532 from ClickHouse/add-separate-mutex-for-factories-info 
Use a separate mutex for query_factories_info in Context.
 2022-05-26 13:03:28 +02:00
Alexander Tokmakov
3d259fa0de
Update Context.cpp 2022-05-25 23:18:07 +03:00
kssenii
b89a4edc04 Merge master 2022-05-25 20:55:04 +02:00
alesapin
6f5c86e55e Merge branch 'master' into i_object_storage 2022-05-25 20:49:01 +02:00
Alexander Tokmakov
779e6ea0b9 make it better, fix on cluster queries 2022-05-25 20:17:49 +02:00
Nikolai Kochetov
7b681fa8ac Fixing build. 2022-05-25 17:15:23 +02:00
Nikolai Kochetov
6370c29049 Use a separate mutex for query_factories_info in Context. 2022-05-25 14:16:59 +00:00
Vitaly Baranov
497c70d786
Merge pull request #37269 from vitlibar/rework-access-control-notifications 
Rework AccessControl's notifications.
 2022-05-24 17:47:45 +02:00
Alexander Tokmakov
d0f998fb88 Merge branch 'master' into fix_trash 2022-05-23 21:25:56 +02:00
alesapin
06c3dd69c0 Move directories 2022-05-22 13:51:48 +02:00
kssenii
9aa20193d5 Merge master 2022-05-22 00:56:20 +02:00
kssenii
ffd8d9723a Continuation 2022-05-22 00:21:40 +02:00
Vitaly Baranov
58f4a86ec7 Rework notifications used in access management. 2022-05-21 10:15:39 +02:00
alesapin
654b27e307 Merge branch 'master' into i_object_storage 2022-05-20 11:56:13 +02:00
alesapin
92c15ec97c Get rid of IDiskRemote 2022-05-19 20:07:15 +02:00
Alexander Tokmakov
3bc39f9929 fixes 2022-05-18 21:57:20 +02:00
Alexander Tokmakov
9772924d06 Merge branch 'master' into fix_trash 2022-05-18 17:27:54 +02:00
Robert Schulze
43945cea1b
Fixing some warnings 2022-05-16 20:59:27 +02:00
kssenii
646cf4cc95 Better 2022-05-14 14:26:04 +02:00
kssenii
58faaf0ea3 Merge master 2022-05-13 18:51:07 +02:00
Azat Khuzhin
a0ba113e5e Mark Context::getInterserverIOHandler() as const 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-12 16:54:01 +03:00
Azat Khuzhin
a04b6ad314 Mark Context::getInterserverCredentials() as const 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-12 16:54:01 +03:00
Kseniia Sumarokova
59bfcd917b
Merge pull request #36876 from kssenii/extend-cache-log 
Log into filesystem_cache_log when cache not even attempted
 2022-05-12 12:38:36 +02:00
Kruglov Pavel
af3db5a992
Merge pull request #36726 from PolyProgrammist/s3clustermacro 
Handle cluster macro in s3Cluster table function
 2022-05-11 18:00:57 +02:00
Vasily Nemkov
de9a06df5b
Merge branch 'master' into fix_session_log_crash 2022-05-09 17:00:24 +04:00
Vasily Nemkov
58953b5f28 Fixed issues with concurrent access to User object 
ContextAccess might be updated from another thread, nullifying UserPtr.
Can happen if user was dropped in concurrent query.

This could crash server under certain conditions.
 2022-05-09 15:56:50 +03:00
kssenii
fd1ab053f8 Merge master 2022-05-06 12:50:33 +02:00
Vitaly Baranov
dd51265218
Merge pull request #36864 from vitlibar/backup-improvements-4 
Backup improvements
 2022-05-05 15:37:51 +02:00
alesapin
aba78e5ed0
Merge pull request #36859 from ClickHouse/revert-36858-revert-35637-memory-overcommit-free 
Revert "Revert "Memory overcommit: continue query execution if memory is available""
 2022-05-04 23:43:00 +02:00
mergify[bot]
64084b5e32
Merge branch 'master' into shared_ptr_helper3 2022-05-03 20:46:16 +00:00
kssenii
0266cdf125 Add entries to cache log when cache was not event attempted 2022-05-03 18:29:41 +02:00
Dmitry Novik
5ba7a55c18
Merge pull request #36650 from bigo-sg/hive_text_parallel_parsing 
Parallel parsing of hive text format
 2022-05-03 15:56:28 +02:00
Vitaly Baranov
5257ce31f8 Improved using ThreadPool for making backup or restoring, changed columns in system.backups. 2022-05-03 11:03:13 +02:00
Dmitry Novik
2ed5a4013a
Revert "Revert "Memory overcommit: continue query execution if memory is available"" 2022-05-03 00:45:13 +02:00
alesapin
f0b7af0aa2
Revert "Memory overcommit: continue query execution if memory is available" 2022-05-03 00:36:50 +02:00
Dmitry Novik
71b6f89166
Merge pull request #35637 from ClickHouse/memory-overcommit-free 
Memory overcommit: continue query execution if memory is available
 2022-05-02 19:00:18 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
KinderRiven
d595df1604 fix 2022-05-01 19:45:16 +08:00
KinderRiven
ce9a6965d0 impl system.log_table 2022-05-01 19:45:16 +08:00
KinderRiven
5a361a0db5 impl cache_log in filecache 2022-05-01 19:45:16 +08:00
KinderRiven
4db54ff6bc add log record 2022-05-01 19:45:16 +08:00
KinderRiven
29f0471caf add cache log 2022-05-01 19:45:16 +08:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase 
Reorganize source files so that base won't depend on Common
 2022-04-30 09:40:58 +03:00
mergify[bot]
e9fde5d067
Merge branch 'master' into memory-overcommit-free 2022-04-29 19:31:59 +00:00
Nikolai Kochetov
5807ca47b0
Merge pull request #36425 from ClickHouse/pool-size-hot-reload 
Allow to increase number of threads and tasks for background executors
 2022-04-29 17:24:12 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Maksim Kita
04429d85db Added user_defined_path config setting 2022-04-28 22:13:33 +02:00
Vadim Volodin
233f0b4b52 Handle cluster macro in s3Cluster table function 2022-04-28 18:38:17 +03:00
taiyang-li
99dee35b6e parallel parsing of hive text format 2022-04-26 14:33:10 +08:00
alesapin
04afe552f8 Trying to avoid segfaults 2022-04-25 22:51:09 +02:00
Vitaly Baranov
ed0af9d4a0 Improve shutdown with async backup/restore. 2022-04-25 16:34:34 +02:00
Nikita Mikhaylov
9baf7b14dc Fix build 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
aaa7112405 Even better 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
36bdee0499 Deleted settings 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
4818b63979 Better 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
8c325c2cdd Better 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
31ccb9c1c3 Done 2022-04-22 13:56:16 +00:00
kssenii
24e376337d Merge with master 2022-04-21 12:33:56 +02:00
Kseniia Sumarokova
766a84e64f
Merge pull request #35915 from kssenii/better-s3-settings 
Better settings configuration for s3
 2022-04-20 10:22:06 +02:00
Antonio Andelic
bbb0be6a44
Merge pull request #36402 from rschu1ze/clang-tidy-contains 
Activate clang-tidy warning "readability-container-contains"
 2022-04-20 08:15:46 +02:00
kssenii
7cfc0ac433 Merge with master 2022-04-19 15:13:45 +02:00
mergify[bot]
3c1dd44b80
Merge branch 'master' into fix-insert-select 2022-04-19 12:17:31 +00:00
kssenii
1f9a597b77 Merge master 2022-04-19 11:31:13 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
avogar
ae01646136 Place new logic of schema inference in insert select from table function under setting 2022-04-14 19:05:20 +00:00
kssenii
0912b038fa Merge master 2022-04-13 15:30:01 +02:00
kssenii
37242a0103 Allow non-evictable file segments based on predicate 2022-04-12 16:55:25 +02:00
kssenii
9804c39de7 Composable cache 2022-04-12 14:33:13 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
kssenii
87855c4ae7 Merge master 2022-04-08 12:32:48 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements 
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
 2022-04-08 12:06:26 +02:00
Alexander Tokmakov
6a46da93ae Merge branch 'master' into mvcc_prototype 2022-04-07 23:22:19 +02:00
Alexander Tokmakov
7f54e7b422 Merge branch 'master' into mvcc_prototype 2022-04-07 15:14:06 +02:00
Kruglov Pavel
73adbb4c15
Merge pull request #35986 from amosbird/better-scalar1 
Fix performance regression of scalar query
 2022-04-07 14:07:59 +02:00
Amos Bird
df06f9f974
Fix performance regression of scalar query 2022-04-06 17:50:22 +08:00
kssenii
4e50da722b Better s3 settings 2022-04-04 16:14:56 +02:00
Alexander Tokmakov
a2167f12b8 Merge branch 'master' into mvcc_prototype 2022-04-04 14:24:23 +02:00
Azat Khuzhin
14538f6456 Add system.processors_profile_log 
This is the system table that will contain Processors level profiling.

v2: one entry per Processor, not 3 (PortFull/NeedData/work())
v3: us over ms
v4: Enable processors_profile_log table by default

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-04 13:19:41 +03:00
kssenii
6c8e073a61 Merge master 2022-04-01 16:48:29 +02:00
Kruglov Pavel
a217e9e9d0
Remove unused header 2022-03-31 14:27:43 +02:00
kssenii
0fc92fe2aa Get rid of all "remote" mentions 2022-03-30 17:15:29 +02:00
avogar
849596c7a2 Improve schema inference for insert select queries 2022-03-30 12:44:53 +00:00
Alexander Tokmakov
287d858fda Merge branch 'master' into mvcc_prototype 2022-03-29 16:24:12 +02:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00
kssenii
59b7394caf WriteSettings 2022-03-21 09:52:48 +01:00
Alexander Tokmakov
c2ac8d4a5c review fixes 2022-03-16 21:05:34 +01:00
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
Alexander Tokmakov
278d779a01 log cleanup, more comments 2022-03-14 21:43:34 +01:00
Azat Khuzhin
d13fc9ec44 Wait for IDiskRemote thread pool properly 
ASan report [1]:

    WARNING: ThreadSanitizer: heap-use-after-free (pid=611)
      Read of size 4 at 0x7b48006f0250 by thread T273:
        0 MemoryTracker::allocImpl() obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:105:49 (clickhouse+0xa05d0ef)
        1 MemoryTracker::allocImpl() obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:239:22 (clickhouse+0xa05d580)
        2 MemoryTracker::alloc(long) obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:246:5 (clickhouse+0xa05dea7)
        3 DB::ThreadStatus::~ThreadStatus() obj-x86_64-linux-gnu/../src/Common/ThreadStatus.cpp:154:28 (clickhouse+0xa059b01)
        4 ThreadFromGlobalPool::ThreadFromGlobalPool<>()::'lambda'()::operator()() obj-x86_64-linux-gnu/../s>

      Previous write of size 8 at 0x7b48006f0250 by main thread:
        8 std::__1::unordered_map<std::__1::basic_string<>, DB::ProcessListForUser, std::__1::hash<>, std::__1>
        9 DB::ProcessList::~ProcessList() obj-x86_64-linux-gnu/../src/Interpreters/ProcessList.h:275:7 (clickhouse+0x17b71c9c)
        10 DB::ContextSharedPart::~ContextSharedPart() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:304:5 (clickhouse+0x17b70712)
        13 DB::SharedContextHolder::reset() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:453:44 (clickhouse+0x17b3daf9)
        14 DB::Server::main()::$_8::o>

      Thread T273 (tid=946, running) created by thread T223 at:
        0 pthread_create <null> (clickhouse+0x9f7d49d)
        15 DB::WriteBufferFromS3::makeSinglepartUpload() obj-x86_64-linux-gnu/../src/IO/WriteBufferFromS3.cpp:332:9 (clickhouse+0x17a81b6a)

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/35072/19be9c8c6433ffd1bbf8eb08975d8afbaebf3d43/stress_test__thread__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-14 12:07:47 +03:00
Alexander Tokmakov
0906b59fba fixes 2022-03-09 21:38:18 +01:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
taiyang-li
22a938fff0 solve issue: https://github.com/ClickHouse/ClickHouse/issues/34767 2022-03-08 19:16:09 +08:00
taiyang-li
d78493eeb2 commit again 2022-03-08 16:41:16 +08:00
taiyang-li
e21ab223c5 finish dev 2022-03-08 16:11:21 +08:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
Azat Khuzhin
081f9caa04 Avoid possible deadlock on server shutdown 
Here is an example for deadlock during shutting down DDLWorker:

Server main thread:

    6  ThreadFromGlobalPool::join () at ../src/Common/ThreadPool.h:217
    7  DB::DDLWorker::shutdown () at ../src/Interpreters/DDLWorker.cpp:123
    8  DB::DDLWorker::~DDLWorker () at ../src/Interpreters/DDLWorker.cpp:131
    9  DB::DDLWorker::~DDLWorker () at ../src/Interpreters/DDLWorker.cpp:130
    10 std::__1::default_delete<DB::DDLWorker>::operator() () at ../contrib/libcxx/include/memory:1397
    11 std::__1::unique_ptr<>::reset (this=0x7f7521d44fd0, __p=0x0) at ../contrib/libcxx/include/memory:1658
    12 DB::ContextSharedPart::shutdown () at ../src/Interpreters/Context.cpp:380 <!-- holds mutex
    13 DB::Context::shutdown () at ../src/Interpreters/Context.cpp:2677

DDLWorker thread:

    7  DB::Context::getLock () at ../src/Interpreters/Context.cpp:472 <-- trying to acquire shared.mutex
    8  DB::Context::getTCPPortSecure () at ../src/Interpreters/Context.cpp:2120
    9  DB::DDLTask::findCurrentHostID () at ../src/Interpreters/DDLTask.cpp:169
    10 DB::DDLWorker::initAndCheckTask () at ../src/Interpreters/DDLWorker.cpp:191
    11 DB::DDLWorker::scheduleTasks () at ../src/Interpreters/DDLWorker.cpp:367
    12 DB::DDLWorker::runMainThread () at ../src/Interpreters/DDLWorker.cpp:1063

v2: replace getTryLockTimed() with getTryLock() since std::recursive_mutex does not have try_lock_for()
v3: stole objects from context
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Fixes: #21318
 2022-03-06 14:31:18 +03:00