thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-04 21:42:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
139,960 Commits 9,185 Branches 1,782 Tags 2.3 GiB
aca46eecc9
Commit Graph

578 Commits

Author SHA1 Message Date
Nikita Mikhaylov
668062dc29
Merge branch 'master' into no-keep-context-lock-while-calculating-access 2023-07-20 14:42:13 +02:00
János Benjamin Antal
e74acda53e
PRQL integration (#50686) 
* Added prql-lib

* Add PRQL parser

* Extend stateless tests

* Add unit tests for `ParserPRQL`

---------

Co-authored-by: Ubuntu <ubuntu@ip-172-31-37-24.eu-central-1.compute.internal>
Co-authored-by: Ubuntu <ubuntu@ip-10-10-10-195.eu-central-1.compute.internal>
Co-authored-by: Александр Нам <47687537+seshWCS@users.noreply.github.com>
 2023-07-20 12:54:42 +02:00
Vitaly Baranov
815a3857de Remove non-const function Context::getClientInfo(). 2023-07-17 15:02:07 +02:00
Raúl Marín
dbdfc07790
Merge branch 'master' into async_to_query_log 2023-07-10 10:32:56 +02:00
Alexey Milovidov
db14b2c54f Remove useless logs 2023-07-06 02:16:38 +02:00
Raúl Marín
11236b3ce7 Merge remote-tracking branch 'blessed/master' into async_to_query_log 2023-06-26 17:13:19 +02:00
Raúl Marín
b1cedf0e42 Simplify handling of AsyncInsertFlush 2023-06-20 11:38:19 +02:00
Raúl Marín
9b899b9351 Fix async insert logs with setting on mergetree 2023-06-20 11:38:19 +02:00
Raúl Marín
cf74aee6c9 Log async query flushes to query_log 2023-06-20 11:38:19 +02:00
Raúl Marín
7ab660acfb Extract logging functions outside of executeQueryImpl 2023-06-20 11:38:19 +02:00
Robert Schulze
5311ebe1d2
Make query cache work effective with nested queries. 2023-06-17 20:27:25 +00:00
Robert Schulze
24717ed908
Factorize check if query cache exists 2023-06-17 18:24:20 +00:00
Robert Schulze
69e7c1cc82
Introduce convenience ctor for Key construction during read 2023-06-17 18:22:15 +00:00
Robert Schulze
6e8af09289
Provide query cache reader with dummy block 2023-06-17 12:48:29 +00:00
Robert Schulze
840e73eb74
Update #includes 2023-06-17 12:28:18 +00:00
Nikita Taranov
a3652f1727 Merge branch 'master' into optimize_reading2 2023-05-29 15:26:18 +02:00
Nikita Taranov
42a9072888 Merge branch 'master' into optimize_reading2 2023-05-28 18:04:44 +02:00
Anton Popov
612173e734 refactoring near alter conversions 2023-05-25 22:54:54 +00:00
Sema Checherinda
23f894b995
Merge pull request #49777 from helifu/master1 
Add 'initial_query_id' field for system.processors_profile_log
 2023-05-25 12:55:32 +02:00
helifu
802b63f2ab Add 'initial_query_id' field for system.processors_profile_log 
Facilitate profile data association and aggregation for the same query
 2023-05-25 09:37:02 +08:00
helifu
2255b0287a Add 'partitions' field for system.query_log 2023-05-24 20:42:31 +08:00
Nikita Taranov
0dd67bacf2
Merge branch 'master' into optimize_reading2 2023-05-17 15:06:41 +02:00
Alexey Milovidov
5a44dc26e7 Fixes for clang-17 2023-05-13 02:57:31 +02:00
Nikita Taranov
ec63a6ee7c
Merge branch 'master' into optimize_reading2 2023-05-10 17:24:37 +02:00
Robert Schulze
0598bfd79e
Mark query cache as non-experimental 2023-05-06 13:58:40 +00:00
Robert Schulze
983e37048c
Merge pull request #48853 from ClickHouse/rs/qc-totals-extremes2 
QueryCache: Support queries with totals and extremes
 2023-05-06 15:41:10 +02:00
Robert Schulze
9cd81400f3
Query cache: disable for internal queries 2023-05-05 19:22:14 +00:00
Robert Schulze
945a5130cf
Simplify simplify simplify 2023-05-05 16:28:51 +00:00
Robert Schulze
aba5db40a1
Merge remote-tracking branch 'origin/master' into rs/qc-totals-extremes2 2023-05-05 14:11:10 +00:00
Nikita Taranov
de03b90570 impl 2023-04-27 17:07:00 +00:00
Anton Popov
c7db58be66 add MergeTree setting async_insert 2023-04-25 01:11:58 +00:00
Robert Schulze
b565257a39
Support totals and extremes in query cache 2023-04-24 22:23:27 +00:00
Robert Schulze
38d5f12a32
Query Cache: Implement per-user quotas 2023-04-23 16:13:51 +00:00
Robert Schulze
b0473098c6
Merge branch 'master' into qc-compression 2023-03-30 10:55:42 +02:00
Nikolay Degterinsky
227425968f
Merge branch 'master' into query_id 2023-03-27 13:41:42 +02:00
Frank Chen
532590773a Fix typo check 2023-03-22 17:46:09 +08:00
Frank Chen
c1927d8f3e Fix check style 2023-03-22 16:39:14 +08:00
Frank Chen
3e05f2c58a Improve comment 2023-03-22 14:35:58 +08:00
Frank Chen
3de73d03cc Improve the robustness 2023-03-22 14:31:16 +08:00
Frank Chen
837638e1c1 Fix style 2023-03-21 15:27:13 +08:00
Frank Chen
91a34fa627 Set query id in case of exceptions during query execution 2023-03-21 14:10:01 +08:00
Robert Schulze
d8493780b2
Query cache: Enable compression and squashing of result blocks 
ClickHouse reads table data in blocks of 'max_block_size' rows. Due to
filtering, aggregation, etc., result blocks are typically much smaller
than 'max_block_size' but there are also cases where they are much
bigger. Setting 'query_cache_squash_partial_results' (enabled by
default) now controls if result blocks are squashed (if they are tiny)
or split (if they are large) into blocks of 'max_block_size' size before
insertion into the query result cache. This reduces performance of
writes into the query cache but improves compressability of cache
entries and provides more natural block granularity when query results
are later served from the query cache.

Entries in the query cache are now also compressed by default. This
reduces the overall memory consumption at the cost of slower writes into
/ reads from the query cache. To disable compression, use setting
'query_cache_compress_entries'.
 2023-03-20 20:11:41 +00:00
Nikolai Kochetov
815c5aa87e
Merge branch 'master' into apply-settings-for-explain-earlier 2023-03-10 21:19:07 +01:00
Nikolai Kochetov
d2c28fb555 Apply settings for explain subquery as well. 2023-03-10 18:29:13 +00:00
Nikolai Kochetov
f6ac225f98 Apply settings for EXPLAIN earlier (in the same way we do for SELECT). 2023-03-10 15:24:47 +00:00
Alexander Tokmakov
9a95e7b4ee
Merge pull request #47312 from ClickHouse/fix_implicit_transactions 
Fix an assertion with implicit transactions in interserver mode
 2023-03-09 16:47:40 +03:00
Alexander Tokmakov
debd69f03a fix an assertion with implicit transactions in interserver mode 2023-03-08 16:53:23 +01:00
Igor Nikonov
838ff88abe
Merge branch 'master' into log-level-change 2023-03-06 18:00:49 +01:00
Alexander Tokmakov
a97e15e36f Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-03-01 14:26:41 +01:00
AndyB
5fb1f20882 LOG_DEBUG over LOG_INFO 2023-02-27 23:16:34 +00:00
Anton Popov
37570f10ea
Merge pull request #46364 from CurtizJ/add-query-id-in-response 
Add `X-ClickHouse-Query-Id` header in all queries
 2023-02-25 14:18:35 +01:00
Alexander Tokmakov
cad1e0b768 fix 2023-02-25 01:18:34 +01:00
Alexander Tokmakov
fe92fd8a61 Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-02-23 22:35:31 +01:00
Anton Popov
d5864fa88e allow to fallback from async insert in case of large amount of data 2023-02-22 21:59:24 +00:00
Alexander Tokmakov
e660c0838c fix multipart requests 2023-02-22 17:54:35 +01:00
Alexander Tokmakov
592af6d652 fix incomplete interst through http 2023-02-22 02:34:03 +01:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
Anton Popov
e9e6d735ef add query_id header in all queries 2023-02-13 16:44:54 +00:00
kssenii
b0b865c32e Resubmit prefetches 2023-02-08 21:26:24 +01:00
Alexey Milovidov
62bfdb349a Merge branch 'master' into fix-trash-query-kind 2023-02-02 04:26:49 +01:00
Alexey Milovidov
9b908a0be0 Fix trash around query kind, part 2 2023-02-01 03:31:31 +01:00
Robert Schulze
325c6bdf3d
Renaming: "Query Result Cache" --> "Query Cache" 
Reasons:

- The cache will at some point store intermediate results as opposed to
  only query results. We should change the terminology now without
  having to worry about backward compat.

- Equivalent caches in MySQL (1) and Starrocks (2) are called "query
  cache".

- The new name is ca. 13.8% more catchy.

(1) https://dev.mysql.com/doc/refman/5.6/en/query-cache.html
(2) https://docs.starrocks.io/en-us/2.5/using_starrocks/query_cache
 2023-01-31 09:54:34 +00:00
Robert Schulze
659a64a1d9
Merge pull request #45679 from ClickHouse/untangle-query-result-cache-settings 
Untangle configuration of query result cache
 2023-01-31 10:44:14 +01:00
Robert Schulze
27b978381b
Use more speaking setting names 2023-01-30 14:20:43 +00:00
Robert Schulze
7d97fab2dc
Untangle query result cache settings 
This PR modularizes the configuration options of the query result cache
(QRC).

1. Add a feature toggle "allow_experimental_query_result_cache = false"
   which controls whether the experimental QRC can be used at all. This
   configuration option will be removed once the QRC becomes
   non-experimental.

2. Modularize the existing two configuration settings
   "enable_experimental_query_result_cache" and
   "enable_experimental_query_result_cache_passive_usage" into three
   orthogonal settings:

   - "use_query_result_cache = false" controls whether a query (or all
     queries of the session) utilize the query result cache

   - settings "use_query_result_cache_passive_usage = true" and
     "use_query_result_cache_active_usage = true" control the way a
     query (or all queries of the session) utilize the query result
     cache: "passive" use is reading, "active" use is writing.
 2023-01-26 21:17:48 +00:00
Alexander Tokmakov
3744fa2c63 format more messages 2023-01-25 21:16:42 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Robert Schulze
340f406553
Merge branch 'master' into query-result-cache 2023-01-22 13:21:36 +01:00
Alexander Tokmakov
7a824af09e fix 2023-01-18 14:30:20 +01:00
Alexander Tokmakov
1413b9537c make error patterns more useful 2023-01-17 20:04:25 +01:00
Robert Schulze
099e30ef2a
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-16 08:04:49 +00:00
Kseniia Sumarokova
db3e0219fc
Merge pull request #41687 from ClickHouse/40907_Parameterized_views_as_table_functions 
40907 Parameterized views as table functions
 2023-01-12 14:24:32 +01:00
Robert Schulze
ad4884b394
Fix review comments, pt. I 2023-01-09 21:42:51 +00:00
Robert Schulze
45dbcf88e5
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-06 20:42:48 +00:00
Robert Schulze
4fcead15c3
Fix handling of exceptions during query runtime 2023-01-04 22:33:04 +00:00
Robert Schulze
0c15e06578
Cosmetics 2023-01-04 22:31:52 +00:00
Robert Schulze
4f3051de19
Avoid entries that never become stale 2023-01-02 10:49:38 +00:00
Robert Schulze
8381d8b467
query_result_cache_keep_seconds_alive --> query_result_cache_ttl 2023-01-02 09:38:40 +00:00
Robert Schulze
ba55fd3ad2
Cosmetics 2023-01-02 09:30:44 +00:00
Robert Schulze
e9e04166d9
Merge remote-tracking branch 'origin/master' into query-result-cache 2023-01-02 08:20:27 +00:00
Kruglov Pavel
894726bd8f
Merge branch 'master' into improve-streaming-engines 2022-12-29 22:59:45 +01:00
SmitaRKulkarni
02ff2ea8ac
Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-12-29 13:56:00 +01:00
Raúl Marín
5de11979ce
Unify query elapsed time measurements (#43455) 
* Unify query elapsed time reporting

* add-test: Make shell tests executable

* Add some tests around query elapsed time

* Style and ubsan
 2022-12-28 21:01:41 +01:00
Robert Schulze
bc6bcdfd7b
Allow sharing of cache entries 2022-12-17 18:04:18 +00:00
Robert Schulze
20a7af926e
Normalize AST before QRC insert/lookup 2022-12-17 16:48:11 +00:00
Robert Schulze
021311be8d
Fix passive usage 2022-12-17 15:48:22 +00:00
Robert Schulze
aa50b9cf48
Invert config parameter name for more clarity 2022-12-17 15:10:04 +00:00
Robert Schulze
d1615ec283
Display metrics for query result cache 2022-12-16 10:49:15 +00:00
Robert Schulze
369046ad3f
Move hasNonCacheableFunctions() into QueryResultCache file 2022-12-16 10:38:52 +00:00
Smita Kulkarni
16d8fd02fb Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-12-16 08:52:48 +01:00
Robert Schulze
4db33f16e3
Make DoS attack on cache less likely 2022-12-15 21:42:51 +00:00
Robert Schulze
bcc54ab9b4
Add cfg for min query duration for caching 2022-12-15 20:10:37 +00:00
Kruglov Pavel
c5b2e4cc23
Merge branch 'master' into improve-streaming-engines 2022-12-15 18:44:35 +01:00
Robert Schulze
537c64aad2
Merge remote-tracking branch 'origin/master' into query-result-cache 
Conflicts:
- tests/queries/0_stateless/02117_show_create_table_system.reference
 2022-12-15 12:17:58 +00:00
Robert Schulze
973d4f75de
More intuitive active/passive enablement of query result cache 2022-12-15 12:16:36 +00:00
Robert Schulze
f1e367179e
Make maximum record count per item configurable 2022-12-15 11:16:44 +00:00
Robert Schulze
e52dfaf45c
Make number of cached items configurable 2022-12-15 10:08:28 +00:00
Robert Schulze
fc972d760f
Make it configurable if queries with non-deterministic functions are cached 2022-12-12 14:03:21 +00:00
Robert Schulze
b78652e138
Query result cache 2022-12-12 10:37:10 +00:00
Alexander Tokmakov
d557f4a200 hold ProcessListEntry a bit longer in case of exception from Interpreter 2022-12-01 17:08:20 +01:00
Smita Kulkarni
953e1068b5 Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-11-28 19:10:32 +01:00
Anton Popov
c811f34a41
Simplify AsynchronousInsertQueue and fix race (#43233) 2022-11-25 15:02:22 +01:00
Smita Kulkarni
c84a50e77c Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-11-16 12:40:04 +01:00
Vitaly Baranov
8d72f75556 Make the password wiping an option of IAST::format(). 2022-11-16 01:35:06 +01:00
Vitaly Baranov
8e99f5fea3 Move maskSensitiveInfoInQueryForLogging() to src/Parsers/ 2022-11-14 18:55:19 +01:00
Vitaly Baranov
e18c97faf7 Remove dependencies maskSensitiveInfo() from Context. 2022-11-14 18:55:19 +01:00
Kruglov Pavel
58b7f6ca04
Merge pull request #42320 from Avogar/improve-schema-inference-insert-select 
Improve using structure from insertion table in table functions
 2022-11-14 14:51:27 +01:00
Smita Kulkarni
4dec8337f0 Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-11-11 19:04:35 +01:00
Kruglov Pavel
83c5c44b11
Merge branch 'master' into improve-schema-inference-insert-select 2022-11-03 18:41:51 +01:00
Kruglov Pavel
b124875257
Merge branch 'master' into improve-streaming-engines 2022-11-03 13:22:06 +01:00
Smita Kulkarni
9aa06e48f6 Merge branch 'master' into 40907_Parameterized_views_as_table_functions 2022-10-31 15:46:52 +01:00
Vitaly Baranov
dcf8724adf Move prepareQueryForLogging() to a separate header. 2022-10-31 10:50:33 +01:00
Vitaly Baranov
5d2a222fe4 Mask sensitive information in logs. 2022-10-31 10:50:33 +01:00
avogar
8e13d1f1ec Improve and refactor Kafka/StorageMQ/NATS and data formats 2022-10-28 16:41:10 +00:00
Azat Khuzhin
56bc85746f Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Interpreters/ProcessList.cpp
 2022-10-22 16:49:08 +02:00
Smita Kulkarni
d8fe1d2d63 Moved is_create_parameterized_view out of Context , and added into ActionsMatcher::Data and SelectQueryInfo - 40907 Parameterized views as table functions 2022-10-21 14:45:28 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Smita Kulkarni
e4ac3d0e18 Added FunctionParameterValuesVisitor, renamed bool to is_create_parameterized_view, added checks for parameterized view and support & test to propagate query parameters - 40907 Parameterized views as table functions 2022-10-19 18:30:03 +02:00
Alexey Milovidov
dfa202a15d Merge branch 'master' into fix-race-condition-finish-cancel 2022-10-19 02:35:42 +02:00
Kruglov Pavel
9588653a6e
Merge branch 'master' into improve-schema-inference-insert-select 2022-10-18 15:17:27 +02:00
Alexey Milovidov
f88ed8195b Fix trash 2022-10-17 04:21:08 +02:00
Anton Popov
ec81d61f48 better logging for async insert 2022-10-16 12:45:02 +00:00
Kruglov Pavel
f1dd247181
Merge branch 'master' into improve-schema-inference-insert-select 2022-10-14 17:16:58 +02:00
avogar
10335aec13 Improve using structure from insertion table in table functions 2022-10-14 15:09:35 +00:00
Kruglov Pavel
d43b5bbcd0
Remove forgotten debug logging 2022-10-14 15:47:46 +02:00
Kseniia Sumarokova
3819a21033
Merge pull request #42078 from azat/insert-progress 
Fix progress for INSERT SELECT
 2022-10-11 10:56:20 +02:00
Azat Khuzhin
5166a1d58e Fix written_bytes/written_rows in INSERT SELECT intermediate progress packets 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-05 19:12:18 +02:00
Anton Popov
148995894f add system table asynchronous_inserts_log 2022-10-03 21:35:55 +00:00
Vitaly Baranov
a5b2a49ab1 Refactor wiping sensitive information from logs. 2022-10-01 17:17:45 +02:00
Yakov Olkhovskiy
7c94f98f6a
Merge pull request #37961 from ClibMouse/Kusto-phase1 
Kusto phase1
 2022-09-27 12:42:15 -04:00
Alexey Milovidov
45afacdae4
Merge pull request #41186 from ClickHouse/fix-three-fourth-of-trash 
Fix more than half of the trash
 2022-09-22 07:28:26 +03:00
Kruglov Pavel
22e11aef2d
Merge pull request #40910 from Avogar/new-json-formats 
Add new JSON formats, add improvements and refactoring
 2022-09-21 14:19:08 +02:00
Alexey Milovidov
45bd3cfc30 Merge branch 'master' into fix-three-fourth-of-trash 2022-09-20 21:27:41 +02:00
Vitaly Baranov
f58f8104b5
Merge pull request #41474 from vitlibar/mask-some-info-in-logs 
Mask some information in logs.
 2022-09-20 12:11:02 +02:00
Vitaly Baranov
f75dd93965 Mask some information in logs. 2022-09-19 15:00:06 +02:00
Alexey Milovidov
ab4db2d0c4 Fix 5/6 of trash 2022-09-19 08:50:53 +02:00
Yong Wang
37ae7a8cca Kusto-phase1 : apply parser comments to kusto, remove unused variable 2022-09-18 20:25:27 -07:00
Yong Wang
4e4a04c08b
Merge branch 'ClickHouse:master' into Kusto-phase1 2022-09-18 21:40:30 -04:00
Frank Chen
d3265150c0 Make sure span is finished in the onFinish callback 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 18:28:21 +08:00
avogar
ad68b7be0f Better 2022-09-09 15:01:45 +00:00
Yong Wang
2ca54d12ee
Merge branch 'ClickHouse:master' into Kusto-phase1 2022-09-05 17:42:38 -04:00
Frank Chen
9d63cbe811 Merge 'origin/master' into tracing_context_propagation to resolve conflicts 2022-09-01 23:18:59 +08:00
Frank Chen
1da35a486c Disable query_span for internal queries 2022-09-01 23:16:05 +08:00
Duc Canh Le
8590cc46c4 implement intersect + except distinct 2022-08-30 18:09:01 +08:00
Frank Chen
92f7ca3616 Move TracingContextOnThread::current() out of class for simplicity 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:23:56 +08:00
Frank Chen
bb00dcc19b Remove using namespace from header 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:20:13 +08:00
Frank Chen
cd19366b44 Move classes into DB::OpenTelemetry namespace 2022-08-24 16:41:40 +08:00
Frank Chen
efc6a60a60 Clean code 2022-08-24 15:59:44 +08:00
Frank Chen
a3b6ad2a65
Merge branch 'master' into tracing_context_propagation 2022-08-18 20:59:07 +08:00
Yong Wang
c2c457ea0e Kusto-phase1: Change the dialect to Enum, rename sql_dialect to dialect, set limit to subquery 2022-08-17 06:03:41 -07:00