thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-01 20:12:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
91,855 Commits 9,193 Branches 1,782 Tags 2.2 GiB
ea0996674f
Commit Graph

27917 Commits

Author SHA1 Message Date
Nikolai Kochetov
b8d27aa8dd
Merge pull request #37469 from azat/projections-optimize_aggregation_in_order 
Implement in order aggregation (optimize_aggregation_in_order) for projections for tables with fully materialized projections
 2022-06-21 12:17:35 +02:00
Dmitry Novik
e10f079bd3 Get rid of allocations in OvercommitTracker 2022-06-21 10:15:33 +00:00
Kseniia Sumarokova
71ee5dbbb2
Merge pull request #38227 from kssenii/diff-reduce 
buffer's getFileSize small changes
 2022-06-21 12:12:22 +02:00
Nikolai Kochetov
e45d552f4d Do not create empty folder for in-memory parts. 2022-06-21 09:59:46 +00:00
mergify[bot]
bdd5b94192
Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-21 08:13:04 +00:00
Nikolai Kochetov
5232c17b89 Fix style 2022-06-21 07:30:45 +00:00
Nikolai Kochetov
3429a5e07a Fix 2022-06-21 07:26:43 +00:00
Saulius Valatka
6bbabf59d9 simplify build, add read method randomized tests, fix typos 2022-06-21 09:59:46 +03:00
Saulius Valatka
68eef98128 check for io_uring support, fail if not supported 2022-06-21 09:59:46 +03:00
Saulius Valatka
0dc7854af1 add initial io_uring support 2022-06-21 09:59:46 +03:00
Saulius Valatka
3f755a2505 add liburing to build 2022-06-21 09:59:46 +03:00
Alexander Tokmakov
75b8800b5c
Merge pull request #37822 from ClickHouse/fix_flaky_tests_with_transactions 
Try fix flaky tests with transactions
 2022-06-21 09:38:25 +03:00
Larry Luo
bbd73ba727 use utility methods to access x509 struct fields. 2022-06-20 21:27:33 -04:00
Dmitry Novik
9ca368ac20 Use do while(false) in macro 2022-06-20 23:10:40 +00:00
Anton Popov
dcc9cd0367
fix style check 2022-06-20 22:50:39 +02:00
Maksim Kita
cb018348cf
Merge pull request #38022 from kitaisreal/sorting-added-batch-queue-variants 
Sorting added batch queue variants
 2022-06-20 22:35:44 +02:00
zvonand
22af00b757 rename variable + fix handling of ENABLE_LIBRARIES 2022-06-20 23:53:47 +05:00
Nikolai Kochetov
dccf90b1ea Cleanup. 2022-06-20 18:18:17 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Dmitry Novik
f6692c34e6
Merge pull request #34632 from excitoon-favorites/optimizedprocessing 
Optimized processing of ORDER BY in window functions
 2022-06-20 20:03:26 +02:00
Dmitry Novik
53d656d89f Fix deadlock in OvercommitTracker logging 2022-06-20 17:35:24 +00:00
Kseniia Sumarokova
2d039b739c
Update ReadBufferFromFileDescriptor.cpp 2022-06-20 19:13:20 +02:00
Kseniia Sumarokova
e48ce50863
Update ArrowBufferedStreams.cpp 2022-06-20 19:12:51 +02:00
Anton Popov
464cb59920 slightly better code 2022-06-20 16:22:36 +00:00
Anton Popov
59a9ce6def fix reading from s3 in some corner cases 2022-06-20 15:24:38 +00:00
kssenii
cea81822bd Fix 2022-06-20 17:19:36 +02:00
mergify[bot]
b440ee84ae
Merge branch 'master' into fix-short-circuit 2022-06-20 15:14:19 +00:00
zvonand
d4e5686b99 minor: fix message for base64 2022-06-20 20:13:09 +05:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
zvonand
78d55d6f46 small fixes 2022-06-20 19:30:54 +05:00
Antonio Andelic
0c76fc1121 Fix style 2022-06-20 14:21:59 +00:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
Antonio Andelic
777ae72259 Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-20 14:06:45 +00:00
Antonio Andelic
df31b562ec Use sync after reconnect to avoid stale reads 2022-06-20 14:05:51 +00:00
Nikolai Kochetov
39b02d4ab8 Continure resolving conflicts. 2022-06-20 15:39:05 +02:00
Nikolai Kochetov
7452d04e3a Merge branch 'master' into refactor-something-in-part-volumes 2022-06-20 15:31:02 +02:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
Nikolai Kochetov
ef8b967c3f Fix more tests. 2022-06-20 13:07:14 +00:00
Antonio Andelic
6c225bb04d
Update src/Common/ZooKeeper/ZooKeeperImpl.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-06-20 14:55:59 +02:00
Dmitry Novik
77177917bd
Merge pull request #37848 from ClickHouse/window-function-expression 
Support expressions with window functions
 2022-06-20 14:21:13 +02:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6 
Backup Improvements 6
 2022-06-20 14:13:30 +02:00
alesapin
7ae775d2db Review fixes 2022-06-20 14:01:05 +02:00
alesapin
e3184d264b Merge branch 'master' into disk_transaction 2022-06-20 13:47:30 +02:00
mergify[bot]
4ed606a275
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 10:17:51 +00:00
Kseniia Sumarokova
a756b4be27
Merge pull request #37391 from azat/insert-profile-events-fix 
Send profile events for INSERT queries (previously only SELECT was supported)
 2022-06-20 12:16:29 +02:00
Kseniia Sumarokova
c83594284d
Merge pull request #38205 from azat/fix-window-view 
(Window View is an experimental feature) Fix LOGICAL_ERROR for WINDOW VIEW with incorrect structure
 2022-06-20 11:47:45 +02:00
Nikolai Kochetov
fab62513aa
Merge pull request #38029 from ClickHouse/fix-possible-crash-after-removing-replica-in-distributed 
Fix possible crash in Distributed async insert in case of removing a replica from config.
 2022-06-20 11:38:35 +02:00
Vitaly Baranov
a6fc0dea4e Fix clang-tidy more. 2022-06-20 11:04:37 +02:00
Vladimir Chebotarev
aef6fe6008 Rebase fix. 2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
bfb67ded2c Rebase fix. 2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
b81df2be83 Update src/Interpreters/ExpressionAnalyzer.cpp 
Co-authored-by: Dmitry Novik <d1mas1k4@yandex.ru>
 2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
92a553fb77 Build fix. 2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
6a363b7429 Build fix. 2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
d41c97ea1d Review fixes. 2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
4f38e01343 Unused code. 2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
cc45f15eae Build fix. 2022-06-20 05:15:08 +03:00
Vladimir Chebotarev
3c2a63b87a Fix test. 2022-06-20 05:15:07 +03:00
Vladimir Chebotarev
e50210969f Style. 2022-06-20 05:15:07 +03:00
Vladimir Chebotarev
7f9557f8a3 Added optimize_read_in_window_order setting. 2022-06-20 05:15:07 +03:00
Vladimir Chebotarev
ec22f6d539 Draft. 2022-06-20 05:15:07 +03:00
Vitaly Baranov
638ea23399 Fix build. 2022-06-20 03:44:59 +02:00
Vitaly Baranov
2c8788266c Fix tests. 2022-06-20 03:44:49 +02:00
Alexey Milovidov
e2fb85c072 Merge branch 'master' into enable-enable_positional_arguments 2022-06-20 00:37:36 +02:00
mergify[bot]
670a63865e
Merge branch 'master' into window-function-expression 2022-06-19 22:14:54 +00:00
alesapin
ae8108d30b Add comments, better interface 2022-06-19 23:48:46 +02:00
zvonand
832fd6e0a9 Added tests + minor updates 2022-06-19 23:10:28 +05:00
Alexey Milovidov
f024956f62 Merge branch 'master' of github.com:ClickHouse/ClickHouse into revert-35914-FIPS_compliance 2022-06-19 18:59:53 +02:00
Alexey Milovidov
7700c26076
Merge pull request #38196 from ClickHouse/revert-38194-revert-37015-zstd_window_log_max 
Revert "Revert "Add a setting to use more memory for zstd decompression""
 2022-06-19 19:58:03 +03:00
Alexey Milovidov
c941d0c8e2
Merge pull request #33653 from ClickHouse/enable-some-settings 
Enable some settings
 2022-06-19 19:52:56 +03:00
Alexey Milovidov
354a10d1e1
Merge pull request #35075 from azat/dist-insert-hung-fix 
Disable send_logs_level for INSERT into Distributed to avoid possible hung
 2022-06-19 19:39:56 +03:00
alesapin
2e6e3f082a Merge branch 'master' into disk_transaction 2022-06-19 18:21:02 +02:00
Azat Khuzhin
bc3e73d776 Fix LOGICAL_ERROR for WINDOW VIEW with incorrect structure 
Caching header of the source table in the WINDOW VIEW should not be
done, since there is no ability to get notification when it had been
changed (ALTER or CREATE/DROP).

And this fires on [CI], when the following tests had been executed in
order in stress tests:
- 01050_window_view_parser_tumble (leaves wm for mt)
- 01748_partition_id_pruning (cache input_header)
- 01188_attach_table_from_path (insert into mt with wm attached and
  incorrect structure)

  [CI]: https://s3.amazonaws.com/clickhouse-test-reports/38056/109980eb275c064d08bc031bfdc14d95b9a7272b/stress_test__undefined__actions_.html

Follow-up for: #37965 (@Vxider)
Fixes: #37815
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-19 19:16:39 +03:00
alesapin
119ce90e5b Fix strange diff 2022-06-19 18:03:43 +02:00
alesapin
08b5dd091b Fix double negative 2022-06-19 17:12:04 +02:00
Vitaly Baranov
8a7c970ce0 Fix style. 2022-06-19 15:58:26 +02:00
alesapin
50801e41c5 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-19 14:05:46 +02:00
Vitaly Baranov
115be82440 DiskAccessStorage is now allowed to backup by default. 2022-06-19 13:16:36 +02:00
Vitaly Baranov
9f197defda Add support for setting 'allow_backup' to skip access entities from putting to backup. 2022-06-19 12:49:50 +02:00
Vitaly Baranov
01aaaf7395 More accurate access checking for RESTORE. 2022-06-19 11:26:41 +02:00
Alexey Milovidov
abced45397 Enable "enable_positional_arguments" 2022-06-19 05:55:12 +02:00
Alexey Milovidov
04b7a34870 Maybe fix error 2022-06-19 05:21:59 +02:00
Alexey Milovidov
0cf88e0950
Revert "ClickHouse's boringssl module updated to the official version of the FIPS compliant." 2022-06-18 23:16:18 +03:00
Alexey Milovidov
67a66721f2 Merge branch 'master' into enable-some-settings 2022-06-18 19:16:23 +02:00
Maksim Kita
b35998d7ea SortCursor improve performance for random data 2022-06-18 18:20:01 +02:00
Maksim Kita
6adf6b1707 Fixed tests 2022-06-18 18:20:01 +02:00
Maksim Kita
282f65c5e2 Fixed performance tests 2022-06-18 18:20:01 +02:00
Maksim Kita
ace900a1f8 Fixed tests 2022-06-18 18:20:01 +02:00
Maksim Kita
dbbf499005 Fixed unit tests 2022-06-18 18:20:01 +02:00
Maksim Kita
33f4b4d834 Fixed tests 2022-06-18 18:20:01 +02:00
Maksim Kita
ef084ad12a Fixed tests 2022-06-18 18:20:00 +02:00
Maksim Kita
fd7e533a09 MergingSortedAlgorithm added batch sorting queue 2022-06-18 18:20:00 +02:00
Maksim Kita
c86191a6b9 SortingTransform add cursor order 2022-06-18 18:20:00 +02:00
Maksim Kita
d82ab22333 Fixed style check 2022-06-18 18:20:00 +02:00
Maksim Kita
3664f02690 Replaced SortingHeap with SortingQueue 2022-06-18 18:20:00 +02:00
Maksim Kita
9670504781 Fixed tests 2022-06-18 18:20:00 +02:00
Maksim Kita
9f55a2cbac Fixed tests 2022-06-18 18:20:00 +02:00
Maksim Kita
d7e3e85d2f Sorting added batch queue variants 2022-06-18 18:20:00 +02:00
Alexey Milovidov
73709b0488
Revert "Revert "Add a setting to use more memory for zstd decompression"" 2022-06-18 15:55:35 +03:00
Antonio Andelic
dd75e9cba5
Merge pull request #38149 from vdimir/multiple_joins_original_names_34697 
Add setting multiple_joins_try_to_keep_original_names
 2022-06-18 14:47:31 +02:00
alesapin
e6b0648f99 Merge branch 'master' into disk_transaction 2022-06-18 14:26:21 +02:00
alesapin
1d7cf28cab
Merge pull request #38190 from ClickHouse/fix_36123 
Check row size to avoid out of bounds access in PostgreSQLSource
 2022-06-18 14:18:35 +02:00
alesapin
16e8b85fbf
Revert "Add a setting to use more memory for zstd decompression" 2022-06-18 14:08:14 +02:00
Vitaly Baranov
de9a07d18d Fix RESTORE ALL for tables without database in backup. 2022-06-18 14:07:01 +02:00
Vitaly Baranov
36475c5b98 Fix handling empty files in backups. 2022-06-18 12:28:32 +02:00
Dmitry Novik
8b917d12f2 Small refactoring 2022-06-18 01:40:30 +00:00
Alexey Milovidov
e20259e9ca
Merge pull request #37015 from wuxiaobai24/zstd_window_log_max 
Add a setting to use more memory for zstd decompression
 2022-06-18 04:19:27 +03:00
Alexey Milovidov
1b9a47f825
Merge pull request #38157 from qoega/remove-trash 
Remove processor description from span attributes - it is not working
 2022-06-18 04:09:13 +03:00
Alexander Gololobov
54e599161d Check row size to avoid out of bounds access 2022-06-18 00:16:45 +02:00
Maksim Kita
7e759cac39
Merge pull request #38184 from kitaisreal/sql-user-defined-function-loader-allow-symlinks 
SQLUserDefinedFunctionsLoader allow symlinks in user_defined directory
 2022-06-17 23:03:52 +02:00
zvonand
f4b3af091d fix zero byte 2022-06-17 23:48:14 +05:00
Antonio Andelic
f72e509b3b
Merge pull request #38052 from amosbird/join_regression_fix 
Fix significant join performance regression
 2022-06-17 19:55:33 +02:00
Vitaly Baranov
a0c558a17e Implement backup/restore for ACL system tables (system.users, system.roles, etc.) 2022-06-17 18:14:31 +02:00
Alexander Tokmakov
83adf56383 fix race 2022-06-17 18:13:57 +02:00
Maksim Kita
ed03bb5dc6
Merge pull request #38127 from iyupeng/2022061601.fix_merge_sorter 
Fix redundant memory reservation for output block in MergeSorter
 2022-06-17 17:50:26 +02:00
Maksim Kita
965b706350 SQLUserDefinedFunctionsLoader allow symlinks in user_defined directory 2022-06-17 17:43:42 +02:00
alesapin
93e465809d Fix style check 2022-06-17 16:49:49 +02:00
Alexander Tokmakov
e9c2157f01 Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-17 15:57:30 +02:00
Alexander Gololobov
f69f24ffbd
Merge pull request #38135 from vdimir/issue_37928 
Try to fix temporary name clash in partial merge join
 2022-06-17 15:10:19 +02:00
mergify[bot]
23e0b16058
Merge branch 'master' into fix-possible-crash-after-removing-replica-in-distributed 2022-06-17 11:50:44 +00:00
Alexander Tokmakov
5ffc38d9b9 fix style check, add comments 2022-06-17 13:47:14 +02:00
alesapin
7524500ca9 Remove logic from disk 2022-06-17 13:45:17 +02:00
avogar
23f48a9fb9 Fix bug with nested short-circuit functions 2022-06-17 11:44:49 +00:00
Maksim Kita
77c519ae08
Merge pull request #38142 from kitaisreal/executable-user-defined-functions-parameters-direct-fix 
Disable parameters for non direct executable user defined functions
 2022-06-17 12:57:27 +02:00
alesapin
e5b405701a Merge branch 'master' into disk_transaction 2022-06-17 12:16:12 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache 
Extract some diff from pr #36171
 2022-06-17 12:07:31 +02:00
Andrey Zvonov
f987f461e5 fix style -- rm unused ErrorCode 2022-06-17 15:00:32 +05:00
Igor Nikonov
baebbc084f
Merge pull request #38027 from ClickHouse/decimal_rounding_fix 
Fix: rounding for Decimal128/Decimal256 with more than 19-digits long scale
 2022-06-17 09:48:18 +02:00
Alexander Gololobov
59d32f8c96
Merge pull request #38132 from Avogar/fix-mapped-array 
Fix logical error in array mapped functions with const nullable column
 2022-06-17 08:49:41 +02:00
mergify[bot]
296905771f
Merge branch 'master' into better-support-gcp 2022-06-17 01:19:11 +00:00
zvonand
c1b2b669ab remove wrong code 2022-06-17 01:52:45 +05:00
Yatsishin Ilya
6fdcac1c9d Remove processor description from span attributes - it is not working anyway. 2022-06-16 19:49:07 +00:00
Maksim Kita
f8f6a95547
Merge pull request #38144 from kitaisreal/sort-description-compile-fix-typo 
SortDescription compile fix typo
 2022-06-16 20:16:14 +02:00
Alexander Tokmakov
39c0219c11 fixes 2022-06-16 19:41:32 +02:00
Dmitry Novik
500895306f Fix typo 2022-06-16 16:06:59 +00:00
Dmitry Novik
644399db8a Add comment and update docs 2022-06-16 15:59:51 +00:00
vdimir
2a9942ee80
Add setting multiple_joins_try_to_keep_original_names 2022-06-16 15:50:03 +00:00
Dmitry Novik
8479d880f3 Remove redundant code 2022-06-16 15:49:49 +00:00
Dmitry Novik
376412e417 Small refactoring 2022-06-16 15:41:04 +00:00
alesapin
ca33ff93cb
Merge pull request #37555 from ClickHouse/revert-37534-revert-37036-keeper-preprocess-operations 
Add support for preprocessing ZooKeeper operations and real-time digest in `clickhouse-keeper`
 2022-06-16 17:29:00 +02:00
Mikhail f. Shiryaev
2bef5257b7
Merge pull request #38147 from ClickHouse/22.7-prepare 
Update version after release
 2022-06-16 17:19:40 +02:00
Mikhail f. Shiryaev
06dd85f921
Update version to 22.7.1.1 2022-06-16 17:15:22 +02:00
Dmitry Novik
f9b71551d0 Cleanup ActionsVisitor 2022-06-16 15:11:16 +00:00
Maksim Kita
9a066ad29a
Merge pull request #38090 from kitaisreal/aggregate-functions-added-restrict-into-batch-methods 
Aggregate functions added restrict into batch methods
 2022-06-16 16:45:10 +02:00
kssenii
797482f132 Better 2022-06-16 16:40:29 +02:00
Maksim Kita
cd1bbc6a1b SortDescription compile fix typo 2022-06-16 16:40:02 +02:00
Maksim Kita
f8d5e908d7
Update SortingTransform.cpp 2022-06-16 16:39:34 +02:00
Maksim Kita
aff66742e7 Disable parameters for non direct executable user defined functions 2022-06-16 16:29:04 +02:00
Anton Popov
13ec7e3092
Merge pull request #37978 from CurtizJ/fix-sparse-s3 
Fix reading of sparse columns from s3
 2022-06-16 15:32:54 +02:00
Dmitry Novik
0663f07e67 Rework expressions with window functions 2022-06-16 13:29:56 +00:00
vdimir
d50c51f262
Try to fix temporary name clash in partial merge join 2022-06-16 12:47:55 +00:00
alesapin
379fff7318 Merge branch 'master' into disk_transaction 2022-06-16 14:36:19 +02:00
alesapin
b11439ff36 Compiles 2022-06-16 14:24:26 +02:00
Roman Vasin
ed3fe84b63 Fix runKinit() is called only for USE_KRB5 2022-06-16 14:45:27 +03:00
Roman Vasin
0ab6bfd5d8 Add warning about using krb5 parameter in StorageKafka.cpp 2022-06-16 14:25:37 +03:00
Roman Vasin
6e28275569 Add warnings about using krb5 parameters 2022-06-16 14:21:04 +03:00
mergify[bot]
f46f7257dd
Merge branch 'master' into fix-nothing-error 2022-06-16 10:58:03 +00:00
mergify[bot]
2557e8ad51
Merge branch 'master' into decimal_rounding_fix 2022-06-16 10:53:49 +00:00
avogar
a3a7cc7a5d Fix logical error in array mapped functions with const nullable column 2022-06-16 10:41:53 +00:00
zvonand
a800158438 wip upload 2022-06-16 15:11:41 +05:00
mergify[bot]
9e98c9db78
Merge branch 'master' into aggregate-functions-added-restrict-into-batch-methods 2022-06-16 09:53:55 +00:00
Roman Vasin
d93fd3bd2d Add complilation support for case when krb5 is not used 2022-06-16 09:30:40 +00:00
Azat Khuzhin
b3bf7589ef Fix possible concurrent access in ProgressIndication 
In case of all of the above:
- clickhouse-local
- input_format_parallel_parsing=true
- write_progress_on_update=true

It is possible concurrent access to the following:
- writeProgress() (class properties) (guarded with progress_mutex)
- thread_data/host_cpu_usage (guarded with profile_events_mutex)

v2: decrease number of rows for INSERT ProfileEvents test (10 times)
    CI: https://s3.amazonaws.com/clickhouse-test-reports/37391/4bd5c335182279dcc5020aa081b13c3044135951/stateless_tests__debug__actions__[1/3].html
v3: decrease number of rows for INSERT ProfileEvents test (10 times) and add a comment
    CI: https://s3.amazonaws.com/clickhouse-test-reports/37391/026d7f732cb166c90d6c287b02824b6c7fdebf0c/stateless_tests_flaky_check__address__actions_/runlog.log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

f
 2022-06-16 11:59:01 +03:00
Azat Khuzhin
4baa7690ae Send profile events for INSERT queries (previously only SELECT was supported) 
Reproducer:

    echo "1" | clickhouse-client --query="insert into function null('foo String') format TSV" --print-profile-events --profile-events-delay-ms=-1

However, clickhouse-local is differnt, it does sent the periodically,
but only if query was long enough, i.e.:

    # yes | head -n100000 | clickhouse-local --query="insert into function null('foo String') format TSV" --print-profile-events --profile-events-delay-ms=-1
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ContextLock: 10 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] DiskReadElapsedMicroseconds: 29 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] IOBufferAllocBytes: 200000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] IOBufferAllocs: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertQuery: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertedBytes: 1000000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] InsertedRows: 100000 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] MemoryTrackerUsage: 1521975 (gauge)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSCPUVirtualTimeMicroseconds: 102148 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSReadChars: 135700 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] OSWriteChars: 8 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] Query: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] RWLockAcquiredReadLocks: 2 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ReadBufferFromFileDescriptorRead: 5 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] ReadBufferFromFileDescriptorReadBytes: 134464 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] RealTimeMicroseconds: 293747 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] SoftPageFaults: 382 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] TableFunctionExecute: 1 (increment)
    [s1.ch] 2022.05.20 15:20:27 [ 0 ] UserTimeMicroseconds: 102148 (increment)

v2: Proper support ProfileEvents in INSERTs (with protocol change)
v3: Receive profile events on INSERT queries
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 11:59:01 +03:00
Alexey Milovidov
f2b665a799
Merge pull request #38120 from wangdh15/delete-unused-field-fix-build-error 
Remove unused class member
 2022-06-16 11:11:26 +03:00
Yu, Peng
518a726c1d Fix redundant memory reservation for output block in MergeSorter 2022-06-16 15:54:30 +08:00
Antonio Andelic
5b9e81bdf9 Revert sync waiter 2022-06-16 07:53:29 +00:00
Antonio Andelic
0b843d316f Merge branch 'master' into keeper-sequential-consistency-reads 2022-06-16 07:52:56 +00:00
Azat Khuzhin
0d4f78639e Interpreters/Aggregator: cleaner interface for block release during merge 
Suggested-by: @amosbird
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
4ff82eb9d5 Implement optimize_aggregation_in_order for projections 
v2: use real column name instead of aliases from GROUP BY
    Fixes the following error in 01710_projection_aggregation_in_order:

        Not found column a in block. There are only columns: toStartOfHour(ts), sum(value). (NOT_FOUND_COLUMN_IN_BLOCK)
v2.1: Get back support for projected and non-projected parts
v2.2: merge tests and rename
v3: Reduce copy-paste for optimize_aggregation_in_order for projections
v4: rebase on top of QueryPlanResourceHolder
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
14f9491619 MergeTreeData: preserve order for group_by_elements_order_descr 
This is required for proper optimize_aggregation_in_order for
projections.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
4694929623 Implement merging only for AggregatingStep 
v2: fill AggregateColumnsConstData only for only_merge
    (fixes 01291_aggregation_in_order and some other tests)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
3559e35b70 AggregatingStep: remove unused forward decl 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
682c93e2e1 Use proper setting compile_aggregate_expressions in MergeTreeDataSelectExecutor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:35 +03:00
Alexey Milovidov
c26abbfdc5
Merge pull request #38093 from danlark1/master 
Optimize most important parts with NEON SIMD
 2022-06-16 08:26:56 +03:00
mergify[bot]
ff9f4af28c
Merge branch 'master' into enable-some-settings 2022-06-16 03:51:59 +00:00
wangdh15
02cce40b3a when using clang12 compile, the unused filed shard_count will cause compile error. So delete it. 2022-06-16 10:43:31 +08:00
Anton Popov
92b7b9789a
try to fix fpc codec 2022-06-16 03:41:09 +02:00
mergify[bot]
7f24574609
Merge branch 'master' into better-support-gcp 2022-06-15 23:18:20 +00:00
Mikhail f. Shiryaev
6fdba20b13
Merge pull request #38098 from ClickHouse/clang-tidy-warning-fix 
Fix: build error
 2022-06-16 01:11:39 +02:00
Robert Schulze
b39b963733
Merge pull request #37993 from ClickHouse/stripping 
Strip less aggressively to make the embedded hash survive
 2022-06-16 00:08:36 +02:00
Robert Schulze
b936abe813
+ comments about keeping stuff in sync 2022-06-16 00:07:59 +02:00
alesapin
11b6664863 WI{ 2022-06-16 00:06:52 +02:00
Maksim Kita
3eea38f078
Merge pull request #38028 from kitaisreal/background-schedule-pool-refactoring 
BackgroundSchedulePool remove Poco::NotificationQueue
 2022-06-15 23:10:49 +02:00
Maksim Kita
7d5af1a79f
Merge pull request #37992 from kitaisreal/partial-sorting-transform-refactoring 
PartialSortingTransform refactoring
 2022-06-15 22:55:04 +02:00
Alexander Tokmakov
6ddfec47d2 maybe fix livelock on queue processing 2022-06-15 22:22:27 +02:00
Alexander Tokmakov
5e2b42c85a Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-15 20:47:02 +02:00
Robert Schulze
61709a674d
Merge pull request #38089 from ClickHouse/fpc-follow-up 
Small follow-up for FPC codec
 2022-06-15 20:33:41 +02:00
Vitaly Baranov
c2c35fad82 Refactoring of the code getting create table queries for backup. 2022-06-15 20:32:35 +02:00
Vitaly Baranov
c0f06c5e16 Require new privilige 'BACKUP' to make a backup. 2022-06-15 20:32:35 +02:00
Vitaly Baranov
0102626532 Disable the 'BACKUP ALL' command (it's not quite clear what to do with predefined databases). 2022-06-15 20:32:35 +02:00
Vitaly Baranov
cb9bf62e77 Change syntax RESTORE ALL DATABASES => RESTORE ALL 2022-06-15 20:32:35 +02:00
Vitaly Baranov
1198e86295 Fix storing temporary tables and skipping system tables while making a backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
6590f82ec4 Fix tryGetCreateTableQuery() for system tables. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
491ab58b3f Remove IF NOT EXISTS before storing create query in DatabaseMemory. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
b2323991d6 BACKUP DATABASE now stores inner tables of matviews only if the corresponding matviews are stored. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
6877b8f864 Fix renaming visitor. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
d78a2cda72 Restore tables regarding their dependencies. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
cf34883000 Use QualifiedTableName instead of DatabaseAndTableName. Remove mode 'ALL TEMPORARY TABLES' 2022-06-15 20:32:34 +02:00
Vitaly Baranov
21f3bed435 Simplify path calculations in backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
592f568f83 Move backup/restore code to storages and databases - part 2. 2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57 Move backup/restore code to storages and databases - part 1. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
ce1836f0d2 Lock tables for share before backup and restore. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21 Rework collecting replicated parts. 2022-06-15 20:28:42 +02:00
Vitaly Baranov
d00b4a7fdb Remove obsolete function IRestoreCoordination::getReplicatedTableDataPath() 2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Vitaly Baranov
e891eba80e Finalize write buffers used in backups. 2022-06-15 20:26:27 +02:00
Danila Kutenin
607dd8d6ca Restart the pipeline, I guess 2022-06-15 18:16:56 +00:00
Daniel Kutenin
a769dea8ef
Merge branch 'ClickHouse:master' into master 2022-06-15 19:12:10 +01:00
Roman Vasin
344fbe8de4 Fix code style 2022-06-15 20:26:42 +03:00
Roman Vasin
89a659e738 Move krb header files from KerberosInit.h to KerberosInit.cpp 2022-06-15 20:08:16 +03:00
Roman Vasin
1c26424371 Change message in StorageKafka; Code style correction 2022-06-15 19:35:21 +03:00
Nikolai Kochetov
25b9990737 Fixing build 2022-06-15 16:32:22 +00:00
Alexey Milovidov
7baa54f92f
Merge pull request #37697 from zvonand/dt64-neg-subseconds 
Fix DateTime64 negative fractional seconds
 2022-06-15 19:30:42 +03:00
Alexander Tokmakov
2ac72319bd
Merge pull request #37185 from amosbird/projection-fix-three 
Fix possible heap-use-after-free error when reading system.projection_parts and system.projection_parts_columns
 2022-06-15 19:00:10 +03:00
Igor Nikonov
b1137c9cba Fix: build error 
+ clang tidy warning fixed
 2022-06-15 15:21:05 +00:00
Danila Kutenin
048f56bf4d Fix some tests and comments 2022-06-15 14:40:21 +00:00
kssenii
500f49972b Extract diff from PR 36171 2022-06-15 16:40:18 +02:00
Alexander Tokmakov
5bfb15262c Revert "More parallel execution for queries with FINAL (#36396)" 
This reverts commit c8afeafe0e.
 2022-06-15 17:25:38 +03:00
Alexander Tokmakov
e4d4d0a854 fix race on queue processing 2022-06-15 16:21:26 +02:00
Roman Vasin
dd5b0ee065 Add kerberosInit() function to call KeberosInit 2022-06-15 17:02:53 +03:00
Antonio Andelic
ac0b7ab20b Fix backwards compatibility with older snapshots 2022-06-15 13:46:27 +00:00
mergify[bot]
c2afc2f6c6
Merge branch 'master' into background-schedule-pool-refactoring 2022-06-15 13:43:31 +00:00
Nikolai Kochetov
82ebf869a7 Fixing style 2022-06-15 13:40:30 +00:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Danila Kutenin
08e3f77a9c Optimize most important parts with NEON SIMD 
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.

That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
 2022-06-15 13:19:29 +00:00
Antonio Andelic
c403ebf4d2 Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-15 12:48:32 +00:00
Antonio Andelic
7e1f64002d Address review comments 2022-06-15 12:48:30 +00:00
Robert Schulze
a0d936cc9f
Small follow-up for FPC codec 
- add paper reference + doxygen

- remove endianness handling (ClickHouse assumes little endian)

- documentation

- other minor stuff
 2022-06-15 14:21:28 +02:00
Yakov Olkhovskiy
e604d31feb
Merge pull request #38074 from ClickHouse/bug-with-fill-extra 
Inconsistency in ORDER BY ... WITH FILL feature.
 2022-06-15 08:19:55 -04:00
Maksim Kita
bf6d155170 Aggregate functions added restrict into batch methods 2022-06-15 14:11:10 +02:00
alesapin
1b66166501 Merge branch 'master' into disk_transaction 2022-06-15 13:54:10 +02:00
mergify[bot]
2cb9579234
Merge branch 'master' into join_regression_fix 2022-06-15 11:53:42 +00:00
alesapin
b557c1dd80 Interm stage 2022-06-15 13:51:21 +02:00
Nikolai Kochetov
2ee349f29c Fix test_encrypted_disk 2022-06-15 11:42:59 +00:00
Nikolai Kochetov
a274bec0fa Fix test_encrypted_disk 2022-06-15 11:41:08 +00:00
mergify[bot]
d704264fae
Merge branch 'master' into decimal_rounding_fix 2022-06-15 10:47:09 +00:00
Nikita Taranov
c8afeafe0e
More parallel execution for queries with FINAL (#36396) 2022-06-15 12:44:20 +02:00
Alexander Tokmakov
56f2121c1a
Revert "Add backoff to merges in replicated queue if always_fetch_merged_part is enabled" 2022-06-15 13:04:01 +03:00
Robert Schulze
9794098ebb
Merge pull request #37553 from koloshmet/fpc_codec 
FPC Codec for floating point data
 2022-06-15 12:03:41 +02:00
Maksim Kita
864fd627e1
Merge pull request #35762 from kitaisreal/merge-tree-multiple-order-by-columns-improve-insert-performance 
MergeTree multiple ORDER BY columns improve insert performance
 2022-06-15 11:01:36 +02:00
Roman Vasin
9bf6b9d491 Add kinit presence handling in StorageKafka; Cleanup code in HDFSCommon 2022-06-15 11:37:02 +03:00
mergify[bot]
d148299c20
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-15 07:11:31 +00:00
zvonand
c149c916ec initial setup 2022-06-15 11:49:55 +05:00
Yakov Olkhovskiy
411695bd97 do not fill 'to' boundary 2022-06-14 22:26:50 -04:00
mergify[bot]
12bd2c0e3f
Merge branch 'master' into dt64-neg-subseconds 2022-06-15 02:24:17 +00:00
Alexey Milovidov
ab9fc572d5
Merge pull request #37667 from ClickHouse/group-by-enum-fix 
Support types with non-standard defaults in ROLLUP, CUBE, GROUPING SETS
 2022-06-15 05:14:33 +03:00
Alexey Milovidov
04e25bc044
Merge pull request #38058 from kitaisreal/unary-logical-functions-improve-performance-dynamic-dispatch 
UnaryLogicalFunctions improve performance using dynamic dispatch
 2022-06-15 04:18:20 +03:00
Anton Popov
a59be0fd5d better support of GCP storage 2022-06-15 00:23:45 +00:00
Alexey Milovidov
0957c885e2 Remove -0. from CPU usage in the client 2022-06-14 23:36:16 +02:00
Alexey Milovidov
5e9e5a4eaf
Merge pull request #37525 from Avogar/avro-structs 
Support Maps and Records, allow to insert null as default in Avro format
 2022-06-15 00:04:29 +03:00