thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
105,439 Commits 9,328 Branches 1,783 Tags 2.1 GiB
8a74238fe0
Commit Graph

4971 Commits

Author SHA1 Message Date
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Robert Schulze
1885bb0524
Make comment consistent accross generated files 2022-09-28 08:11:09 +00:00
Alexey Milovidov
be6689ef08
Merge pull request #41372 from HarryLeeIBM/hlee-s390x-siphash 
Another fix for SipHash Endian issue on s390x
 2022-09-28 01:05:05 +03:00
Antonio Andelic
d0457addbd Support filtered list 2022-09-27 15:02:26 +00:00
serxa
b057d07977 IO scheduling subsystem 2022-09-27 13:26:41 +00:00
Antonio Andelic
265c8b3d26 Merge branch 'master' into keeper-multiread 2022-09-27 12:37:22 +00:00
Antonio Andelic
7a9afc4bdd Fix zookeeper_log 2022-09-27 12:36:59 +00:00
Alexander Tokmakov
3d4a5a493b
Merge pull request #41484 from FrankChen021/on_cluster_dll 
Add OpenTelemetry support to ON CLUSTER DDL
 2022-09-27 15:15:52 +03:00
Robert Schulze
aa7e62ad5f
Add functio ntryBase58Decode() 
- makes it consistent with tryBase64Decode(), follow-up to #39292

- additionally the following minor changes:

  - rename Common/base58.h|cpp to Common/Base58.h|cpp for constency with
    Common/Base64.h|cpp

  - check that (encode|decode|tryDecode)Base64() gets just one argument
 2022-09-27 10:18:36 +00:00
Antonio Andelic
937d534cd4 Add support for simple list 2022-09-26 08:05:02 +00:00
Antonio Andelic
f833366555 Merge branch 'master' into keeper-multiread 2022-09-26 07:16:45 +00:00
Alexey Milovidov
e3a6f2381b Revert "Merge pull request #40033 from ClickHouse/reenable-avx512-vbmi-columnvector-filter" 
This reverts commit 70f63d2aae, reversing
changes made to a0693c3a84.
 2022-09-25 17:03:52 +02:00
Nikita Mikhaylov
e897816572
Evict stale DNS entries from cache in case of network errors (#41707) 2022-09-25 14:06:13 +02:00
Nikita Mikhaylov
dbcba1490c
Try to make MergeTreeRestartingThread more reactive in case of expired sessions (#41092) 2022-09-23 14:40:04 +02:00
Frank Chen
2344e0738e Keep compatibility during upgrading 2022-09-23 11:32:21 +08:00
Frank Chen
40f9e0b69a Address review comments 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-23 10:08:42 +08:00
Roman Vasin
cdd20241bf Finish toStartOfDay; fix toStartOfMinute 2022-09-22 13:16:46 +00:00
Alexey Milovidov
2c0eb994db Fix build after merge 2022-09-22 06:53:33 +02:00
Alexey Milovidov
45afacdae4
Merge pull request #41186 from ClickHouse/fix-three-fourth-of-trash 
Fix more than half of the trash
 2022-09-22 07:28:26 +03:00
Frank Chen
99c74d182c
Merge branch 'master' into on_cluster_dll 2022-09-22 09:58:10 +08:00
Nikita Taranov
100c055510
Prefetching in aggregation (#39304) 
* impl

* stash

* clean up

* do not apply when HT is small

* make branch static

* also in merge

* do not hardcode look ahead value

* fix

* apply to methods with cheap key calculation

* more tests

* silence tidy

* fix build

* support HashMethodKeysFixed

* apply during merge only for cheap

* stash

* fixes

* rename method

* add feature flag

* cache prefetch threshold value

* fix

* fix

* Update HashMap.h

* fix typo

* 256KB as default l2 size

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2022-09-21 18:59:07 +02:00
HarryLeeIBM
77153533ac Fix endian issue in integer hex string conversion 2022-09-21 09:21:16 -04:00
Nikolai Kochetov
9e1bb19559
Purge jemalloc arenas in case of high memory usage. (#40277) 
* Purge jemalloc arenas in case of high memory usage.

* Purge jemalloc arenas in case of high memory usage.

* Get RSS before jemalloc counters. Try to avoid negative RSS.

* Try to avoid negative RSS.

* muzzy -> dirty

* Another fix.

* Update MemoryTracker.cpp

* Wait for purged memory.

* Revert "Wait for purged memory."

This reverts commit 53a2621a2d.
 2022-09-20 23:56:10 +02:00
Alexey Milovidov
45bd3cfc30 Merge branch 'master' into fix-three-fourth-of-trash 2022-09-20 21:27:41 +02:00
Alexey Milovidov
f55a67c7b2 Fix build 2022-09-20 21:27:18 +02:00
Alexey Milovidov
133318d8bd
Merge branch 'master' into hlee-s390x-siphash 2022-09-20 06:10:57 +03:00
Sergei Trifonov
cf2db48c29
Merge pull request #40631 from ClickHouse/readonly-settings-allow 
Allow to modify constrained settings in readonly mode
 2022-09-20 02:18:14 +02:00
Sergei Trifonov
7902922ae4
Merge branch 'master' into readonly-settings-allow 2022-09-19 21:00:12 +02:00
serxa
7a1d4a404a add DiskS3* profile events per S3 API calls 2022-09-19 18:10:47 +00:00
serxa
f8aa738511 more conventional profile events names 2022-09-19 17:23:22 +00:00
serxa
6ed53585ab rearrange and fix profile event descriptions 2022-09-19 17:03:48 +00:00
Sergei Trifonov
ff2e1ff8d8
Merge branch 'master' into support_s3_api_event_metrics 2022-09-19 14:35:45 +02:00
Sergei Trifonov
f7a1ed3141
Merge branch 'master' into readonly-settings-allow 2022-09-19 12:52:36 +02:00
Alexey Milovidov
000b6ac81c Fix error 2022-09-19 09:30:48 +02:00
Alexey Milovidov
730655d4fd Fix 8/9 of trash 2022-09-19 08:53:20 +02:00
Alexey Milovidov
91baedf03a Fix 6/7 of trash 2022-09-19 08:53:20 +02:00
Alexey Milovidov
ab4db2d0c4 Fix 5/6 of trash 2022-09-19 08:50:53 +02:00
Alexey Milovidov
84f42e0874 Fix 3/4 of trash 2022-09-19 08:50:53 +02:00
Alexey Milovidov
d4b9fe41be
Merge pull request #41457 from ClickHouse/remove-trash-5 
Remove trash from Field
 2022-09-19 06:36:48 +03:00
Frank Chen
2ae43bb4e8 Add test case 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-19 11:18:35 +08:00
Alexey Milovidov
09ae1a4bf4
Merge branch 'master' into hlee-s390x-siphash 2022-09-18 11:09:02 +03:00
Alexey Milovidov
7e6da853cd Merge branch 'master' into pod-array-overflow 2022-09-18 09:24:23 +02:00
Alexey Milovidov
791de6592b Remove trash from Field 2022-09-18 05:16:08 +02:00
HarryLeeIBM
f3e8738145 Fixed issues in code review 2022-09-17 19:48:08 -07:00
Alexey Milovidov
dc76677eb2
Merge pull request #41440 from ClickHouse/remove-cruft 
Remove cruft about gcc
 2022-09-18 03:15:26 +03:00
Alexey Milovidov
6c74c54cb5 Merge branch 'master' into pod-array-overflow 2022-09-18 01:24:52 +02:00
Alexey Milovidov
9ca4e2ec7c Fix overflow in PODArray 2022-09-17 22:26:39 +02:00
Alexey Milovidov
f4a48cd4d6 Remove cruft 2022-09-17 22:16:31 +02:00
Alexey Milovidov
35cce03125 Remove dlopen 2022-09-17 03:02:34 +02:00
Alexey Milovidov
31ebd34f5b
Merge pull request #41193 from azat/DWARF-5-parser-v3 
Add support for DWARF-5 (without emitting them in binaries)
 2022-09-17 02:58:02 +03:00
Antonio Andelic
81f7cf3ca6 Add support for MultiRead in Keeper 2022-09-16 12:00:35 +00:00
Robert Schulze
b32b02d844
Merge pull request #40897 from ClickHouse/catboost-bridge-resurrected 
Move CatBoost evaluation into clickhouse-library-bridge
 2022-09-16 13:12:09 +02:00
Antonio Andelic
5662e0050c Define client for multi read 2022-09-16 09:55:02 +00:00
Alexey Milovidov
da01982652
Merge pull request #41046 from azat/build/llvm-15 
Switch to llvm/clang 15
 2022-09-16 07:31:06 +03:00
HarryLeeIBM
d62ba01e93 Fix SipHash Endian issue for s390x 2022-09-15 06:25:23 -07:00
serxa
4971c1f04d review fixes 2022-09-15 11:34:14 +00:00
Frank Chen
f576d8191c Merge branch 'master' into on_cluster_dll 2022-09-15 18:05:36 +08:00
Robert Schulze
b64af5989e
Merge branch 'master' into opentelemetry_stress 2022-09-15 09:43:11 +02:00
Frank Chen
77d0971efa Serialize tracing context 2022-09-15 14:46:55 +08:00
mateng0915
5badb1b186 resolve the review comments 2022-09-14 19:19:40 +08:00
teng.ma
a85c917488 fix master conflict 2022-09-14 19:18:31 +08:00
teng.ma
ebea9f9359 Feature: Support Event Tracing when calling Aws S3 API 
Description:
============
In  Computing && Storage Architecture,  using S3 as remote / Shared storage, the Method to access S3 is using AWS S3 API
There is a gap between ClickHouse DB with Ozone Operation
In ClickHouse, operation is SQL and background task
In S3 , the operation should be AWS S3 API
And one sql maybe can mapped to multiple  API

Solution:
=========
Added Calling API as event into system.events table
 2022-09-14 19:18:31 +08:00
Robert Schulze
e8e6dddc76
Merge pull request #41034 from FrankChen021/distributed 
Improve the observability of INSERT on distributed table
 2022-09-13 20:55:11 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage 
KeeperMap storage engine
 2022-09-13 14:15:09 +02:00
Kseniia Sumarokova
b2c9c04c7b
Merge pull request #40821 from kssenii/improve-marks-cache-loading 
Allow to load marks in threadpool in advance
 2022-09-13 12:31:58 +02:00
Frank Chen
7303ae1796 Make sure span holder will be destructed only once 2022-09-13 18:27:29 +08:00
Robert Schulze
fac1be9700
chore: restore SYSTEM RELOAD MODEL(S) and moniting view SYSTEM.MODELS 
- This commit restores statements "SYSTEM RELOAD MODEL(S)" which provide
  a mechanism to update a model explicitly. It also saves potentially
  unnecessary reloads of a model from disk after it's initial load.

  To keep the complexity low, the semantics of "SYSTEM RELOAD MODEL(S)
  was changed from eager to lazy. This means that both statements
  previously immedately reloaded the specified/all models, whereas now
  the statements only trigger an unload and the first call to
  catboostEvaluate() does the actual load.

- Monitoring view SYSTEM.MODELS is also restored but with some obsolete
  fields removed. The view was not documented in the past and for now it
  remains undocumented. The commit is thus not considered a breach of
  ClickHouse's public interface.
 2022-09-12 19:33:02 +00:00
Sergei Trifonov
c31818260f renames and refactoring 2022-09-12 21:03:06 +02:00
Frank Chen
20191932df Fix style 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 00:41:05 +08:00
Frank Chen
7e1f2901da Fix 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 00:06:17 +08:00
Sergei Trifonov
fe88a7991d
Merge branch 'master' into readonly-settings-allow 2022-09-12 16:20:40 +02:00
Frank Chen
ebaa24ecae Fix flaky tests 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-12 22:15:30 +08:00
Antonio Andelic
1cd43d8e5c Merge branch 'master' into keeper-storage 2022-09-12 13:42:51 +00:00
Azat Khuzhin
7e130aeb69 Add support for DWARF-5 (without emitting them in binaries) 
ClickHouse changes to the folly parser:
- use camel_case
- add NOLINT
- avoid using folly:: (use std:: instead)
- avoid using boost:: (use std:: instead)

But note, now it has not been enabled by default (like it was
initially), because you may need recent debugger to support DWARF-5
correctly, and to make debugging easier, let's do this later.

A good example is gdb 10, even though it looks like it should support
it, it still produce some errors, like here [1]:

    Dwarf Error: DW_FORM_strx1 found in non-DWO CU [in module /usr/bin/clickhouse]

  [1]: https://github.com/ClickHouse/ClickHouse/pull/40772#issuecomment-1236331323

And not only it complains, apparently it can "activate" SDT probes
(replace "nop" with "int3"), and I believe this is what happens here
[2].

  [2]: https://github.com/ClickHouse/ClickHouse/pull/41063#issuecomment-1242992314

There you got int3 in the case when ClickHouse got SIGTRAP:

<details>

```
    0x7f494705e093 <+1139>: jne    0x7f494705e450            ; <+2096> [inlined] update_tls_slotinfo at dl-open.c:732
    0x7f494705e099 <+1145>: testl  %r13d, %r13d
    0x7f494705e09c <+1148>: je     0x7f494705e09f            ; <+1151> at dl-open.c:744:6
    0x7f494705e09e <+1150>: int3
->  0x7f494705e09f <+1151>: movl   -0x54(%rbp), %eax
    0x7f494705e0a2 <+1154>: testl  %eax, %eax
    0x7f494705e0a4 <+1156>: jne    0x7f494705e410            ; <+2032> at dl-open.c:745:5

But if I repeat the query it does not:

    0x7ffff7fe5093 <+1139>: jne    0x7ffff7fe5450            ; <+2096> [inlined] update_tls_slotinfo at dl-open.c:732
    0x7ffff7fe5099 <+1145>: testl  %r13d, %r13d
    0x7ffff7fe509c <+1148>: je     0x7ffff7fe509f            ; <+1151> at dl-open.c:744:6
    0x7ffff7fe509e <+1150>: nop
->  0x7ffff7fe509f <+1151>: movl   -0x54(%rbp), %eax
    0x7ffff7fe50a2 <+1154>: testl  %eax, %eax
    0x7ffff7fe50a4 <+1156>: jne    0x7ffff7fe5410            ; <+2032> at dl-open.c:745:5
```

</details>

Test command was:

    clickhouse local --stacktrace -q "select * from file('data.capnp', 'CapnProto', 'val1 char') settings format_schema='nonexist:Message'

*P.S. I did this, because I have libraries compiled with DWARF5 (i.e. glibc), and dwarf parser simply fails on my dev env.*

Refs: 490b287ca3
(cherry picked from commit ee5696bb32)
(cherry picked from commit e03870bc8b)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-11 21:06:14 +02:00
Alexey Milovidov
8b5328f7e5 Merge branch 'master' into fix-trash-base58 2022-09-11 20:43:54 +02:00
Alexey Milovidov
3e6c4e3f16 Fix clang-tidy 2022-09-11 20:43:36 +02:00
Alexey Milovidov
9a0892c40c
Merge pull request #41171 from ClickHouse/fix-half-of-trash 
Fix half of trash
 2022-09-11 21:40:22 +03:00
kssenii
b7d751b782 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-11 13:23:30 +02:00
Alexey Milovidov
5802c2fdd2
Merge pull request #40713 from ClickHouse/remove-useless-line 
Remove one line from XML, because we do not care
 2022-09-11 09:53:30 +03:00
Alexey Milovidov
0896e6b9b6
Merge pull request #41183 from ClickHouse/fix-parallel-hash-join-for-floats 
Better parallel hash JOIN for floats
 2022-09-11 09:13:57 +03:00
Alexey Milovidov
e0a9ae0496 Fix base58 trash 2022-09-11 08:09:14 +02:00
Alexey Milovidov
3d8a2130cf Continue fixing the trash 2022-09-11 06:31:15 +02:00
Alexey Milovidov
9fbd361a2b Merge branch 'master' into fix-trash-base58 2022-09-11 06:23:01 +02:00
Alexey Milovidov
7f1e7b5967 Merge branch 'master' into fix-half-of-trash 2022-09-11 06:20:47 +02:00
Alexey Milovidov
512fb998f1 Fix ridiculous trash in Base58 2022-09-11 03:54:40 +02:00
Alexey Milovidov
cc38ac3764 Better parallel hash JOIN for floats 2022-09-11 03:04:44 +02:00
Alexey Milovidov
ae6c74a916 Merge branch 'master' into fix-half-of-trash 2022-09-11 02:30:56 +02:00
Alexey Milovidov
b75d56ad9b Merge branch 'master' into remove-some-methods 2022-09-11 01:55:02 +02:00
Alexey Milovidov
91338ea771
Merge branch 'master' into remove-useless-line 2022-09-11 02:03:39 +03:00
Azat Khuzhin
4e3135383b Fix clang-tidy warnings (from clang-15) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
e0a8f19cfb Fix -Wzero-as-null-pointer-constant in MemoryStatisticsOS::get for FreeBSD 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Alexey Milovidov
871ee15a9c
Merge pull request #40953 from ClibMouse/sensitive_data_masker_unittest_issue 
Rearrange sensitive data masker unit test
 2022-09-10 20:03:30 +03:00
Alexey Milovidov
61278c81e4 Merge branch 'master' into remove-useless-line 2022-09-10 05:07:10 +02:00
Alexey Milovidov
fd235919aa Remove some methods 2022-09-10 05:04:40 +02:00
Suzy Wang
d60340eb40
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-09 22:59:50 -04:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Azat Khuzhin
8c5583d7a5 Fix stack-use-after-return in GetPriorityForLoadBalancing::getPriorityFunc() 
clang-15 reports [1]:

<details>

<summary>ASan report</summary>

```
    ==1==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f1d04c4eb20 at pc 0x000031c4803c bp 0x7f1d05e19a00 sp 0x7f1d05e199f8
    READ of size 8 at 0x7f1d04c4eb20 thread T200 (QueryPullPipeEx)
        #0 0x31c4803b in DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3::operator()(unsigned long) const build_docker/../src/Common/GetPriorityForLoadBalancing.cpp:42:40
        #1 0x31c4803b in decltype(static_cast<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&>(fp)(static_cast<unsigned long>(fp0))) std::__1::__invoke<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/type_traits:3640:23
        #2 0x31c4803b in unsigned long std::__1::__invoke_void_return_wrapper<unsigned long, false>::__call<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/__functional/invoke.h:30:16
        #3 0x31c4803b in std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>::operator()(unsigned long&&) build_docker/../contrib/libcxx/include/__functional/function.h:230:12
        #4 0x31c4803b in unsigned long std::__1::__function::__policy_invoker<unsigned long (unsigned long)>::__call_impl<std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>>(std::__1::__function::__policy_storage const*, unsigned long) build_docker/../contrib/libcxx/include/__functional/function.h:711:16
        #5 0x31c38b07 in std::__1::__function::__policy_func<unsigned long (unsigned long)>::operator()(unsigned long&&) const build_docker/../contrib/libcxx/include/__functional/function.h:843:16
        #6 0x31c38b07 in std::__1::function<unsigned long (unsigned long)>::operator()(unsigned long) const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12
        #7 0x31c38b07 in PoolWithFailoverBase<DB::IConnectionPool>::getShuffledPools(unsigned long, std::__1::function<unsigned long (unsigned long)> const&) build_docker/../src/Common/PoolWithFailoverBase.h:174:39

      This frame has 2 object(s):
        [32, 40) 'pool_size.addr' <== Memory access at offset 32 is inside this variable
        [64, 88) 'ref.tmp' (line 18)
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/integration_tests__asan__[1/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-09 20:06:48 +02:00
Antonio Andelic
32483aeec8 Merge branch 'master' into keeper-storage 2022-09-09 12:24:01 +00:00
Antonio Andelic
8ae7fb14b3
Merge pull request #40918 from ClickHouse/fix-keeper-system-path-check 
Fix check for Keeper system path modification
 2022-09-09 09:12:49 +02:00
Alexey Milovidov
af8ed9c166
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-09 00:05:53 +03:00
Sergei Trifonov
f92b6a6176
Merge branch 'master' into readonly-settings-allow 2022-09-08 17:44:32 +02:00
Sergei Trifonov
62541ab764 fix more tests and clang tidy build 2022-09-08 17:43:09 +02:00
Antonio Andelic
3502718a2c Less complications 2022-09-08 13:52:14 +00:00
Robert Schulze
60f9f6855d
feat: implement catboost in library-bridge 
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes #27870
 2022-09-08 09:01:32 +00:00
Antonio Andelic
09c0bf2931 Add unit tests for match path 2022-09-08 08:16:38 +00:00
Antonio Andelic
5d837e4f41 Merge branch 'master' into fix-keeper-system-path-check 2022-09-08 07:41:04 +00:00
Robert Schulze
68808858a5
Merge pull request #41050 from FrankChen021/exception_safe 
Fix failed stress test (OpenTelemetry)
 2022-09-08 09:19:54 +02:00
Suzy Wang
61b2e1c32c
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-07 16:49:03 -04:00
Kseniia Sumarokova
a270eeef91
Merge pull request #41008 from kssenii/refactor-merge-tree-read 
Small refactoring around merge tree readers (get rid of data part ptr)
 2022-09-07 18:27:33 +02:00
Dmitry Novik
499e479892
Merge pull request #40873 from azat/build/fix-debug-symbols-quirk 
Fix debug symbols
 2022-09-07 17:31:35 +02:00
Suzy Wang
e0b8f26839
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-07 10:38:21 -04:00
Vitaly Baranov
31ed722572
Merge pull request #41044 from vitlibar/more-conventional-conversion-yaml-to-xmk 
More conventional conversion yaml to xml
 2022-09-07 13:46:32 +02:00
vdimir
6d4b6c452a
Merge branch 'master' into grace_hash_join 2022-09-07 08:00:14 +00:00
Frank Chen
fc05b05be3 Fix style and typo 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-07 15:14:43 +08:00
Frank Chen
de8f6bdce7 More safe 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-07 13:39:12 +08:00
Sergei Trifonov
35524d0175 typo 2022-09-06 20:51:33 +02:00
Sergei Trifonov
77ee4c04aa fix stateless tests 2022-09-06 20:28:50 +02:00
Vitaly Baranov
63e992d52d Edit test configs. 2022-09-06 17:09:26 +02:00
Frank Chen
6ced4131ca exception safe 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-06 22:11:47 +08:00
Kseniia Sumarokova
3558361a05
Merge branch 'master' into refactor-merge-tree-read 2022-09-06 16:00:43 +02:00
Suzy Wang
cc88679a52
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-06 09:36:53 -04:00
Vitaly Baranov
6b55f4dd68 Use pretty-print to output preprocessed configs in readable form. 2022-09-06 15:01:47 +02:00
Vitaly Baranov
e9b75deeba Make conversion YAML->XML more conventional. 2022-09-06 15:01:41 +02:00
Sergei Trifonov
f77809ddbc
Merge pull request #40900 from ClickHouse/s3-detailed-metrics 
S3 detailed metrics
 2022-09-06 13:20:57 +02:00
Alexey Milovidov
940a53e519
Merge pull request #40984 from Lucky-Chang/typo_fix 
Fix some typos and clang-tidy warnings
 2022-09-06 02:37:50 +03:00
kssenii
83514fa2ef Refactor 2022-09-05 20:08:22 +02:00
Sergei Trifonov
1c247308f1
Merge branch 'master' into s3-detailed-metrics 2022-09-05 15:55:41 +02:00
Alexander Tokmakov
abffd4595e
Merge branch 'master' into zookeeper_client_fault_injection 2022-09-05 15:03:26 +03:00
Antonio Andelic
e07539d1b9 Merge branch 'master' into keeper-storage 2022-09-05 09:54:13 +00:00
Luck-Chang
0800c0fd4c fix some typos 2022-09-05 14:09:42 +08:00
Luck-Chang
1ac8e739c9 fix some typos and clang-tidy warnings 2022-09-05 09:50:24 +08:00
Alexey Milovidov
d20d49f39f
Merge pull request #40958 from arenadata/ADQM-566 
stylecheck renewed and version in docker set explicitely
 2022-09-05 03:23:38 +03:00
kssenii
48dc32faf8 Better test 2022-09-04 19:54:34 +02:00
Alexey Milovidov
dbca269ec7
Merge pull request #40321 from azat/mem/untracked_memory_limit_increase 
Remove ThreadStatus::untracked_memory_limit_increase
 2022-09-04 04:42:24 +03:00
Alexey Milovidov
660c1439ec Fix build 2022-09-04 03:27:55 +02:00
Alexey Milovidov
13a129bee7 Merge branch 'master' into remove-useless-line 2022-09-04 03:24:40 +02:00
Alexey Milovidov
22f7bfdcb5
Update gtest_sensitive_data_masker.cpp 2022-09-04 03:42:58 +03:00
Alexey Milovidov
c6f7ddc5e6
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-04 03:42:24 +03:00
Azat Khuzhin
9479e2143f Do not try to load empty debug files 
This will avoid CANNOT_PARSE_ELF error for builds that has empty debug
file in clickhouse-common-static-dbg package, i.e. debug build.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-02 23:23:05 +02:00
Azat Khuzhin
597197c01d Improve error messages in Elf loader 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-02 23:22:25 +02:00
Azat Khuzhin
45afaa6fb8 Fix loading external symbols 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-02 23:22:25 +02:00
Alexander Tokmakov
2044470895 review fixes 2022-09-02 19:18:44 +02:00
Suzy Wang
83b029e716 re arrange sensitive data masker unit test scenario order 2022-09-02 10:11:57 -07:00
Sergei Trifonov
067e712fdf
Merge branch 'master' into s3-detailed-metrics 2022-09-02 16:59:52 +02:00
Ilya Golshtein
74561b40ae stylecheck renewed 2022-09-02 17:42:18 +03:00
Antonio Andelic
b816d46520
Merge branch 'master' into fix-keeper-system-path-check 2022-09-02 13:42:16 +02:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage 2022-09-02 12:13:07 +02:00
Alexander Tokmakov
3582a51a72
Merge pull request #40920 from ClickHouse/fix-typos 
Fix typos found by new `codespell`
 2022-09-02 13:11:16 +03:00
Alexander Tokmakov
5b6a8677ef
Merge pull request #40890 from arthurpassos/patch-1 
CaresPTRResolver small safety improvement
 2022-09-02 12:29:23 +03:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Antonio Andelic
b7eebfc626 Correctly check if the node is using system path 2022-09-02 07:47:12 +00:00
Robert Schulze
319d8b00a7
Merge pull request #39010 from FrankChen021/tracing_context_propagation 
Improve the opentelemetry tracing context propagation across threads
 2022-09-02 07:56:43 +02:00
Sergei Trifonov
29e922f05c support all profile events for disk s3 2022-09-02 02:22:52 +02:00
Sergei Trifonov
ac9dc53c73 Merge branch 'master' into s3-detailed-metrics 2022-09-01 18:31:02 +02:00
Sergei Trifonov
bcb6475c4a add separate s3 profile events for disk s3 2022-09-01 18:30:55 +02:00
Dmitry Novik
2ca619d40f
Merge pull request #40320 from azat/memory-tracker-fix-peak-race 
Fix rare race in updating peak of memory tracker
 2022-09-01 18:07:33 +02:00
Frank Chen
9d63cbe811 Merge 'origin/master' into tracing_context_propagation to resolve conflicts 2022-09-01 23:18:59 +08:00
Alexander Tokmakov
04d431793b
Update src/Common/ZooKeeper/ZooKeeperImpl.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-09-01 17:12:22 +03:00
Alexander Tokmakov
36454a340d
Update src/Common/ZooKeeper/ZooKeeperArgs.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-09-01 17:12:16 +03:00
Antonio Andelic
62ff61404f Improve remove recursive 2022-09-01 12:16:13 +00:00
Arthur Passos
fb42afbbac
CaresPTRResolver small safety improvement 
Previous to #40769, only `hostent::h_aliases` was being accessed. After that PR got merged, `hostent::h_name` started being accessed as well. This PR moves the first `hostent::h_aliases != nullptr` check that could prevent `hostent::h_name` from being accessed. During debugging, I observed that even when there are not aliases, `hostent::h_aliases` is not null. That's why it hasn't caused any problems, but proposing this change to be on the safe side.
 2022-09-01 08:59:14 -03:00
kssenii
82879543d3 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-01 13:13:01 +02:00
Antonio Andelic
c7d1dd942c Address PR comments 2022-09-01 07:56:06 +00:00
Frank Chen
d9a516288c Resolve comments 2022-09-01 11:56:10 +08:00
kssenii
10a7e73d52 Fix unit test 2022-08-31 21:07:39 +02:00
kssenii
0f747e7987 Move cache files 2022-08-31 17:50:55 +02:00
Robert Schulze
ca01286028
Merge pull request #39629 from ClickHouse/catboost-bridge 
Move CatBoost evaluation into clickhouse-library-bridge
 2022-08-31 16:16:11 +02:00
Antonio Andelic
9dd1a9859d Merge branch 'master' into keeper-storage 2022-08-31 10:12:11 +00:00
Kseniia Sumarokova
9abcfce529
Merge pull request #40759 from kssenii/fix-write-through-cache-logical-error 
Fix logical error from write-though cache from stress test
 2022-08-31 11:55:18 +02:00
kssenii
a5f69517ae Allow to load marks in threadpool 2022-08-30 20:20:05 +02:00
Alexander Tokmakov
6a50c20734
Merge branch 'master' into zookeeper_client_fault_injection 2022-08-30 15:43:30 +03:00
Alexander Tokmakov
022f440ad0
Merge pull request #40769 from arthurpassos/caresptrresolver-hosts-patch 
Fix CaresPTRResolver not reading hosts file
 2022-08-30 14:35:10 +03:00
Kseniia Sumarokova
54f6f5bb2c
Merge branch 'master' into fix-write-through-cache-logical-error 2022-08-30 13:30:13 +02:00
Kseniia Sumarokova
5eb188311a
Merge pull request #40586 from kssenii/fix-race 
Fix possible race in fs cache
 2022-08-30 12:58:31 +02:00
Alexander Tokmakov
0ec7f068cc
Merge pull request #40677 from ClickHouse/query-insert-oom 
Avoid deadlock in case of new query and OOM
 2022-08-30 13:54:34 +03:00
Antonio Andelic
53da469147 Merge branch 'master' into keeper-storage 2022-08-30 10:22:04 +00:00
Antonio Andelic
80365e0448 Simplify drop and create 2022-08-30 09:19:59 +00:00
Frank Chen
922a0dee74 Fix style 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-30 15:14:12 +08:00
Frank Chen
f17d56b528 Merge branch 'master' into tracing_context_propagation 2022-08-30 14:24:36 +08:00
Frank Chen
83cbdef3c6 Fix context re-initialization for ThreadPool 2022-08-30 12:26:23 +08:00
Robert Schulze
6b2b3c1eb3
feat: implement catboost in library-bridge 
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes #27870
 2022-08-29 20:26:45 +00:00
Dmitry Novik
865ee5d0d6 Refactor code 2022-08-29 20:24:35 +02:00
Arthur Passos
961365c7a4 Fix CaresPTRResolver not reading hosts file 2022-08-29 15:11:39 -03:00
Kseniia Sumarokova
c5c48e44ea
Merge branch 'master' into fix-mysql-timeouts 2022-08-29 19:33:29 +02:00
kssenii
545c6c8be4 Fix 2022-08-29 17:50:27 +02:00
kssenii
b1dab84d97 Review fixes 2022-08-29 16:23:14 +02:00
kssenii
0a6c4b9265 Fix 2022-08-29 16:20:53 +02:00
kssenii
877ade9a50 Merge remote-tracking branch 'upstream/master' into fix-race 2022-08-29 16:05:27 +02:00
Alexander Tokmakov
8bdb589c2b Merge branch 'master' into zookeeper_client_fault_injection 2022-08-29 13:34:57 +02:00
Alexander Tokmakov
69387acffa
Revert "Support for DWARF-5 in in house DWARF parser" 2022-08-29 14:25:53 +03:00
Alexey Milovidov
ae97e880d7
Merge pull request #40710 from azat/DWARF-5 
Support for DWARF-5 in in house DWARF parser
 2022-08-29 00:13:33 +03:00
Alexey Milovidov
f462160bd1 Merge remote-tracking branch 'origin/master' into remove-useless-line 2022-08-28 21:00:29 +02:00
Alexey Milovidov
00074a5e14
Merge pull request #40711 from ClickHouse/all_new_functions_must_be_documented 
All new functions must be documented
 2022-08-28 17:05:05 +03:00
Azat Khuzhin
e03870bc8b Common/Dwarf: fix clang-tidy 2022-08-28 15:51:46 +02:00
Alexey Milovidov
d3a7945b1b
Merge pull request #40646 from ClickHouse/force-documentation-3 
Every profile event must have documentation
 2022-08-28 04:10:11 +03:00
Alexey Milovidov
b42d26acfe Remove one line from XML, because we do not care 2022-08-28 02:44:02 +02:00
Azat Khuzhin
ee5696bb32 Add support for DWARF-5 
I have to do this, since I have libraries compiled with DWARF5 (i.e.
glibc).

ClickHouse changes:
- use camel_case
- add NOLINT
- avoid using folly:: (use std:: instead)
- avoid using boost:: (use std:: instead)

Refs: 490b287ca3
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-27 22:28:15 +02:00
Alexey Milovidov
6139cc8f7a Add Documentation to FunctionFactory 2022-08-27 22:06:57 +02:00
Alexey Milovidov
e7054debc1
Merge pull request #40649 from ClickHouse/embedded-documentation 
A prototype for embedded documentation
 2022-08-27 21:28:11 +03:00
Alexey Milovidov
a6387ac0f7
Merge branch 'master' into force-documentation-3 2022-08-27 19:31:44 +03:00
Alexey Milovidov
e04f69717b
Merge pull request #40673 from ClickHouse/vdimir/tmp-file-metrics-doc 
Add doc to ExternalSort/Agg/Join ProfileEvents
 2022-08-27 16:52:40 +03:00
Alexander Gololobov
89be7a5a25
Update ProfileEvents.cpp 2022-08-26 22:55:42 +02:00
Kseniia Sumarokova
11b82ffc5e
Merge branch 'master' into fix-race 2022-08-26 22:55:14 +02:00
Kseniia Sumarokova
0cf76dab77
Merge pull request #40608 from ClickHouse/kssenii-patch-4 
Fix stress test after #40420
 2022-08-26 22:54:37 +02:00
Alexey Milovidov
c9858099ea
Update ProfileEvents.cpp 2022-08-26 22:10:10 +03:00
Alexander Gololobov
4fef408104
Merge branch 'master' into force-documentation-3 2022-08-26 20:29:06 +02:00
vdimir
c5a9b64080
Add doc to ExternalSort/Agg/Join ProfileEvents 2022-08-26 13:41:53 +00:00
Alexey Milovidov
46594e47bc Add comment 2022-08-26 15:28:18 +02:00
kssenii
470e18677f Ger rid of cache lock 2022-08-26 14:13:36 +02:00
Vladimir C
e067629e0d
Merge pull request #40239 from vdimir/vdimir/tmp-file-metrics 
More metrics for on-disk temporary data
 2022-08-26 11:28:01 +02:00
Robert Schulze
1c8c83ccf3
Merge pull request #40620 from zvonand/zvonand-b58 
Base58 fix handling leading 0 / '1'
 2022-08-26 10:18:26 +02:00
Alexander Gololobov
c7a792d929
small grammar fix 2022-08-26 10:03:43 +02:00
Alexander Gololobov
856f83caa1
Removed unnecessary commas 2022-08-26 10:00:20 +02:00
Frank Chen
cf1081eada Exception safe from destructor 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-26 11:18:16 +08:00
Alexey Milovidov
a029116fe3 Add a comment 2022-08-26 04:01:34 +02:00
Alexey Milovidov
8afbfebd25 A prototype for embedded documentation 2022-08-26 03:41:52 +02:00
Alexey Milovidov
91bd6f1d46 Write the documentation for all ProfileEvents. Enforce that documentation exists. 2022-08-26 01:35:03 +02:00
Kseniia Sumarokova
a8abf58f13
Update FileSegment.cpp 2022-08-25 23:15:24 +02:00
Alexander Tokmakov
eae2667a1c
Merge pull request #40148 from ClickHouse/replicate-tables-readonly 
`ReplicatedMergeTree` tables as readonly until attach is finished
 2022-08-25 19:39:19 +03:00
zvonand
3b164f0b12 style fix 2022-08-25 17:07:41 +03:00
Vladimir C
ddde5096ef
Merge branch 'master' into vdimir/tmp-file-metrics 2022-08-25 15:23:35 +02:00
Antonio Andelic
9964513155 Merge branch 'master' into keeper-storage 2022-08-25 12:57:03 +00:00
zvonand
b3360c169d Fix handling zeros/'1's at the beginning of input 2022-08-25 15:35:39 +03:00
Frank Chen
92f7ca3616 Move TracingContextOnThread::current() out of class for simplicity 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:23:56 +08:00
Frank Chen
bb00dcc19b Remove using namespace from header 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:20:13 +08:00
Kseniia Sumarokova
fac87d9c4d
Update FileSegment.cpp 2022-08-25 13:18:03 +02:00
Kseniia Sumarokova
2089ce14dd
Merge branch 'master' into fix-race 2022-08-25 13:03:29 +02:00
Kseniia Sumarokova
44577ea8f6
Update FileSegment.cpp 2022-08-25 13:02:41 +02:00
Kseniia Sumarokova
7e0ff0dfaf
Update FileSegment.cpp 2022-08-25 12:53:51 +02:00
Kseniia Sumarokova
b0f5fd9bf6
Update FileSegment.cpp 2022-08-25 12:41:31 +02:00
alesapin
c4f8811701
Merge pull request #40395 from ClickHouse/data_source_description 
Add copy shortcut for backups if dest and source support it
 2022-08-25 12:36:30 +02:00
Kseniia Sumarokova
f6eea03120
Merge pull request #40569 from kssenii/add-comments 
Add some logs to fs cache
 2022-08-25 12:06:48 +02:00
Frank Chen
221a71f801 Fix typo 2022-08-25 15:05:07 +08:00
Frank Chen
558b179cb1 Fix typo 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 10:14:18 +08:00
Frank Chen
99c37ce6c6
Merge branch 'master' into tracing_context_propagation 2022-08-25 10:07:16 +08:00
Antonio Andelic
da0d6056f4 Merge branch 'master' into replicate-tables-readonly 2022-08-24 17:29:33 +00:00
kssenii
ee2f9caa16 Fix 2022-08-24 18:50:22 +02:00
vdimir
7194df1184
Move back TemporaryFile -> TemporaryFileOnDisk 2022-08-24 16:14:11 +00:00
vdimir
1e24a731f7
Move Disks/TemporaryFileOnDisk -> Common/TemporaryFile 2022-08-24 16:14:10 +00:00
vdimir
8701a909a2
enoughSpaceInDirectory works if file is not created yet 2022-08-24 16:14:09 +00:00
vdimir
51c44424cc
More metrics for temp files 2022-08-24 16:14:09 +00:00
vdimir
cd4038d511
Use TemporaryFileOnDisk instead of Poco::TemporaryFile 2022-08-24 16:14:08 +00:00
vdimir
7e0c9062c7
Add ProfileEvents::ExternalSort(Un)CompressedBytes 2022-08-24 16:14:07 +00:00
Kseniia Sumarokova
3b982f6b8b
Update FileCache.cpp 2022-08-24 17:29:43 +02:00
Kseniia Sumarokova
aa0ed28a0b
Update FileSegment.cpp 2022-08-24 16:43:44 +02:00
Kseniia Sumarokova
985276f8e6
Merge branch 'master' into add-comments 2022-08-24 16:33:35 +02:00
Kseniia Sumarokova
6d0907ebef
Update FileSegment.cpp 2022-08-24 15:55:07 +02:00
Kseniia Sumarokova
2cea295a7a
More logs 2022-08-24 15:53:54 +02:00
alesapin
eeb0a3584e Merge branch 'master' into data_source_description 2022-08-24 15:26:18 +02:00
Kseniia Sumarokova
3e8a2d9093
Update FileSegment.cpp 2022-08-24 14:16:53 +02:00
kssenii
2b792a2bdd Add more logs 2022-08-24 12:40:44 +02:00
Frank Chen
1cf3659f26 Fix style checking 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-24 18:03:14 +08:00
Frank Chen
dd0f2cbe6b Fix style checking 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-24 17:04:02 +08:00
Frank Chen
03007606ba Fix format 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-24 17:01:31 +08:00
Frank Chen
cd19366b44 Move classes into DB::OpenTelemetry namespace 2022-08-24 16:41:40 +08:00
Frank Chen
efc6a60a60 Clean code 2022-08-24 15:59:44 +08:00
Antonio Andelic
fc33941959 Add support for shared drop/create 2022-08-23 13:15:31 +00:00
Kseniia Sumarokova
d814a56277
Merge pull request #40420 from kssenii/fix-cache-bug 
Fix incorrect fallback to skip cache in case of very high concurrency level
 2022-08-23 14:18:38 +02:00
Robert Schulze
635cc7617e
Merge pull request #40464 from ClickHouse/less-string-ref 
Reduce some usage of StringRef
 2022-08-23 12:16:00 +02:00
Maksim Kita
11ba15dde6
Merge branch 'master' into dns-resolver-remove-hints 2022-08-22 22:23:35 +02:00
Alexey Milovidov
0252695c6e
Merge pull request #40224 from ClickHouse/alexey-milovidov-patch-4 
Check what will be if I enable `concurrency-mt-unsafe` in `clang-tidy`
 2022-08-22 23:12:01 +03:00
Maksim Kita
191f39ae8d DNSResolver remove AI_V4MAPPED, AI_ALL hints 2022-08-22 21:03:25 +02:00
Robert Schulze
ec9ef741fc
Merge branch 'master' into less-string-ref 2022-08-22 20:48:16 +02:00
Kseniia Sumarokova
f85e5f8808
Merge branch 'master' into fix-cache-bug 2022-08-22 16:54:07 +02:00
Azat Khuzhin
ec2e671d20 Remove ThreadStatus::untracked_memory_limit_increase 
It looks useless nowadays, because operator new cannot throw
MEMORY_LIMIT_EXCEEDED today, and so any code that works on Exception is
likely safe.

Refs: #40249
Refs: #24483
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-22 13:26:18 +03:00
kssenii
74087d57c4 Less notify 2022-08-22 12:00:14 +02:00
kssenii
26384b2543 Fix 2022-08-22 00:02:05 +02:00
Alexey Milovidov
996aa2d126 Remove some trash 2022-08-21 20:46:33 +02:00
Robert Schulze
e8b3f56733
Limit suppression to a specific warning 2022-08-21 18:24:17 +00:00
Robert Schulze
4caf2c4c33
Reduce some usage of StringRef 
See #39535 and #39300
 2022-08-21 18:10:32 +00:00
Antonio Andelic
f29ef68936 Merge branch 'master' into keeper-storage 2022-08-21 17:09:36 +00:00
alesapin
78ba732b31 Merge branch 'master' into data_source_description 2022-08-21 12:11:21 +02:00
Alexey Milovidov
0db45afb08
Merge branch 'master' into alexey-milovidov-patch-4 2022-08-20 19:58:31 +03:00
Alexey Milovidov
3649aac593 Fix style 2022-08-20 17:59:51 +02:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
Frank Chen
54a530a4e0 Make sure right toString is linked 2022-08-20 14:22:33 +08:00
Frank Chen
0569608192 Use toString 2022-08-20 12:30:39 +08:00
pkubaj
17f757a94c Add FreeBSD/powerpc64le support 2022-08-19 21:41:03 +02:00
Azat Khuzhin
28f2607d50 Fix rare race in updating peak of memory tracker 
Previously it was possible that allocation, that had been done with
locked MEMORY_LIMIT_EXCEEDED (LockMemoryExceptionInThread), will update
the peak to the amount that will include some other allocations.

Consider the following example (T1/T2 - different threads):

Initial state:

    MemoryTracker::amount = 0

Possible race:

    LockMemoryExceptionInThread locker;
T1: MemoryTracker::allocImpl(1MiB)      T2: MemoryTracker::allocImpl(1PiB)
                                            will_be = 1T, amount = 1T
    will_be = 1T+1MiB, amount=1T+1MiB
    peak = will_be
                                            amount = 0 // revert due to exceeding limit
                                            throw(MEMORY_LIMIT_EXCEEDED)

So as you can see T1 includes allocations from T2 into the peak.

Note, that almost every place that has memory_usage column uses
MemoryTracker::peak (not MemoryTracker::amount), this is true for
query_log (for part_log column name is peak_memory_usage, and obviously
it uses MemoryTracker::peak).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-19 20:57:12 +02:00
kssenii
e641664115 Refactor 2022-08-19 18:15:18 +02:00
alesapin
d8664c3227 Add shortcut for backups 2022-08-19 16:58:30 +02:00
Antonio Andelic
5e2c8869a9 Merge branch 'master' into replicate-tables-readonly 2022-08-19 09:41:11 +00:00
Alexander Tokmakov
3d253ec51b
Merge branch 'master' into zookeeper_client_fault_injection 2022-08-18 21:23:50 +03:00
Frank Chen
a3b6ad2a65
Merge branch 'master' into tracing_context_propagation 2022-08-18 20:59:07 +08:00
Nikolai Kochetov
5a85531ef7
Merge pull request #38286 from Avogar/schema-inference-cache 
Add schema inference cache for s3/hdfs/file/url
 2022-08-18 13:07:50 +02:00
Kseniia Sumarokova
d8fd1f9c37
Merge pull request #36171 from kssenii/make-cache-composable 
Make cache composable, allow not to evict certain files (regarding .idx, .mrk, ..), delete old cache version
 2022-08-18 12:37:34 +02:00
Vladimir C
7ac0eb1af1
Merge pull request #40035 from ClickHouse/vdimir/cluster-discovery-observer 
Add observer mode to (zoo)keeper cluster discovery feature
 2022-08-18 10:53:49 +02:00
kssenii
d63d34f7bb Merge master 2022-08-17 15:49:32 +02:00
avogar
8dd54c043d Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache 2022-08-17 11:47:40 +00:00
Alexander Tokmakov
747d74121c
Merge branch 'master' into zookeeper_client_fault_injection 2022-08-17 14:41:02 +03:00
Nikolai Kochetov
8ec3f0821c
Merge pull request #40249 from azat/fix-memory-account-on-error 
Fix memory accounting in case of MEMORY_LIMIT_EXCEEDED errors
 2022-08-17 12:51:40 +02:00
Alexander Tokmakov
ae000e9125
Merge branch 'master' into zookeeper_client_fault_injection 2022-08-17 12:48:54 +03:00
Alexander Tokmakov
4482465ce3
Merge pull request #40134 from arthurpassos/fix_cares_reverse_dns_race_condition 
Fix CaresPTRResolver race condition
 2022-08-16 21:06:17 +03:00
Nikolai Kochetov
feab1eb0a3
Update CurrentMemoryTracker.cpp 2022-08-16 18:40:06 +02:00
Nikolai Kochetov
8e4c967fc9 Try to account untracked memory (per thread) more correctly. 2022-08-16 18:09:05 +02:00
Azat Khuzhin
ded70fbd53 Fix memory accounting in case of MEMORY_LIMIT_EXCEEDED errors 
In case of memory allocation had been failed, it should not be take into
account anywhere, otherwise it will report incorrect memory usage for
query and in logs.

I found this while was digging into OOM in stress tests, and saw memory
allocations that had been significantly greater the memory limit (even
with overcommit).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-16 18:08:56 +02:00
kssenii
43991729ef Fix 2022-08-16 17:08:12 +02:00
kssenii
93816e7366 Review fixes 2022-08-16 15:11:55 +02:00
kssenii
3901778d49 Add more current metrics for cache 2022-08-16 11:32:23 +02:00
Antonio Andelic
2ed42bca7e Merge branch 'master' into replicate-tables-readonly 2022-08-16 08:17:53 +00:00
kssenii
eb26b219b9 Merge master 2022-08-16 00:56:27 +02:00
Alexander Tokmakov
c9bb91c31f
Merge pull request #40232 from ClickHouse/fix_a_bug_with_symlinks_detection 
Fix a bug with symlinks detection
 2022-08-15 20:28:30 +03:00
Robert Schulze
70f63d2aae
Merge pull request #40033 from ClickHouse/reenable-avx512-vbmi-columnvector-filter 
Revert the revert of "ColumnVector: optimize filter with AVX512 VBMI2 compress store"
 2022-08-15 18:39:57 +02:00
Alexander Tokmakov
0e22b4b800
Update src/Common/filesystemHelpers.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-08-15 14:25:25 +03:00
Alexander Tokmakov
6f5a7c3bf7 fix a bug with symlinks detection 2022-08-15 12:30:47 +02:00
Nikita Mikhaylov
2282f2f414
Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-08-15 12:25:45 +02:00
Alexey Milovidov
903021f35a
Merge branch 'master' into profile-events-incomplete-result 2022-08-14 11:04:19 +03:00
Alexey Milovidov
1a8ddf2956 Addition to prev. revision 2022-08-14 09:35:22 +02:00
Alexey Milovidov
001aca3b47 ProfileEvents for incomplete data due to query complexity settings 2022-08-14 09:17:02 +02:00
Alexey Milovidov
216c1d8efa
Revert "Use separate counter for RSS in global memory tracker." 2022-08-14 09:50:12 +03:00
Alexey Milovidov
6b8f4f1d59 Merge branch 'master' into server-side-time 2022-08-12 21:36:40 +02:00
Maksim Kita
098436f012
Merge pull request #40129 from kitaisreal/pod-array-assign-empty-array-fix 
PODArray assign empty array fix
 2022-08-12 18:34:19 +02:00
Kseniia Sumarokova
a6cfc7bc3b
Merge pull request #34651 from alexX512/master 
New caching strategies
 2022-08-12 17:23:37 +02:00
Alexey Milovidov
8c3d1f2b5b
Merge branch 'master' into server-side-time 2022-08-12 17:40:02 +03:00
Arthur Passos
07163eb79a Remove c-ares lib de-initialization 2022-08-12 10:49:39 -03:00
vdimir
cef6c4f8ce
Add observer mode to (zoo)keeper cluster discovery feature 2022-08-12 13:26:12 +00:00
Kruglov Pavel
4c7222d938
Merge pull request #40020 from canhld94/ch_canh_fix_hash 
fix HashMethodOneNumber with const column
 2022-08-12 14:40:24 +02:00
Nikita Mikhaylov
00d2c57a68
Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-08-12 13:32:24 +02:00
Antonio Andelic
1c0f85ccdb Attach table in background 2022-08-12 09:32:13 +00:00
kssenii
fd6362d2f2 Merge master 2022-08-12 11:22:39 +02:00
Kseniia Sumarokova
09ad1115d7
Merge pull request #40138 from ClickHouse/kssenii-patch-4 
fs cache: minor change
 2022-08-12 11:18:03 +02:00
Maksim Kita
e2ae0d585b
Merge pull request #40078 from Algunenano/threadlocal_perf 
Optimize CurrentMemoryTracker alloc and free
 2022-08-12 10:24:54 +02:00
Maksim Kita
5dd4f4b10f
Merge branch 'master' into pod-array-assign-empty-array-fix 2022-08-12 10:20:33 +02:00
alexX512
06cb8737a8 Delete TODO 2022-08-11 22:38:13 +00:00
kssenii
b7c91b9cdb Merge remote-tracking branch 'origin/make-cache-composable' into make-cache-composable 2022-08-11 23:09:35 +02:00
kssenii
768a207176 Merge master 2022-08-11 22:25:53 +02:00
Kseniia Sumarokova
004a4d4947
Update FileCache.cpp 2022-08-11 22:15:33 +02:00
Kseniia Sumarokova
687ea5dd76
Update FileCache.cpp 2022-08-11 22:13:22 +02:00
Arthur Passos
59108345e9 Use one CaresPTRResolver per thread 2022-08-11 13:28:07 -03:00
Kseniia Sumarokova
be69169f97
Merge pull request #38048 from KinderRiven/decoupling_cache_function_and_algorithm 
Decoupling local cache function and cache algorithm
 2022-08-11 16:49:56 +02:00
Maksim Kita
f9326b00c8 PODArray assign empty array fix 2022-08-11 16:26:41 +02:00
Raúl Marín
11a274e990 Clean up constinit usage and add a comment 2022-08-11 13:27:53 +02:00
Nikolai Kochetov
971d94e492
Merge pull request #38682 from ClickHouse/separate-counter-for-rss-in-memory-tracker 
Use separate counter for RSS in global memory tracker.
 2022-08-11 10:11:18 +02:00
Duc Canh Le
84cd867aa8 materialize column instead of handling column in hash method 2022-08-11 10:46:06 +08:00
Alexey Milovidov
8374f31306
Merge pull request #39425 from arenadata/ADQM-485 
Add support of dates from year 1900 to 2299 for Date32 and DateTime64
 2022-08-11 05:01:53 +03:00
Raúl Marín
d7a545e30d Try to optimize CurrentMemoryTracker alloc and free 2022-08-10 22:05:09 +02:00
Nikolai Kochetov
c9353cbae8
Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-08-10 21:39:50 +02:00
Alexey Milovidov
0b24dbaec8 Merge 2022-08-10 19:41:44 +02:00
KinderRiven
1aa7bbcbbd fix unique_ptr 2022-08-11 00:11:41 +08:00
KinderRiven
9b7f87677d fix 2022-08-11 00:11:41 +08:00
KinderRiven
9ba94e64f9 fix 2022-08-11 00:11:41 +08:00
KinderRiven
2ae02a4921 fix style 2022-08-11 00:11:40 +08:00
KinderRiven
76e0aad69e fix 2022-08-11 00:11:40 +08:00
KinderRiven
fbaa70b313 fix 2022-08-11 00:11:40 +08:00
KinderRiven
f6a58bff4c fix build 2022-08-11 00:11:40 +08:00
KinderRiven
d2b5581632 fix 2022-08-11 00:11:40 +08:00
KinderRiven
1b01cc8ed9 fix 2022-08-11 00:11:40 +08:00
KinderRiven
164fa1ab0e fix build and style 2022-08-11 00:11:40 +08:00
KinderRiven
61b580aba4 add note 2022-08-11 00:11:40 +08:00
KinderRiven
081cd4938a fix style 2022-08-11 00:11:40 +08:00
KinderRiven
9d83b93e88 fix rebase 2022-08-11 00:11:40 +08:00
KinderRiven
50fd740ec3 fix 2022-08-11 00:11:40 +08:00
KinderRiven
c5f9022510 fix 2022-08-11 00:11:40 +08:00
KinderRiven
43cf771657 better 2022-08-11 00:11:40 +08:00
KinderRiven
ffaf44c1c1 fix style 2022-08-11 00:11:40 +08:00
KinderRiven
7f54fa726b Decoupling cache functions and algorithms 2022-08-11 00:11:40 +08:00
Kseniia Sumarokova
a4d513657c
Merge pull request #40010 from ClickHouse/catboost-further-preparation 
Further preparation for catboost integration into library-bridge
 2022-08-10 17:35:12 +02:00
kssenii
0ff5609738 Finish support for persistent files 2022-08-10 16:54:59 +02:00
Nikita Mikhaylov
8e8e845a89
Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-08-10 16:44:24 +02:00
Duc Canh Le
bf5e9173ad
fix hashMethodOneNumber constructor 2022-08-10 21:01:09 +08:00
Duc Canh Le
ec49f7b3a3 fix style 2022-08-10 19:28:08 +08:00
Duc Canh Le
17fa8076e3 better implementation 2022-08-10 18:59:13 +08:00
Duc Canh Le
7910a93b02 fix other hash methods and add tests 2022-08-10 18:59:13 +08:00
Duc Canh Le
1a08161dbf simplify solution 2022-08-10 18:59:13 +08:00
Duc Canh Le
cdd5b32d29 fix HashMethodOneNumber with const column 2022-08-10 18:59:13 +08:00
kssenii
7b66b96ec5 Minor change 2022-08-10 12:20:54 +02:00
kssenii
5c3227ba56 Merge master 2022-08-10 12:00:34 +02:00
Antonio Andelic
4023d4a37a Add support for drop/truncate and tests 2022-08-10 07:24:56 +00:00
Alexey Milovidov
33e43cc6f9
Merge branch 'master' into server-side-time 2022-08-10 07:28:15 +03:00
Robert Schulze
05ea33c897
Revert "Revert "ColumnVector: optimize filter with AVX512VBMI2 compress store"" 
This reverts commit 5524706b78.
 2022-08-09 17:00:13 +00:00
Kruglov Pavel
5ebf230dbb
Merge pull request #39426 from kitaisreal/shell-command-wait-pid-refactoring 
ShellCommand wait pid refactoring
 2022-08-09 11:48:20 +02:00
Roman Vasin
96598e3574 Replace normalizeDayNum() by std::clamp() 2022-08-09 08:52:02 +00:00
alexX512
21624708aa Fix tidy build 2022-08-09 08:36:49 +00:00
alexX512
c9c26d4898 Fix review 2022-08-08 21:34:24 +00:00
Robert Schulze
e0d5020a92
Add simple versioning to the *-bridge-to-server protocol 
- In general, it is expected that clickhouse-*-bridges and
  clickhouse-server were build from the same source version (e.g. are
  upgraded "atomically"). If that is not the case, we should at least
  be able to detect the mismatch and abort.

- This commit adds a URL parameter "version", defined in a header shared
  by the server and bridges. The bridge returns an error in case of
  mismatch.

- The version is *not* send and checked for "ping" requests (used for
  handshake), only for regular requests send after handshake. This is
  because the internally thrown server-side exception due to HTTP
  failure does not propagate the exact HTTP error (it only stores the
  error as text), and as a result, the server-side handshake code
  simply retries in case of error with exponential backoff and finally
  fails with a "timeout error". This is reasonable as pings typically
  fail due to time out. However, without a rework of HTTP exceptions,
  version mismatch during ping would also appear as "timeout" which is
  too misleading. The behavior may be changed later if needed.

- Note that introducing a version parameter does not represent a
  protocol upgrade itself. Bridges older than the server will simply
  ignore the field. Only servers older than the bridges receive an error
  but such a situation should never occur in practice.
 2022-08-08 19:40:37 +00:00
Alexey Milovidov
db85ebfaa7 Add server-side time to Progress 2022-08-08 06:55:41 +02:00
Alexey Milovidov
5524706b78
Revert "ColumnVector: optimize filter with AVX512VBMI2 compress store" 2022-08-08 05:03:15 +03:00
alexX512
f16c668a0c Delete Args from CacheBase 2022-08-07 20:44:26 +00:00
alexX512
1c9d40e4a8 Fix build error 2022-08-07 20:37:11 +00:00
alexX512
269631cfbe Fix typo 2022-08-07 20:02:32 +00:00
alexX512
62f84b88eb Review fixes 2022-08-07 20:02:21 +00:00
alexX512
8adf9f2448 fix compile error 2022-08-07 19:59:39 +00:00
alexX512
2be72560f4 Add loading of mark cache policy 2022-08-07 19:59:39 +00:00
alexX512
a057e1607a Fix remove operation, add evict ion i from cachef number of elements greater than max_elements_size 2022-08-07 19:59:39 +00:00
alexX512
8be62770aa Test with SLRU and debug logs 2022-08-07 19:59:39 +00:00
alexX512
c044fee7a0 Test with LRU 2022-08-07 19:59:39 +00:00
alexX512
a238e349cd Unblocking get max_size 2022-08-07 19:59:39 +00:00
alexX512
daab74925c Test with default cache policy = SLRU 2022-08-07 19:59:39 +00:00
alexX512
994a9fc005 Minor fixes 2022-08-07 19:59:38 +00:00
alexX512
50a8887ab9 Add [[maybe_unused]] locks and delete all objects from cache queue 2022-08-07 19:59:38 +00:00
alexX512
69cc89e36e Check with default cache policy = LRU 2022-08-07 19:59:38 +00:00
alexX512
a94950d3ac Test only with LRU 2022-08-07 19:59:38 +00:00
alexX512
9940b69bd7 Include fix in ICachePolicy.h 2022-08-07 19:59:38 +00:00
alexX512
910645a52a Include fix in SLRUCachePolicy.h 2022-08-07 19:59:38 +00:00
alexX512
4f52efd4bb Build fix 2022-08-07 19:59:38 +00:00
alexX512
6bf29cb610 Change class LRUCache to class CachBase. Check running CacheBase with default pcahce policy SLRU 2022-08-07 19:59:30 +00:00
alexX512
c606ccc75d Fix formatting problems 2022-08-07 19:16:40 +00:00
alexX512
1133e42367 Add CacheBase clas instead of LRUCache and SLRUCache fo simpler configuration oache policiesf 2022-08-07 19:16:40 +00:00
alexX512
a0d2da7261 Add tests for SLRU cache 2022-08-07 19:16:40 +00:00
AlPerevyshin
d7bdf9e601 Add SLRU cache 2022-08-07 19:16:38 +00:00
avogar
9b1a267203 Refactor, remove TTL, add size limit, add system table and system query 2022-08-05 16:20:15 +00:00
kssenii
91f4143b6e Merge master 2022-08-05 12:14:08 +02:00
Alexey Milovidov
3130aa7218
Merge pull request #39732 from HarryLeeIBM/hlee-s390x-unaligned 
Fix Endian issue in SipHash for s390x
 2022-08-05 07:24:57 +03:00
kssenii
7a9b0bc47f Merge master 2022-08-05 01:48:52 +02:00
Alexey Milovidov
24ee944986
Update TaskStatsInfoGetter.cpp 2022-08-04 07:34:36 +03:00
Fangyuan Deng
0bf5c3d1a1
Merge branch 'master' into fix_netlink_leak 2022-08-04 11:42:17 +08:00
pzhdfy
848a349e42 fix potential netlink leak when init fail in TaskStatsInfoGetter 2022-08-04 11:37:27 +08:00
Alexey Milovidov
f474eb957d
Merge pull request #39758 from ClickHouse/tsan_clang_15 
Try clang-15 for build with tsan
 2022-08-04 01:56:52 +03:00
Nikita Mikhaylov
f144eae388
Fix typo and extra dots in exception messages from OverCommitTracker (#39858) 2022-08-03 15:23:07 +02:00
Robert Schulze
00a7c8733b
Merge pull request #39633 from guowangy/filter-vbmi2 
ColumnVector: optimize filter with AVX512VBMI2 compress store
 2022-08-03 09:23:24 +02:00
Frank Chen
035896fa16 Fix build check 2022-08-02 20:18:42 +08:00
Frank Chen
1c9b462c80 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 20:18:22 +08:00
Alexander Tokmakov
b4c4448f0f fix 2022-08-02 12:07:38 +02:00
Alexander Tokmakov
82b50e79cf
Merge branch 'master' into tsan_clang_15 2022-08-02 13:00:55 +03:00
Frank Chen
57dde8c250 Clean up header inclusion 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-02 15:43:19 +08:00
Alexey Milovidov
316528817b
Merge pull request #39179 from azat/fsync-profile-events 
Add profile events for fsync
 2022-08-02 05:44:35 +03:00
Wangyang Guo
6a7213291b Merge master and resolve conflict 2022-08-02 10:40:40 +08:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
Alexander Tokmakov
ecf7ce1f74 Merge branch 'master' into zookeeper_client_fault_injection 2022-08-01 20:49:01 +02:00
Alexander Tokmakov
0d68b1c67f fix build with clang-15 2022-08-01 18:00:54 +02:00
Robert Schulze
bf574b9154
Merge pull request #39760 from ClickHouse/bit-fiddling 
Use std::popcount, ::countl_zero, ::countr_zero functions
 2022-08-01 17:04:51 +02:00
Alexander Tokmakov
80f9ba9186
Merge pull request #39690 from ClickHouse/show-addresses-in-stack-traces 
Configuration parameter to hide addresses in stack traces
 2022-08-01 14:20:37 +03:00
Wangyang Guo
b05be56eef ColumnVector: naming style fix 2022-08-01 10:16:15 +08:00
Robert Schulze
d39259a4c0
More conversions 2022-07-31 22:06:02 +00:00
Azat Khuzhin
3e627e2861 Add profile events for fsync 
The following new provile events had been added:

- FileSync - Number of times the F_FULLFSYNC/fsync/fdatasync function was called for files.
- DirectorySync - Number of times the F_FULLFSYNC/fsync/fdatasync function was called for directories.
- FileSyncElapsedMicroseconds - Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for files.
- DirectorySyncElapsedMicroseconds - Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for directories.

v2: rewrite test to sh with retries
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-31 23:19:30 +03:00
kssenii
e5f4a619ed Merge master 2022-07-31 20:24:40 +03:00
Robert Schulze
a7734672b9
Use std::popcount, ::countl_zero, ::countr_zero functions 
- Introduced with the C++20 <bit> header

- The problem with __builtin_c(l|t)z() is that 0 as input has an
  undefined result (*) and the code did not always check. The std::
  versions do not have this issue.

- In some cases, we continue to use buildin_c(l|t)z(), (e.g. in
  src/Common/BitHelpers.h) because the std:: versions only accept
  unsigned inputs (and they also check that) and the casting would be
  ugly.

(*) https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
 2022-07-31 15:16:51 +00:00
Robert Schulze
52d08d9db4
Merge pull request #39520 from ClickHouse/no-split-binary 
Remove SPLIT_BINARY
 2022-07-31 14:23:31 +02:00
Alexey Milovidov
d285648a37
Merge branch 'master' into shell-command-wait-pid-refactoring 2022-07-31 02:57:46 +03:00
Robert Schulze
85773e0926
Merge pull request #38615 from liyinsg/simplified_function_registration_interface 
Simplified function registration interface
 2022-07-31 00:18:37 +02:00
Alexey Milovidov
c0d7b6efc3 Suggestions from @tavplubix 2022-07-30 01:45:06 +02:00
Alexey Milovidov
4aa96767d5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into show-addresses-in-stack-traces 2022-07-30 01:44:04 +02:00
HarryLeeIBM
cc73b53116 Fix Endian issue in SipHash for s390x 2022-07-29 13:21:50 -07:00
Robert Schulze
3d1797f75f
Merge remote-tracking branch 'origin/master' into no-split-binary 2022-07-29 12:17:43 +00:00
Robert Schulze
a3c52f1140
Merge pull request #39679 from azat/fix-utils 
Fix utils build on CI
 2022-07-29 14:02:53 +02:00
Azat Khuzhin
498c8b3c52 Fix clang-tidy in utils/examples 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-29 11:36:51 +03:00
Li Yin
4088c0a7f3 Automated function registration 
Automated register all functions with below naming convention by
iterating through the symbols:
void DB::registerXXX(DB::FunctionFactory &)
 2022-07-29 15:39:50 +08:00
Alexey Milovidov
552b517e59 Allow to hide addresses in stack traces 2022-07-29 00:13:02 +02:00
Azat Khuzhin
e1f8cff3b4 Add duration_ms into system.zookeeper_log 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-28 22:34:51 +03:00
Robert Schulze
199e254777
Merge remote-tracking branch 'origin/master' into no-split-binary 2022-07-28 15:54:22 +00:00
Antonio Andelic
15e999f8bc
Merge pull request #39096 from ClickHouse/keeper-version-check 
Add version for Keeper API
 2022-07-28 10:10:49 +02:00
Sergei Trifonov
edf6b02723
Merge pull request #37558 from ClickHouse/concurrency-control 
add concurrency control for pipeline executors
 2022-07-28 08:56:24 +02:00
Yakov Olkhovskiy
f1bc767ccf
Merge pull request #39627 from HarryLeeIBM/hlee-s390x-wide_int 
Fix wide integer unit tests on s390x platform
 2022-07-27 21:30:12 -04:00
Alexander Tokmakov
8fc075a527
Merge pull request #39619 from ClickHouse/try-do-enable-global-allocations-for-attach 
Replace MemoryTrackerBlockerInThread to LockMemoryExceptionInThread
 2022-07-27 13:13:04 +03:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache 2022-07-27 11:35:36 +02:00
Antonio Andelic
904a05ac21
Merge pull request #39496 from azat/custom-tld-exclamation-asterisk 
Add support of !/* (exclamation/asterisk) in custom TLDs
 2022-07-27 08:55:49 +02:00
Wangyang Guo
e6752d687f TargetSpecific: add AVX512VBMI2 support 2022-07-27 13:30:43 +08:00
Wangyang Guo
50fdbcdc12 CpuId: add AVX512VBMI2 detection 2022-07-27 13:30:43 +08:00
HarryLeeIBM
d475086100 Fix wide integer unit test 2022-07-26 12:57:45 -07:00
Nikolai Kochetov
f1818a9103 Make only one possible public ctor in MemoryTrackerBlockerInThread 2022-07-26 19:04:20 +00:00
Nikolai Kochetov
dcb86eca0b Make only one possible public ctor in MemoryTrackerBlockerInThread 2022-07-26 19:03:26 +00:00
Nikolai Kochetov
fa0707b4e7 Review fixes. 2022-07-26 17:51:09 +00:00
Nikolai Kochetov
33ce662d3e Replace MemoryTrackerBlockerInThread to LockMemoryExceptionInThread in some places. Reduced MemoryTrackerBlockerInThread level to User. 2022-07-26 15:22:00 +00:00
Kruglov Pavel
c683cb252f
Merge pull request #39227 from amosbird/rename-log1 
Rename log when rename merge tree tables
 2022-07-26 17:12:44 +02:00
Sergei Trifonov
24ab5fbb86 fix finish() condition to account only active threads 2022-07-26 11:32:45 +02:00
Antonio Andelic
35b61cc94e Merge branch 'master' into keeper-version-check 2022-07-26 07:47:57 +00:00
Azat Khuzhin
1d4a7c7290 Add support of !/* (exclamation/asterisk) in custom TLDs 
Public suffix list may contain special characters (you may find format
here - [1]):
- asterisk (*)
- exclamation mark (!)

  [1]: https://github.com/publicsuffix/list/wiki/Format

It is easier to describe how it should be interpreted with an examples.

Consider the following part of the list:

    *.sch.uk
    *.kawasaki.jp
    !city.kawasaki.jp

And here are the results for `cutToFirstSignificantSubdomainCustom()`:

If you have only asterisk (*):

    foo.something.sheffield.sch.uk -> something.sheffield.sch.uk
    sheffield.sch.uk               -> sheffield.sch.uk

If you have exclamation mark (!) too:

    foo.kawasaki.jp                -> foo.kawasaki.jp
    foo.foo.kawasaki.jp            -> foo.foo.kawasaki.jp
    city.kawasaki.jp               -> city.kawasaki.jp
    some.city.kawasaki.jp          -> city.kawasaki.jp

TLDs had been verified wit the following script [2], to match with
python publicsuffix2 module.

  [2]: https://gist.github.com/azat/c1a7a9f1e3519793134ef4b1df5461a6

v2: fix StringHashTable padding requirements
Fixes: #39468
Follow-up for: #17748
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-26 08:34:30 +03:00
Roman Vasin
b462366415 Merge branch 'master' of github.com:ClickHouse/ClickHouse into ADQM-485 2022-07-25 17:55:47 +00:00
Roman Vasin
b412ea5f6d Improve generateRandom() for Date32; fix tests 01087_table_function_generate, 01277_fromUnixTimestamp64, 01691_DateTime64_clamp and 01702_toDateTime_from_string_clamping 2022-07-25 17:06:11 +00:00
Sergei Trifonov
d9d1381c3f Merge branch 'master' into concurrency-control 2022-07-25 14:16:46 +02:00
mergify[bot]
f667eff025
Merge branch 'master' into shell-command-wait-pid-refactoring 2022-07-25 11:42:05 +00:00
Robert Schulze
24c74da473
Merge pull request #39430 from azat/getauxval-fix 
Fix LSan by fixing getauxval() (resubmit v4)
 2022-07-25 10:43:09 +02:00
Alexey Milovidov
388d06fda1
Merge pull request #39535 from ClickHouse/stringref 
Less usage of StringRef
 2022-07-25 04:06:11 +03:00
Azat Khuzhin
93f876b58a Add a test for LSan 
(cherry picked from commit 51e7c41883)
v2: fix type check
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-25 01:21:40 +03:00
Robert Schulze
4333750985
Less usage of StringRef 
... replaced by std::string_view, see #39262
 2022-07-24 18:33:52 +00:00
Robert Schulze
73c04b64e3
Merge pull request #39493 from nicelulu/fix_build_osx 
Fix build on mac osx
 2022-07-24 18:11:34 +02:00
Robert Schulze
c788e05c77
Merge pull request #39292 from zvonand/zvonand-b58-datatype 
Simplify Base58 encoding/decoding
 2022-07-24 18:09:40 +02:00
Andrey Zvonov
032fcab70e set char to be explicitely signed 2022-07-24 14:23:22 +02:00
Alexey Milovidov
071374b152 Remove SPLIT_BINARY 2022-07-24 01:15:54 +02:00
Constantine Peresypkin
d645d03e89 fix -DENABLE_EXAMPLES=1 in master 2022-07-23 19:00:52 +02:00
Antonio Andelic
6aff87d4b5 Ignore system paths from snapshots and logstore 2022-07-23 13:46:59 +00:00
Antonio Andelic
46a6fbd7aa Small polish 2022-07-22 11:03:07 +00:00
zhangxiao871
0d4d1e66b5 Fix build 2022-07-22 18:53:44 +08:00
Alexander Tokmakov
bed2206ae9
Merge pull request #39460 from ClickHouse/remove_some_dead_and_commented_code 
Remove some dead and commented code
 2022-07-22 13:24:34 +03:00
Roman Vasin
0b102c6d1f Fix code style 2022-07-22 08:24:05 +00:00
Antonio Andelic
3040ff0959 Merge branch 'master' into keeper-version-check 2022-07-22 08:07:55 +00:00
Roman Vasin
74d2bf8d95 Fix gTest for DateLUTTest 2022-07-22 07:49:02 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache 2022-07-21 18:59:14 +02:00
Robert Schulze
ea0a3bf600
Merge branch 'master' into stringref-to-string_view 2022-07-21 18:33:06 +02:00
Maksim Kita
0159fe1274
Merge branch 'master' into shell-command-wait-pid-refactoring 2022-07-21 16:38:28 +02:00
Andrey Zvonov
e473606cd1
Merge branch 'master' into zvonand-b58-datatype 2022-07-21 15:55:05 +02:00
Alexander Tokmakov
9e9969cea7
Merge pull request #37827 from arthurpassos/host_regexp_multiple_domains 
Test host_regexp against all PTR records instead of only one
 2022-07-21 16:43:05 +03:00
Alexander Tokmakov
a8da5d96fc remove some dead and commented code 2022-07-21 15:05:48 +02:00
Maksim Kita
7f4a1b8bb8 ShellCommand wait pid refactoring 2022-07-21 11:45:09 +02:00
Nikolai Kochetov
e15967e9db
Merge pull request #38475 from ClickHouse/additional-filters 
Additional filters for a table (from setting)
 2022-07-21 07:52:04 +02:00
Alexey Milovidov
4305eacd92
Merge pull request #39434 from ClickHouse/revert-39299-fix-lsan-v3 
Revert "[RFC] Fix LSan by fixing getauxval()"
 2022-07-21 06:47:54 +03:00
Alexey Milovidov
f4f5a5e044
Merge pull request #39397 from azat/fix-extractAll 
Fix UB (stack-use-after-scope) in extactAll()
 2022-07-21 05:21:18 +03:00
Alexey Milovidov
72dda35108
Revert "[RFC] Fix LSan by fixing getauxval()" 2022-07-21 04:34:04 +03:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Kseniia Sumarokova
3759ee76a7
Merge pull request #39386 from ClickHouse/kssenii-patch-4 
Rethrow exception on cache initialisation on server startup, better error message
 2022-07-20 21:26:25 +02:00
Roman Vasin
e3192cf753 Correct docs to reflect new range 1900..2299 for Date32 and DateTime64; Cleanup code 2022-07-20 15:19:02 +00:00
zvonand
11b8d788ca small improvements 2022-07-20 13:36:47 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache 2022-07-20 13:30:42 +02:00
Kseniia Sumarokova
0840dfe2e0
Update LRUFileCache.cpp 2022-07-20 11:42:24 +02:00
Maksim Kita
d3ce4ffc88
Merge pull request #38929 from pkit/pkit/wait_pid_fix 
fix long wait for process exit in ShellCommand
 2022-07-20 11:26:38 +02:00
Azat Khuzhin
18eb8b6d48 Fix UB (stack-use-after-scope) in extactAll() 
After #37544 OptimizedRegularExpressionImpl started to be moved, but
StringSearcher is not copyable since it holds pointers that goes out of
scope after move (before Regexps::get() returns std::shared_ptr<Regexp>
but it had been replaced with Regexps::createRegexp() that returns
Regexp object).

<details>

<summary>ASan report</summary>

    ==48348==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7fff577239a9 at pc 0x00001518209b bp 0x7fff57723820 sp 0x7fff57723818
    READ of size 1 at 0x7fff577239a9 thread T0
        0 0x1518209a in char8_t const* DB::StringSearcher<true, true>::search<char8_t>(char8_t const*, char8_t const*) const /bld/./src/Common/StringSearcher.h:730:41
        1 0x1518dd3f in char8_t const* DB::StringSearcher<true, true>::search<char8_t>(char8_t const*, unsigned long) const /bld/./src/Common/StringSearcher.h:751:16
        2 0x1518dd3f in OptimizedRegularExpressionImpl<false>::match(char const*, unsigned long, std::__1::vector<OptimizedRegularExpressionDetails::Match, std::__1::allocator<OptimizedRegularExpressionDetails::Match> >&, unsigned int) const /bld/./src/Common/OptimizedRegularExpression.cpp:463:54
        3 0x1811cb42 in DB::ExtractAllImpl::get(char const*&, char const*&) /bld/./src/Functions/FunctionsStringArray.h:588:18
        4 0x1811aa62 in DB::FunctionTokens<DB::ExtractAllImpl>::executeImpl(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const /bld/./src/Functions/FunctionsStringArray.h:704:30
        5 0x14fe17b4 in DB::IFunction::executeImplDryRun(std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long) const /bld/./src/Functions/IFunction.h:409:16

    Address 0x7fff577239a9 is located in stack of thread T0 at offset 201 in frame
        0 0x1518d98f in OptimizedRegularExpressionImpl<false>::match(char const*, unsigned long, std::__1::vector<OptimizedRegularExpressionDetails::Match, std::__1::allocator<OptimizedRegularExpressionDetails::Match> >&, unsigned int) const /bld/./src/Common/OptimizedRegularExpression.cpp:439

</details>

CI: https://s3.amazonaws.com/clickhouse-test-reports/39342/c6f7698f9ad6ae22199182ebf7c3b2dac77d69d8/fuzzer_astfuzzerasan,actions//report.html
Fixes: #37544 (cc @rschu1ze)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-20 10:18:44 +03:00
Roman Vasin
5f9c293963 Fix addDays() and addWeeks() in upper and lower limits of Date and Date32 2022-07-19 17:29:08 +00:00
Kseniia Sumarokova
891dbccdc7
Update LRUFileCache.cpp 2022-07-19 19:18:42 +02:00
Robert Schulze
02ed126533
Update src/Common/TraceSender.cpp 
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
 2022-07-19 18:44:31 +02:00