thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 03:12:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
97,850 Commits 9,326 Branches 1,783 Tags 2.1 GiB
02e08fa301
Commit Graph

31268 Commits

Author SHA1 Message Date
Igor Nikonov
d8acb0e7ad Tiny improvements along the code 
- less allocations (potentially)
 2022-09-13 12:06:56 +00:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze 
Fix SYSTEM UNFREEZE for ordinary database
 2022-09-13 14:55:11 +03:00
Alexander Tokmakov
2845494624
Merge pull request #41120 from azat/async-insert-queue-term 
Terminate AsynchronousInsertQueue w/o waiting for timeout
 2022-09-13 14:52:50 +03:00
Alexander Tokmakov
54001d3871
Revert "Fix uncaught DNS_ERROR on failed connection to replicas" 2022-09-13 13:51:09 +03:00
Vladimir C
cc7292df74
Merge pull request #41105 from ClickHouse/vdimir/upd_storage_rdb_sample_block 2022-09-13 12:37:12 +02:00
Kseniia Sumarokova
b2c9c04c7b
Merge pull request #40821 from kssenii/improve-marks-cache-loading 
Allow to load marks in threadpool in advance
 2022-09-13 12:31:58 +02:00
alesapin
9af591a328
Revert "Sharding s3 key names (2)" 2022-09-13 12:29:02 +02:00
Frank Chen
d3265150c0 Make sure span is finished in the onFinish callback 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 18:28:21 +08:00
Frank Chen
7303ae1796 Make sure span holder will be destructed only once 2022-09-13 18:27:29 +08:00
Kseniia Sumarokova
3dd28bb79a
Merge pull request #41042 from kssenii/refactor-cache-for-async-download 
Cache refactoring for async download
 2022-09-13 12:26:35 +02:00
Kseniia Sumarokova
5a76440078
Merge pull request #41232 from kssenii/fix-logical-error-in-write-through-cache 
Fix logical error in write-through cache found by stress test
 2022-09-13 12:25:21 +02:00
Kseniia Sumarokova
651d850652
Merge pull request #41230 from kssenii/minor-change 
minor change in MergeTreeDataPartCompact
 2022-09-13 12:23:18 +02:00
Kseniia Sumarokova
6655f1933a
Merge pull request #41233 from josemimbrero-tinybird/fix-filesystem_cache_log 
Fix String log entry in system.filesystem_cache_log
 2022-09-13 12:21:18 +02:00
alesapin
69d08b1007
Merge pull request #41075 from ClickHouse/keeper-fix-possible-segfault 
Fix possible segfault during Keeper shutdown
 2022-09-13 12:13:24 +02:00
Kruglov Pavel
7d7606a1e3
Merge pull request #41162 from Avogar/fix-http-redirect 
Support relative path in Location header after http redirect
 2022-09-13 12:10:44 +02:00
Antonio Andelic
a6b5ffec5d Polishing 2022-09-13 09:51:46 +00:00
zhenjial
5841d9e9b0 sync before destruct 2022-09-13 15:53:24 +08:00
zhenjial
67c08e3e22 sync before destruct 2022-09-13 15:06:22 +08:00
zhenjial
16c8cd0bd3 wait write finish 2022-09-13 14:19:40 +08:00
Robert Schulze
fac1be9700
chore: restore SYSTEM RELOAD MODEL(S) and moniting view SYSTEM.MODELS 
- This commit restores statements "SYSTEM RELOAD MODEL(S)" which provide
  a mechanism to update a model explicitly. It also saves potentially
  unnecessary reloads of a model from disk after it's initial load.

  To keep the complexity low, the semantics of "SYSTEM RELOAD MODEL(S)
  was changed from eager to lazy. This means that both statements
  previously immedately reloaded the specified/all models, whereas now
  the statements only trigger an unload and the first call to
  catboostEvaluate() does the actual load.

- Monitoring view SYSTEM.MODELS is also restored but with some obsolete
  fields removed. The view was not documented in the past and for now it
  remains undocumented. The commit is thus not considered a breach of
  ClickHouse's public interface.
 2022-09-12 19:33:02 +00:00
Kseniia Sumarokova
5d1e9822f8
Merge branch 'master' into fix-filesystem_cache_log 2022-09-12 20:32:42 +02:00
Antonio Andelic
583c530d0f
Merge pull request #41177 from azat/http-handlers-reload 
Apply changes to http handlers on fly without server restart
 2022-09-12 20:25:00 +02:00
Alexander Tokmakov
05b4303234
Merge pull request #41146 from ClickHouse/followup_39933 
Stop background operations before converting database from Ordinary
 2022-09-12 21:20:25 +03:00
Antonio Andelic
b2cc6a8cc6 Use promise/future 2022-09-12 18:19:41 +00:00
Kruglov Pavel
702ddff5f6
Fix style 2022-09-12 19:38:34 +02:00
avogar
8ac2fc7b26 Don't outout totals/extremes in all row formats, update docs 2022-09-12 17:21:40 +00:00
Jose
1880a745b2 Fix String in log entry 2022-09-12 18:53:54 +02:00
kssenii
a07f3d4bea Fix 2022-09-12 18:45:48 +02:00
Frank Chen
20191932df Fix style 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 00:41:05 +08:00
kssenii
06f80d78f8 Fix 2022-09-12 18:16:53 +02:00
Frank Chen
7e1f2901da Fix 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 00:06:17 +08:00
Frank Chen
7d6903bdc0 Fix 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 00:06:03 +08:00
Azat Khuzhin
b698a4ff65 Apply changes to http handlers on fly without server restart 
This has been implemented by simply restarting http servers in case of
http_handlers directive in configuration xml had been changed.

But, for this I have to change the handlers interface to accept
configuration separatelly, since the configuration that contains in the
server is the configuration with which server had been started.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Co-authored-by: Antonio Andelic
 2022-09-12 17:34:51 +02:00
vdimir
ba506043ce
Fix exponential query rewrite in case of lots of cross joins with where 2022-09-12 15:05:34 +00:00
Alexander Tokmakov
da4fbb552d
Merge branch 'master' into fix_parts_removal 2022-09-12 17:49:29 +03:00
Antonio Andelic
deee0d639f Address PR comments 2022-09-12 14:43:52 +00:00
Alexey Milovidov
2aedd41023
Remove strange code (#40195) 
* Remove strange code

* Even more code removal

* Fix style

* Remove even more code

* Simplify code by making it slower

* Attempt to do something

* Attempt to do something

* Well do something with this horrible trash

* Add a test
 2022-09-12 16:29:23 +02:00
Frank Chen
ebaa24ecae Fix flaky tests 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-12 22:15:30 +08:00
Azat Khuzhin
5cb37eb1a0 Terminate AsynchronousInsertQueue w/o waiting for timeout 
Previously it simply sleeps for async_insert_busy_timeout_ms delay on
shutdown, which is not great, and produces false-positive hung checks on
CI [1].

  [1]: https://pastila.nl/?0087a100/db80915eac58e67eb05516bf459e5510

Refs: #29390 (cc @tavplubix, @CurtizJ)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-12 16:11:54 +02:00
kssenii
413fbb6507 Self-review fixes 2022-09-12 15:46:11 +02:00
Antonio Andelic
1cd43d8e5c Merge branch 'master' into keeper-storage 2022-09-12 13:42:51 +00:00
Anton Popov
76af7efba9
Merge branch 'master' into optimize_compact_part_mutation 2022-09-12 15:35:47 +02:00
Dmitry Novik
c236706703
Merge branch 'master' into grouping-comp 2022-09-12 15:19:51 +02:00
Antonio Andelic
c2dfabe51c Use shared_ptr for Event 2022-09-12 12:25:39 +00:00
Antonio Andelic
e985ee3354 Close sessions on Keeper shutdown 2022-09-12 12:22:48 +00:00
Alexander Tokmakov
ed3c95c28f review fix 2022-09-12 13:53:43 +02:00
Kseniia Sumarokova
df22e8d30c
Merge pull request #41190 from kssenii/fix-clean-up-thread 
Fix cleaning up broken detached part after timeout
 2022-09-12 13:21:45 +02:00
kssenii
88b589917c Update defaults 2022-09-12 12:48:47 +02:00
Kruglov Pavel
6535301888
Merge branch 'master' into Avogar-patch-1 2022-09-12 12:23:28 +02:00
Alexander Tokmakov
7f462a8762
Update src/Storages/StorageReplicatedMergeTree.cpp 2022-09-12 13:22:08 +03:00
Kseniia Sumarokova
d0c058f273
Merge pull request #41132 from Algunenano/views_log_window 
Fix query_views_log with Window views
 2022-09-12 11:56:22 +02:00
Alexey Milovidov
9c5f9f1815
Merge pull request #41158 from den-crane/bug/trivial_count_optimization_with_array_join 
Fix: trivial count optimization with array join. revert revert #39466
 2022-09-12 08:56:10 +03:00
Alexey Milovidov
94d6e9ce7d
Merge pull request #41189 from kssenii/fix-positional-args 
Fix for positional arguments
 2022-09-12 03:21:39 +03:00
Azat Khuzhin
7e130aeb69 Add support for DWARF-5 (without emitting them in binaries) 
ClickHouse changes to the folly parser:
- use camel_case
- add NOLINT
- avoid using folly:: (use std:: instead)
- avoid using boost:: (use std:: instead)

But note, now it has not been enabled by default (like it was
initially), because you may need recent debugger to support DWARF-5
correctly, and to make debugging easier, let's do this later.

A good example is gdb 10, even though it looks like it should support
it, it still produce some errors, like here [1]:

    Dwarf Error: DW_FORM_strx1 found in non-DWO CU [in module /usr/bin/clickhouse]

  [1]: https://github.com/ClickHouse/ClickHouse/pull/40772#issuecomment-1236331323

And not only it complains, apparently it can "activate" SDT probes
(replace "nop" with "int3"), and I believe this is what happens here
[2].

  [2]: https://github.com/ClickHouse/ClickHouse/pull/41063#issuecomment-1242992314

There you got int3 in the case when ClickHouse got SIGTRAP:

<details>

```
    0x7f494705e093 <+1139>: jne    0x7f494705e450            ; <+2096> [inlined] update_tls_slotinfo at dl-open.c:732
    0x7f494705e099 <+1145>: testl  %r13d, %r13d
    0x7f494705e09c <+1148>: je     0x7f494705e09f            ; <+1151> at dl-open.c:744:6
    0x7f494705e09e <+1150>: int3
->  0x7f494705e09f <+1151>: movl   -0x54(%rbp), %eax
    0x7f494705e0a2 <+1154>: testl  %eax, %eax
    0x7f494705e0a4 <+1156>: jne    0x7f494705e410            ; <+2032> at dl-open.c:745:5

But if I repeat the query it does not:

    0x7ffff7fe5093 <+1139>: jne    0x7ffff7fe5450            ; <+2096> [inlined] update_tls_slotinfo at dl-open.c:732
    0x7ffff7fe5099 <+1145>: testl  %r13d, %r13d
    0x7ffff7fe509c <+1148>: je     0x7ffff7fe509f            ; <+1151> at dl-open.c:744:6
    0x7ffff7fe509e <+1150>: nop
->  0x7ffff7fe509f <+1151>: movl   -0x54(%rbp), %eax
    0x7ffff7fe50a2 <+1154>: testl  %eax, %eax
    0x7ffff7fe50a4 <+1156>: jne    0x7ffff7fe5410            ; <+2032> at dl-open.c:745:5
```

</details>

Test command was:

    clickhouse local --stacktrace -q "select * from file('data.capnp', 'CapnProto', 'val1 char') settings format_schema='nonexist:Message'

*P.S. I did this, because I have libraries compiled with DWARF5 (i.e. glibc), and dwarf parser simply fails on my dev env.*

Refs: 490b287ca3
(cherry picked from commit ee5696bb32)
(cherry picked from commit e03870bc8b)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-11 21:06:14 +02:00
Alexey Milovidov
8b5328f7e5 Merge branch 'master' into fix-trash-base58 2022-09-11 20:43:54 +02:00
Alexey Milovidov
3e6c4e3f16 Fix clang-tidy 2022-09-11 20:43:36 +02:00
Alexey Milovidov
9a0892c40c
Merge pull request #41171 from ClickHouse/fix-half-of-trash 
Fix half of trash
 2022-09-11 21:40:22 +03:00
Alexey Milovidov
bd40d0de7d
Merge pull request #41187 from ClickHouse/faster-build 
Faster build
 2022-09-11 21:39:36 +03:00
Denny Crane
0e4e868f29
Merge branch 'master' into bug/trivial_count_optimization_with_array_join 2022-09-11 15:23:10 -03:00
kssenii
9c2bbcf4a6 Fix 2022-09-11 17:38:51 +02:00
kssenii
02c3d8b0a0 Fix 2022-09-11 16:42:56 +02:00
alesapin
6eead4ea1e Fix fasttest build 2022-09-11 15:16:54 +02:00
kssenii
2f3bfc5411 Fix 2022-09-11 13:54:25 +02:00
alesapin
21ab379026 Fix comment 2022-09-11 13:47:04 +02:00
alesapin
ac53830000 Fix hung check 2022-09-11 13:37:39 +02:00
kssenii
b7d751b782 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-11 13:23:30 +02:00
kssenii
37bdba98b1 Fix 2022-09-11 12:58:04 +02:00
Alexey Milovidov
7f5a1cbe16 Faster build 2022-09-11 09:29:24 +02:00
Alexey Milovidov
5802c2fdd2
Merge pull request #40713 from ClickHouse/remove-useless-line 
Remove one line from XML, because we do not care
 2022-09-11 09:53:30 +03:00
Alexey Milovidov
5592c8e775 Faster build 2022-09-11 08:50:44 +02:00
Alexey Milovidov
88e9178830 Faster build 2022-09-11 08:46:05 +02:00
Alexey Milovidov
0896e6b9b6
Merge pull request #41183 from ClickHouse/fix-parallel-hash-join-for-floats 
Better parallel hash JOIN for floats
 2022-09-11 09:13:57 +03:00
Alexey Milovidov
e0a9ae0496 Fix base58 trash 2022-09-11 08:09:14 +02:00
Alexey Milovidov
3d8a2130cf Continue fixing the trash 2022-09-11 06:31:15 +02:00
Alexey Milovidov
9fbd361a2b Merge branch 'master' into fix-trash-base58 2022-09-11 06:23:01 +02:00
Alexey Milovidov
7f1e7b5967 Merge branch 'master' into fix-half-of-trash 2022-09-11 06:20:47 +02:00
Alexey Milovidov
05d0f48d17
Merge pull request #41173 from ClickHouse/remove-some-methods 
Remove some methods
 2022-09-11 07:19:52 +03:00
Alexey Milovidov
90fd507292 Fix test 2022-09-11 06:18:43 +02:00
Alexey Milovidov
512fb998f1 Fix ridiculous trash in Base58 2022-09-11 03:54:40 +02:00
Alexey Milovidov
cc38ac3764 Better parallel hash JOIN for floats 2022-09-11 03:04:44 +02:00
Alexey Milovidov
ae6c74a916 Merge branch 'master' into fix-half-of-trash 2022-09-11 02:30:56 +02:00
Alexey Milovidov
6f513dab6f
Merge pull request #41169 from ClickHouse/remove-trash-4 
Fix strange code in DatabaseReplicated
 2022-09-11 03:30:08 +03:00
Alexey Milovidov
2b6724c5e3 Fix example 2022-09-11 01:58:55 +02:00
Alexey Milovidov
b75d56ad9b Merge branch 'master' into remove-some-methods 2022-09-11 01:55:02 +02:00
Alexey Milovidov
91338ea771
Merge branch 'master' into remove-useless-line 2022-09-11 02:03:39 +03:00
Azat Khuzhin
c1e70169d2 Suppress clang-analyzer-cplusplus.NewDelete in MsgPackRowInputFormat 
Appartently there is some issue with clang-15, since even the following
example shows error [1].

  [1]: https://gist.github.com/azat/027f0e949ea836fc2e6269113ceb8752

clang-tidy report [1]:

    FAILED: src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o                                                                                                            /usr/bin/cmake -E __run_co_compile --launcher="prlimit;--as=10000000000;--data=5000000000;--cpu=1000;/usr/bin/ccache" --tidy=/usr/bin/clang-tidy-15 --source=/ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp -- /usr/bin/clang++-15 --target=x86_64-linux-gnu --sysroot=/ch/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc  -DAWS_SDK_VERSION_MAJOR=1 -DAWS_SDK_VERSION_MINOR=7 -DAWS_SDK_VERSION_PATCH=231 -DBOOST_ASIO_HAS_STD_INVOKE_RESULT=1 -DBOOST_ASIO_STANDALONE=1 -DCARES_STATICLIB -DCONFIGDIR=\"\" -DENABLE_MULTITARGET_CODE=1 -DENABLE_OPENSSL_ENCRYPTION -DHAS_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DLIBSASL_EXPORTS=1 -DLZ4_DISABLE_DEPRECATE_WARNINGS=1 -DOBSOLETE_CRAM_ATTR=1 -DOBSOLETE_DIGEST_ATTR=1 -DPLUGINDIR=\"\" -DPOCO_ENABLE_CPP11 -DPOCO_HAVE_FD_EPOLL -DPOCO_OS_FAMILY_UNIX -DSASLAUTHD_CONF_FILE_DEFAULT=\"\" -DSNAPPY_CODEC_AVAILABLE -DSTD_EXCEPTION_HAS_STACK_TRACE=1 -DUNALIGNED_OK -DWITH_COVERAGE=0 -DWITH_GZFILEOP -DX86_64 -DZLIB_COMPAT -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -Iincludes/configs -I/ch/src -Isrc -Isrc/Core/include -I/ch/base/glibc-compatibility/memcpy -I/ch/base/base/.. -Ibase/base/.. -I/ch/contrib/cctz/include -I/ch/base/pcg-random/. -I/ch/contrib/miniselect/include -I/ch/contrib/zstd/lib -Icontrib/cyrus-sasl-cmake -I/ch/contrib/lz4/lib -I/ch/src/Common/mysqlxx/. -Icontrib/c-ares -I/ch/contrib/c-ares -I/ch/contrib/c-ares/include -isystem /ch/contrib/libcxx/include -isystem /ch/contrib/libcxxabi/include -isystem /ch/contrib/libunwind/include -isystem /ch/contrib/libdivide/. -isystem /ch/contrib/jemalloc-cmake/include -isystem /ch/contrib/llvm/llvm/include -isystem contrib/llvm/llvm/include -isystem /ch/contrib/abseil-cpp -isystem /ch/contrib/croaring/cpp -isystem /ch/contrib/croaring/include -isystem /ch/contrib/cityhash102/include -isystem /ch/contrib/boost -isystem /ch/contrib/poco/Net/include -isystem /ch/contrib/poco/Foundation/include -isystem /ch/contrib/poco/NetSSL_OpenSSL/include -isystem /ch/contrib/poco/Crypto/include -isystem /ch/contrib/boringssl/include -isystem /ch/contrib/poco/Util/include -isystem /ch/contrib/poco/JSON/include -isystem /ch/contrib/poco/XML/include -isystem /ch/contrib/replxx/include -isystem /ch/contrib/fmtlib-cmake/../fmtlib/include -isystem /ch/contrib/magic_enum/include -isystem /ch/contrib/double-conversion -isystem /ch/contrib/dragonbox/include -isystem /ch/contrib/re2 -isystem contrib/re2-cmake -isystem /ch/contrib/zlib-ng -isystem contrib/zlib-ng-cmake -isystem /ch/contrib/pdqsort -isystem /ch/contrib/xz/src/liblzma/api -isystem /ch/contrib/aws-c-common/include -isystem /ch/contrib/aws-c-event-stream/include -isystem /ch/contrib/aws/aws-cpp-sdk-s3/include -isystem /ch/contrib/aws/aws-cpp-sdk-core/include -isystem contrib/aws-s3-cmake/include -isystem /ch/contrib/snappy -isystem contrib/snappy-cmake -isystem /ch/contrib/msgpack-c/include -isystem /ch/contrib/fast_float/include -isystem /ch/contrib/librdkafka-cmake/include -isystem /ch/contrib/librdkafka/src -isystem contrib/librdkafka-cmake/auxdir -isystem /ch/contrib/cppkafka/include -isystem /ch/contrib/nats-io/src -isystem /ch/contrib/nats-io/src/adapters -isystem /ch/contrib/nats-io/src/include -isystem /ch/contrib/nats-io/src/unix -isystem /ch/contrib/libuv/include -isystem /ch/contrib/krb5/src/include -isystem contrib/krb5-cmake/include -isystem /ch/contrib/NuRaft/include -isystem /ch/contrib/poco/MongoDB/include -isystem contrib/mariadb-connector-c-cmake/include-public -isystem /ch/contrib/mariadb-connector-c/include -isystem /ch/contrib/mariadb-connector-c/libmariadb -isystem /ch/contrib/icu/icu4c/source/i18n -isystem /ch/contrib/icu/icu4c/source/common -isystem /ch/contrib/capnproto/c++/src -isystem /ch/contrib/arrow/cpp/src -isystem /ch/contrib/arrow-cmake/cpp/src -isystem contrib/arrow-cmake/cpp/src -isystem contrib/arrow-cmake/../orc/c++/include -isystem /ch/contrib/orc/c++/include -isystem contrib/avro-cmake/include -isystem /ch/contrib/avro/lang/c++/api -isystem /ch/contrib/openldap-cmake/linux_x86_64/include -isystem /ch/contrib/openldap/include -isystem /ch/contrib/sparsehash-c11 -isystem /ch/contrib/protobuf/src -isystem src/Server/grpc_protos -isystem /ch/contrib/grpc/include -isystem /ch/contrib/libhdfs3/include -isystem /ch/contrib/hive-metastore -isystem /ch/contrib/thrift/lib/cpp/src -isystem contrib/thrift-cmake -isystem /ch/contrib/azure/sdk/core/azure-core/inc-isystem /ch/contrib/azure/sdk/identity/azure-identity/inc -isystem /ch/contrib/azure/sdk/storage/azure-storage-common/inc -isystem /ch/contrib/azure/sdk/storage/azure-storage-blobs/inc -isystem /ch/contrib/s2geometry/src -isystem /ch/contrib/AMQP-CPP/include -isystem /ch/contrib/AMQP-CPP -isystem /ch/contrib/sqlite-amalgamation -isystem /ch/contrib/rocksdb/include -isystem /ch/contrib/libpqxx/include -isystem /ch/contrib/libpq -isystem /ch/contrib/libpq/include -isystem /ch/contrib/libstemmer_c/include -isystem /ch/contrib/wordnet-blast -isystem /ch/contrib/lemmagen-c/include -isystem /ch/contrib/simdjson/include -isystem /ch/contrib/rapidjson/include -isystem /ch/contrib/consistent-hashing --gcc-toolchain=/ch/cmake/linux/../../contrib/sysroot/linux-x86_64 -std=c++20 -fdiagnostics-color=always -Xclang -fuse-ctor-homing -fsized-deallocation  -UNDEBUG -gdwarf-aranges -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -fasynchronous-unwind-tables -falign-functions=32 -mbranches-within-32B-boundaries -fdiagnostics-absolute-paths -fstrict-vtable-pointers -fexperimental-new-pass-manager -Wall -Wextra -Weverything -Wpedantic -Wno-zero -length-array -Wno-c++98-compat-pedantic -Wno-c++98-compat -Wno-c++20-compat -Wno-conversion -Wno-ctad-maybe-unsupported -Wno-disabled-macro-expansion -Wno-documentation-unknown-command -Wno-double-promotion -Wno-exit-time-destructors -Wno-float-equal -Wno-global-constructors -Wno-missing-prototypes -Wno-missing-variable-declarations -Wno-padded -Wno-switch-enum -Wno-undefined-func-template -Wno-unused-template -Wno-vla -Wno-weak-template-vtables -Wno-weak-vtables -Wno-thread-safety-negative -g -O0 -g -gdwarf-4 -fno-inline  -D_LIBCPP_DEBUG=0   -D OS_LINUX -I/ch/base -I/ch/contrib/magic_enum/include -include /ch/src/Core/iostream_debug_helpers.h -Werror -nostdinc++ -std=gnu++2a -MD -MT src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o -MF src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o.d -o src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o -c /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp

    /ch/contrib/msgpack-c/include/msgpack/v1/detail/cpp11_zone.hpp:195:9: error: Attempt to free released memory [clang-analyzer-cplusplus.NewDelete,-warnings-as-errors]
            ::free(p);
            ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:509:5: note: Taking false branch
        if (buf.eof())
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:514:24: note: Assuming 'i' is not equal to field 'number_of_columns'
        for (size_t i = 0; i != number_of_columns; ++i)
                           ^~~~~~~~~~~~~~~~~~~~~~
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:514:5: note: Loop condition is true.  Entering loop body
        for (size_t i = 0; i != number_of_columns; ++i)
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:516:30: note: Calling 'MsgPackSchemaReader::readObject'
            auto object_handle = readObject();
                                 ^~~~~~~~~~~~
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:426:5: note: Taking false branch
        if (buf.eof())
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:433:5: note: Loop condition is true.  Entering loop body
        while (need_more_data)
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:438:29: note: Calling 'unpack'
                object_handle = msgpack::unpack(buf.position(), buf.buffer().end() - buf.position(), offset);
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:52:12: note: Calling 'unpack'
        return msgpack::v3::unpack(data, len, off, referenced, f, user_data, limit);
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:35:5: note: Control jumps to the 'default' case at line 40
        switch(ret) {
        ^
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:41:9: note:  Execution continues on line 43
            break;
            ^
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:43:35: note: Calling '~unique_ptr'
        return msgpack::object_handle();
                                      ^
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:269:19: note: Calling 'unique_ptr::reset'
      ~unique_ptr() { reset(); }
                      ^~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:314:9: note: '__tmp' is non-null
        if (__tmp)
            ^~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:314:5: note: Taking true branch
        if (__tmp)
        ^
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:315:7: note: Calling 'default_delete::operator()'
          __ptr_.second()(__tmp);
          ^~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:54:5: note: Memory is released
        delete __ptr;
        ^~~~~~~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:54:5: note: Calling 'zone::operator delete'
        delete __ptr;
        ^~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v1/detail/cpp11_zone.hpp:195:9: note: Attempt to free released memory
            ::free(p);
            ^~~~~~~~~

  [1]: https://s3.amazonaws.com/clickhouse-builds/41046/9677898b3b234a5ba0371edaf719ea8890d084ff/binary_tidy/build_log.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
4e3135383b Fix clang-tidy warnings (from clang-15) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
ba4174402a Temporary disable complation of compileAddIntoAggregateStatesFunctions 
Fixes the following use-of-uninitialized-value in llvm [1]:

    ==696==WARNING: MemorySanitizer: use-of-uninitialized-value
        0 0x498141d9 in llvm::ilist_traits<llvm::MachineInstr>::removeNodeFromList(llvm::MachineInstr*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:155:24
        1 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::remove(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>&) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:253:11
        2 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:22
        3 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>, llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:305:15
        4 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::clear() build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:309:18
        5 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::~iplist_impl() build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:210:20
        6 0x498145ff in llvm::MachineBasicBlock::~MachineBasicBlock() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:56:1
        7 0x49969c1f in llvm::MachineFunction::DeleteMachineBasicBlock(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:426:8
        8 0x49969c1f in llvm::ilist_alloc_traits<llvm::MachineBasicBlock>::deleteNode(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:127:21
        9 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineBasicBlock, false, false, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:11
        10 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:272:39
        11 0x4983f4d6 in llvm::MachineFunction::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/CodeGen/MachineFunction.h:767:53
        12 0x4983f4d6 in llvm::MachineBasicBlock::eraseFromParent() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:1317:16
        13 0x4a0c9a27 in llvm::TailDuplicator::removeDeadBlock(llvm::MachineBasicBlock*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:1051:8
        14 0x4a0c1e41 in llvm::TailDuplicator::tailDuplicateAndUpdate(bool, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:189:5
        15 0x4a0ca16e in llvm::TailDuplicator::tailDuplicateBlocks() build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:288:19
        16 0x4a0be9f9 in (anonymous namespace)::TailDuplicateBase::runOnMachineFunction(llvm::MachineFunction&) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplication.cpp:98:21
        17 0x499a2777 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunctionPass.cpp:72:13
        18 0x4dbba34d in llvm::FPPassManager::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1435:27
        19 0x4dbe3761 in llvm::FPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1481:16
        20 0x4dbbebbb in (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1550:27
        21 0x4dbbebbb in llvm::legacy::PassManagerImpl::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:541:44
        22 0x4dbe454b in llvm::legacy::PassManager::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1677:14
        23 0x414405df in DB::JITCompiler::compile(llvm::Module&) build_docker/../src/Interpreters/JIT/CHJIT.cpp:78:22
        24 0x4143bb7d in DB::CHJIT::compileModule(std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module>>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:378:29
        25 0x4143aded in DB::CHJIT::compileModule(std::__1::function<void (llvm::Module&)>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:359:24
        26 0x4147b25e in DB::compileAggregateFunctions(DB::CHJIT&, std::__1::vector<DB::AggregateFunctionWithOffset, std::__1::allocator<DB::AggregateFunctionWithOffset>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) build_docker/../src/Interpreters/JIT/compileFunction.cpp:738:32
        27 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:661:49
        28 0x3de0a23a in std::__1::pair<std::__1::shared_ptr<DB::CompiledExpressionCacheEntry>, bool> DB::CacheBase<wide::integer<128ul, unsigned int>, DB::CompiledExpressionCacheEntry, UInt128Hash, DB::CompiledFunctionWeightFunction>::getOrSet<DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0>(wide::integer<128ul, unsigned int> const&, DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0&&) build_docker/../src/Common/CacheBase.h:125:24
        29 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded() build_docker/../src/Interpreters/Aggregator.cpp:657:70

      Memory was marked as uninitialized
        0 0xb988ded in __sanitizer_dtor_callback (/usr/bin/clickhouse+0xb988ded) (BuildId: c4a880b742797a1f37bc4f5ed869f055cc86486b)
        1 0x498145da in llvm::MachineBasicBlock::~MachineBasicBlock() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:56:1
        2 0x49969c1f in llvm::MachineFunction::DeleteMachineBasicBlock(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:426:8
        3 0x49969c1f in llvm::ilist_alloc_traits<llvm::MachineBasicBlock>::deleteNode(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:127:21
        4 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineBasicBlock, false, false, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:11
        5 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:272:39
        6 0x4983f4d6 in llvm::MachineFunction::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/CodeGen/MachineFunction.h:767:53
        7 0x4983f4d6 in llvm::MachineBasicBlock::eraseFromParent() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:1317:16
        8 0x4a0c9a27 in llvm::TailDuplicator::removeDeadBlock(llvm::MachineBasicBlock*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:1051:8
        9 0x4a0c1e41 in llvm::TailDuplicator::tailDuplicateAndUpdate(bool, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:189:5
        10 0x4a0ca16e in llvm::TailDuplicator::tailDuplicateBlocks() build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:288:19
        11 0x4a0be9f9 in (anonymous namespace)::TailDuplicateBase::runOnMachineFunction(llvm::MachineFunction&) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplication.cpp:98:21
        12 0x499a2777 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunctionPass.cpp:72:13
        13 0x4dbba34d in llvm::FPPassManager::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1435:27
        14 0x4dbe3761 in llvm::FPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1481:16
        15 0x4dbbebbb in (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1550:27
        16 0x4dbbebbb in llvm::legacy::PassManagerImpl::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:541:44
        17 0x4dbe454b in llvm::legacy::PassManager::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1677:14
        18 0x414405df in DB::JITCompiler::compile(llvm::Module&) build_docker/../src/Interpreters/JIT/CHJIT.cpp:78:22
        19 0x4143bb7d in DB::CHJIT::compileModule(std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module>>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:378:29
        20 0x4143aded in DB::CHJIT::compileModule(std::__1::function<void (llvm::Module&)>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:359:24
        21 0x4147b25e in DB::compileAggregateFunctions(DB::CHJIT&, std::__1::vector<DB::AggregateFunctionWithOffset, std::__1::allocator<DB::AggregateFunctionWithOffset>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) build_docker/../src/Interpreters/JIT/compileFunction.cpp:738:32
        22 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:661:49
        23 0x3de0a23a in std::__1::pair<std::__1::shared_ptr<DB::CompiledExpressionCacheEntry>, bool> DB::CacheBase<wide::integer<128ul, unsigned int>, DB::CompiledExpressionCacheEntry, UInt128Hash, DB::CompiledFunctionWeightFunction>::getOrSet<DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0>(wide::integer<128ul, unsigned int> const&, DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0&&) build_docker/../src/Common/CacheBase.h:125:24
        24 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded() build_docker/../src/Interpreters/Aggregator.cpp:657:70

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/490a2c75610c4bc3191d55226f8454b3c3d3919a/stateful_tests__msan_.html

Note, that it is safe to do, but only for this method, since it had been
disabled anyway, back in #27574, and I guess this MSan report may be
related.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
e0a8f19cfb Fix -Wzero-as-null-pointer-constant in MemoryStatisticsOS::get for FreeBSD 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
e8d7403a38 Suppress warning in FormatFactory::getFormatFromFileDescriptor() for FreeBSD 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Alexey Milovidov
871ee15a9c
Merge pull request #40953 from ClibMouse/sensitive_data_masker_unittest_issue 
Rearrange sensitive data masker unit test
 2022-09-10 20:03:30 +03:00
Alexey Milovidov
89bae332a0
Merge pull request #41170 from ClickHouse/fix-weird-lowcardinality 
Fix something terribly wrong in LowCardinality
 2022-09-10 18:08:35 +03:00
Kruglov Pavel
2ca5846595
Merge pull request #41159 from azat/load-balancing-fix 
Fix stack-use-after-return in GetPriorityForLoadBalancing::getPriorityFunc()
 2022-09-10 15:11:31 +02:00
Alexey Milovidov
61278c81e4 Merge branch 'master' into remove-useless-line 2022-09-10 05:07:10 +02:00
Alexey Milovidov
fd235919aa Remove some methods 2022-09-10 05:04:40 +02:00
Suzy Wang
d60340eb40
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-09 22:59:50 -04:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Alexey Milovidov
daff201566 Fix something terribly wrong in LowCardinality 2022-09-10 03:28:37 +02:00
Alexey Milovidov
4301b362ab Fix strange code in DatabaseReplicated 2022-09-10 02:34:37 +02:00
Igor Nikonov
726639484a
Revert "Query plan optimization setting: read in window order" 2022-09-09 22:15:38 +02:00
avogar
d8cbd51333 Support relative path in Location header after http redirect 2022-09-09 19:26:14 +00:00
Azat Khuzhin
8c5583d7a5 Fix stack-use-after-return in GetPriorityForLoadBalancing::getPriorityFunc() 
clang-15 reports [1]:

<details>

<summary>ASan report</summary>

```
    ==1==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f1d04c4eb20 at pc 0x000031c4803c bp 0x7f1d05e19a00 sp 0x7f1d05e199f8
    READ of size 8 at 0x7f1d04c4eb20 thread T200 (QueryPullPipeEx)
        #0 0x31c4803b in DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3::operator()(unsigned long) const build_docker/../src/Common/GetPriorityForLoadBalancing.cpp:42:40
        #1 0x31c4803b in decltype(static_cast<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&>(fp)(static_cast<unsigned long>(fp0))) std::__1::__invoke<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/type_traits:3640:23
        #2 0x31c4803b in unsigned long std::__1::__invoke_void_return_wrapper<unsigned long, false>::__call<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/__functional/invoke.h:30:16
        #3 0x31c4803b in std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>::operator()(unsigned long&&) build_docker/../contrib/libcxx/include/__functional/function.h:230:12
        #4 0x31c4803b in unsigned long std::__1::__function::__policy_invoker<unsigned long (unsigned long)>::__call_impl<std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>>(std::__1::__function::__policy_storage const*, unsigned long) build_docker/../contrib/libcxx/include/__functional/function.h:711:16
        #5 0x31c38b07 in std::__1::__function::__policy_func<unsigned long (unsigned long)>::operator()(unsigned long&&) const build_docker/../contrib/libcxx/include/__functional/function.h:843:16
        #6 0x31c38b07 in std::__1::function<unsigned long (unsigned long)>::operator()(unsigned long) const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12
        #7 0x31c38b07 in PoolWithFailoverBase<DB::IConnectionPool>::getShuffledPools(unsigned long, std::__1::function<unsigned long (unsigned long)> const&) build_docker/../src/Common/PoolWithFailoverBase.h:174:39

      This frame has 2 object(s):
        [32, 40) 'pool_size.addr' <== Memory access at offset 32 is inside this variable
        [64, 88) 'ref.tmp' (line 18)
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/integration_tests__asan__[1/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-09 20:06:48 +02:00
Denny Crane
be90cecbd2 Revert "Revert "Fix trivial count optimization with array join"" 
This reverts commit 17de7b2876.
 2022-09-09 14:56:38 -03:00
Azat Khuzhin
763bb18f98 Fix SIGSEGV in SortedBlocksWriter in case of empty block 
CI found one issue [1].

Here is the stack trace for invalid read:

<details>

<summary>stack trace</summary>

```
    0: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long(this="") const at string:1445:22
    1: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(this="", __str="") at string:1927
    2: DB::TemporaryFileLazySource::TemporaryFileLazySource(this=0x00007f3aec105f58, path_="", header_=0x00007f38ffd93b40) at TemporaryFileLazySource.cpp:11
    3: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] DB::TemporaryFileLazySource* std::__1::construct_at<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, DB::TemporaryFileLazySource*>(__args=0x00007f38ffd91560) at construct_at.h:38:50
    4: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] void std::__1::allocator_traits<std::__1::allocator<DB::TemporaryFileLazySource> >::construct<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void, void>(__args=0x00007f38ffd91560) at allocator_traits.h:298
    5: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::__shared_ptr_emplace<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block>(this=0x00007f3aec105f40, __args=0x00007f38ffd91560) at shared_ptr.h:293
    6: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::allocate_shared<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:954
    7: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::make_shared<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:963
    8: DB::SortedBlocksWriter::streamFromFile(this=<unavailable>, file=<unavailable>) const at SortedBlocksWriter.cpp:238
    9: DB::SortedBlocksWriter::premerge(this=<unavailable>) at SortedBlocksWriter.cpp:209:32
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/stress_test__asan_.html

So the problem here is that there was empty unique_ptr<> reference to
temporary file, because of empty block that accepted by
SortedBlocksWriter::insert(), but insert() is not a problem the problem
is premerge() that steals blocks from insert() and do not have check
that there are some rows. However this check exists in
SortedBlocksWriter::flush(), and in that case temporary file is not
created.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-09 19:52:38 +02:00
Azat Khuzhin
57146c9361 Fix typos in SortedBlocksWriter 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-09 19:52:38 +02:00
Anton Popov
a1b5582a7c
Merge pull request #37152 from CurtizJ/fix-filling-of-missed-nested 
Fix filling of missed Nested columns with multiple levels
 2022-09-09 19:29:01 +02:00
Alexander Tokmakov
9bcec2d49a stop background ops before converting db engine 2022-09-09 18:27:19 +02:00
zhenjial
51badf3d0a fix 2022-09-10 00:08:50 +08:00
Dmitry Novik
64c4023737
Merge branch 'master' into grouping-comp 2022-09-09 18:05:01 +02:00
Dmitry Novik
877776e569
Update src/Functions/grouping.h 2022-09-09 18:04:29 +02:00
Kruglov Pavel
c33aa54032
Fix 2022-09-09 17:53:26 +02:00
zhenjial
bd9fabc3f7 code optimization, add test 2022-09-09 23:27:42 +08:00
Nikolay Degterinsky
12e7cd6b19
Merge pull request #41068 from jthmath/bugfix 
fix ignore projection logic error
 2022-09-09 17:10:03 +02:00
alesapin
86a3719bef
Merge pull request #41128 from ClickHouse/less_errors 
Fix bad warnings in part fetches
 2022-09-09 16:58:09 +02:00
Roman Vasin
3aa33f39ac Fix ErrorCodes for style check 2022-09-09 14:17:46 +00:00
Roman Vasin
da3e938ad6 Enable Date32 results for toStartOfWeek 2022-09-09 13:45:51 +00:00
Roman Vasin
b03a0a724f Enable Date32 results for toStartOfYear, toStartOfISOYear, toStartOfQuarter, toStartOfMonth, toMonday, toLastDayOfMonth 2022-09-09 13:25:52 +00:00
Alexander Tokmakov
98509e62e8
Update src/Storages/Freeze.cpp 2022-09-09 16:20:24 +03:00
Kseniia Sumarokova
5bb40e5856
Change cache setting do_not_evict_index_and_mark_files default to 0. (#41139) 2022-09-09 15:04:32 +02:00
Antonio Andelic
32483aeec8 Merge branch 'master' into keeper-storage 2022-09-09 12:24:01 +00:00
kssenii
5cba9b789c Merge remote-tracking branch 'upstream/master' into refactor-cache-for-async-download 2022-09-09 14:14:04 +02:00
Igor Nikonov
5379f37331
Merge pull request #41015 from ClickHouse/window_func_optimization_setting 
Query plan optimization setting: read in window order
 2022-09-09 14:00:41 +02:00
Raúl Marín
b33c1fa60f Fix query_views_log with Window views 2022-09-09 13:46:53 +02:00
alesapin
01e745bbbf Fix parts removal 2022-09-09 13:26:13 +02:00
Alexander Tokmakov
48927ba0ac
Merge branch 'master' into no-hardlinks-while-making-backup-of-mergetree-in-atomic-db 2022-09-09 14:24:44 +03:00
alesapin
23d95e6723 Fix parts removal in case of failures 2022-09-09 13:24:22 +02:00
kssenii
c2c35278ba Refactor cache for async download 2022-09-09 13:12:47 +02:00
alesapin
e3fc78aa5f Fix bad warnings in part fetches 2022-09-09 12:14:42 +02:00
Antonio Andelic
8ae7fb14b3
Merge pull request #40918 from ClickHouse/fix-keeper-system-path-check 
Fix check for Keeper system path modification
 2022-09-09 09:12:49 +02:00
lthaooo
d4dd6b05b2 optimize compact part mutation 2022-09-09 10:22:35 +08:00
Dmitry Novik
323fdeff0b
Merge branch 'master' into fix-odr-vialation 2022-09-09 00:42:14 +02:00
Alexey Milovidov
af8ed9c166
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-09 00:05:53 +03:00
Dmitry Novik
29b5d023ee Replace WITH_TEXT_LOG by WITHOUT_TEXT_LOG 2022-09-08 19:16:37 +02:00
zhenjial
469ceaa156 code optimization 2022-09-09 00:47:43 +08:00
Vitaly Baranov
9acc73d811
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2022-09-08 17:51:29 +02:00
alesapin
62c911924b
Merge pull request #41108 from ClickHouse/more_logging 
More logging for S3
 2022-09-08 17:10:55 +02:00
alesapin
5e00124f8a More logging for S3 2022-09-08 16:18:21 +02:00
Antonio Andelic
3502718a2c Less complications 2022-09-08 13:52:14 +00:00
Antonio Andelic
b6f8e06708
Typo fix 2022-09-08 14:57:34 +02:00
Alexander Tokmakov
507ad0c4d9
Merge pull request #40691 from azat/mergetree/transaction-wal-fix 
Move committing InMemory parts to WAL out of NOEXCEPT_SCOPE()
 2022-09-08 15:55:30 +03:00
Antonio Andelic
b99996961d Address PR comments 2022-09-08 12:45:36 +00:00
vdimir
8121904dc4
Minor update StorageEmbeddedRocksDB::getSampleBlock 2022-09-08 12:43:38 +00:00
alesapin
15dc68340f
Merge pull request #41103 from ClickHouse/return_key_to_message 
Improve logging one time
 2022-09-08 14:00:06 +02:00
Vitaly Baranov
122009a2bd Use table lock if database is ordinary and zero-copy-replication is enabled. 2022-09-08 13:54:59 +02:00
Kruglov Pavel
04aea19b31
Merge pull request #40881 from coelho/patch-1 
Fix uncaught DNS_ERROR on failed connection to replicas
 2022-09-08 13:32:45 +02:00
Maksim Kita
1f9ea15aab
Merge pull request #41049 from canhld94/ch_canh_gs_core 
Remove parser logic for WITH GROUPING SETS
 2022-09-08 13:32:27 +02:00
Kruglov Pavel
96cb0e54d8
Merge branch 'master' into Avogar-patch-1 2022-09-08 13:24:08 +02:00
alesapin
6da37b92e1 Improve logging one time 2022-09-08 13:11:53 +02:00
Vladimir C
90647cb0c5
Merge pull request #39970 from SachinSetiya/majority_quorum 
Add majority_insert_quorum setting
 2022-09-08 11:31:33 +02:00
Frank Chen
a986380522
Update src/Storages/Distributed/DirectoryMonitor.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-09-08 17:25:29 +08:00
Robert Schulze
c16707ff00
chore: delete obsolete modelEvaluate() function + SYSTEM.MODELS view 
- The deleted function modelEvaluate() was superseded by
  catboostEvaluate().

- Also delete the external model repository, as modelEvaluate() was it's
  last user. Additionally remove the system view SYSTEM.MODELS for
  inspecting the repository.

- SYSTEM RELOAD MODELS is also obsolete. HOWEVER, it was retained and
  made a no-op instead of deleted.

  Why?
  The reason is that RBAC in distributed setups works by storing
  privileges (granted and revoked) as plain SQL statements in Keeper.
  Nodes read these statements at startup and parse them. If a privilege
  for SYSTEM RELOAD MODELS exists but parser doesn't recognize it
  nodes would fail to come up.

  Considered but rejected alternatives:
  - Ignore SYSTEM RELOAD MODELS during parsing RBAC privileges and
    return an error for regular SYSTEM RELOAD MODELS SQL. Special-case
    of no-op behavior, too brittle.
  - Remove SYSTEM RELOAD MODELS manually from Keeper via command-line
    manipulation of Keeper nodes or via SQL by dropping the privileges.
    Needs user intervention during upgrade.
 2022-09-08 09:10:11 +00:00
Robert Schulze
60f9f6855d
feat: implement catboost in library-bridge 
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes #27870
 2022-09-08 09:01:32 +00:00
Antonio Andelic
09c0bf2931 Add unit tests for match path 2022-09-08 08:16:38 +00:00
Antonio Andelic
5d837e4f41 Merge branch 'master' into fix-keeper-system-path-check 2022-09-08 07:41:04 +00:00
Robert Schulze
68808858a5
Merge pull request #41050 from FrankChen021/exception_safe 
Fix failed stress test (OpenTelemetry)
 2022-09-08 09:19:54 +02:00
Robert Schulze
9d4de0cbaa
Merge pull request #40999 from ClickHouse/sse2-special-build 
Add special x86-SSE2-only build
 2022-09-08 09:06:29 +02:00
Frank Chen
329f31e7ab Address review comments 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-08 11:38:10 +08:00
Anton Popov
7c12b448b8 Merge remote-tracking branch 'upstream/master' into HEAD 2022-09-08 01:52:52 +00:00
Anton Popov
5a790b15b4 try to fix filling of missed Nested columns with multiple levels 2022-09-08 00:20:11 +00:00
Suzy Wang
61b2e1c32c
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-07 16:49:03 -04:00
Igor Nikonov
e74bb00e90 Fix: EXPLAIN PLAN - make settings in SETTING clause effective 2022-09-07 19:11:25 +00:00
Kseniia Sumarokova
a270eeef91
Merge pull request #41008 from kssenii/refactor-merge-tree-read 
Small refactoring around merge tree readers (get rid of data part ptr)
 2022-09-07 18:27:33 +02:00
Dmitry Novik
499e479892
Merge pull request #40873 from azat/build/fix-debug-symbols-quirk 
Fix debug symbols
 2022-09-07 17:31:35 +02:00
Suzy Wang
e0b8f26839
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-07 10:38:21 -04:00
alesapin
365438d617
Merge pull request #41016 from ClickHouse/one_more_logging 
Slightly improve diagnostics and remove assertions
 2022-09-07 15:23:17 +02:00
Antonio Andelic
951ee44229 Process read requests with snapshot install 2022-09-07 11:51:56 +00:00
Vitaly Baranov
31ed722572
Merge pull request #41044 from vitlibar/more-conventional-conversion-yaml-to-xmk 
More conventional conversion yaml to xml
 2022-09-07 13:46:32 +02:00
vdimir
6234d564d1
Fix ReplicatedMergeTreeSink, add more info to logs 2022-09-07 11:19:28 +00:00
vdimir
000ba8a60d
Use insert_qurum = auto instead of majority_insert_quorum 2022-09-07 11:19:27 +00:00
Sachin
ade4337978
Add majority_insert_quorum setting 
majority_insert_quorum is defined as (number_of_replicas/2)+1. Insert
will be successful only if majority of quorum have applied it. If
insert_quorum and majority_insert_quorum both are specified, max of
both will be used.
 2022-09-07 11:19:24 +00:00
Kruglov Pavel
8513af5c1b
Merge pull request #40638 from azat/mergetree/insert-perf 
Do not obtain storage snapshot for each INSERT block (slightly improves performance)
 2022-09-07 12:53:09 +02:00
Robert Schulze
c07f234f09
fix: disable ENABLE_MULTITARGET_CODE for SSE2 builds 2022-09-07 10:52:31 +00:00
Antonio Andelic
61398cee80 Merge branch 'master' into keeper-linearizable-reads 2022-09-07 10:25:35 +00:00
Robert Schulze
bf8fed8be8
Revert "fix: don't force-inline SSE3 code into generic code" 
This reverts commit d054ffd110.
 2022-09-07 09:45:07 +00:00
Vitaly Baranov
9c847ceec9 No hardlinks while making backup of MergeTree in atomic database. 2022-09-07 11:44:50 +02:00
Antonio Andelic
a377a5f1e9 Change assert to LOGICAL_ERROR 2022-09-07 09:24:50 +00:00
Antonio Andelic
4cdf18cc81 Shutdown storage after RAFT 2022-09-07 09:20:47 +00:00
Antonio Andelic
c6b17bd7f9 Don't preprocess if storage finalized 2022-09-07 09:14:03 +00:00
Antonio Andelic
d15fdae8dc
Merge pull request #40853 from ClickHouse/embeddedrocksdb-delete-update-support 
Support for `UPDATE` and `DELETE` in `EmbeddedRocksDB`
 2022-09-07 11:03:49 +02:00
alesapin
81c98dadd2 Remove redundant change 2022-09-07 11:01:06 +02:00
Frank Chen
fc05b05be3 Fix style and typo 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-07 15:14:43 +08:00
Frank Chen
de8f6bdce7 More safe 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-07 13:39:12 +08:00
jthmath
478d2ad06a fix ignore projection 2022-09-07 10:23:52 +08:00
Nikolay Degterinsky
981e9dbce2
Merge pull request #40997 from canhld94/ch_canh_fix_grouping_set 
Fix grouping set with group_by_use_nulls
 2022-09-07 00:08:31 +02:00
alesapin
6ded03c000 Disable fetch shortcut for zero copy replication 2022-09-07 00:00:10 +02:00
Robert Schulze
d054ffd110
fix: don't force-inline SSE3 code into generic code 
Force-inlining code compiled for SSE3 into "generic"
(non-platform-specific) code works for standard x86 builds where
everything is compiled with SSE 4.2 (and smaller). It no longer works
if we compile everything only with SSE 2.
 2022-09-06 18:23:19 +00:00
Dmitry Novik
b92c383d4b Try to fix ODR for Loggers class 2022-09-06 19:58:31 +02:00
alesapin
43493389b9 Merge branch 'master' into one_more_logging 2022-09-06 19:40:24 +02:00
Anton Popov
9def867795 fix reading of subcolumns from in-memory parts 2022-09-06 17:38:51 +00:00
alesapin
b778b9f37f Improve logging better 2022-09-06 19:25:58 +02:00
alesapin
09e97a6381 Fix style 2022-09-06 18:38:34 +02:00
alesapin
ceed9f418b Return better errors handling 2022-09-06 18:22:44 +02:00
Vitaly Baranov
63e992d52d Edit test configs. 2022-09-06 17:09:26 +02:00
Anton Popov
ce9b76f416 fix Nested in in-memory parts 2022-09-06 15:01:47 +00:00
kssenii
487bc0fba3 Fix heap use after free 2022-09-06 16:50:58 +02:00
Frank Chen
6ced4131ca exception safe 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-06 22:11:47 +08:00
Kseniia Sumarokova
3558361a05
Merge branch 'master' into refactor-merge-tree-read 2022-09-06 16:00:43 +02:00
Anton Popov
f76c1482bd try to fix filling of missed Nested columns with multiple levels 2022-09-06 13:56:32 +00:00
Duc Canh Le
4fe0457e59
Merge branch 'master' into ch_canh_gs_core 2022-09-06 21:40:02 +08:00
Duc Canh Le
46f281f9ea remove parser logic for WITH GROUPING SETS 2022-09-06 21:36:58 +08:00