Commit Graph

4011 Commits

Author SHA1 Message Date
Robert Schulze
72b3ebd180
Merge pull request #66136 from rschu1ze/bump-s2
Bump s2geometry again
2024-07-05 14:05:31 +00:00
Robert Schulze
d3f23c2753
Bump s2geometry again 2024-07-05 13:29:34 +00:00
Vitaly Baranov
f315b36323
Merge pull request #66069 from vitlibar/add-prometheus-protocols-protobufs
Add protobufs for Prometheus remote-write/remote-read protocols
2024-07-05 10:41:04 +00:00
Robert Schulze
3621b35591
Merge pull request #66094 from rschu1ze/bump-s2geometry
Bump s2geometry to latest master
2024-07-05 10:30:20 +00:00
Robert Schulze
4dca031e9d
Merge pull request #66100 from rschu1ze/bump-azure
Bump Azure to 1.12
2024-07-05 09:11:00 +00:00
Vitaly Baranov
1ef5bca591 Fix cmake function PROTOBUF_GENERATE_CPP(): now it returns correct paths in SRCS and HDRS
even if input ".proto" files are located in sibling directories.
2024-07-04 23:07:19 +02:00
Vitaly Baranov
e428542b2e Add prometheus protobufs. 2024-07-04 23:06:14 +02:00
Antonio Andelic
ec6739120c
Merge pull request #66041 from ClickHouse/try-disabling-jemalloc-background-threads
Try disabling jemalloc background threads
2024-07-04 20:30:39 +00:00
Robert Schulze
ffe1f8fea0
Bump Azure to 1.12 2024-07-04 19:24:30 +00:00
Robert Schulze
597810b69d
Fix s390x build 2024-07-04 18:46:09 +00:00
Robert Schulze
632f292fd1
Merge pull request #66082 from rschu1ze/bmp-vectorscan11
Bump vectorscan to 5.4.11
2024-07-04 18:11:10 +00:00
Robert Schulze
1bd515cc1f
Merge remote-tracking branch 'ClickHouse/master' into bump-s2geometry 2024-07-04 15:43:47 +00:00
Robert Schulze
11a30d6d60
Bump s2geometry to latest master 2024-07-04 14:20:06 +00:00
Antonio Andelic
613ed1ebbf Merge branch 'master' into try-disabling-jemalloc-background-threads 2024-07-04 13:58:26 +02:00
Antonio Andelic
c93d8cbb66 Fixes 2024-07-04 13:57:47 +02:00
Nikita Mikhaylov
e2a494fa60
Merge pull request #66064 from rschu1ze/mysql-openssl-tsan
Maybe fix tsan assert in `test_mysql_killed_while_insert_8_0`
2024-07-04 11:17:42 +00:00
Robert Schulze
33b7afc1b4
Bump vectorscan to 5.4.11 2024-07-04 10:30:24 +00:00
Robert Schulze
dbac2212a0
Merge pull request #66072 from rschu1ze/llvm-15-burning-some-trash
Clean-up custom LLVM 15 patches
2024-07-04 09:06:38 +00:00
Robert Schulze
43b088dbd0
Merge pull request #66056 from rschu1ze/bmp-vectorscan
Bump vectorscan to 5.4.10.1
2024-07-04 08:59:50 +00:00
Robert Schulze
eb7ab5128d
Clean-up custom LLVM 15 patches 2024-07-03 21:07:59 +00:00
Robert Schulze
9737c5bab4
Probably fix tsan assert in test_mysql_killed_while_insert_8_0 2024-07-03 17:38:53 +00:00
Robert Schulze
8b14754005
Fix ARM build (upgrade sysroot) 2024-07-03 16:29:47 +00:00
Robert Schulze
39a371b27d
Bump vectorscan 2024-07-03 15:07:31 +00:00
Antonio Andelic
5fd36059e4 Try disabling background threads 2024-07-03 15:28:01 +02:00
Azat Khuzhin
9c7db6711b Fix possible issues with MySQL client protocol TLS connections
Occasionally, 02479_mysql_connect_to_self fails on CI [1].

  [1]: https://github.com/ClickHouse/ClickHouse/issues/50911

The problem was indeed query profiler and EINTR, but not in a way you
may think.

For such failures you may see the following trace in trace_log:

    contrib/openssl/crypto/bio/bss_sock.c:127::sock_read
    contrib/openssl/crypto/bio/bio_meth.c:121::bread_conv
    contrib/openssl/crypto/bio/bio_lib.c:285::bio_read_intern
    contrib/openssl/crypto/bio/bio_lib.c:311::BIO_read
    contrib/openssl/ssl/record/methods/tls_common.c:398::tls_default_read_n
    contrib/openssl/ssl/record/methods/tls_common.c:575::tls_get_more_records
    contrib/openssl/ssl/record/methods/tls_common.c:1122::tls_read_record
    contrib/openssl/ssl/record/rec_layer_s3.c:645::ssl3_read_bytes
    contrib/openssl/ssl/s3_lib.c:4527::ssl3_read_internal
    contrib/openssl/ssl/s3_lib.c:4551::ssl3_read
    contrib/openssl/ssl/ssl_lib.c:2343::ssl_read_internal
    contrib/openssl/ssl/ssl_lib.c:2357::SSL_read
    contrib/mariadb-connector-c/libmariadb/secure/openssl.c:729::ma_tls_read
    contrib/mariadb-connector-c/libmariadb/ma_tls.c:90::ma_pvio_tls_read
    contrib/mariadb-connector-c/libmariadb/ma_pvio.c:250::ma_pvio_read
    contrib/mariadb-connector-c/libmariadb/ma_pvio.c:297::ma_pvio_cache_read
    contrib/mariadb-connector-c/libmariadb/ma_net.c:373::ma_real_read
    contrib/mariadb-connector-c/libmariadb/ma_net.c:427::ma_net_read
    contrib/mariadb-connector-c/libmariadb/mariadb_lib.c:192::ma_net_safe_read
    contrib/mariadb-connector-c/libmariadb/mariadb_lib.c:2138::mthd_my_read_query_result
    contrib/mariadb-connector-c/libmariadb/mariadb_lib.c:2212::mysql_real_query
    src/Common/mysqlxx/Query.cpp:56::mysqlxx::Query::executeImpl()
    src/Common/mysqlxx/Query.cpp:73::mysqlxx::Query::use()
    src/Processors/Sources/MySQLSource.cpp:50::DB::MySQLSource::Connection::Connection()

After which the connection will fail with:

    Code: 1000. DB::Exception: Received from localhost:9000. DB::Exception: mysqlxx::ConnectionLost: Lost connection to MySQL server during query (127.0.0.1:9004). (POCO_EXCEPTION)

But, if you will take a look at ma_tls_read() you will see that it has
proper retries for SSL_ERROR_WANT_READ (and EINTR is just a special case
of it), but still, for some reason it fails.

And the reason is the units of the read/write timeout, ma_tls_read()
calls poll(read_timeout) in case of SSL_ERROR_WANT_READ, but, it
incorrectly assume that the timeout is in milliseconds, but that timeout
was in seconds, this bug had been fixed in [2], and now it works like a
charm!

  [2]: https://github.com/ClickHouse/mariadb-connector-c/pull/17

I've verified it with patching openssl library:

    diff --git a/crypto/bio/bss_sock.c b/crypto/bio/bss_sock.c
    index 82f7be85ae..3d2f3926a0 100644
    --- a/crypto/bio/bss_sock.c
    +++ b/crypto/bio/bss_sock.c
    @@ -124,7 +124,24 @@ static int sock_read(BIO *b, char *out, int outl)
                 ret = ktls_read_record(b->num, out, outl);
             else
     # endif
    -            ret = readsocket(b->num, out, outl);
    +        {
    +            static int i = 0;
    +            static int j = 0;
    +            if (!(++j % 5))
    +            {
    +                fprintf(stderr, "sock_read: inject EAGAIN with ret=0\n");
    +                ret = 0;
    +                errno = EAGAIN;
    +            }
    +            else if (!(++i % 3))
    +            {
    +                fprintf(stderr, "sock_read: inject EAGAIN with ret=-1\n");
    +                ret = -1;
    +                errno = EAGAIN;
    +            }
    +            else
    +                ret = readsocket(b->num, out, outl);
    +        }
             BIO_clear_retry_flags(b);
             if (ret <= 0) {
                 if (BIO_sock_should_retry(ret))

And before this patch (well, not the patch itself, but the referenced
patch in mariadb-connector-c) if you will pass read_write_timeout=1 it
will fail:

    SELECT * FROM mysql('127.0.0.1:9004', system, one, 'default', '', SETTINGS connect_timeout = 100, connection_wait_timeout = 100, read_write_timeout=1)
    Code: 1000. DB::Exception: Received from localhost:9000. DB::Exception: mysqlxx::ConnectionLost: Lost connection to MySQL server during query (127.0.0.1:9004). (POCO_EXCEPTION)

But after, it always works:

    $ ch benchmark -c10 -q "SELECT * FROM mysql('127.0.0.1:9004', system, one, 'default', '', SETTINGS connection_pool_size=1, connect_timeout = 100, connection_wait_timeout = 100, read_write_timeout=1)"
    ^CStopping launch of queries. SIGINT received.

    Queries executed: 478.

    localhost:9000, queries: 478, QPS: 120.171, RPS: 120.171, MiB/s: 0.001, result RPS: 120.171, result MiB/s: 0.001.

    0.000%          0.014 sec.
    10.000%         0.058 sec.
    20.000%         0.065 sec.
    30.000%         0.073 sec.
    40.000%         0.079 sec.
    50.000%         0.087 sec.
    60.000%         0.089 sec.
    70.000%         0.091 sec.
    80.000%         0.095 sec.
    90.000%         0.100 sec.
    95.000%         0.102 sec.
    99.000%         0.105 sec.
    99.900%         0.140 sec.
    99.990%         0.140 sec.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-07-01 18:39:43 +02:00
Antonio Andelic
8f9beffc65 No jemalloc profiler for non-Linux 2024-06-28 16:04:45 +02:00
Antonio Andelic
c9fa974472 Use background thread 2024-06-27 08:46:10 +02:00
Antonio Andelic
35820eaa7b Build jemalloc with profiler 2024-06-26 11:13:18 +02:00
Robert Schulze
605d51700a
Merge pull request #65529 from rschu1ze/openssl-temp-official-fix
OpenSSL: Replace temporary fix for unsynchronized access by official fix
2024-06-23 15:24:28 +00:00
Robert Schulze
0a60dc1672
OpenSSL: Replace temporary fix for unsynchronized access by official fix 2024-06-21 12:57:44 +00:00
Robert Schulze
df857ff17a
Bump re2 to latest HEAD 2024-06-21 10:34:30 +00:00
Robert Schulze
fd52e19b3e
Suppress leaksan false positive in OpenSSL 2024-06-19 08:30:47 +00:00
Robert Schulze
abe424b24b
Merge pull request #65111 from rschu1ze/reenable-session-caching
Re-enable OpenSSL session caching
2024-06-18 16:39:38 +00:00
Alexander Tokmakov
b150b0f5fa
Revert "Fix AWS ECS" 2024-06-17 23:11:59 +02:00
Robert Schulze
cdcfcb99fc
Merge remote-tracking branch 'rschu1ze/master' into reenable-session-caching 2024-06-17 20:51:11 +00:00
Robert Schulze
186bd0cc3d
Temporary fix for tsan issue openssl#24629 2024-06-17 12:20:54 +00:00
Alexey Milovidov
2c1918908a Revert "Debug AWS"
This reverts commit 20ca8f8714.
2024-06-16 17:12:55 +02:00
Alexey Milovidov
b41bb96605 Revert "Debug AWS"
This reverts commit 642dc35a76.
2024-06-16 17:12:19 +02:00
Alexey Milovidov
37eb815b17 Fix error 2024-06-16 17:12:05 +02:00
Alexey Milovidov
642dc35a76 Debug AWS 2024-06-16 15:21:17 +02:00
Alexey Milovidov
20ca8f8714 Debug AWS 2024-06-16 15:20:26 +02:00
Alexey Milovidov
cb390cc3d9 Fix alloc/dealloc mismatch or prevent false-positive ASan error in the AWS SDK 2024-06-16 13:14:59 +02:00
Raúl Marín
0d8bd133e6
Merge pull request #64959 from canhld94/bump_fmt_9
Update fmtlib version to 9.1.0
2024-06-14 14:05:04 +00:00
Robert Schulze
d92a2f3ef6
Remove obsolete fix from aws submodule 2024-06-12 13:20:50 +00:00
Robert Schulze
97f96a7b84
Merge pull request #65125 from rschu1ze/bug-63792
Fix false positives leaky memory warnings in OpenSSL
2024-06-12 12:40:50 +00:00
Robert Schulze
92ca8b30b3
Merge pull request #65048 from rschu1ze/bump-absl
Bump abseil to latest HEAD
2024-06-12 08:26:58 +00:00
Robert Schulze
848054e85f
Fix another false positive leak 2024-06-11 21:06:45 +00:00
Robert Schulze
e0279e856f
Fix #63792 (hopefully) 2024-06-11 19:42:20 +00:00
Robert Schulze
af83bc92ce
Switch to same HEAD as before but with s390x-breaking commit reverted 2024-06-11 14:16:38 +00:00
Robert Schulze
e5dcf75968
Add forked abseil submodule back 2024-06-11 14:09:56 +00:00