Commit Graph

72 Commits

Author SHA1 Message Date
Alexey Milovidov
cd7f9d981c Remove ya.make 2021-09-25 04:22:54 +03:00
alexey-milovidov
20d8523a2e
Merge pull request #28888 from azat/mysql-in-fix
Fix queries to external databases (i.e. MySQL) with multiple columns in IN ( i.e. `(k,v) IN ((1, 2))` )
2021-09-18 15:53:40 +03:00
alexey-milovidov
c4b3add859
Update Query.cpp 2021-09-18 15:46:48 +03:00
Sergei Semin
f74350c148 fix special build 2021-09-12 02:49:31 +03:00
Azat Khuzhin
c77c4e22ae Add MySQL query into after transformation in case of BadQuery
Since it may differs from the user input
2021-09-11 14:06:19 +03:00
Sergei Semin
a46d23fa79 ignore reserved-macro-identifier when building lib mysqlxx 2021-09-09 23:51:47 +03:00
Dmitrii Kovalkov
b073d5a4b4 Default argument for Pool.get() 2021-09-08 13:11:38 +03:00
Azat Khuzhin
ed70ed6f71 Introduce connection_no_wait setting for MySQL engine.
This will allow to avoid superfluous sleep during query execution, since
this not only not desired behavoiur, but also may hang the server, since
if you will execute enough queries that will use MySQL database but will
not allow enough connections (or your MySQL server is too slow) then you
may run out of threads in the global thread pool.

Also note that right now it is possible to get deadlock when the mysql
pool is full, consider the following scenario:

- you have m1 and m2 mysql tables
- you have q1 and q2 queries, bot queries join m1 and m2
- q1 allocated connection for m1 but cannot allocate connection for m2
- q2 allocated connection for m2 but cannot allocate connection for m1
- but to resolve the lock one should give up on the locking while it is not possible right now...

And then you got no free threads and this:

    # grep -h ^202 /proc/$(pgrep clickhouse-serv)/task/*/syscall  | cut -d' ' -f2 | sort | uniq -c | sort -nr | head
       1554 0x7ffb60b92fe8 # mutex in mysqlxx::PoolWithFailover::get
       1375 0x7ffb9f1c4748 # mutex in ::PoolEntryHelper::~PoolEntryHelper from DB::MultiplexedConnections::invalidateReplica
       1160 0x7ffb612918b8 # mutex in mysqlxx::PoolWithFailover::get
         42 0x7ffb9f057984 # mutex in ThreadPoolImpl<std::__1::thread>::worker

    *NOTE: 202 is a `futex` with WAIT*

(Went with `syscall` because debugging 10k+ threads is not easy, and
eventually it may TRAP)
2021-09-02 22:23:37 +03:00
tavplubix
c36fe210b8
Revert "Add test for #13398" 2021-08-27 21:07:10 +03:00
kssenii
962f415740 Add test and settings 2021-08-23 22:18:04 +00:00
Alexey Milovidov
f063e44131 Whitespaces 2021-08-15 09:00:28 +03:00
Evgeniy Gatov
2487063be3 METR-41529 2021-06-21 22:31:42 +03:00
Azat Khuzhin
4f41ebcae3 Add settings (for connections) for MySQL storage engine
Default settings are not very efficient, since they do not even reuse
connections.
And when each query requires connection you can have only ~80 QPS, while
by simply enabling connection reuse (connection_auto_close=false) you
can have ~500 QPS (and by increasing connection_pool_size you can have
better QPS throughput).

So this patch allows to pass through some connection related settings
for the StorageMySQL engine, like:
- connection_pool_size=16
- connection_max_tries=3
- connection_auto_close=true

v2: remove connection_pool_default_size
v3: remove num_tries_on_connection_loss
2021-05-20 22:44:52 +03:00
Alexey Milovidov
4ff812db7f Maybe better support for paths with whitespaces 2021-04-24 22:47:52 +03:00
Alexander Tokmakov
3b4bd18809 fix prewhere in materialize mysql 2021-04-16 18:51:27 +03:00
kssenii
2c70183652 Fix settings 2021-04-01 11:41:22 +00:00
kssenii
747967a9bb Better 2021-04-01 10:27:24 +00:00
kssenii
4e849f3b77 Less logging 2021-03-31 14:06:24 +00:00
kssenii
ba5ddd5c32 Do not parse replicas in PoolWithFailover 2021-03-31 14:03:52 +00:00
kssenii
d409f089fc Fix split build 2021-03-28 14:56:02 +00:00
kssenii
1bd1a705ce Fix clang tidy 2021-03-28 12:45:49 +00:00
kssenii
0a66f11f69 Fix style, fix build 2021-03-28 10:27:37 +00:00
kssenii
c17748a0ba Add proper settings 2021-03-27 21:57:35 +00:00
kssenii
ef537b802f Better comments 2021-03-27 21:10:44 +00:00
kssenii
95e8a8b9f0 Support shards 2021-03-27 14:40:07 +00:00
kssenii
ae868208c2 Use pool with failover in mysql storage 2021-03-27 14:39:45 +00:00
Alexey Milovidov
eadf0248d0 Build fixes 2021-03-16 14:07:53 +03:00
Haavard Kvaalen
efa94c5735 Fix use after free in mysqlxx::Pool::tryGet()
tryGet() created an Entry object that referenced a Connection, but then
deleted the Connection just before the Entry went out of scope.
2021-03-04 16:51:27 +01:00
Alexander Kazakov
63b95c7451
MySQL dictionary source: A mechanism for retrying unexpected connection loss during communication with mysql server (#21237)
* Added a new type of Exception

for better recognition of connection failures

* Added more logging for mysql communication

* Retries on connection loss during query.

Make MySQL dictionary source resilient to unexpected loss of connection
during querying MySQL. This behavior is controlled with
".fail_on_connection_loss" config parameter, which defaults to false.

* Stripped some debugging leftover garbage

* Minor followup corrections

* Corrections after PR comments

* Yet more fixes
2021-02-27 11:18:28 +03:00
Alexander Kazakov
66f603512c
A followup correction to #19998 (#21221) 2021-02-26 11:42:37 +03:00
Alexander Kazakov
4bcaed98d8
Added "opt_reconnect" parameter to config for controlling MYSQL_OPT_RECONNECT option (#19998)
* Support for MYSQL_OPT_RECONNECT option

It can now be controlled by .opt_reconnect = {true|false} parameter in
the mysql replica config section
2021-02-26 09:49:49 +03:00
Alexander Kazakov
196d454114 Initialize random engine with clock and thread_id
Thread-safe and fairly descent approach in our case
2021-02-24 14:55:16 +03:00
Alexander Kazakov
deb566a40b Resorted from std::random_device to std::rand() 2021-02-24 11:48:04 +03:00
Alexander Kazakov
0296d7d026 Added some explanations on randomization 2021-02-17 21:51:05 +03:00
Alexander Kazakov
acb5fb8179 Randomly shuffle replicas withing the same priority 2021-02-17 20:58:04 +03:00
Alexey Milovidov
940b337eec Remove useless files 2021-02-03 10:03:32 +03:00
Alexey Milovidov
f8592f4539 Remove useless files 2021-02-03 10:03:13 +03:00
Alexey Milovidov
12485eee6b Fix some of the issues found by Coverity 2021-02-02 22:07:23 +03:00
Alexey Milovidov
f7fffe173c Set charset to utf8mb4 in mariadb-connector 2021-01-29 10:57:22 +03:00
tavplubix
f13de96afc
Merge pull request #17366 from zhang2014/fix/ISSUES-16835
ISSUES-16835 try fix miss match header with MySQL SHOW statement
2020-12-09 23:12:47 +03:00
zhang2014
2f735aa61d ISSUES-16835 try fix review comment 2020-12-09 16:19:09 +08:00
Alexander Kazakov
0e41af11c7
Fixed typo in message format (#17900) 2020-12-08 14:28:35 +03:00
Winter Zhang
972cb41fae
Merge branch 'master' into fix/ISSUES-16835 2020-12-08 15:32:58 +08:00
Alexander Kazakov
81d9d3237f
In mysqlxx::Pool: fix for reconnection problem (#17681)
Remove stalled unrecoverable connections from myqsl connections pool
2020-12-03 23:36:10 +03:00
zhang2014
a74235397e ISSUES-16835 try fix miss match header with MySQL SHOW statement 2020-11-24 23:31:43 +08:00
alesapin
16fa71a4c9 Missed comment 2020-10-08 20:46:59 +03:00
alesapin
70f2a3328d Better atomic operations on refcount 2020-10-08 20:45:18 +03:00
alesapin
de722cd57c Fix race condition in MySQL pool 2020-10-08 18:50:09 +03:00
Vasily Nemkov
3973a17530
MySql datatypes dateTime64 and decimal (#11512) 2020-09-09 15:18:02 +03:00
Yatsishin Ilya
a47ffd191d Change my_bool to bool.
The my_bool type no longer exists, and people may have been using it in their own code since some mysql.h functions have been returning that. The recommended fix is to use bool or int, which will work both in 8.0 and older versions.

https://bugs.mysql.com/bug.php?id=85131
2020-08-28 11:59:50 +03:00