Commit Graph

20 Commits

Author SHA1 Message Date
Azat Khuzhin
ed70ed6f71 Introduce connection_no_wait setting for MySQL engine.
This will allow to avoid superfluous sleep during query execution, since
this not only not desired behavoiur, but also may hang the server, since
if you will execute enough queries that will use MySQL database but will
not allow enough connections (or your MySQL server is too slow) then you
may run out of threads in the global thread pool.

Also note that right now it is possible to get deadlock when the mysql
pool is full, consider the following scenario:

- you have m1 and m2 mysql tables
- you have q1 and q2 queries, bot queries join m1 and m2
- q1 allocated connection for m1 but cannot allocate connection for m2
- q2 allocated connection for m2 but cannot allocate connection for m1
- but to resolve the lock one should give up on the locking while it is not possible right now...

And then you got no free threads and this:

    # grep -h ^202 /proc/$(pgrep clickhouse-serv)/task/*/syscall  | cut -d' ' -f2 | sort | uniq -c | sort -nr | head
       1554 0x7ffb60b92fe8 # mutex in mysqlxx::PoolWithFailover::get
       1375 0x7ffb9f1c4748 # mutex in ::PoolEntryHelper::~PoolEntryHelper from DB::MultiplexedConnections::invalidateReplica
       1160 0x7ffb612918b8 # mutex in mysqlxx::PoolWithFailover::get
         42 0x7ffb9f057984 # mutex in ThreadPoolImpl<std::__1::thread>::worker

    *NOTE: 202 is a `futex` with WAIT*

(Went with `syscall` because debugging 10k+ threads is not easy, and
eventually it may TRAP)
2021-09-02 22:23:37 +03:00
tavplubix
c36fe210b8
Revert "Add test for #13398" 2021-08-27 21:07:10 +03:00
kssenii
962f415740 Add test and settings 2021-08-23 22:18:04 +00:00
Azat Khuzhin
4f41ebcae3 Add settings (for connections) for MySQL storage engine
Default settings are not very efficient, since they do not even reuse
connections.
And when each query requires connection you can have only ~80 QPS, while
by simply enabling connection reuse (connection_auto_close=false) you
can have ~500 QPS (and by increasing connection_pool_size you can have
better QPS throughput).

So this patch allows to pass through some connection related settings
for the StorageMySQL engine, like:
- connection_pool_size=16
- connection_max_tries=3
- connection_auto_close=true

v2: remove connection_pool_default_size
v3: remove num_tries_on_connection_loss
2021-05-20 22:44:52 +03:00
kssenii
2c70183652 Fix settings 2021-04-01 11:41:22 +00:00
kssenii
747967a9bb Better 2021-04-01 10:27:24 +00:00
kssenii
4e849f3b77 Less logging 2021-03-31 14:06:24 +00:00
kssenii
ba5ddd5c32 Do not parse replicas in PoolWithFailover 2021-03-31 14:03:52 +00:00
kssenii
1bd1a705ce Fix clang tidy 2021-03-28 12:45:49 +00:00
kssenii
0a66f11f69 Fix style, fix build 2021-03-28 10:27:37 +00:00
kssenii
c17748a0ba Add proper settings 2021-03-27 21:57:35 +00:00
kssenii
ef537b802f Better comments 2021-03-27 21:10:44 +00:00
kssenii
ae868208c2 Use pool with failover in mysql storage 2021-03-27 14:39:45 +00:00
Alexander Kazakov
196d454114 Initialize random engine with clock and thread_id
Thread-safe and fairly descent approach in our case
2021-02-24 14:55:16 +03:00
Alexander Kazakov
deb566a40b Resorted from std::random_device to std::rand() 2021-02-24 11:48:04 +03:00
Alexander Kazakov
0296d7d026 Added some explanations on randomization 2021-02-17 21:51:05 +03:00
Alexander Kazakov
acb5fb8179 Randomly shuffle replicas withing the same priority 2021-02-17 20:58:04 +03:00
Alexey Milovidov
a576a4fbcd Enable extra warnings for base, utils, programs 2020-05-10 01:59:34 +03:00
alexey-milovidov
c7863e1ac6
Clang Tidy, part 7 (#9799)
* Attempt to enable identifier naming check

* Attempt to enable identifier naming check, continue

* Progress on identifier names

* Adopt identifier names check

* Fixed error

* Merge with master

* Fixed build

* Fixed build

* Fixed build

* Fixed build

* Fixed build

* Fixed error

* Fixed error

* Fixed error
2020-03-23 05:12:31 +03:00
Ivan
b7ef5a699c
Move FastMemcpy to contribs (#9219)
* Get rid of non-existent vectorclass
* Move FastMemcpy to contribs
* Restore comments
* Disable FastMemcpy on non-Linux
* Fix cmake file
* Don't build FastMemcpy for ARM64
* Replace FastMemcpy submodule with its contents
* Fix cmake file
* Move widechar_width to contrib/
* Move sumbur to contrib/
* Move consistent-hashing to contrib/
* Fix UBSan tests
2020-03-13 01:26:16 +03:00