The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
previously allowed.
Hence, this change
- removes shared_ptr_helper and as a result all inherited create() methods,
- instead, Storage objects are now created using make_shared<>() by the
caller (for that to work, many constructors had to be made public), and
- all Storage classes were marked as noncopyable using boost::noncopyable.
In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
1. Dropped support for DatabaseOrdinary for MaterializeMySQL. It
is marked as experimental, and dropping support makes the code
more maintaible, and speeds up integration tests by 50%.
2. Get rid of thread name logic for StorageMaterializeMySQL wrapping,
use setInternalQuery instead (similar to MaterializedPostgreSQL).
This will fix race with DatabaseCatalog::loadMarkedAsDroppedTables(),
since MaterializeMySQL, and MaterializedMySQLSyncThread in background,
will be started earlier then
DatabaseCatalog::loadMarkedAsDroppedTables() and will move those tables
to metadata_dropped, and after loadMarkedAsDroppedTables() will start
and try to load partially dropped tables and will hit UUID overlap:
12:02:51.536783 [ 3026034 ] {} <Information> Application: starting up
12:02:53.019282 [ 3026034 ] {} <Information> DatabaseMaterializeMySQL<Atomic> (mysql): Total 9 tables and 0 dictionaries.
12:02:53.041699 [ 3026200 ] {} <Debug> mysql.data (7143b65f-6982-4600-b143-b65f6982e600): Loading data parts
12:02:53.041740 [ 3026200 ] {} <Debug> mysql.data (7143b65f-6982-4600-b143-b65f6982e600): There are no data parts
12:02:53.620382 [ 3026034 ] {} <Information> DatabaseMaterializeMySQL<Atomic> (mysql): Starting up tables.
12:03:00.669730 [ 3026183 ] {} <Debug> executeQuery: (internal) /*Materialize MySQL step 1: execute MySQL DDL for dump data*/ DROP TABLE mysql.data
12:03:00.741894 [ 3026269 ] {} <Information> DatabaseCatalog: Trying load partially dropped table mysql.data (7143b65f-6982-4600-b143-b65f6982e600) from /var/lib/clickhouse/metadata_dropped/mysql.data.7143b65f-6982-4600-b143-b65f6982e600.sql
12:03:00.742582 [ 3026269 ] {} <Debug> mysql.data (7143b65f-6982-4600-b143-b65f6982e600): Loading data parts
12:03:00.742650 [ 3026269 ] {} <Debug> mysql.data (7143b65f-6982-4600-b143-b65f6982e600): There are no data parts
12:03:00.773137 [ 3026034 ] {} <Error> Application: Caught exception while loading metadata: Code: 57, e.displayText() = DB::Exception: Mapping for table with UUID=7143b65f-6982-4600-b143-b65f6982e600 already exists. It happened due to UUID collision, most likely because some not random UUIDs were manually specified in CREATE queries., Stack trace (when copying this message, always include the lines below):
12:03:01.224557 [ 3026034 ] {} <Error> Application: DB::Exception: Mapping for table with UUID=7143b65f-6982-4600-b143-b65f6982e600 already exists. It happened due to UUID collision, most likely because some not random UUIDs were manually specified in CREATE queries.
Cc: @zhang2014
TODO (suggested by Nikolai)
1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies
3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)
Also need to figure out how prewhere works for projections, and
row_filter_policies.
wip
This is not the problem for plain DROP DATABASE query since it first
remove tables.
(gdb) bt
0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
1 0x00007f353c262774 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x11049288 <DB::DatabaseCatalog::init(DB::Context const*)::database_catalog+8>) at ../nptl/pthread_mutex_lock.c:80
2 0x0000000004fd5d1e in DB::pthread_mutex_lock (arg=arg@entry=0x11049288 <DB::DatabaseCatalog::init(DB::Context const*)::database_catalog+8>) at ../src/Common/ThreadFuzzer.cpp:253
3 0x000000000c70b3a9 in std::__1::__libcpp_mutex_lock (__m=__m@entry=0x11049288 <DB::DatabaseCatalog::init(DB::Context const*)::database_catalog+8>) at ../contrib/libcxx/include/__threading_support:322
4 std::__1::mutex::lock (this=this@entry=0x11049288 <DB::DatabaseCatalog::init(DB::Context const*)::database_catalog+8>) at ../contrib/libcxx/src/mutex.cpp:33
5 0x0000000008acd4e2 in std::__1::lock_guard<std::__1::mutex>::lock_guard (__m=..., this=<synthetic pointer>) at ../contrib/libcxx/include/__mutex_base:90
6 DB::DatabaseCatalog::removeDependency (this=this@entry=0x11049280 <DB::DatabaseCatalog::init(DB::Context const*)::database_catalog>, from=..., where=...) at ../src/Interpreters/DatabaseCatalog.cpp:388
7 0x000000000975044b in DB::StorageMaterializedView::shutdown (this=0x7f338d178a00) at ../src/Storages/StorageMaterializedView.cpp:362
8 0x0000000008a79602 in DB::DatabaseWithOwnTablesBase::shutdown (this=0x7f353be3cc60) at ../contrib/libcxx/include/__tree:184
9 0x0000000009546196 in DB::DatabaseWithDictionaries::shutdown (this=0x7f353be3cc60) at ../src/Databases/DatabaseWithDictionaries.cpp:265
10 0x0000000008acecdb in DB::DatabaseCatalog::detachDatabase (this=0x11049280 <DB::DatabaseCatalog::init(DB::Context const*)::database_catalog>, database_name=..., drop=drop@entry=false, check_empty=check_empty@entry=false) at ../contrib/libcxx/include/memory:3826
11 0x0000000004bf0aa5 in DB::InterpreterCreateQuery::createDatabase (this=0x7ffd6e1bab80, create=...) at ../src/Interpreters/InterpreterCreateQuery.cpp:162
12 0x0000000008b04327 in DB::InterpreterCreateQuery::execute (this=this@entry=0x7ffd6e1bab80) at ../src/Interpreters/InterpreterCreateQuery.cpp:722
13 0x0000000008d0fdaa in DB::executeCreateQuery (has_force_restore_data_flag=false, file_name=..., database=..., context=..., query=...) at ../src/Interpreters/loadMetadata.cpp:48
14 DB::loadDatabase (context=..., database=..., database_path=..., force_restore_data=<optimized out>) at ../src/Interpreters/loadMetadata.cpp:72
15 0x0000000008d103c3 in DB::loadMetadata (context=...) at ../src/Interpreters/loadMetadata.cpp:111
16 0x0000000004f4d25e in DB::Server::main (this=<optimized out>) at ../contrib/libcxx/include/memory:2582
17 0x000000000bbc8963 in Poco::Util::Application::run (this=this@entry=0x7ffd6e1bcc60) at ../contrib/poco/Util/src/Application.cpp:334
18 0x0000000004ffe1de in DB::Server::run (this=0x7ffd6e1bcc60) at ../programs/server/Server.cpp:178
19 0x0000000004ff36cc in mainEntryClickHouseServer (argc=3, argv=0x7f353be58bc0) at ../programs/server/Server.cpp:1060
20 0x0000000004f49b99 in main (argc_=<optimized out>, argv_=<optimized out>) at ../contrib/libcxx/include/vector:655
