Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on.
Following syntax variants are now supported:
OPTIMIZE TABLE table DEDUPLICATE; -- the old one
OPTIMIZE TABLE table DEDUPLICATE BY *;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT (colX, colY);
OPTIMIZE TABLE table DEDUPLICATE BY col1,col2,col3;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex');
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT (colX, colY);
Note that * behaves just like in SELECT: MATERIALIZED, and ALIAS columns are not used for expansion.
Also, it is an error to specify empty list of columns, or write an expression that results in an empty list of columns, or deduplicate by an ALIAS column.
Column transformers other than EXCEPT are not supported.
Materialized view drop the inner table (if any) drop the drop() method,
and this will lead to essure recursive drop() from the drop worker
thread, which will hang:
3 std::__1::condition_variable::wait<> ()
4 DB::DatabaseCatalog::waitTableFinallyDropped (this=0x7ffff7831500, uuid=...) at ../src/Interpreters/DatabaseCatalog.cpp:828
5 DB::InterpreterDropQuery::executeToTable (this=this@entry=0x7fff16ff88a0, table_id_=..., query=...) at ../src/Interpreters/InterpreterDropQuery.cpp:135
6 DB::InterpreterDropQuery::execute (this=this@entry=0x7fff16ff88a0) at ../contrib/libcxx/include/string:1474
7 DB::executeDropQuery (kind=kind@entry=DB::ASTDropQuery::Drop, global_context=..., target_table_id=...) at ../src/Storages/StorageMaterializedView.cpp:156
8 DB::StorageMaterializedView::drop (this=0x7ffefc348f40) at ../src/Storages/StorageMaterializedView.cpp:169
9 DB::DatabaseCatalog::dropTableFinally (this=this@entry=0x7ffff7831500, table=...) at ../src/Interpreters/DatabaseCatalog.cpp:775
10 DB::DatabaseCatalog::dropTableDataTask (this=0x7ffff7831500) at ../src/Interpreters/DatabaseCatalog.cpp:745
Fix this by dropping the inner table just before scheduling job the the
drop worker thread.
* "lock_acquire_timeout" controls for how long a query will continue to
acquire each lock on its argument tables
* "lock_acquire_timeout_for_background_operations" is a per-table
setting for storages of *MergeTree family
