Commit Graph

322 Commits

Author SHA1 Message Date
Alexander Tokmakov
d1568c1614 fix rename distributed 2019-12-19 22:51:12 +03:00
Azat Khuzhin
81aeff2d2a Add _shard_num virtual column for the Distributed engine
With JOIN from system.clusters one can figure out from which server data
came.

TODO:
- optimization to avoid communicating with unrelated shards (for queries
  like "AND _shard_num = n")
- fix aliases (see tests with serverError expected)

v0: AddingConstColumnBlockInputStream
v2: VirtualColumnUtils::rewriteEntityInAst
v3: fix remote(Distributed) by appending _shard_num only if has been requested
2019-11-06 15:26:12 +03:00
Alexander Kuzmenkov
29052b6a37
Merge pull request #7377 from azat/INSERT-Distributed-MATERIALIZED-cols
* Fix INSERT into Distributed non local node with MATERIALIZED columns

Previous patch e527def18a ("Fix INSERT
into Distributed() table with MATERIALIZED column") fixes it only for
cases when the node is local, i.e. direct insert.

This patch address the problem when the node is not local
(`is_local == false`), by erasing materialized columns on INSERT into
Distributed.

And this patch fixes two cases, depends on `insert_distributed_sync`
setting:

- `insert_distributed_sync=0`

    ```
    Not found column value in block. There are only columns: date. Stack trace:

    2. 0x7ffff7be92e0 DB::Exception::Exception() dbms/src/Common/Exception.h:27
    3. 0x7fffec5d6cf6 DB::Block::getByName(...) dbms/src/Core/Block.cpp:187
    4. 0x7fffec2fe067 DB::NativeBlockInputStream::readImpl() dbms/src/DataStreams/NativeBlockInputStream.cpp:159
    5. 0x7fffec2d223f DB::IBlockInputStream::read() dbms/src/DataStreams/IBlockInputStream.cpp:61
    6. 0x7ffff7c6d40d DB::TCPHandler::receiveData() dbms/programs/server/TCPHandler.cpp:971
    7. 0x7ffff7c6cc1d DB::TCPHandler::receivePacket() dbms/programs/server/TCPHandler.cpp:855
    8. 0x7ffff7c6a1ef DB::TCPHandler::readDataNext(unsigned long const&, int const&) dbms/programs/server/TCPHandler.cpp:406
    9. 0x7ffff7c6a41b DB::TCPHandler::readData(DB::Settings const&) dbms/programs/server/TCPHandler.cpp:437
    10. 0x7ffff7c6a5d9 DB::TCPHandler::processInsertQuery(DB::Settings const&) dbms/programs/server/TCPHandler.cpp:464
    11. 0x7ffff7c687b5 DB::TCPHandler::runImpl() dbms/programs/server/TCPHandler.cpp:257
    ```

- `insert_distributed_sync=1`

    ```
    2019.10.18 13:23:22.114578 [ 44 ] {a78f669f-0b08-4337-abf8-d31e958f6d12} <Error> executeQuery: Code: 171, e.displayText() = DB::Exception: Block structure mismatch in RemoteBlockOutputStream stream: different number of columns:
    date Date UInt16(size = 1), value Date UInt16(size = 1)
    date Date UInt16(size = 0): Insertion status:
    Wrote 1 blocks and 0 rows on shard 0 replica 0, 127.0.0.1:59000 (average 0 ms per block)
    Wrote 0 blocks and 0 rows on shard 1 replica 0, 127.0.0.2:59000 (average 2 ms per block)
     (version 19.16.1.1) (from [::1]:3624) (in query: INSERT INTO distributed_00952 VALUES ), Stack trace:

    2. 0x7ffff7be92e0 DB::Exception::Exception() dbms/src/Common/Exception.h:27
    3. 0x7fffec5da4e9 DB::checkBlockStructure<void>(...)::{...}::operator()(...) const dbms/src/Core/Block.cpp:460
    4. 0x7fffec5da671 void DB::checkBlockStructure<void>(...) dbms/src/Core/Block.cpp:467
    5. 0x7fffec5d8d58 DB::assertBlocksHaveEqualStructure(...) dbms/src/Core/Block.cpp:515
    6. 0x7fffec326630 DB::RemoteBlockOutputStream::write(DB::Block const&) dbms/src/DataStreams/RemoteBlockOutputStream.cpp:68
    7. 0x7fffe98bd154 DB::DistributedBlockOutputStream::runWritingJob(DB::DistributedBlockOutputStream::JobReplica&, DB::Block const&)::{lambda()#1}::operator()() const dbms/src/Storages/Distributed/DistributedBlockOutputStream.cpp:280
    <snip>
    ````

Fixes: #7365
Fixes: #5429
Refs: #6891

* Cover INSERT into Distributed with MATERIALIZED columns and !is_local node

I guess that adding new cluster into server-test.xml is not required,
but it won't harm.

* Update DistributedBlockOutputStream.cpp
2019-10-24 12:35:09 +03:00
Amos Bird
295864e6e0
better scalar query 2019-10-23 21:37:54 +08:00
Azat Khuzhin
260a9fba2b StorageDistributed: Drop extra inclusion of materializeBlock.h 2019-10-18 23:38:33 +03:00
Alexey Milovidov
285d66cee1 Improve table locks (incomplete) 2019-08-27 23:43:08 +03:00
alexey-milovidov
561b173ddc
Merge pull request #6675 from yandex/simplify-shared-ptr-helper
Simplify shared_ptr_helper
2019-08-27 00:36:52 +03:00
Alexey Milovidov
0eec226c9a Simplify shared_ptr_helper 2019-08-26 22:07:29 +03:00
alesapin
cf31187254 Resolve conflicts 2019-08-26 21:08:58 +03:00
alesapin
0d0ebf885f Remove two redundant arguments from alter 2019-08-26 17:50:34 +03:00
Alexey Milovidov
479b4d81f3 Fixed error 2019-08-26 16:46:07 +03:00
Alexey Milovidov
60d660e263 Merge branch 'master' into merge-constraints 2019-08-26 16:13:44 +03:00
alesapin
062034dfd8 Eventually merged with master 2019-08-26 13:07:32 +03:00
Alexey Milovidov
d798cbde2f Fixed double whitespaces 2019-08-25 01:28:06 +03:00
Alexey Milovidov
3fef8e92ad Merging CONSTRAINTS support 2019-08-25 00:20:20 +03:00
alesapin
70ca7f4156 Merge with master 2019-08-21 11:57:35 +03:00
alexey-milovidov
308baa182e
Merge branch 'master' into table-constraints 2019-08-21 06:33:08 +03:00
alexey-milovidov
3f607b21fe
Merge pull request #6375 from yandex/try-to-remove-dry-run
Try to remove dry run
2019-08-21 03:52:41 +03:00
Alexander Kazakov
fe5cd47068
Process prewhere clause in "skip unused shards" optimization (#6521)
* Process prewhere clause in optimize_skip_unused_shards

* Better diagnostics and logging
2019-08-19 23:28:24 +03:00
Nikolai Kochetov
287eb633b8 Merged with master 2019-08-14 18:16:52 +03:00
Nikolai Kochetov
0635188104 Allow constants for RemoteBlockInputStream. 2019-08-14 18:14:32 +03:00
alesapin
428c753ed7 Merge with master 2019-08-13 11:37:47 +03:00
Gleb Novikov
381ea6cb6e Merge branch 'master' into table-constraints 2019-08-11 12:38:34 +03:00
kreuzerkrieg
112fc71276 adding -Wshadow for GCC 2019-08-09 23:58:16 +03:00
Nikolai Kochetov
a1023bebff Fix tests. 2019-08-09 16:37:42 +03:00
alesapin
2803fcc2ba Make atomic settings 2019-08-09 16:07:19 +03:00
alesapin
e0d18c0fe8 Merge branch 'master' into merge_tree_settings_alter 2019-08-08 18:16:22 +03:00
CurtizJ
7e22f78ce4 fix segfault with enabled 'optimize_skip_unused_shards' and missing sharding key 2019-08-07 18:47:25 +03:00
alesapin
52b9658bcf Merge with master 2019-08-06 15:55:12 +03:00
Alexey Milovidov
f3921ce3e0 Changed boost::filesystem to std::filesystem 2019-08-01 01:37:41 +03:00
alesapin
d9a2ac123a Fix small typo and merge 2019-07-29 12:17:37 +03:00
chertus
9f9ba3a06f restore cropped names in JOIN ON section for distributed queries 2019-07-26 20:43:42 +03:00
alesapin
8a00ce5ff1 Intermediate step 2019-07-24 15:56:39 +03:00
Ivan Lezhankin
9f01fa7aaa Implement getDatabaseName() and rename() for all storages 2019-07-09 18:40:21 +03:00
Gleb Novikov
a146b7383e Merge branch 'master' into table-constraints 2019-06-30 11:32:02 +02:00
alesapin
fd3abbe93d
Merge pull request #4935 from zhang2014/feature/support_system_replicas
Support system replicas queries for distributed
2019-06-17 15:30:30 +03:00
Gleb Novikov
e7293486bd Added ALTER TABLE support to constraints (ADD CONSTRAINT, DROP CONSTRAINT) 2019-06-02 17:41:12 +03:00
Vitaly Baranov
421c6a90e2
Use function getSampleBlockNonMaterialized() to simplify the solution 2019-05-31 21:31:09 +03:00
zhang2014
80788cd7a8 Rename system sync distributed to system flush distributed 2019-05-29 10:43:52 +08:00
zhang2014
1a33840964 fix review 2019-05-29 10:43:52 +08:00
zhang2014
c44f608868 support replicas system queries for distributed 2019-05-29 10:43:52 +08:00
Azat Khuzhin
e527def18a
Fix INSERT into Distributed() table with MATERIALIZED column
By just skipping MATERIALIZED columns during processing.

P.S. you cannot use insert_allow_materialized_columns since it works
only for Buffer() engine.

Fixes: #4015
Fixes: #3673
Fixes: 01501fa8db ("correct column list
for rewritten INSERT query into Distributed [#CLICKHOUSE-4161]")
2019-05-27 23:32:05 +03:00
alexey-milovidov
8ef7f3589a
Merge pull request #4936 from TCeason/feature/support_settings_for_async
Suport settings for async distributed inserts (#4852)
2019-05-09 23:59:29 +03:00
Alexey Milovidov
270a31abb5 Rename method for consistency 2019-05-02 19:07:23 +03:00
tai
0d39968233 Suport settings for async(#4852)
There is no hash operation on Settings. Because it is less valuable for hash operations.

And I dont know how to write a testcase.
2019-04-29 17:02:23 +08:00
chertus
82dc1fb1db rewrite ASTSelectQuery (consistent children) 2019-04-09 17:22:35 +03:00
chertus
b6e2697101 Merge branch 'master' into ast 2019-03-18 13:36:13 +03:00
Ivan Lezhankin
61954d855d More reverts 2019-03-15 20:09:14 +03:00
chertus
ffc3974934 SelectQueryOptions v2 2019-03-15 18:57:18 +03:00
chertus
53abdccb95 refactoring: extract SelectQueryOptions 2019-03-15 16:49:58 +03:00