kssenii
9ec92ec514
Fix tests, less manual concatination of paths
2021-05-05 18:39:30 +03:00
Anton Popov
aea93d9ae5
Merge remote-tracking branch 'upstream/master' into HEAD
2021-04-20 15:16:12 +03:00
Azat Khuzhin
2561a67fd8
Replace !__clang__ with !defined(__clang) to fix gcc builds
...
$ gg 'if !__clang__' | cut -d: -f1 | sort -u | xargs sed -i 's/#if !__clang__/#if !defined(__clang__)/g'
2021-04-18 23:37:50 +03:00
Anton Popov
2afa1590e0
ColumnSparse: fix MergeTree in old syntax
2021-04-17 04:06:59 +03:00
Anton Ivashkin
09379b2b8a
Fix Zero-Copy replication with several S3 volumes (issue 22679)
2021-04-16 12:34:48 +03:00
Anton Popov
6ce875175b
Merge remote-tracking branch 'upstream/master' into HEAD
2021-04-16 02:08:20 +03:00
Anton Popov
298251e55d
fix merges with sparse columns and disable sparse for some data types
2021-04-12 02:33:53 +03:00
Anton Popov
d46958a8d2
Merge remote-tracking branch 'upstream/master' into HEAD
2021-04-06 00:54:49 +03:00
alesapin
8d5a787f6b
Merge branch 'master' into merge_tree_deduplication
2021-04-05 10:40:03 +03:00
Alexey Milovidov
8b5d0a598f
Minor improvement in index deserialization
2021-04-04 12:17:54 +03:00
alesapin
0204f5dd35
Merge branch 'master' into merge_tree_deduplication
2021-04-03 15:24:26 +03:00
alesapin
a555d078a2
Add exception handling
2021-04-02 19:56:02 +03:00
Mike Kot
c947280dfc
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-04-01 21:38:51 +03:00
alesapin
c15d7e009d
Some initial code
2021-03-31 18:20:30 +03:00
Anton Popov
372a1b1fe7
Merge remote-tracking branch 'upstream/master' into HEAD
2021-03-29 19:57:49 +03:00
Anton Popov
577d571300
ColumnSparse: initial implementation
2021-03-29 19:54:24 +03:00
Anton Popov
ea82e7725f
Merge pull request #21562 from CurtizJ/serialization-refactoring-4
...
Refactoring of data types serialization
2021-03-29 16:36:44 +03:00
alexey-milovidov
f895bc895c
Merge pull request #22011 from ClickHouse/min_max_time_system_parts_datetime64
...
Expose DateTime64 minmax part index in system.parts and system.parts_columns
2021-03-25 16:02:33 +03:00
Anton Popov
6a15431be7
Merge remote-tracking branch 'upstream/master' into HEAD
2021-03-25 15:57:35 +03:00
Mike Kot
285af08949
Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local
2021-03-24 22:34:20 +03:00
alexey-milovidov
612d4fb073
Update IMergeTreeDataPart.cpp
2021-03-24 02:03:14 +03:00
Pavel Kovalenko
a92cf30b67
Code review fixes.
2021-03-23 13:33:07 +03:00
Alexey Milovidov
8d0210b510
Expose DateTime64 minmax part index in system.parts and system.parts_columns #18244
2021-03-23 01:16:41 +03:00
Mike Kot
c55a73b752
Added the solution to handle the corruption case
...
When the part data (e.g. data.bin) is corrupted, but the checksums.txt
is present -- explicitly deleting the checksums.txt.
Removed the extra logging, changes some exceptions message.
2021-03-22 17:23:43 +03:00
Anton Popov
173d2ea1f4
Merge remote-tracking branch 'upstream/master' into HEAD
2021-03-16 02:50:14 +03:00
Anton Popov
1b07d28043
fix unwanted changes
2021-03-13 02:59:42 +03:00
Anton Popov
bc417cf54a
refactoring of serializations
2021-03-09 17:46:52 +03:00
alesapin
5b3161e0b5
Get rid of const_cast
2021-03-05 20:24:06 +03:00
Anton Ivashkin
d08b481660
Fixes by review responces
2021-03-05 19:20:38 +03:00
Anton Ivashkin
e69124a0a6
Merge master
2021-03-04 13:26:40 +03:00
alesapin
9ebf1b4fad
Get rid of separate minmax index fields
2021-03-02 13:33:54 +03:00
Anton Ivashkin
c891cf4557
Fixes by review response
2021-02-26 12:48:57 +03:00
Anton Ivashkin
4d44d75bc7
Fix build after merge one more time
2021-02-08 14:45:10 +03:00
Anton Ivashkin
e64c63c611
Merge master
2021-02-05 20:10:06 +03:00
Anton Ivashkin
df6c882aab
Fix build after merge
2021-02-05 18:52:40 +03:00
Anton Popov
a8f3078ce9
Merge remote-tracking branch 'upstream/master' into HEAD
2021-01-27 19:48:55 +03:00
Anton Popov
c7070da85a
better abstractions in disk interface
2021-01-26 17:49:35 +03:00
Anton Popov
658f24dcff
Merge pull request #19358 from CurtizJ/fix-subcolumns
...
Fix several cases, while reading subcolumns
2021-01-25 20:26:07 +03:00
Azat Khuzhin
cb951c2116
Add metrics for MergeTree parts types
...
- PartsWide
- PartsCompact
- PartsInMemory
2021-01-21 21:17:00 +03:00
Anton Popov
ac3de63a71
fix several cases, while reading subcolumns
2021-01-21 15:34:11 +03:00
Anton Ivashkin
357d98eb36
Merge master
2021-01-20 12:23:03 +03:00
Anton Ivashkin
eba98b04b0
Zero copy replication over S3: Hybrid storage support
2021-01-18 19:16:45 +03:00
Alexey Milovidov
029302d766
Merge with master
2021-01-16 17:09:44 +03:00
Alexey Milovidov
24c8e53440
Merge branch 'master' into multiple-nested
2021-01-16 16:28:40 +03:00
alexey-milovidov
2e2988e5d8
Merge pull request #19146 from azat/server-memory-limit-blocking
...
MemoryTracker: Do not ignore server memory limits during blocking by default
2021-01-16 11:09:19 +03:00
alexey-milovidov
5f189c5756
Merge pull request #19122 from ClickHouse/data-part-better-code
...
Add metrics for part number in MergeTree in ClickHouse
2021-01-16 00:20:15 +03:00
Azat Khuzhin
61b2d0ce42
MemoryTracker: Do not ignore server memory limits during blocking by default
2021-01-15 22:46:58 +03:00
alexey-milovidov
b97beea22a
Merge pull request #19101 from ClickHouse/check_compression_codec_read
...
Fix compression codec read for empty files
2021-01-15 20:55:58 +03:00
Alexey Milovidov
e238fd64ac
Add part metrics
2021-01-15 15:28:53 +03:00
Alexey Milovidov
6a2a5e53ed
Slightly better code of IMergeTreeDataPart #18955
2021-01-15 15:15:13 +03:00
alesapin
e106df2ad0
Fix comment
2021-01-15 12:10:03 +03:00
alesapin
0662d6bd7d
Fix compression codec read for empty files
2021-01-15 12:04:23 +03:00
Alexey Milovidov
8276a1c8d2
Faster parts removal, more safe and efficient interface of IDisk
2021-01-14 19:24:13 +03:00
Anton Popov
0e903552a0
fix TTLs with WHERE
2021-01-13 17:04:27 +03:00
Anton Popov
d7200ee2ed
minor changes
2021-01-13 02:20:32 +03:00
Anton Popov
15ead18673
Merge remote-tracking branch 'upstream/master' into HEAD
2021-01-12 19:46:10 +03:00
Anton Popov
5822ee1f01
allow multiple rows TTL with WHERE expression
2021-01-12 02:07:21 +03:00
Anton Popov
36ae0e4d35
Merge remote-tracking branch 'upstream/master' into HEAD
2021-01-11 13:51:12 +03:00
Azat Khuzhin
b1f08f5c27
Rename FileSyncGuard to DirectorySyncGuard
2021-01-07 20:26:18 +03:00
Azat Khuzhin
513a824f30
Fix fsync_part_directory for parts renames
2021-01-07 19:30:25 +03:00
Alexey Milovidov
36e1361cf8
Miscellaneous
2021-01-07 15:29:34 +03:00
Anton Popov
11283e3d81
Merge remote-tracking branch 'upstream/master' into HEAD
2020-12-25 21:25:59 +03:00
Anton Popov
b60c00ba74
refactoring of TTL stream
2020-12-25 18:46:13 +03:00
alesapin
54455b4740
Add test for already working code
2020-12-23 14:53:49 +03:00
Anton Popov
a42b00c9aa
Merge remote-tracking branch 'upstream/master' into HEAD
2020-12-17 20:43:23 +03:00
Anton Ivashkin
0f0500ca0c
Merge master
2020-12-16 18:31:13 +03:00
alesapin
b307e545a9
Fix check
2020-12-09 14:46:04 +03:00
alesapin
0f4056fd95
Add additional size check in debug mode
2020-12-09 14:23:37 +03:00
Anton Popov
66e0add2ba
fix nested
2020-12-07 16:35:12 +03:00
Anton Popov
b384beb564
Merge remote-tracking branch 'upstream/master' into HEAD
2020-11-23 17:46:51 +03:00
Nicolae Vartolomei
040aba9f85
Add uuid.txt to checksums for parts stored on disk
...
We are breaking backwards compatibility anyway (but agted by a setting)
2020-11-20 13:49:17 +00:00
Nicolae Vartolomei
425dc4b11b
Add unique identifiers IMergeTreeDataPart structure
...
For now uuids are not generated at all, they are present only if the
part is updated manually (as you can see in the integration test).
The only place where they can be seen today by an end user is in
`system.parts` table. I was looking for hiding this column behind an
option but couldn't find an easy way to do that.
Likely this is also required for WAL, but need to think how not to break
compatibility.
Relates to #13574 , https://github.com/ClickHouse/ClickHouse/issues/13574
Next 1: In the upcoming PR the plan is to integrate de-duplication based on
these fingerprints in the query pipeline.
Next 2: We'll enable automatic generation of uuids and come up with a
way for conditionally sending uuids when processing distributed queries
only when part movement is in progress.
2020-11-19 13:14:25 +00:00
Anton Popov
245c395a68
Merge remote-tracking branch 'upstream/master' into HEAD
2020-11-06 22:00:32 +03:00
Anton Ivashkin
1742fb3256
Merge master
2020-11-03 12:27:16 +03:00
Anton Ivashkin
78021714f1
S3 zero copy replication: more simple s3 check
2020-11-03 12:20:26 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell
2020-10-27 12:04:03 +01:00
Anton Popov
a249f0c95e
Merge remote-tracking branch 'upstream/master' into HEAD
2020-10-23 22:05:00 +03:00
Anton Ivashkin
e3879afa69
S3 zero copy replication: fix virtual method default parameter
2020-10-23 16:27:50 +03:00
Anton Ivashkin
652c56e74e
Fix style, fix build
2020-10-22 16:07:20 +03:00
Anton Ivashkin
f833501c77
Merge branch 'master' into s3_zero_copy_replication
2020-10-22 11:17:21 +03:00
Alexey Milovidov
2619efadc8
Fix multiple issues with memory tracking
2020-10-21 03:31:34 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration ( #13956 )
...
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-10-20 18:10:24 +03:00
Anton Ivashkin
14a78f87b0
Zero copy replication over S3: fetch instead of merge
2020-10-15 19:14:47 +03:00
Pavel Kovalenko
ed61c5681b
Use 'moving' directory instead of 'detached' when move part to another disk/volume.
2020-10-15 16:55:13 +03:00
Anton Popov
cbe12a532e
allow to extract subcolumns from column
2020-10-13 22:39:22 +03:00
Anton Ivashkin
9272ed06b4
Move Zookeeper lock for S3 shared part in IMergeTreeDataPart
2020-10-09 17:24:10 +03:00
Anton Ivashkin
766dbfd2be
Merge master
2020-10-09 15:00:14 +03:00
Anton Ivashkin
acf86568a7
S3 zero copy replication proof of concept
2020-10-08 18:45:10 +03:00
Pavel Kovalenko
69c126f1f1
Possibility to move part to another disk/volume if first attempt was failed.
2020-10-07 14:35:28 +03:00
Pavel Kovalenko
69b4bc6f33
Proper error handling during insert into MergeTree with S3.
2020-10-06 12:38:00 +03:00
Anton Popov
edc2fe2226
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-18 17:51:54 +03:00
Anton Popov
cb4801e3be
allow to read subcolumns of complex types
2020-09-18 02:12:43 +03:00
Anton Popov
3b9be70c8c
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-05 18:53:55 +03:00
alesapin
acc0ee0657
Apply TTL if it's not calculated for part
2020-09-03 11:59:41 +03:00
Anton Popov
4fe57fc169
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-02 01:29:44 +03:00
alesapin
067eb4599d
Fix empty columns case
2020-08-31 16:39:27 +03:00
alesapin
77faf9587f
Better interface
2020-08-28 12:07:20 +03:00
alesapin
d426751e26
Default codec if part is not stored on disk
2020-08-27 16:32:23 +03:00
alesapin
1225c09a02
Fix default codec for in-memory parts
2020-08-27 11:35:55 +03:00
alesapin
44eb702fbe
Remove redundant code
2020-08-26 22:41:57 +03:00
alesapin
3a680dc8fb
Fix removal bug
2020-08-26 22:40:04 +03:00
alesapin
1fb93a8bf2
Remove read helpers
2020-08-26 18:57:06 +03:00
alesapin
18eb0dbcaa
Better remove
2020-08-26 18:39:11 +03:00
alesapin
2fc80189af
Add default compression codec to merge tree data part
2020-08-26 18:29:46 +03:00
Anton Popov
4df5c72878
Merge remote-tracking branch 'upstream/master' into HEAD
2020-08-24 12:57:27 +03:00
alexey-milovidov
7b53a0ef33
Revert "Added allow_merges option for volumes in multi-disk configuration ( #13402 )"
...
This reverts commit 1e2616542a
2020-08-21 18:44:29 +03:00
Vladimir Chebotarev
1e2616542a
Added allow_merges option for volumes in multi-disk configuration ( #13402 )
2020-08-21 12:04:13 +03:00
alesapin
1707f84a44
Less strict check and rare rename bug
2020-07-27 12:42:37 +03:00
alesapin
78d357f0d2
Add a test for sticking mutations bug
2020-07-25 17:42:20 +03:00
alesapin
b3ec3b1ed1
Fix loading parts without checksums
2020-07-16 13:54:49 +03:00
Anton Popov
ca346ea13c
rename fsync-related settings
2020-07-03 02:41:37 +03:00
Anton Popov
53e955c6dd
several fixes
2020-06-29 23:36:18 +03:00
Anton Popov
b2aa565a37
allow to turn on fsync on inserts, merges and fetches
2020-06-27 17:52:16 +03:00
alesapin
6f1824f0ea
Correct merge with master
2020-06-26 14:30:23 +03:00
alesapin
e9c47dc89c
Merge branch 'master' into CurtizJ-polymorphic-parts
2020-06-26 14:27:19 +03:00
alesapin
dffdece350
getColumns in StorageInMemoryMetadta (only compilable)
2020-06-17 19:39:58 +03:00
alesapin
1afdebeebd
Primary key in storage metadata
2020-06-17 15:39:20 +03:00
alesapin
ab61abccc1
Partition key in StorageInMemoryMetadata
2020-06-17 13:34:23 +03:00
Anton Popov
df3dfd5b81
fix clang-tidy build
2020-06-04 01:00:02 +03:00
Anton Popov
11c4e9dde3
in-memory parts: fix 'check' query
2020-06-03 22:19:49 +03:00
Anton Popov
1ce09e1faa
Merge remote-tracking branch 'upstream/master' into polymorphic-parts
2020-06-03 16:27:54 +03:00
Anton Popov
caf1e4e8cc
in-memory-parts: fixes
2020-06-03 12:51:23 +03:00
Alexey Milovidov
b78c48310b
Whitespaces
2020-05-30 01:37:44 +03:00
Anton Popov
c919840722
in-memory parts: partition commands
2020-05-29 18:02:12 +03:00
Anton Popov
d8342e5b12
in-memory parts: better restore and clear stale wal files
2020-05-27 23:05:55 +03:00
alesapin
c7cda399c7
Merge branch 'master' into consistent_metadata2
2020-05-27 12:51:56 +03:00
alesapin
8614487a5a
Fix primary key check
2020-05-26 16:46:19 +03:00
Alexey Milovidov
c858f4d89c
Clarifications
2020-05-26 02:47:21 +03:00
alesapin
c3a6571036
Compilable code
2020-05-25 20:22:20 +03:00
alesapin
6281dd6893
Merge pull request #11115 from ClickHouse/consistent_metadata
...
Refactoring in storage metadata (more consistent keys)
2020-05-25 13:09:56 +03:00
Alexey Milovidov
7e1813825b
Return old names of macros
2020-05-24 01:24:01 +03:00
Alexey Milovidov
e1695feb7f
Apply all transformations again
2020-05-23 23:40:32 +03:00
Alexey Milovidov
85f84550ba
Progress on task
2020-05-23 23:37:37 +03:00
Alexey Milovidov
7e2fb9ad65
Apply all transformations again
2020-05-23 22:38:30 +03:00
Alexey Milovidov
eacff92d0e
Progress on task
2020-05-23 22:35:08 +03:00
Alexey Milovidov
29762240de
Remove duplicate whitespaces (preparation)
2020-05-23 22:31:54 +03:00
Alexey Milovidov
9d2a0d2dd7
Apply all transformations again
2020-05-23 21:59:49 +03:00
Alexey Milovidov
a2ad11897f
Remove duplicate whitespaces (preparation)
2020-05-23 21:53:58 +03:00
Alexey Milovidov
1f13515a65
Make all LOG in single line (preparation)
2020-05-23 21:31:37 +03:00
Alexey Milovidov
8042e5febe
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}\5{}", \4, \6);/'
2020-05-23 19:58:15 +03:00
Alexey Milovidov
ee4ffbc332
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}", \4);/'
2020-05-23 19:47:56 +03:00
alesapin
327d17ac6a
Better
2020-05-21 22:46:03 +03:00
alesapin
616902a995
Sorting and primary key (broken)
2020-05-20 21:11:38 +03:00
alesapin
8bc527eecd
Compile partition key
2020-05-20 15:16:55 +03:00
Gleb Novikov
1a25ac6e1f
Merge branch 'master' into refactor-reservations
2020-05-16 23:34:45 +03:00
alesapin
26606dd640
Fix polymorphic parts
2020-05-15 13:36:35 +03:00
Gleb Novikov
390b39b272
VolumePtr instead of DiskPtr in MergeTreeData*
2020-05-10 00:24:15 +03:00
Anton Popov
aa723d245f
Merge remote-tracking branch 'upstream/master' into polymorphic-parts
2020-04-30 18:27:39 +03:00
Anton Popov
4069dbcc58
in-memory parts: add waiting for insert
2020-04-20 04:38:38 +03:00
Anton Popov
391f7c34be
in memory parts: basic read/write
2020-04-17 20:30:46 +03:00
Alexey Milovidov
f704c465a7
Make aware of one special case of imcompletely written parts #9940
2020-04-16 00:52:21 +03:00
Anton Popov
79024d73a2
improve performance of index analysis with monotonic functions
2020-04-06 13:37:34 +03:00
Ivan Lezhankin
06446b4f08
dbms/ → src/
2020-04-03 18:14:31 +03:00
