thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-10-05 08:00:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
43,544 Commits 8,978 Branches 1,768 Tags 1.9 GiB
c39eb8f71b
Commit Graph

55 Commits

Author SHA1 Message Date
alesapin
ebb36bec8a Merge branch 'master' into atomic_metadata5 2020-06-18 11:57:16 +03:00
alesapin
1ddeb3d149 Buildable getSampleBlock in StorageInMemoryMetadata 2020-06-16 18:51:29 +03:00
Azat Khuzhin
c139a05370 Forward declaration in StorageDistributed 2020-06-14 01:09:21 +03:00
Azat Khuzhin
d2383f0f5d Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication 2020-06-08 21:58:56 +03:00
Azat Khuzhin
86c5465bf8 Rewrite StorageSystemDistributionQueue interfaces 2020-06-04 03:04:32 +03:00
Azat Khuzhin
f0050adc51 Make system.distribution_queue metrics non racy 2020-06-04 02:36:16 +03:00
Azat Khuzhin
09c3ca9c6c Add last_exception into system.distribution_queue 2020-06-04 02:36:16 +03:00
Azat Khuzhin
389f78ceee Add system.distribution_queue 
system.distribution_queue contains the following columns:
- database
- table
- data_path
- is_blocked
- error_count
- data_files
- data_compressed_bytes
 2020-06-04 02:36:16 +03:00
alesapin
6253e9b97b Revert disabled tests 2020-06-02 21:41:29 +03:00
Alexey Milovidov
25f941020b Remove namespace pollution 2020-05-31 00:57:37 +03:00
Alexey Milovidov
146370934a Keep the value of DistributedFilesToInsert metric on exceptions 2020-05-27 13:07:38 +03:00
Alexey Milovidov
7e1813825b Return old names of macros 2020-05-24 01:24:01 +03:00
Alexey Milovidov
ce0619dabf Progress on task 2020-05-24 00:26:45 +03:00
Alexey Milovidov
2d7d5a1547 Apply all transformations again 2020-05-24 00:16:27 +03:00
Alexey Milovidov
bab24879e9 Progress on task 2020-05-24 00:16:05 +03:00
Alexey Milovidov
e1695feb7f Apply all transformations again 2020-05-23 23:40:32 +03:00
Alexey Milovidov
85f84550ba Progress on task 2020-05-23 23:37:37 +03:00
Alexey Milovidov
7e2fb9ad65 Apply all transformations again 2020-05-23 22:38:30 +03:00
Alexey Milovidov
eacff92d0e Progress on task 2020-05-23 22:35:08 +03:00
Alexey Milovidov
29762240de Remove duplicate whitespaces (preparation) 2020-05-23 22:31:54 +03:00
Alexey Milovidov
7fed65cbe2 Remove duplicate whitespaces (preparation) 2020-05-23 22:14:58 +03:00
Alexey Milovidov
ab0562a574 Make all LOG in single line (preparation) 2020-05-23 22:05:41 +03:00
Alexey Milovidov
a2ad11897f Remove duplicate whitespaces (preparation) 2020-05-23 21:53:58 +03:00
Alexey Milovidov
1f13515a65 Make all LOG in single line (preparation) 2020-05-23 21:31:37 +03:00
Alexey Milovidov
8042e5febe find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}\5{}", \4, \6);/' 2020-05-23 19:58:15 +03:00
Alexey Milovidov
e391b77d81 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/' 2020-05-23 19:56:05 +03:00
Alexey Milovidov
8d2e80a5e2 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+"\)' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+, "[^"]+")\)/\1_FORMATTED(\2)/' 2020-05-23 19:42:39 +03:00
Azat Khuzhin
d93b9a57f6 Forward declaration for Context as much as possible. 
Now after changing Context.h 488 modules will be recompiled instead of 582.
 2020-05-21 01:53:18 +03:00
alexey-milovidov
7cf3538840
Merge pull request #10270 from ClickHouse/quota-key-in-client 
Support quota_key for Native client
 2020-05-17 14:09:40 +03:00
alexey-milovidov
7ee35f102d
Merge pull request #10867 from azat/dist-INSERT-load-balancing 
Respect prefer_localhost_replica/load_balancing on INSERT into Distributed
 2020-05-17 11:11:35 +03:00
Alexey Milovidov
397859ccb8 Fix error 2020-05-17 08:45:20 +03:00
Azat Khuzhin
2498041cc1 Avoid sending partially written files by the DistributedBlockOutputStream 2020-05-16 01:00:42 +03:00
Azat Khuzhin
52d73c7f45 Fix prefer_localhost_replica=0 and load_balancing for Distributed INSERT 2020-05-14 03:29:03 +03:00
Azat Khuzhin
e1d4837753 Fix list of possible nodes for Distributed INSERT for internal_replication=0 2020-05-14 03:28:08 +03:00
Azat Khuzhin
cdf3845e43 Respect load_balancing in DirectoryMonitor, to fix w/o internal_replication 2020-05-14 01:33:25 +03:00
Azat Khuzhin
085bafad05 Handle prefer_localhost_replica on INSERT into Distributed 
Right now it will issue remote send even if finally the local replica
will be selected - not good I guess.

This should also fix load_balancing.
 2020-05-13 01:38:03 +03:00
Azat Khuzhin
889f54b549 Fix ENOENT exception on current_batch.txt in DirectoryMonitor 
current_batch.txt will not exist if there was no send, this is the case
when all batches that was pending has been marked as pending.
 2020-05-13 01:23:18 +03:00
alexey-milovidov
ddc84163a7
Merge pull request #10486 from azat/dist-send-on-INSERT 
Fix distributed send that are scheduled by INSERT query
 2020-05-11 06:28:35 +03:00
Azat Khuzhin
5c89cdbe61 Fix distributed send retries on distributed_directory_monitor_{max_,}sleep_time_ms > 5min 
In this case error_count can be decreased before checking it for
rescheduling send.

And actually this can be a problem not only when
distributed_directory_monitor_{max_,}sleep_time_ms > 5min, because all
threads can be occupated and the real timeout between sends will be > 5min.
 2020-05-10 12:37:38 +03:00
Gleb Novikov
c637d99e07 Volumes and storages refactoring: 
1. Moved Volume to separate file
  2. Created IVolume interface and implemented current behaviour in implementation of new interface — VolumeJBOD
  3. Replaced all old volume usages with new VolumeJBOD. Where it is unnecessary to have JBOD — left just IVolume.
  4. Removed old Volume completely
  5. Moved StoragePolicy to separated files
  6. Moved DiskSelector to separated files
  7. Removed DiskSpaceMonitor file
 2020-05-04 23:15:38 +03:00
Azat Khuzhin
6ffdd53b6a Share auto-increment for first batch and tmp file in DistributedBlockOutputStream 2020-05-03 14:47:59 +03:00
Azat Khuzhin
53c470cab4 Fix directory monitor initialization from INSERT into Distributed 
This also fixes hardlink code (when one file should be sent to multiple
servers, i.e. internal_replication == false) of writeToShard() with
distributed_storage_policy (i.e. when StorageDistributed::getPath() will
path to different filesystems).

Plus also cleanup DistributedBlockOutputStream::writeToShard() a little.
 2020-05-03 14:47:51 +03:00
Azat Khuzhin
e97e1f06db Do not schedule distributed send if there were no error 
Since in this case it will be scheduled from the
DistributedBlockOutputStream with the
distributed_directory_monitor_max_sleep_time_ms, and this will overwrite
timer that was set by the DistributedBlockOutputStream, not good.
 2020-05-03 14:46:44 +03:00
Azat Khuzhin
947b3942dd Schedule distributed sends after the file has been written 2020-05-03 14:46:43 +03:00
Azat Khuzhin
0157fd5d93 Fix distributed send that are scheduled by INSERT query 
Before this patch each INSERT query re-schedule distributed send, thus
each time it resets the timer, while this is not the expected behaviour,
since in on frequent INSERT distributed sends will not be triggered at
all.

Fix this by not resetting the timer.
 2020-05-03 14:46:42 +03:00
Azat Khuzhin
6bb39dafc3 Drop decreated code (cond var and note for thread) in DirectoryMonitor 2020-05-03 14:46:41 +03:00
Azat Khuzhin
63d8ab8f03 Make createSelector() static (in storage) and const (in stream) 2020-05-01 11:31:05 +03:00
Azat Khuzhin
f22ba15b4a Reduce copy-paste of DistributedBlockOutputStream::createSelector 
This will make it less error prone.
 2020-05-01 02:59:40 +03:00
Alexey Milovidov
1e325a9fd9 Checkpoint 2020-04-22 09:22:14 +03:00
Azat Khuzhin
5d11118cc9 Use thread pool (background_distributed_schedule_pool_size) for distributed sends 
After #8756 the problem with 1 thread for each (distributed table, disk)
for distributed sends became even worse (since there can be multiple
disks), so use predefined thread pool for this tasks, that can be
controlled with background_distributed_schedule_pool_size knob.
 2020-04-19 12:01:56 +03:00