mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-14 18:32:29 +00:00
171f5179e8
* CLICKHOUSE-4063: less manual html @ index.md * CLICKHOUSE-4063: recommend markdown="1" in README.md * CLICKHOUSE-4003: manually purge custom.css for now * CLICKHOUSE-4064: expand <details> before any print (including to pdf) * CLICKHOUSE-3927: rearrange interfaces/formats.md a bit * CLICKHOUSE-3306: add few http headers * Remove copy-paste introduced in #3392 * Hopefully better chinese fonts #3392 * get rid of tabs @ custom.css * Apply comments and patch from #3384 * Add jdbc.md to ToC and some translation, though it still looks badly incomplete * minor punctuation * Add some backlinks to official website from mirrors that just blindly take markdown sources * Do not make fonts extra light * find . -name '*.md' -type f | xargs -I{} perl -pi -e 's//g' {} * find . -name '*.md' -type f | xargs -I{} perl -pi -e 's/ sql/g' {} * Remove outdated stuff from roadmap.md * Not so light font on front page too * Refactor Chinese formats.md to match recent changes in other languages * Update some links on front page * Remove some outdated comment * Add twitter link to front page * More front page links tuning * Add Amsterdam meetup link * Smaller font to avoid second line * Add Amsterdam link to README.md * Proper docs nav translation * Back to 300 font-weight except Chinese * fix docs build * Update Amsterdam link * remove symlinks * more zh punctuation * apply lost comment by @zhang2014 * Apply comments by @zhang2014 from #3417 * Remove Beijing link * rm incorrect symlink * restore content of docs/zh/operations/table_engines/index.md * CLICKHOUSE-3751: stem terms while searching docs * CLICKHOUSE-3751: use English stemmer in non-English docs too * CLICKHOUSE-4135 fix * Remove past meetup link * Add blog link to top nav * Add ContentSquare article link * Add form link to front page + refactor some texts * couple markup fixes * minor * Introduce basic ODBC driver page in docs * More verbose 3rd party libs disclaimer * Put third-party stuff into a separate folder * Separate third-party stuff in ToC too * Update links * Move stuff that is not really (only) a client library into a separate page * Add clickhouse-hdfs-loader link * Some introduction for "interfaces" section * Rewrite tcp.md * http_interface.md -> http.md * fix link * Remove unconvenient error for now * try to guess anchor instead of failing * remove symlink * Remove outdated info from introduction * remove ru roadmap.md * replace ru roadmap.md with symlink * Update roadmap.md * lost file * Title case in toc_en.yml * Sync "Functions" ToC section with en * Remove reference to pretty old ClickHouse release from docs * couple lost symlinks in fa * Close quote in proper place * Rewrite en/getting_started/index.md * Sync en<>ru getting_started/index.md * minor changes * Some gui.md refactoring * Translate DataGrip section to ru * Translate DataGrip section to zh * Translate DataGrip section to fa * Translate DBeaver section to fa * Translate DBeaver section to zh * Split third-party GUI to open-source and commercial * Mention some RDBMS integrations + ad-hoc translation fixes * Add rel="external nofollow" to outgoing links from docs * Lost blank lines * Fix class name * More rel="external nofollow" * Apply suggestions by @sundy-li * Mobile version of front page improvements * test * test 2 * test 3 * Update LICENSE * minor docs fix * Highlight current article as suggested by @sundy-li * fix link destination * Introduce backup.md (only "en" for now) * Mention INSERT+SELECT in backup.md * Some improvements for replication.md * Add backup.md to toc * Mention clickhouse-backup tool * Mention LightHouse in third-party GUI list * Introduce interfaces/third-party/proxy.md * Add clickhouse-bulk to proxy.md * Major extension of integrations.md contents * fix link target * remove unneeded file * better toc item name * fix markdown * better ru punctuation * Add yet another possible backup approach * Simplify copying permalinks to headers * Support non-eng link anchors in docs + update some deps * Generate anchors for single-page mode automatically * Remove anchors to top of pages * Remove anchors that nobody links to * build fixes * fix few links * restore css * fix some links * restore gifs * fix lost words * more docs fixes * docs fixes * NULL anchor * update urllib3 dependency * more fixes * Remove excessive content from print version * Try short license again * Back to long license for now * Introduce anchor integrity checks for single-page docs * Add --save-raw-single-page option to build.py (helps to debug incorrect anchors) * fix kafka engine links * fix one class of broken anchors * fix some broken links * Add https://github.com/hatarist/clickhouse-cli to third-party section (in gui.md for now, maybe will add cli.md later) * fix one more class of links to nowhere * less duplicate anchors * get rid of weird anchors * fix anchor * fix link * fix couple links * rearrange integrations.md a bit + sync zh version * Mention nagios plugin in other languages * port summingmergetree.md fix to zh * Make doc links to nowhere fatal * additional check in markdown extension * add option to skip pdf * 2019 have started
650 lines
26 KiB
HTML
650 lines
26 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="ru">
|
||
<head>
|
||
<meta charset="utf-8"/>
|
||
<title>ClickHouse Quick Start Guide</title>
|
||
|
||
<link rel="shortcut icon" href="favicon.ico"/>
|
||
|
||
<meta name="description" content="Quick start guide to ClickHouse — open-source distributed column-oriented DBMS"/>
|
||
<meta name="keywords"
|
||
content="tutorial, ClickHouse, DBMS, OLAP, relational, analytics, analytical, big data, open-source, SQL, web-analytics"/>
|
||
|
||
<link href="index.css" media="all" rel="stylesheet" />
|
||
</head>
|
||
<body>
|
||
|
||
<div class="page">
|
||
|
||
<div>
|
||
<div id="tutorial_logo">
|
||
<a href="/">
|
||
<svg xmlns="http://www.w3.org/2000/svg" width="90" height="80" viewBox="0 0 9 8">
|
||
<path class="red" d="M0,7 h1 v1 h-1 z"></path>
|
||
<path class="orange" d="M0,0 h1 v7 h-1 z"></path>
|
||
<path class="orange" d="M2,0 h1 v8 h-1 z"></path>
|
||
<path class="orange" d="M4,0 h1 v8 h-1 z"></path>
|
||
<path class="orange" d="M6,0 h1 v8 h-1 z"></path>
|
||
<path class="orange" d="M8,3.25 h1 v1.5 h-1 z"></path>
|
||
</svg>
|
||
</a>
|
||
</div>
|
||
|
||
<h1 id="tutorial_title">ClickHouse</h1>
|
||
<h2 id="tutorial_subtitle">Tutorial</h2>
|
||
</div>
|
||
|
||
<p>Let's get started with sample dataset from open sources. We will use USA civil flights data since 1987 till 2015.
|
||
It's hard to call this sample a Big Data (contains 166 millions rows, 63 Gb of uncompressed data) but this
|
||
allows us to quickly get to work. Dataset is available for download <a href="https://yadi.sk/d/pOZxpa42sDdgm">here</a>.
|
||
Also you may download it from the original datasource <a
|
||
href="docs/en/getting_started/example_datasets/ontime/"
|
||
rel="external nofollow">as described here</a>.</p>
|
||
|
||
<p>Firstly we will deploy ClickHouse to a single server. Below that we will also review the process of deployment to
|
||
a cluster with support for sharding and replication.</p>
|
||
|
||
<p>On Ubuntu and Debian Linux ClickHouse can be installed from <a href="/#quick-start">packages</a>.
|
||
For other Linux distributions you can <a href="docs/en/development/build/"
|
||
rel="external nofollow">compile
|
||
ClickHouse from sources</a> and then install.</p>
|
||
|
||
<p><b>clickhouse-client</b> package contains <a
|
||
href="docs/en/interfaces/cli/">clickhouse-client</a> application —
|
||
interactive ClickHouse client. <b>clickhouse-common</b> contains a clickhouse-server binary file. <b>clickhouse-server</b>
|
||
— contains config files for the clickhouse-server.</p>
|
||
|
||
<p>Server config files are located in /etc/clickhouse-server/. Before getting to work please notice the <b>path</b>
|
||
element in config. <b>Path</b> determines the location for data storage. It's not really handy to directly
|
||
edit <b>config.xml</b> file considering package updates. Recommended way is to override the config elements in
|
||
<a href="docs/en/operations/configuration_files/">files of config.d directory</a>.
|
||
Also you may want to <a href="docs/en/operations/access_rights/">set up access
|
||
rights</a> at the start.</p>
|
||
|
||
<p><b>clickhouse-server</b> won't be launched automatically after package installation. It won't be automatically
|
||
restarted after updates either. Start the server with:
|
||
<pre>sudo service clickhouse-server start</pre>
|
||
Default location for server logs is /var/log/clickhouse-server/
|
||
Server is ready to handle client connections once "Ready for connections" message was logged.</p>
|
||
|
||
<p>Use <b>clickhouse-client</b> to connect to the server.</p>
|
||
|
||
<div class="spoiler"><a class="spoiler_title">Tips for clickhouse-client</a>
|
||
<div class="spoiler_body">
|
||
Interactive mode:
|
||
<pre>
|
||
clickhouse-client
|
||
clickhouse-client --host=... --port=... --user=... --password=...
|
||
</pre>
|
||
Enable multiline queries:
|
||
<pre>
|
||
clickhouse-client -m
|
||
clickhouse-client --multiline
|
||
</pre>
|
||
Run queries in batch-mode:
|
||
<pre>
|
||
clickhouse-client --query='SELECT 1'
|
||
echo 'SELECT 1' | clickhouse-client
|
||
</pre>
|
||
Insert data from file of a specified format:
|
||
<pre>
|
||
clickhouse-client --query='INSERT INTO table VALUES' < data.txt
|
||
clickhouse-client --query='INSERT INTO table FORMAT TabSeparated' < data.tsv
|
||
</pre>
|
||
</div>
|
||
</div>
|
||
|
||
<h3>Create table for sample dataset</h3>
|
||
<div class="spoiler"><a class="spoiler_title">Create table query</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
$ clickhouse-client --multiline
|
||
ClickHouse client version 0.0.53720.
|
||
Connecting to localhost:9000.
|
||
Connected to ClickHouse server version 0.0.53720.
|
||
|
||
:) CREATE TABLE ontime
|
||
(
|
||
Year UInt16,
|
||
Quarter UInt8,
|
||
Month UInt8,
|
||
DayofMonth UInt8,
|
||
DayOfWeek UInt8,
|
||
FlightDate Date,
|
||
UniqueCarrier FixedString(7),
|
||
AirlineID Int32,
|
||
Carrier FixedString(2),
|
||
TailNum String,
|
||
FlightNum String,
|
||
OriginAirportID Int32,
|
||
OriginAirportSeqID Int32,
|
||
OriginCityMarketID Int32,
|
||
Origin FixedString(5),
|
||
OriginCityName String,
|
||
OriginState FixedString(2),
|
||
OriginStateFips String,
|
||
OriginStateName String,
|
||
OriginWac Int32,
|
||
DestAirportID Int32,
|
||
DestAirportSeqID Int32,
|
||
DestCityMarketID Int32,
|
||
Dest FixedString(5),
|
||
DestCityName String,
|
||
DestState FixedString(2),
|
||
DestStateFips String,
|
||
DestStateName String,
|
||
DestWac Int32,
|
||
CRSDepTime Int32,
|
||
DepTime Int32,
|
||
DepDelay Int32,
|
||
DepDelayMinutes Int32,
|
||
DepDel15 Int32,
|
||
DepartureDelayGroups String,
|
||
DepTimeBlk String,
|
||
TaxiOut Int32,
|
||
WheelsOff Int32,
|
||
WheelsOn Int32,
|
||
TaxiIn Int32,
|
||
CRSArrTime Int32,
|
||
ArrTime Int32,
|
||
ArrDelay Int32,
|
||
ArrDelayMinutes Int32,
|
||
ArrDel15 Int32,
|
||
ArrivalDelayGroups Int32,
|
||
ArrTimeBlk String,
|
||
Cancelled UInt8,
|
||
CancellationCode FixedString(1),
|
||
Diverted UInt8,
|
||
CRSElapsedTime Int32,
|
||
ActualElapsedTime Int32,
|
||
AirTime Int32,
|
||
Flights Int32,
|
||
Distance Int32,
|
||
DistanceGroup UInt8,
|
||
CarrierDelay Int32,
|
||
WeatherDelay Int32,
|
||
NASDelay Int32,
|
||
SecurityDelay Int32,
|
||
LateAircraftDelay Int32,
|
||
FirstDepTime String,
|
||
TotalAddGTime String,
|
||
LongestAddGTime String,
|
||
DivAirportLandings String,
|
||
DivReachedDest String,
|
||
DivActualElapsedTime String,
|
||
DivArrDelay String,
|
||
DivDistance String,
|
||
Div1Airport String,
|
||
Div1AirportID Int32,
|
||
Div1AirportSeqID Int32,
|
||
Div1WheelsOn String,
|
||
Div1TotalGTime String,
|
||
Div1LongestGTime String,
|
||
Div1WheelsOff String,
|
||
Div1TailNum String,
|
||
Div2Airport String,
|
||
Div2AirportID Int32,
|
||
Div2AirportSeqID Int32,
|
||
Div2WheelsOn String,
|
||
Div2TotalGTime String,
|
||
Div2LongestGTime String,
|
||
Div2WheelsOff String,
|
||
Div2TailNum String,
|
||
Div3Airport String,
|
||
Div3AirportID Int32,
|
||
Div3AirportSeqID Int32,
|
||
Div3WheelsOn String,
|
||
Div3TotalGTime String,
|
||
Div3LongestGTime String,
|
||
Div3WheelsOff String,
|
||
Div3TailNum String,
|
||
Div4Airport String,
|
||
Div4AirportID Int32,
|
||
Div4AirportSeqID Int32,
|
||
Div4WheelsOn String,
|
||
Div4TotalGTime String,
|
||
Div4LongestGTime String,
|
||
Div4WheelsOff String,
|
||
Div4TailNum String,
|
||
Div5Airport String,
|
||
Div5AirportID Int32,
|
||
Div5AirportSeqID Int32,
|
||
Div5WheelsOn String,
|
||
Div5TotalGTime String,
|
||
Div5LongestGTime String,
|
||
Div5WheelsOff String,
|
||
Div5TailNum String
|
||
)
|
||
ENGINE = MergeTree(FlightDate, (Year, FlightDate), 8192);
|
||
</pre>
|
||
</div>
|
||
</div>
|
||
|
||
<p>Now we have a table of <a href="docs/en/table_engines/mergetree/">MergeTree type</a>.
|
||
MergeTree table type is recommended for usage in production. Table of this kind has a primary key used for
|
||
incremental sort of table data. This allows fast execution of queries in ranges of a primary key.</p>
|
||
|
||
|
||
<p><b>Note</b>
|
||
We store ad network banners impressions logs in ClickHouse. Each table entry looks like:
|
||
[Advertiser ID, Impression ID, attribute1, attribute2, …].
|
||
Let assume that our aim is to provide a set of reports for each advertiser. Common and frequently demanded query
|
||
would be to count impressions for a specific Advertiser ID. This means that table primary key should start with
|
||
Advertiser ID. In this case ClickHouse needs to read smaller amount of data to perform the query for a
|
||
given Advertiser ID.
|
||
</p>
|
||
|
||
<h3>Load data</h3>
|
||
<pre>xz -v -c -d < ontime.csv.xz | clickhouse-client --query="INSERT INTO ontime FORMAT CSV"</pre>
|
||
<p>ClickHouse INSERT query allows to load data in any <a href="docs/en/formats/">supported
|
||
format</a>. Data load requires just O(1) RAM consumption. INSERT query can receive any data volume as input.
|
||
It's strongly recommended to insert data with <a
|
||
href="docs/en/introduction/performance/#performance-when-inserting-data">not too small
|
||
size blocks</a>. Notice that insert of blocks with size up to max_insert_block_size (= 1 048 576
|
||
rows by default) is an atomic operation: data block will be inserted completely or not inserted at all. In case
|
||
of disconnect during insert operation you may not know if the block was inserted successfully. To achieve
|
||
exactly-once semantics ClickHouse supports idempotency for <a
|
||
href="docs/en/table_engines/replication/">replicated tables</a>. This means
|
||
that you may retry insert of the same data block (possibly on a different replicas) but this block will be
|
||
inserted just once. Anyway in this guide we will load data from our localhost so we may not take care about data
|
||
blocks generation and exactly-once semantics.</p>
|
||
|
||
<p>INSERT query into tables of MergeTree type is non-blocking (so does a SELECT query). You can execute SELECT
|
||
queries right after of during insert operation.</p>
|
||
|
||
<p>Our sample dataset is a bit not optimal. There are two reasons.</p>
|
||
|
||
<p>The first is that String data type is used in cases when <a
|
||
href="docs/en/data_types/enum/">Enum</a> or numeric type would fit best.</p>
|
||
|
||
<p class="tip"><b>⚖</b> When set of possible values is determined and known to be small. (E.g. OS name, browser
|
||
vendors etc.) it's recommended to use Enums or numbers to improve performance.
|
||
When set of possible values is not limited (search query, URL, etc.) just go ahead with String.</p>
|
||
|
||
<p>The second is that dataset contains redundant fields like Year, Quarter, Month, DayOfMonth, DayOfWeek. In fact a
|
||
single FlightDate would be enough. Most likely they have been added to improve performance for other DBMS'es
|
||
which DateTime handling functions may be not efficient.</p>
|
||
|
||
<p class="tip"><b>✯</b> ClickHouse <a
|
||
href="docs/en/functions/date_time_functions/">functions
|
||
for operating with DateTime fields</a> are well-optimized so such redundancy is not required. Anyway much
|
||
columns is not a reason to worry — ClickHouse is a <a href="https://en.wikipedia.org/wiki/Column-oriented_DBMS"
|
||
rel="external nofollow">column-oriented
|
||
DBMS</a>. This allows you to have as much fields as you need. Hundreds of columns in a table is fine for
|
||
ClickHouse.</p>
|
||
|
||
<h3>Querying the sample dataset</h3>
|
||
|
||
<p>Here are some examples of the queries from our test data.</p>
|
||
|
||
<ul>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">the most popular destinations in 2015;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT
|
||
OriginCityName,
|
||
DestCityName,
|
||
count(*) AS flights,
|
||
bar(flights, 0, 20000, 40)
|
||
FROM ontime WHERE Year = 2015 GROUP BY OriginCityName, DestCityName ORDER BY flights DESC LIMIT 20
|
||
</pre>
|
||
<img src="images/tutorial/a8518a200d6d405a95ee80ea1c8e1c90.png"/>
|
||
<pre>
|
||
SELECT
|
||
OriginCityName < DestCityName ? OriginCityName : DestCityName AS a,
|
||
OriginCityName < DestCityName ? DestCityName : OriginCityName AS b,
|
||
count(*) AS flights,
|
||
bar(flights, 0, 40000, 40)
|
||
FROM ontime WHERE Year = 2015 GROUP BY a, b ORDER BY flights DESC LIMIT 20
|
||
</pre>
|
||
<img src="images/tutorial/d3578db55e304bd7b5eba818abdb53f5.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">the most popular cities of departure;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT OriginCityName, count(*) AS flights
|
||
FROM ontime GROUP BY OriginCityName ORDER BY flights DESC LIMIT 20
|
||
</pre>
|
||
<img src="images/tutorial/ef4141f348234773a5349c4bd3e8f804.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">cities of departure which offer maximum variety of
|
||
destinations;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT OriginCityName, uniq(Dest) AS u
|
||
FROM ontime GROUP BY OriginCityName ORDER BY u DESC LIMIT 20
|
||
</pre>
|
||
<img src="images/tutorial/2409f49d11fb4aa1b8b5ff34cf9ca75d.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">flight delay dependence on the day of week;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT DayOfWeek, count() AS c, avg(DepDelay > 60) AS delays
|
||
FROM ontime GROUP BY DayOfWeek ORDER BY DayOfWeek
|
||
</pre>
|
||
<img src="images/tutorial/885e507930e34b7c8f788d25e7ca2bcf.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">cities of departure with most frequent delays for 1 hour or
|
||
longer;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT OriginCityName, count() AS c, avg(DepDelay > 60) AS delays
|
||
FROM ontime
|
||
GROUP BY OriginCityName
|
||
HAVING c > 100000
|
||
ORDER BY delays DESC
|
||
LIMIT 20
|
||
</pre>
|
||
<img src="images/tutorial/ac292656d03946d0aba35c75783a31f2.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">flights of maximum duration;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT OriginCityName, DestCityName, count(*) AS flights, avg(AirTime) AS duration
|
||
FROM ontime
|
||
GROUP BY OriginCityName, DestCityName
|
||
ORDER BY duration DESC
|
||
LIMIT 20
|
||
</pre>
|
||
<img src="images/tutorial/7b3c2e685832439b8c373bf2015131d2.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">distribution of arrival time delays split by aircompanies;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT Carrier, count() AS c, round(quantileTDigest(0.99)(DepDelay), 2) AS q
|
||
FROM ontime GROUP BY Carrier ORDER BY q DESC
|
||
</pre>
|
||
<img src="images/tutorial/49c332e3d93146ba8f46beef6b2b02b0.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">aircompanies who stopped flights operation;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT Carrier, min(Year), max(Year), count()
|
||
FROM ontime GROUP BY Carrier HAVING max(Year) < 2015 ORDER BY count() DESC
|
||
</pre>
|
||
<img src="images/tutorial/24956f1a2efc48d78212586958aa036c.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">most trending destination cities in 2015;</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT
|
||
DestCityName,
|
||
sum(Year = 2014) AS c2014,
|
||
sum(Year = 2015) AS c2015,
|
||
c2015 / c2014 AS diff
|
||
FROM ontime
|
||
WHERE Year IN (2014, 2015)
|
||
GROUP BY DestCityName
|
||
HAVING c2014 > 10000 AND c2015 > 1000 AND diff > 1
|
||
ORDER BY diff DESC
|
||
</pre>
|
||
<img src="images/tutorial/f3132f4d1c0d42eab26d9111afe7771a.png"/></div>
|
||
</div>
|
||
</li>
|
||
<li>
|
||
<div class="spoiler"><a class="spoiler_title">destination cities with maximum popularity-season
|
||
dependency.</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
SELECT
|
||
DestCityName,
|
||
any(total),
|
||
avg(abs(monthly * 12 - total) / total) AS avg_month_diff
|
||
FROM
|
||
(
|
||
SELECT DestCityName, count() AS total
|
||
FROM ontime GROUP BY DestCityName HAVING total > 100000
|
||
)
|
||
ALL INNER JOIN
|
||
(
|
||
SELECT DestCityName, Month, count() AS monthly
|
||
FROM ontime GROUP BY DestCityName, Month HAVING monthly > 10000
|
||
)
|
||
USING DestCityName
|
||
GROUP BY DestCityName
|
||
ORDER BY avg_month_diff DESC
|
||
LIMIT 20
|
||
</pre>
|
||
<img src="images/tutorial/26b2c7aae21a4c76800cb1c7a33a374d.png"/></div>
|
||
</div>
|
||
</li>
|
||
</ul>
|
||
|
||
<h3>ClickHouse deployment to cluster</h3>
|
||
<p>ClickHouse cluster is a homogenous cluster. Steps to set up:
|
||
<ol>
|
||
<li>Install ClickHouse server on all machines of the cluster</li>
|
||
<li>Set up cluster configs in configuration file</li>
|
||
<li>Create local tables on each instance</li>
|
||
<li>Create a <a href="docs/en/table_engines/distributed/">Distributed table</a></li>
|
||
</ol>
|
||
</p>
|
||
|
||
<p><a href="docs/en/table_engines/distributed/">Distributed-table</a> is actually a kind of
|
||
"view" to local tables of ClickHouse cluster. SELECT query from a distributed table will be executed using
|
||
resources of all cluster's shards. You may specify configs for multiple clusters and create multiple
|
||
Distributed-tables providing views to different clusters.</p>
|
||
|
||
<div class="spoiler"><a class="spoiler_title">Config for cluster of three shards. Each shard stores data on a single
|
||
replica</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
<remote_servers>
|
||
<perftest_3shards_1replicas>
|
||
<shard>
|
||
<replica>
|
||
<host>example-perftest01j.yandex.ru</host>
|
||
<port>9000</port>
|
||
</replica>
|
||
</shard>
|
||
<shard>
|
||
<replica>
|
||
<host>example-perftest02j.yandex.ru</host>
|
||
<port>9000</port>
|
||
</replica>
|
||
</shard>
|
||
<shard>
|
||
<replica>
|
||
<host>example-perftest03j.yandex.ru</host>
|
||
<port>9000</port>
|
||
</replica>
|
||
</shard>
|
||
</perftest_3shards_1replicas>
|
||
</remote_servers>
|
||
</pre>
|
||
</div>
|
||
</div>
|
||
Creating a local table:
|
||
<pre>CREATE TABLE ontime_local (...) ENGINE = MergeTree(FlightDate, (Year, FlightDate), 8192);</pre>
|
||
Creating a distributed table providing a view into local tables of the cluster:
|
||
<pre>CREATE TABLE ontime_all AS ontime_local
|
||
ENGINE = Distributed(perftest_3shards_1replicas, default, ontime_local, rand());</pre>
|
||
|
||
<p>You can create a Distributed table on all machines in the cluster. This would allow to run distributed queries on
|
||
any machine of the cluster. Besides distributed table you can also use <a
|
||
href="docs/en/table_functions/remote/">*remote* table function</a>.</p>
|
||
|
||
<p>Let's run <a href="docs/en/query_language/queries/#insert">INSERT SELECT</a> into Distributed table
|
||
to spread the table to multiple servers.</p>
|
||
|
||
<pre>INSERT INTO ontime_all SELECT * FROM ontime;</pre>
|
||
|
||
<p class="tip"><b>⚠</b> Worth to notice that the approach given above wouldn't fit for sharding of large
|
||
tables.</p>
|
||
|
||
<p>As you could expect heavy queries are executed N times faster being launched on 3 servers instead of one.</p>
|
||
<div class="spoiler"><a class="spoiler_title">See here</a>
|
||
<div class="spoiler_body">
|
||
<img src="images/tutorial/ece020129fdf4a18a6e75daf2e699cb9.png"/>
|
||
|
||
<p>You may have noticed that quantiles calculation are slightly different. This happens due to <a
|
||
href="https://github.com/tdunning/t-digest/raw/master/docs/t-digest-paper/histo.pdf">t-digest</a>
|
||
algorithm implementation which is non-deterministic — it depends on the order of data processing.</p>
|
||
</div>
|
||
</div>
|
||
|
||
<p>In this case we have used a cluster with 3 shards each contains a single replica.</p>
|
||
|
||
<p>To provide for resilience in production environment we recommend that each shard should contain 2-3 replicas
|
||
distributed between multiple data-centers. Note that ClickHouse supports unlimited number of replicas.</p>
|
||
|
||
<div class="spoiler"><a class="spoiler_title">Config for cluster of one shard containing three replicas</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
<remote_servers>
|
||
...
|
||
<perftest_1shards_3replicas>
|
||
<shard>
|
||
<replica>
|
||
<host>example-perftest01j.yandex.ru</host>
|
||
<port>9000</port>
|
||
</replica>
|
||
<replica>
|
||
<host>example-perftest02j.yandex.ru</host>
|
||
<port>9000</port>
|
||
</replica>
|
||
<replica>
|
||
<host>example-perftest03j.yandex.ru</host>
|
||
<port>9000</port>
|
||
</replica>
|
||
</shard>
|
||
</perftest_1shards_3replicas>
|
||
</remote_servers>
|
||
</pre>
|
||
</div>
|
||
</div>
|
||
|
||
<p>To enable replication <a href="http://zookeeper.apache.org/" rel="external nofollow">ZooKeeper</a> is required.
|
||
ClickHouse will take care of data consistency on all replicas and run restore procedure after failure
|
||
automatically. It's recommended to deploy ZooKeeper cluster to separate servers.</p>
|
||
|
||
<p>ZooKeeper is not a requirement — in some simple cases you can duplicate the data by writing it into all the
|
||
replicas from your application code. This approach is not recommended — in this case ClickHouse is not able to
|
||
guarantee data consistency on all replicas. This remains the responsibility of your application.</p>
|
||
|
||
<div class="spoiler"><a class="spoiler_title">Set ZooKeeper locations in configuration file</a>
|
||
<div class="spoiler_body">
|
||
<pre>
|
||
<zookeeper-servers>
|
||
<node>
|
||
<host>zoo01.yandex.ru</host>
|
||
<port>2181</port>
|
||
</node>
|
||
<node>
|
||
<host>zoo02.yandex.ru</host>
|
||
<port>2181</port>
|
||
</node>
|
||
<node>
|
||
<host>zoo03.yandex.ru</host>
|
||
<port>2181</port>
|
||
</node>
|
||
</zookeeper-servers>
|
||
</pre>
|
||
</div>
|
||
</div>
|
||
|
||
<p>Also we need to set macros for identifying shard and replica — it will be used on table creation</p>
|
||
<pre>
|
||
<macros>
|
||
<shard>01</shard>
|
||
<replica>01</replica>
|
||
</macros>
|
||
</pre>
|
||
<p>If there are no replicas at the moment on replicated table creation — a new first replica will be instantiated.
|
||
If there are already live replicas — new replica will clone the data from existing ones. You have an option to
|
||
create all replicated tables first and that insert data to it. Another option is to create some replicas and add
|
||
the others after or during data insertion.</p>
|
||
|
||
<pre>
|
||
CREATE TABLE ontime_replica (...)
|
||
ENGINE = ReplicatedMergeTree(
|
||
'/clickhouse_perftest/tables/{shard}/ontime',
|
||
'{replica}',
|
||
FlightDate,
|
||
(Year, FlightDate),
|
||
8192);
|
||
</pre>
|
||
<p>Here we use <a href="docs/en/table_engines/replication/#replicatedmergetree">ReplicatedMergeTree</a>
|
||
table type. In parameters we specify ZooKeeper path containing shard and replica identifiers.</p>
|
||
|
||
<pre>INSERT INTO ontime_replica SELECT * FROM ontime;</pre>
|
||
<p>Replication operates in multi-master mode. Data can be loaded into any replica — it will be synced with other
|
||
instances automatically. Replication is asynchronous so at a given moment of time not all replicas may contain
|
||
recently inserted data. To allow data insertion at least one replica should be up. Others will sync up data and
|
||
repair consistency once they will become active again. Please notice that such scheme allows for the possibility
|
||
of just appended data loss.</p>
|
||
|
||
<p class="warranty"><a href="https://github.com/yandex/ClickHouse/blob/master/LICENSE"
|
||
rel="external nofollow" target="_blank">
|
||
ClickHouse source code is published under Apache 2.0 License.</a> Software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||
KIND, either express or implied.</p>
|
||
|
||
<p id="footer">© 2016–2019 <a href="https://yandex.com/company/" rel="external nofollow">YANDEX</a> LLC</p>
|
||
|
||
</div>
|
||
|
||
<script type="text/javascript" src="https://yastatic.net/jquery/3.1.1/jquery.min.js"></script>
|
||
<script type="text/javascript">
|
||
$('.spoiler_title').click(function () {
|
||
$(this).next('.spoiler_body').toggle(100);
|
||
});
|
||
|
||
var name = document.getElementById('main_title').textContent.trim().toLowerCase();
|
||
var feedback_address = name + '-feedback' + '@yandex-team.com';
|
||
var feedback_email = document.getElementById('feedback_email');
|
||
feedback_email.setAttribute('href', 'mailto:' + feedback_address);
|
||
feedback_email.textContent = feedback_address;
|
||
</script>
|
||
<!-- Yandex.Metrika counter -->
|
||
<script type="text/javascript">
|
||
(function (d, w, c) {
|
||
(w[c] = w[c] || []).push(function() {
|
||
try {
|
||
w.yaCounter18343495 = new Ya.Metrika2({
|
||
id:18343495,
|
||
clickmap:true,
|
||
trackLinks:true,
|
||
accurateTrackBounce:true,
|
||
webvisor:true
|
||
});
|
||
} catch(e) { }
|
||
});
|
||
|
||
var n = d.getElementsByTagName("script")[0],
|
||
s = d.createElement("script"),
|
||
f = function () { n.parentNode.insertBefore(s, n); };
|
||
s.type = "text/javascript";
|
||
s.async = true;
|
||
s.src = "https://mc.yandex.ru/metrika/tag.js";
|
||
|
||
if (w.opera == "[object Opera]") {
|
||
d.addEventListener("DOMContentLoaded", f, false);
|
||
} else { f(); }
|
||
})(document, window, "yandex_metrika_callbacks2");
|
||
</script>
|
||
<noscript>
|
||
<div><img src="https://mc.yandex.ru/watch/18343495" style="position:absolute; left:-9999px;" alt=""/></div>
|
||
</noscript>
|
||
<!-- /Yandex.Metrika counter -->
|
||
|
||
|
||
</body>
|
||
</html>
|