2020-12-18 09:46:50 +00:00
---
2021-02-27 00:01:02 +00:00
toc_priority: 7
2020-12-18 09:46:50 +00:00
toc_title: S3
---
2021-03-05 08:31:16 +00:00
# S3 Table Engine {#table-engine-s3}
2020-12-18 09:46:50 +00:00
2021-03-05 08:31:16 +00:00
This engine provides integration with [Amazon S3 ](https://aws.amazon.com/s3/ ) ecosystem. This engine is similar to the [HDFS ](../../../engines/table-engines/special/file.md#table_engines-hdfs ) engine, but provides S3-specific features.
2020-12-18 09:46:50 +00:00
2021-03-05 08:31:16 +00:00
## Create Table {#creating-a-table}
2020-12-18 09:46:50 +00:00
``` sql
2021-06-01 14:23:46 +00:00
CREATE TABLE s3_engine_table (name String, value UInt32)
2021-04-22 19:57:33 +00:00
ENGINE = S3(path, [aws_access_key_id, aws_secret_access_key,] format, [compression])
2020-12-18 09:46:50 +00:00
```
2021-03-05 08:31:16 +00:00
**Engine parameters**
2020-12-18 09:46:50 +00:00
2021-03-21 14:23:16 +00:00
- `path` — Bucket url with path to file. Supports following wildcards in readonly mode: `*` , `?` , `{abc,def}` and `{N..M}` where `N` , `M` — numbers, `'abc'` , `'def'` — strings. For more information see [below ](#wildcards-in-path ).
2020-12-18 12:12:06 +00:00
- `format` — The [format ](../../../interfaces/formats.md#formats ) of the file.
2021-04-25 19:01:38 +00:00
- `aws_access_key_id` , `aws_secret_access_key` - Long-term credentials for the [AWS ](https://aws.amazon.com/ ) account user. You can use these to authenticate your requests. Parameter is optional. If credentials are not specified, they are used from the configuration file. For more information see [Using S3 for Data Storage ](../mergetree-family/mergetree.md#table_engine-mergetree-s3 ).
2021-03-25 14:24:42 +00:00
- `compression` — Compression type. Supported values: `none` , `gzip/gz` , `brotli/br` , `xz/LZMA` , `zstd/zst` . Parameter is optional. By default, it will autodetect compression by file extension.
2020-12-18 09:46:50 +00:00
2021-03-05 14:31:55 +00:00
**Example**
2020-12-18 09:46:50 +00:00
2021-03-05 14:31:55 +00:00
1. Set up the `s3_engine_table` table:
2020-12-18 09:46:50 +00:00
``` sql
2021-04-22 19:57:33 +00:00
CREATE TABLE s3_engine_table (name String, value UInt32) ENGINE=S3('https://storage.yandexcloud.net/my-test-bucket-768/test-data.csv.gz', 'CSV', 'gzip');
2020-12-18 09:46:50 +00:00
```
2021-03-05 14:31:55 +00:00
2. Fill file:
2020-12-18 09:46:50 +00:00
``` sql
2021-02-28 19:27:22 +00:00
INSERT INTO s3_engine_table VALUES ('one', 1), ('two', 2), ('three', 3);
2020-12-18 09:46:50 +00:00
```
2021-03-05 14:31:55 +00:00
3. Query the data:
2020-12-18 09:46:50 +00:00
``` sql
2021-02-28 19:27:22 +00:00
SELECT * FROM s3_engine_table LIMIT 2;
2020-12-18 09:46:50 +00:00
```
2021-03-04 12:08:35 +00:00
```text
2020-12-18 09:46:50 +00:00
┌─name─┬─value─┐
│ one │ 1 │
│ two │ 2 │
└──────┴───────┘
```
2021-03-05 08:31:16 +00:00
## Virtual columns {#virtual-columns}
- `_path` — Path to the file.
- `_file` — Name of the file.
For more information about virtual columns see [here ](../../../engines/table-engines/index.md#table_engines-virtual_columns ).
2020-12-18 09:46:50 +00:00
## Implementation Details {#implementation-details}
- Reads and writes can be parallel
- Not supported:
- `ALTER` and `SELECT...SAMPLE` operations.
- Indexes.
- Replication.
2021-03-21 14:23:16 +00:00
## Wildcards In Path {#wildcards-in-path}
2020-12-18 09:46:50 +00:00
2021-03-21 14:23:16 +00:00
`path` argument can specify multiple files using bash-like wildcards. For being processed file should exist and match to the whole path pattern. Listing of files is determined during `SELECT` (not at `CREATE` moment).
2020-12-18 09:46:50 +00:00
- `*` — Substitutes any number of any characters except `/` including empty string.
- `?` — Substitutes any single character.
- `{some_string,another_string,yet_another_one}` — Substitutes any of strings `'some_string', 'another_string', 'yet_another_one'` .
2020-12-18 18:45:21 +00:00
- `{N..M}` — Substitutes any number in range from N to M including both borders. N and M can have leading zeroes e.g. `000..078` .
2020-12-18 09:46:50 +00:00
Constructions with `{}` are similar to the [remote ](../../../sql-reference/table-functions/remote.md ) table function.
**Example**
2021-03-04 12:08:35 +00:00
1. Suppose we have several files in CSV format with the following URIs on S3:
2020-12-18 09:46:50 +00:00
- ‘ https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_1.csv’
- ‘ https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_2.csv’
- ‘ https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_3.csv’
- ‘ https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_1.csv’
- ‘ https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_2.csv’
- ‘ https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_3.csv’
2021-03-05 14:31:55 +00:00
There are several ways to make a table consisting of all six files:
2020-12-18 09:46:50 +00:00
2021-03-05 14:31:55 +00:00
The first way:
2020-12-18 09:46:50 +00:00
``` sql
2021-02-28 19:27:22 +00:00
CREATE TABLE table_with_range (name String, value UInt32) ENGINE = S3('https://storage.yandexcloud.net/my-test-bucket-768/{some,another}_prefix/some_file_{1..3}', 'CSV');
2020-12-18 09:46:50 +00:00
```
2021-03-05 14:31:55 +00:00
Another way:
2020-12-18 09:46:50 +00:00
``` sql
2021-02-28 19:27:22 +00:00
CREATE TABLE table_with_question_mark (name String, value UInt32) ENGINE = S3('https://storage.yandexcloud.net/my-test-bucket-768/{some,another}_prefix/some_file_?', 'CSV');
2020-12-18 09:46:50 +00:00
```
2021-03-05 14:31:55 +00:00
Table consists of all the files in both directories (all files should satisfy format and schema described in query):
2020-12-18 09:46:50 +00:00
``` sql
2021-02-28 19:27:22 +00:00
CREATE TABLE table_with_asterisk (name String, value UInt32) ENGINE = S3('https://storage.yandexcloud.net/my-test-bucket-768/{some,another}_prefix/*', 'CSV');
2020-12-18 09:46:50 +00:00
```
2021-03-30 13:08:19 +00:00
If the listing of files contains number ranges with leading zeros, use the construction with braces for each digit separately or use `?` .
2020-12-18 09:46:50 +00:00
**Example**
Create table with files named `file-000.csv` , `file-001.csv` , … , `file-999.csv` :
``` sql
2021-02-28 19:27:22 +00:00
CREATE TABLE big_table (name String, value UInt32) ENGINE = S3('https://storage.yandexcloud.net/my-test-bucket-768/big_prefix/file-{000..999}.csv', 'CSV');
2020-12-18 09:46:50 +00:00
```
## Virtual Columns {#virtual-columns}
- `_path` — Path to the file.
- `_file` — Name of the file.
2021-01-26 08:33:21 +00:00
**See Also**
- [Virtual columns ](../../../engines/table-engines/index.md#table_engines-virtual_columns )
2020-12-18 09:46:50 +00:00
## S3-related settings {#settings}
The following settings can be set before query execution or placed into configuration file.
2021-03-05 14:31:55 +00:00
- `s3_max_single_part_upload_size` — The maximum size of object to upload using singlepart upload to S3. Default value is `64Mb` .
2021-03-10 06:26:55 +00:00
- `s3_min_upload_part_size` — The minimum size of part to upload during multipart upload to [S3 Multipart upload ](https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html ). Default value is `512Mb` .
2021-03-05 14:31:55 +00:00
- `s3_max_redirects` — Max number of S3 redirects hops allowed. Default value is `10` .
2021-06-01 14:23:46 +00:00
- `s3_single_read_retries` — The maximum number of attempts during single read. Default value is `4` .
2020-12-18 09:46:50 +00:00
2020-12-18 18:02:07 +00:00
Security consideration: if malicious user can specify arbitrary S3 URLs, `s3_max_redirects` must be set to zero to avoid [SSRF ](https://en.wikipedia.org/wiki/Server-side_request_forgery ) attacks; or alternatively, `remote_host_filter` must be specified in server configuration.
2021-03-11 10:09:45 +00:00
## Endpoint-based Settings {#endpoint-settings}
2021-01-25 06:33:53 +00:00
The following settings can be specified in configuration file for given endpoint (which will be matched by exact prefix of a URL):
2021-03-10 06:26:55 +00:00
- `endpoint` — Specifies prefix of an endpoint. Mandatory.
- `access_key_id` and `secret_access_key` — Specifies credentials to use with given endpoint. Optional.
2021-05-10 20:04:35 +00:00
- `use_environment_credentials` — If set to `true` , S3 client will try to obtain credentials from environment variables and [Amazon EC2 ](https://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud ) metadata for given endpoint. Optional, default value is `false` .
2021-05-01 17:41:31 +00:00
- `region` — Specifies S3 region name. Optional.
2021-03-30 00:32:26 +00:00
- `use_insecure_imds_request` — If set to `true` , S3 client will use insecure IMDS request while obtaining credentials from Amazon EC2 metadata. Optional, default value is `false` .
2021-03-10 06:26:55 +00:00
- `header` — Adds specified HTTP header to a request to given endpoint. Optional, can be speficied multiple times.
- `server_side_encryption_customer_key_base64` — If specified, required headers for accessing S3 objects with SSE-C encryption will be set. Optional.
2021-05-19 21:42:25 +00:00
- `max_single_read_retries` — The maximum number of attempts during single read. Default value is `4` . Optional.
2021-01-25 06:33:53 +00:00
2021-03-05 08:00:49 +00:00
**Example:**
2021-01-25 06:33:53 +00:00
2021-03-05 09:22:15 +00:00
``` xml
2021-01-25 06:33:53 +00:00
< s3 >
< endpoint-name >
< endpoint > https://storage.yandexcloud.net/my-test-bucket-768/< / endpoint >
<!-- <access_key_id>ACCESS_KEY_ID</access_key_id> -->
<!-- <secret_access_key>SECRET_ACCESS_KEY</secret_access_key> -->
2021-05-01 17:41:31 +00:00
<!-- <region>us - west - 1</region> -->
2021-01-25 06:33:53 +00:00
<!-- <use_environment_credentials>false</use_environment_credentials> -->
2021-03-30 00:32:26 +00:00
<!-- <use_insecure_imds_request>false</use_insecure_imds_request> -->
2021-01-25 06:33:53 +00:00
<!-- <header>Authorization: Bearer SOME - TOKEN</header> -->
2021-01-28 06:32:41 +00:00
<!-- <server_side_encryption_customer_key_base64>BASE64 - ENCODED - KEY</server_side_encryption_customer_key_base64> -->
2021-05-19 21:42:25 +00:00
<!-- <max_single_read_retries>4</max_single_read_retries> -->
2021-01-25 06:33:53 +00:00
< / endpoint-name >
< / s3 >
2021-01-26 08:33:21 +00:00
```
2021-03-18 07:36:11 +00:00
2021-03-05 08:00:49 +00:00
## Usage {#usage-examples}
Suppose we have several files in TSV format with the following URIs on HDFS:
2020-12-18 09:46:50 +00:00
2021-03-05 09:10:35 +00:00
- 'https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_1.csv'
- 'https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_2.csv'
- 'https://storage.yandexcloud.net/my-test-bucket-768/some_prefix/some_file_3.csv'
- 'https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_1.csv'
- 'https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_2.csv'
- 'https://storage.yandexcloud.net/my-test-bucket-768/another_prefix/some_file_3.csv'
2020-12-18 09:46:50 +00:00
2021-03-05 09:10:35 +00:00
1. There are several ways to make a table consisting of all six files:
2020-12-18 09:46:50 +00:00
``` sql
2021-06-01 14:23:46 +00:00
CREATE TABLE table_with_range (name String, value UInt32)
2021-03-05 09:22:15 +00:00
ENGINE = S3('https://storage.yandexcloud.net/my-test-bucket-768/{some,another}_prefix/some_file_{1..3}', 'CSV');
2020-12-18 09:46:50 +00:00
```
2021-03-05 09:10:35 +00:00
2. Another way:
2020-12-18 09:46:50 +00:00
``` sql
2021-06-01 14:23:46 +00:00
CREATE TABLE table_with_question_mark (name String, value UInt32)
2021-03-05 09:22:15 +00:00
ENGINE = S3('https://storage.yandexcloud.net/my-test-bucket-768/{some,another}_prefix/some_file_?', 'CSV');
2020-12-18 09:46:50 +00:00
```
2021-03-05 09:10:35 +00:00
3. Table consists of all the files in both directories (all files should satisfy format and schema described in query):
2020-12-18 09:46:50 +00:00
``` sql
2021-06-01 14:23:46 +00:00
CREATE TABLE table_with_asterisk (name String, value UInt32)
2021-03-05 09:22:15 +00:00
ENGINE = S3('https://storage.yandexcloud.net/my-test-bucket-768/{some,another}_prefix/*', 'CSV');
2020-12-18 09:46:50 +00:00
```
!!! warning "Warning"
If the listing of files contains number ranges with leading zeros, use the construction with braces for each digit separately or use `?` .
2021-03-05 09:10:35 +00:00
4. Create table with files named `file-000.csv` , `file-001.csv` , … , `file-999.csv` :
2020-12-18 09:46:50 +00:00
``` sql
2021-06-01 14:23:46 +00:00
CREATE TABLE big_table (name String, value UInt32)
2021-03-05 09:22:15 +00:00
ENGINE = S3('https://storage.yandexcloud.net/my-test-bucket-768/big_prefix/file-{000..999}.csv', 'CSV');
2020-12-18 09:46:50 +00:00
```
2021-03-18 07:36:11 +00:00
2021-03-05 08:00:49 +00:00
## See also
2020-12-18 09:46:50 +00:00
2021-03-05 08:00:49 +00:00
- [S3 table function ](../../../sql-reference/table-functions/s3.md )