2020-04-03 13:23:32 +00:00
---
2022-04-09 13:29:05 +00:00
sidebar_position: 80
sidebar_label: URL
2020-04-03 13:23:32 +00:00
---
2022-06-02 10:55:18 +00:00
# URL Table Engine
2018-07-27 10:21:04 +00:00
2020-06-18 08:24:31 +00:00
Queries data to/from a remote HTTP/HTTPS server. This engine is similar to the [File ](../../../engines/table-engines/special/file.md ) engine.
2018-07-27 10:21:04 +00:00
2022-02-11 09:46:33 +00:00
Syntax: `URL(URL [,Format] [,CompressionMethod])`
2022-02-13 13:38:43 +00:00
- The `URL` parameter must conform to the structure of a Uniform Resource Locator. The specified URL must point to a server that uses HTTP or HTTPS. This does not require any additional headers for getting a response from the server.
2022-02-11 09:46:33 +00:00
2022-02-13 13:38:43 +00:00
- The `Format` must be one that ClickHouse can use in `SELECT` queries and, if necessary, in `INSERTs` . For the full list of supported formats, see [Formats ](../../../interfaces/formats.md#formats ).
2022-02-11 09:46:33 +00:00
2022-06-02 10:55:18 +00:00
- `CompressionMethod` indicates that whether the HTTP body should be compressed. If the compression is enabled, the HTTP packets sent by the URL engine contain 'Content-Encoding' header to indicate which compression method is used.
2022-02-11 09:46:33 +00:00
2022-02-13 13:38:43 +00:00
To enable compression, please first make sure the remote HTTP endpoint indicated by the `URL` parameter supports corresponding compression algorithm.
2022-06-02 10:55:18 +00:00
2022-02-13 13:38:43 +00:00
The supported `CompressionMethod` should be one of following:
- gzip or gz
- deflate
- brotli or br
- lzma or xz
- zstd or zst
- lz4
- bz2
- snappy
- none
2020-06-10 20:11:41 +00:00
## Usage {#using-the-engine-in-the-clickhouse-server}
2018-07-27 10:21:04 +00:00
2018-09-04 11:18:59 +00:00
`INSERT` and `SELECT` queries are transformed to `POST` and `GET` requests,
respectively. For processing `POST` requests, the remote server must support
[Chunked transfer encoding ](https://en.wikipedia.org/wiki/Chunked_transfer_encoding ).
2018-07-27 10:21:04 +00:00
2020-10-13 17:23:29 +00:00
You can limit the maximum number of HTTP GET redirect hops using the [max_http_get_redirects ](../../../operations/settings/settings.md#setting-max_http_get_redirects ) setting.
2019-12-04 06:46:31 +00:00
2020-06-18 08:24:31 +00:00
## Example {#example}
2018-07-27 10:21:04 +00:00
2018-09-04 11:18:59 +00:00
**1.** Create a `url_engine_table` table on the server :
2018-07-27 10:21:04 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2018-07-27 10:21:04 +00:00
CREATE TABLE url_engine_table (word String, value UInt64)
ENGINE=URL('http://127.0.0.1:12345/', CSV)
```
2018-09-04 11:18:59 +00:00
**2.** Create a basic HTTP server using the standard Python 3 tools and
start it:
2018-07-27 10:21:04 +00:00
2020-03-20 10:10:48 +00:00
``` python3
2018-07-27 10:21:04 +00:00
from http.server import BaseHTTPRequestHandler, HTTPServer
class CSVHTTPServer(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type', 'text/csv')
self.end_headers()
self.wfile.write(bytes('Hello,1\nWorld,2\n', "utf-8"))
if __name__ == "__main__":
server_address = ('127.0.0.1', 12345)
HTTPServer(server_address, CSVHTTPServer).serve_forever()
```
2020-03-20 10:10:48 +00:00
``` bash
2019-09-23 15:31:46 +00:00
$ python3 server.py
2018-07-27 10:21:04 +00:00
```
2018-09-04 11:18:59 +00:00
**3.** Request data:
2018-07-27 10:21:04 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2018-07-27 10:21:04 +00:00
SELECT * FROM url_engine_table
```
2020-03-20 10:10:48 +00:00
``` text
2018-07-27 10:21:04 +00:00
┌─word──┬─value─┐
│ Hello │ 1 │
│ World │ 2 │
└───────┴───────┘
```
2020-04-30 18:19:18 +00:00
## Details of Implementation {#details-of-implementation}
2018-07-27 10:21:04 +00:00
2020-03-21 04:11:51 +00:00
- Reads and writes can be parallel
- Not supported:
- `ALTER` and `SELECT...SAMPLE` operations.
- Indexes.
- Replication.
2018-10-16 10:47:17 +00:00