ClickHouse/benchmark/timescaledb/usability.md

This is a "usability testing" of TimescaleDB. I did not use TimescaleDB before. I will try to install it, load the data and conduct benchmarks. And record every obstacle that I will face.
Usability testing need to be conducted by the most clueless person in the room. Doing this "usability testing" requires a bit of patience and courage (to publish all the struggles as is).

Note: insted of using clear VM, I have to run benchmark on exactly the same baremetal server where all other benchmarks were run.


## Installation

Install as following:
https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/#installation-apt-ubuntu

I've noticed that TimescaleDB documentation website does not have favicon in contrast to the main page.
In other means, it is quite neat.

```
sudo apt install postgresql-common
sudo sh /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo sh -c "echo 'deb [signed-by=/usr/share/keyrings/timescale.keyring] https://packagecloud.io/timescale/timescaledb/ubuntu/ $(lsb_release -c -s) main' > /etc/apt/sources.list.d/timescaledb.list"
wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/timescale.keyring
sudo apt-get update
sudo apt install timescaledb-2-postgresql-13
```

It recommends to tune it:

```
sudo apt install timescaledb-tune

sudo timescaledb-tune --quiet --yes
Using postgresql.conf at this path:
/etc/postgresql/13/main/postgresql.conf

Writing backup to:
/tmp/timescaledb_tune.backup202110292328

Recommendations based on 125.88 GB of available memory and 32 CPUs for PostgreSQL 13
shared_preload_libraries = 'timescaledb'        # (change requires restart)
shared_buffers = 32226MB
effective_cache_size = 96678MB
maintenance_work_mem = 2047MB
work_mem = 10312kB
timescaledb.max_background_workers = 8
max_worker_processes = 43
max_parallel_workers_per_gather = 16
max_parallel_workers = 32
wal_buffers = 16MB
min_wal_size = 512MB
default_statistics_target = 500
random_page_cost = 1.1
checkpoint_completion_target = 0.9
max_locks_per_transaction = 512
autovacuum_max_workers = 10
autovacuum_naptime = 10
effective_io_concurrency = 256
timescaledb.last_tuned = '2021-10-29T23:28:49+03:00'
timescaledb.last_tuned_version = '0.12.0'
Saving changes to: /etc/postgresql/13/main/postgresql.conf
```

```
sudo service postgresql restart
```

Post-install setup:
https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/post-install-setup/

```
$ psql -U postgres -h localhost
Password for user postgres:
psql: error: connection to server at "localhost" (::1), port 5432 failed: fe_sendauth: no password supplied
```

How to set up password?

```
milovidov@mtlog-perftest03j:~$ psql -U postgres -h localhost
Password for user postgres:
psql: error: connection to server at "localhost" (::1), port 5432 failed: fe_sendauth: no password supplied
milovidov@mtlog-perftest03j:~$ psql
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  role "milovidov" does not exist
milovidov@mtlog-perftest03j:~$ sudo psql
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  role "root" does not exist
milovidov@mtlog-perftest03j:~$ psql -U postgres
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  Peer authentication failed for user "postgres"
milovidov@mtlog-perftest03j:~$ psql -U postgres -h localost
psql: error: could not translate host name "localost" to address: Name or service not known
milovidov@mtlog-perftest03j:~$ sudo psql -U postgres -h localost
psql: error: could not translate host name "localost" to address: Name or service not known
milovidov@mtlog-perftest03j:~$ sudo psql -U postgres -h localhost
Password for user postgres:
psql: error: connection to server at "localhost" (::1), port 5432 failed: fe_sendauth: no password supplied
milovidov@mtlog-perftest03j:~$ sudo -u postgres psql -h localhost
Password for user postgres:
psql: error: connection to server at "localhost" (::1), port 5432 failed: fe_sendauth: no password supplied
```

I found an answer here: https://stackoverflow.com/questions/12720967/how-to-change-postgresql-user-password

```
$ sudo -u postgres psql
psql (13.4 (Ubuntu 13.4-4.pgdg18.04+1), server 9.5.25)
Type "help" for help.

postgres=# \password postgres
Enter new password:
Enter it again:
postgres=#

CREATE database tutorial;

postgres=# CREATE EXTENSION IF NOT EXISTS timescaledb;
ERROR:  could not open extension control file "/usr/share/postgresql/9.5/extension/timescaledb.control": No such file or directory
```

Looks like I have old PostgreSQL.

```
$ ls -l /usr/share/postgresql/
10/  11/  13/  9.5/
```

But there is also newer PostgreSQL.

```
$ psql --version
psql (PostgreSQL) 13.4 (Ubuntu 13.4-4.pgdg18.04+1)

psql is new, so what is wrong?
```

Looks like I have all versions running simultaneously?

https://askubuntu.com/questions/17823/how-to-list-all-installed-packages

```
$ ps auxw | grep postgres
postgres  718818  0.0  0.5 33991600 730184 ?     Ss   23:29   0:00 /usr/lib/postgresql/13/bin/postgres -D /var/lib/postgresql/13/main -c config_file=/etc/postgresql/13/main/postgresql.conf
postgres  718825  0.0  0.0 320356 27660 ?        S    23:29   0:00 /usr/lib/postgresql/10/bin/postgres -D /var/lib/postgresql/10/main -c config_file=/etc/postgresql/10/main/postgresql.conf
postgres  718826  0.0  0.0 320712 27900 ?        S    23:29   0:00 /usr/lib/postgresql/11/bin/postgres -D /var/lib/postgresql/11/main -c config_file=/etc/postgresql/11/main/postgresql.conf
postgres  718829  0.0  0.0 320468  7092 ?        Ss   23:29   0:00 postgres: 10/main: checkpointer process
postgres  718830  0.0  0.0 320356  4300 ?        Ss   23:29   0:00 postgres: 10/main: writer process
postgres  718831  0.0  0.0 320356  9204 ?        Ss   23:29   0:00 postgres: 10/main: wal writer process
postgres  718832  0.0  0.0 320776  6964 ?        Ss   23:29   0:00 postgres: 10/main: autovacuum launcher process
postgres  718833  0.0  0.0 175404  3596 ?        Ss   23:29   0:00 postgres: 10/main: stats collector process
postgres  718834  0.0  0.0 320640  5052 ?        Ss   23:29   0:00 postgres: 10/main: bgworker: logical replication launcher
postgres  718835  0.0  0.0 320820  5592 ?        Ss   23:29   0:00 postgres: 11/main: checkpointer
postgres  718836  0.0  0.0 320712  4164 ?        Ss   23:29   0:00 postgres: 11/main: background writer
postgres  718837  0.0  0.0 320712  9040 ?        Ss   23:29   0:00 postgres: 11/main: walwriter
postgres  718838  0.0  0.0 321116  6824 ?        Ss   23:29   0:00 postgres: 11/main: autovacuum launcher
postgres  718839  0.0  0.0 175752  3652 ?        Ss   23:29   0:00 postgres: 11/main: stats collector
postgres  718840  0.0  0.0 321120  6640 ?        Ss   23:29   0:00 postgres: 11/main: logical replication launcher
postgres  718842  0.0  0.1 33991700 263860 ?     Ss   23:29   0:00 postgres: 13/main: checkpointer
postgres  718843  0.0  0.2 33991600 264096 ?     Ss   23:29   0:00 postgres: 13/main: background writer
postgres  718844  0.0  0.0 33991600 22044 ?      Ss   23:29   0:00 postgres: 13/main: walwriter
postgres  718845  0.0  0.0 33992284 7040 ?       Ss   23:29   0:00 postgres: 13/main: autovacuum launcher
postgres  718846  0.0  0.0 177920  4320 ?        Ss   23:29   0:00 postgres: 13/main: stats collector
postgres  718847  0.0  0.0 33992136 7972 ?       Ss   23:29   0:00 postgres: 13/main: TimescaleDB Background Worker Launcher
postgres  718848  0.0  0.0 33992164 7248 ?       Ss   23:29   0:00 postgres: 13/main: logical replication launcher
postgres  718857  0.0  0.0 304492 26284 ?        S    23:29   0:00 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
postgres  718859  0.0  0.0 304592  6480 ?        Ss   23:29   0:00 postgres: checkpointer process
postgres  718860  0.0  0.0 304492  5656 ?        Ss   23:29   0:00 postgres: writer process
postgres  718861  0.0  0.0 304492  4144 ?        Ss   23:29   0:00 postgres: wal writer process
postgres  718862  0.0  0.0 304928  6896 ?        Ss   23:29   0:00 postgres: autovacuum launcher process
postgres  718863  0.0  0.0 159744  4156 ?        Ss   23:29   0:00 postgres: stats collector process
milovid+  724277  0.0  0.0  14364  1024 pts/17   S+   23:41   0:00 grep --color=auto postgres

$ apt list --installed | grep postgres

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

postgresql-10/now 10.16-1.pgdg18.04+1 amd64 [installed,upgradable to: 10.18-1.pgdg18.04+1]
postgresql-11/now 11.11-1.pgdg18.04+1 amd64 [installed,upgradable to: 11.13-1.pgdg18.04+1]
postgresql-11-postgis-3/now 3.1.1+dfsg-1.pgdg18.04+1 amd64 [installed,upgradable to: 3.1.4+dfsg-1.pgdg18.04+1]
postgresql-11-postgis-3-scripts/now 3.1.1+dfsg-1.pgdg18.04+1 all [installed,upgradable to: 3.1.4+dfsg-1.pgdg18.04+1]
postgresql-13/bionic-pgdg,now 13.4-4.pgdg18.04+1 amd64 [installed,automatic]
postgresql-9.5/bionic-pgdg,now 9.5.25-1.pgdg18.04+1 amd64 [installed]
postgresql-9.5-postgis-2.2-scripts/now 2.2.2+dfsg-4.pgdg14.04+1.yandex all [installed,local]
postgresql-client-10/now 10.16-1.pgdg18.04+1 amd64 [installed,upgradable to: 10.18-1.pgdg18.04+1]
postgresql-client-11/now 11.11-1.pgdg18.04+1 amd64 [installed,upgradable to: 11.13-1.pgdg18.04+1]
postgresql-client-13/bionic-pgdg,now 13.4-4.pgdg18.04+1 amd64 [installed,automatic]
postgresql-client-9.5/bionic-pgdg,now 9.5.25-1.pgdg18.04+1 amd64 [installed]
postgresql-client-common/bionic-pgdg,now 231.pgdg18.04+1 all [installed]
postgresql-common/bionic-pgdg,now 231.pgdg18.04+1 all [installed]
timescaledb-2-loader-postgresql-13/bionic,now 2.5.0~ubuntu18.04 amd64 [installed,automatic]
timescaledb-2-postgresql-13/bionic,now 2.5.0~ubuntu18.04 amd64 [installed]
```

Let's remove all older packages.

```
sudo apt remove postgresql-10 postgresql-11 postgresql-9.5 postgresql-client-10 postgresql-client-11 postgresql-client-9.5
```

Just in case:

```
sudo service postgresql restart
```

Now it stopped to work:

```
$ sudo -u postgres psql
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?

$ sudo -u postgres psql -h localhost
psql: error: connection to server at "localhost" (::1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
```

But it's running:

```
$ ps auxw | grep postgres
postgres  726158  0.5  0.5 33991600 730084 ?     Ss   23:45   0:00 /usr/lib/postgresql/13/bin/postgres -D /var/lib/postgresql/13/main -c config_file=/etc/postgresql/13/main/postgresql.conf
postgres  726160  0.0  0.0 33991600 4256 ?       Ss   23:45   0:00 postgres: 13/main: checkpointer
postgres  726161  0.1  0.1 33991600 150048 ?     Ss   23:45   0:00 postgres: 13/main: background writer
postgres  726162  0.0  0.0 33991600 22044 ?      Ss   23:45   0:00 postgres: 13/main: walwriter
postgres  726163  0.0  0.0 33992284 6976 ?       Ss   23:45   0:00 postgres: 13/main: autovacuum launcher
postgres  726164  0.0  0.0 177920  4384 ?        Ss   23:45   0:00 postgres: 13/main: stats collector
postgres  726165  0.0  0.0 33992136 7840 ?       Ss   23:45   0:00 postgres: 13/main: TimescaleDB Background Worker Launcher
postgres  726166  0.0  0.0 33992164 7244 ?       Ss   23:45   0:00 postgres: 13/main: logical replication launcher
milovid+  726578  0.0  0.0  14364  1100 pts/17   S+   23:46   0:00 grep --color=auto postgres
```

But it does not listen 5432:

```
$ netstat -n | grep 5432
```

Let's look at the config:

```
sudo mcedit /etc/postgresql/13/main/postgresql.conf
```

```
# - Connection Settings -

#listen_addresses = 'localhost'
```

Looks like I need to uncomment it.

```
sudo service postgresql restart
```

But it did not help:

```
$ sudo -u postgres psql -h localhost
psql: error: connection to server at "localhost" (::1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
```

Let's consult https://stackoverflow.com/questions/31091748/postgres-server-not-listening

It is mentioning some pg_hba.conf. BTW what is HBA*? Let's find this file...

```
sudo mcedit /etc/postgresql/13/main/pg_hba.conf
```

\* host based authentication rules - it is explained inside this file.

Nothing wrong in this file...

```
$ sudo service postgresql status
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2021-10-29 23:50:14 MSK; 6min ago
  Process: 728545 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 728545 (code=exited, status=0/SUCCESS)

Oct 29 23:50:14 mtlog-perftest03j systemd[1]: postgresql.service: Changed dead -> start
Oct 29 23:50:14 mtlog-perftest03j systemd[1]: Starting PostgreSQL RDBMS...
Oct 29 23:50:14 mtlog-perftest03j systemd[728545]: postgresql.service: Executing: /bin/true
Oct 29 23:50:14 mtlog-perftest03j systemd[1]: postgresql.service: Child 728545 belongs to postgresql.service.
Oct 29 23:50:14 mtlog-perftest03j systemd[1]: postgresql.service: Main process exited, code=exited, status=0/SUCCESS
Oct 29 23:50:14 mtlog-perftest03j systemd[1]: postgresql.service: Changed start -> exited
Oct 29 23:50:14 mtlog-perftest03j systemd[1]: postgresql.service: Job postgresql.service/start finished, result=done
Oct 29 23:50:14 mtlog-perftest03j systemd[1]: Started PostgreSQL RDBMS.
Oct 29 23:50:14 mtlog-perftest03j systemd[1]: postgresql.service: Failed to send unit change signal for postgresql.service: Connection reset by peer
```

It's quite cryptic. What does it mean "Failed to send unit change signal"? Is it good or bad?
What is the "unit"? Maybe it is "SystemD Unit" - the phrase that I've heard many times but don't really understand.

Almost gave up... Wow, I found the culprit! In `/etc/postgresql/13/main/postgresql.conf`:

```
port = 5435
```

Most likely this has happened, because multiple versions of PostgreSQL were installed.

Let's change to 5432.

```
sudo mcedit /etc/postgresql/13/main/postgresql.conf
sudo service postgresql restart
```

But now it does not accept password:

```
milovidov@mtlog-perftest03j:~$ sudo -u postgres psql -h 127.0.0.1
Password for user postgres:
psql: error: connection to server at "127.0.0.1", port 5432 failed: fe_sendauth: no password supplied
milovidov@mtlog-perftest03j:~$ sudo -u postgres psql -h 127.0.0.1 --password ''
Password:
psql: error: connection to server at "127.0.0.1", port 5432 failed: fe_sendauth: no password supplied
milovidov@mtlog-perftest03j:~$ sudo -u postgres psql -h 127.0.0.1
Password for user postgres:
psql: error: connection to server at "127.0.0.1", port 5432 failed: fe_sendauth: no password supplied
```

Works this way:

```
$ sudo -u postgres psql
psql (13.4 (Ubuntu 13.4-4.pgdg18.04+1))
Type "help" for help.

postgres=# \password
Enter new password:
Enter it again:
```

It works with fine ASCII arc:

```
postgres=# CREATE database tutorial;
CREATE DATABASE
postgres=# \c tutorial
You are now connected to database "tutorial" as user "postgres".
tutorial=# CREATE EXTENSION IF NOT EXISTS timescaledb;
WARNING:
WELCOME TO
 _____ _                               _     ____________
|_   _(_)                             | |    |  _  \ ___ \
  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
               Running version 2.5.0
For more information on TimescaleDB, please visit the following links:

 1. Getting started: https://docs.timescale.com/timescaledb/latest/getting-started
 2. API reference documentation: https://docs.timescale.com/api/latest
 3. How TimescaleDB is designed: https://docs.timescale.com/timescaledb/latest/overview/core-concepts

Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescale.com/timescaledb/latest/how-to-guides/configuration/telemetry.

CREATE EXTENSION
```


## Creating Table

Continuing to https://docs.timescale.com/timescaledb/latest/how-to-guides/hypertables/create/

Create table:

```
CREATE TABLE  hits_100m_obfuscated (
WatchID BIGINT,
JavaEnable SMALLINT,
Title TEXT,
GoodEvent SMALLINT,
EventTime TIMESTAMP,
EventDate Date,
CounterID INTEGER,
ClientIP INTEGER,
RegionID INTEGER,
UserID BIGINT,
CounterClass SMALLINT,
OS SMALLINT,
UserAgent SMALLINT,
URL TEXT,
Referer TEXT,
Refresh SMALLINT,
RefererCategoryID SMALLINT,
RefererRegionID INTEGER,
URLCategoryID SMALLINT,
URLRegionID INTEGER,
ResolutionWidth SMALLINT,
ResolutionHeight SMALLINT,
ResolutionDepth SMALLINT,
FlashMajor SMALLINT,
FlashMinor SMALLINT,
FlashMinor2 TEXT,
NetMajor SMALLINT,
NetMinor SMALLINT,
UserAgentMajor SMALLINT,
UserAgentMinor CHAR(2),
CookieEnable SMALLINT,
JavascriptEnable SMALLINT,
IsMobile SMALLINT,
MobilePhone SMALLINT,
MobilePhoneModel TEXT,
Params TEXT,
IPNetworkID INTEGER,
TraficSourceID SMALLINT,
SearchEngineID SMALLINT,
SearchPhrase TEXT,
AdvEngineID SMALLINT,
IsArtifical SMALLINT,
WindowClientWidth SMALLINT,
WindowClientHeight SMALLINT,
ClientTimeZone SMALLINT,
ClientEventTime TIMESTAMP,
SilverlightVersion1 SMALLINT,
SilverlightVersion2 SMALLINT,
SilverlightVersion3 INTEGER,
SilverlightVersion4 SMALLINT,
PageCharset TEXT,
CodeVersion INTEGER,
IsLink SMALLINT,
IsDownload SMALLINT,
IsNotBounce SMALLINT,
FUniqID BIGINT,
OriginalURL TEXT,
HID INTEGER,
IsOldCounter SMALLINT,
IsEvent SMALLINT,
IsParameter SMALLINT,
DontCountHits SMALLINT,
WithHash SMALLINT,
HitColor CHAR,
LocalEventTime TIMESTAMP,
Age SMALLINT,
Sex SMALLINT,
Income SMALLINT,
Interests SMALLINT,
Robotness SMALLINT,
RemoteIP INTEGER,
WindowName INTEGER,
OpenerName INTEGER,
HistoryLength SMALLINT,
BrowserLanguage TEXT,
BrowserCountry TEXT,
SocialNetwork TEXT,
SocialAction TEXT,
HTTPError SMALLINT,
SendTiming INTEGER,
DNSTiming INTEGER,
ConnectTiming INTEGER,
ResponseStartTiming INTEGER,
ResponseEndTiming INTEGER,
FetchTiming INTEGER,
SocialSourceNetworkID SMALLINT,
SocialSourcePage TEXT,
ParamPrice BIGINT,
ParamOrderID TEXT,
ParamCurrency TEXT,
ParamCurrencyID SMALLINT,
OpenstatServiceName TEXT,
OpenstatCampaignID TEXT,
OpenstatAdID TEXT,
OpenstatSourceID TEXT,
UTMSource TEXT,
UTMMedium TEXT,
UTMCampaign TEXT,
UTMContent TEXT,
UTMTerm TEXT,
FromTag TEXT,
HasGCLID SMALLINT,
RefererHash BIGINT,
URLHash BIGINT,
CLID INTEGER
);
```

I remember PostgreSQL does not support unsigned integers. It also does not support TINYINT.
And it does not support zero bytes in TEXT fields. We will deal with it...

```
tutorial=# SELECT create_hypertable('hits_100m_obfuscated', 'EventTime');
ERROR:  column "EventTime" does not exist
```

WTF?

Maybe it because column names are lowercased?

```
tutorial=# SELECT create_hypertable('hits_100m_obfuscated', 'eventtime');
NOTICE:  adding not-null constraint to column "eventtime"
DETAIL:  Time dimensions cannot have NULL values.
         create_hypertable
-----------------------------------
 (1,public,hits_100m_obfuscated,t)
(1 row)
```

Looks like I forgot to specify NOT NULL for every column.
Let's repeat...

```
tutorial=# DROP TABLE hits_100m_obfuscated
tutorial-# ;
DROP TABLE
tutorial=# CREATE TABLE  hits_100m_obfuscated (
tutorial(# WatchID BIGINT NOT NULL,
tutorial(# JavaEnable SMALLINT NOT NULL,
tutorial(# Title TEXT NOT NULL,
tutorial(# GoodEvent SMALLINT NOT NULL,
tutorial(# EventTime TIMESTAMP NOT NULL,
tutorial(# EventDate Date NOT NULL,
tutorial(# CounterID INTEGER NOT NULL,
tutorial(# ClientIP INTEGER NOT NULL,
tutorial(# RegionID INTEGER NOT NULL,
tutorial(# UserID BIGINT NOT NULL,
tutorial(# CounterClass SMALLINT NOT NULL,
tutorial(# OS SMALLINT NOT NULL,
tutorial(# UserAgent SMALLINT NOT NULL,
tutorial(# URL TEXT NOT NULL,
tutorial(# Referer TEXT NOT NULL,
tutorial(# Refresh SMALLINT NOT NULL,
tutorial(# RefererCategoryID SMALLINT NOT NULL,
tutorial(# RefererRegionID INTEGER NOT NULL,
tutorial(# URLCategoryID SMALLINT NOT NULL,
tutorial(# URLRegionID INTEGER NOT NULL,
tutorial(# ResolutionWidth SMALLINT NOT NULL,
tutorial(# ResolutionHeight SMALLINT NOT NULL,
tutorial(# ResolutionDepth SMALLINT NOT NULL,
tutorial(# FlashMajor SMALLINT NOT NULL,
tutorial(# FlashMinor SMALLINT NOT NULL,
tutorial(# FlashMinor2 TEXT NOT NULL,
tutorial(# NetMajor SMALLINT NOT NULL,
tutorial(# NetMinor SMALLINT NOT NULL,
tutorial(# UserAgentMajor SMALLINT NOT NULL,
tutorial(# UserAgentMinor CHAR(2) NOT NULL,
tutorial(# CookieEnable SMALLINT NOT NULL,
tutorial(# JavascriptEnable SMALLINT NOT NULL,
tutorial(# IsMobile SMALLINT NOT NULL,
tutorial(# MobilePhone SMALLINT NOT NULL,
tutorial(# MobilePhoneModel TEXT NOT NULL,
tutorial(# Params TEXT NOT NULL,
tutorial(# IPNetworkID INTEGER NOT NULL,
tutorial(# TraficSourceID SMALLINT NOT NULL,
tutorial(# SearchEngineID SMALLINT NOT NULL,
tutorial(# SearchPhrase TEXT NOT NULL,
tutorial(# AdvEngineID SMALLINT NOT NULL,
tutorial(# IsArtifical SMALLINT NOT NULL,
tutorial(# WindowClientWidth SMALLINT NOT NULL,
tutorial(# WindowClientHeight SMALLINT NOT NULL,
tutorial(# ClientTimeZone SMALLINT NOT NULL,
tutorial(# ClientEventTime TIMESTAMP NOT NULL,
tutorial(# SilverlightVersion1 SMALLINT NOT NULL,
tutorial(# SilverlightVersion2 SMALLINT NOT NULL,
tutorial(# SilverlightVersion3 INTEGER NOT NULL,
tutorial(# SilverlightVersion4 SMALLINT NOT NULL,
tutorial(# PageCharset TEXT NOT NULL,
tutorial(# CodeVersion INTEGER NOT NULL,
tutorial(# IsLink SMALLINT NOT NULL,
tutorial(# IsDownload SMALLINT NOT NULL,
tutorial(# IsNotBounce SMALLINT NOT NULL,
tutorial(# FUniqID BIGINT NOT NULL,
tutorial(# OriginalURL TEXT NOT NULL,
tutorial(# HID INTEGER NOT NULL,
tutorial(# IsOldCounter SMALLINT NOT NULL,
tutorial(# IsEvent SMALLINT NOT NULL,
tutorial(# IsParameter SMALLINT NOT NULL,
tutorial(# DontCountHits SMALLINT NOT NULL,
tutorial(# WithHash SMALLINT NOT NULL,
tutorial(# HitColor CHAR NOT NULL,
tutorial(# LocalEventTime TIMESTAMP NOT NULL,
tutorial(# Age SMALLINT NOT NULL,
tutorial(# Sex SMALLINT NOT NULL,
tutorial(# Income SMALLINT NOT NULL,
tutorial(# Interests SMALLINT NOT NULL,
tutorial(# Robotness SMALLINT NOT NULL,
tutorial(# RemoteIP INTEGER NOT NULL,
tutorial(# WindowName INTEGER NOT NULL,
tutorial(# OpenerName INTEGER NOT NULL,
tutorial(# HistoryLength SMALLINT NOT NULL,
tutorial(# BrowserLanguage TEXT NOT NULL,
tutorial(# BrowserCountry TEXT NOT NULL,
tutorial(# SocialNetwork TEXT NOT NULL,
tutorial(# SocialAction TEXT NOT NULL,
tutorial(# HTTPError SMALLINT NOT NULL,
tutorial(# SendTiming INTEGER NOT NULL,
tutorial(# DNSTiming INTEGER NOT NULL,
tutorial(# ConnectTiming INTEGER NOT NULL,
tutorial(# ResponseStartTiming INTEGER NOT NULL,
tutorial(# ResponseEndTiming INTEGER NOT NULL,
tutorial(# FetchTiming INTEGER NOT NULL,
tutorial(# SocialSourceNetworkID SMALLINT NOT NULL,
tutorial(# SocialSourcePage TEXT NOT NULL,
tutorial(# ParamPrice BIGINT NOT NULL,
tutorial(# ParamOrderID TEXT NOT NULL,
tutorial(# ParamCurrency TEXT NOT NULL,
tutorial(# ParamCurrencyID SMALLINT NOT NULL,
tutorial(# OpenstatServiceName TEXT NOT NULL,
tutorial(# OpenstatCampaignID TEXT NOT NULL,
tutorial(# OpenstatAdID TEXT NOT NULL,
tutorial(# OpenstatSourceID TEXT NOT NULL,
tutorial(# UTMSource TEXT NOT NULL,
tutorial(# UTMMedium TEXT NOT NULL,
tutorial(# UTMCampaign TEXT NOT NULL,
tutorial(# UTMContent TEXT NOT NULL,
tutorial(# UTMTerm TEXT NOT NULL,
tutorial(# FromTag TEXT NOT NULL,
tutorial(# HasGCLID SMALLINT NOT NULL,
tutorial(# RefererHash BIGINT NOT NULL,
tutorial(# URLHash BIGINT NOT NULL,
tutorial(# CLID INTEGER NOT NULL
tutorial(# );
CREATE TABLE
tutorial=# SELECT create_hypertable('hits_100m_obfuscated', 'eventtime');
         create_hypertable
-----------------------------------
 (2,public,hits_100m_obfuscated,t)
(1 row)

tutorial=#
```

Now ok.


## Loading Data

Next - importing data:
https://docs.timescale.com/timescaledb/latest/how-to-guides/migrate-data/import-csv/#csv-import

```
SELECT WatchID::Int64, JavaEnable, toValidUTF8(Title), GoodEvent, EventTime, EventDate, CounterID::Int32, ClientIP::Int32, RegionID::Int32, UserID::Int64, CounterClass, OS, UserAgent, toValidUTF8(URL), toValidUTF8(Referer), Refresh, RefererCategoryID::Int16, RefererRegionID::Int32, URLCategoryID::Int16, URLRegionID::Int32, ResolutionWidth::Int16, ResolutionHeight::Int16, ResolutionDepth, FlashMajor, FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor::Int16, UserAgentMinor, CookieEnable, JavascriptEnable, IsMobile, MobilePhone, toValidUTF8(MobilePhoneModel), toValidUTF8(Params), IPNetworkID::Int32, TraficSourceID, SearchEngineID::Int16, toValidUTF8(SearchPhrase), AdvEngineID, IsArtifical, WindowClientWidth::Int16, WindowClientHeight::Int16, ClientTimeZone, ClientEventTime, SilverlightVersion1, SilverlightVersion2, SilverlightVersion3::Int32, SilverlightVersion4::Int16, toValidUTF8(PageCharset), CodeVersion::Int32, IsLink, IsDownload, IsNotBounce, FUniqID::Int64, toValidUTF8(OriginalURL), HID::Int32, IsOldCounter, IsEvent, IsParameter, DontCountHits, WithHash, HitColor, LocalEventTime, Age, Sex, Income, Interests::Int16, Robotness, RemoteIP::Int32, WindowName, OpenerName, HistoryLength, BrowserLanguage, BrowserCountry, toValidUTF8(SocialNetwork), toValidUTF8(SocialAction), HTTPError, SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming, SocialSourceNetworkID, toValidUTF8(SocialSourcePage), ParamPrice, toValidUTF8(ParamOrderID), ParamCurrency, ParamCurrencyID::Int16, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash::Int64, URLHash::Int64, CLID::Int32
FROM hits_100m_obfuscated
INTO OUTFILE 'dump.csv'
FORMAT CSV
```

https://github.com/ClickHouse/ClickHouse/issues/30872
https://github.com/ClickHouse/ClickHouse/issues/30873

```
$ wc -c dump.csv
80865718769 dump.csv
```

```
milovidov@mtlog-perftest03j:~$ timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV"
panic: could not connect: pq: password authentication failed for user "postgres"

goroutine 12 [running]:
main.processBatches(0xc00001e3c0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:238 +0x887
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
milovidov@mtlog-perftest03j:~$ sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV"
panic: could not connect: pq: password authentication failed for user "postgres"

goroutine 25 [running]:
main.processBatches(0xc00019a350, 0xc00019e660)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:238 +0x887
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb


milovidov@mtlog-perftest03j:~$ sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" --host localhost
flag provided but not defined: -host
Usage of timescaledb-parallel-copy:
  -batch-size int
        Number of rows per insert (default 5000)
  -columns string
        Comma-separated columns present in CSV
  -connection string
        PostgreSQL connection url (default "host=localhost user=postgres sslmode=disable")
  -copy-options string
        Additional options to pass to COPY (e.g., NULL 'NULL') (default "CSV")
  -db-name string
        Database where the destination table exists
  -file string
        File to read from rather than stdin
  -header-line-count int
        Number of header lines (default 1)
  -limit int
        Number of rows to insert overall; 0 means to insert all
  -log-batches
        Whether to time individual batches.
  -reporting-period duration
        Period to report insert stats; if 0s, intermediate results will not be reported
  -schema string
        Destination table's schema (default "public")
  -skip-header
        Skip the first line of the input
  -split string
        Character to split by (default ",")
  -table string
        Destination table for insertions (default "test_table")
  -token-size int
        Maximum size to use for tokens. By default, this is 64KB, so any value less than that will be ignored (default 65536)
  -truncate
        Truncate the destination table before insert
  -verbose
        Print more information about copying statistics
  -version
        Show the version of this tool
  -workers int
        Number of parallel requests to make (default 1)


milovidov@mtlog-perftest03j:~$ sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost'
panic: could not connect: pq: password authentication failed for user "postgres"

goroutine 14 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:238 +0x887
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
panic: could not connect: pq: password authentication failed for user "postgres"

goroutine 13 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:238 +0x887
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
panic: could not connect: pq: password authentication failed for user "postgres"

goroutine 12 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:238 +0x887
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb


milovidov@mtlog-perftest03j:~$ sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password 12345'
panic: could not connect: cannot parse `host=localhost password 12345`: failed to parse as DSN (invalid dsn)

goroutine 13 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:238 +0x887
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb


milovidov@mtlog-perftest03j:~$ sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password=12345'
panic: pq: invalid byte sequence for encoding "UTF8": 0xe0 0x22 0x2c

goroutine 34 [running]:
main.processBatches(0xc000132350, 0xc000136660)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
panic: pq: invalid byte sequence for encoding "UTF8": 0xe0 0x22 0x2c

goroutine 30 [running]:
main.processBatches(0xc000132350, 0xc000136660)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
```

Ok, now I've got something meaningful.
But it does not show, what line has error...

```
$ echo -e '\xe0\x22\x2c'
<EFBFBD>",
```

Let's recreate the dump:

```
rm dump.csv

SELECT WatchID::Int64, JavaEnable, toValidUTF8(Title), GoodEvent, EventTime, EventDate, CounterID::Int32, ClientIP::Int32, RegionID::Int32,
    UserID::Int64, CounterClass, OS, UserAgent, toValidUTF8(URL), toValidUTF8(Referer), Refresh, RefererCategoryID::Int16, RefererRegionID::Int32,
    URLCategoryID::Int16, URLRegionID::Int32, ResolutionWidth::Int16, ResolutionHeight::Int16, ResolutionDepth, FlashMajor, FlashMinor,
    FlashMinor2, NetMajor, NetMinor, UserAgentMajor::Int16, toValidUTF8(UserAgentMinor::String), CookieEnable, JavascriptEnable, IsMobile, MobilePhone,
    toValidUTF8(MobilePhoneModel), toValidUTF8(Params), IPNetworkID::Int32, TraficSourceID, SearchEngineID::Int16, toValidUTF8(SearchPhrase),
    AdvEngineID, IsArtifical, WindowClientWidth::Int16, WindowClientHeight::Int16, ClientTimeZone, ClientEventTime,
    SilverlightVersion1, SilverlightVersion2, SilverlightVersion3::Int32, SilverlightVersion4::Int16, toValidUTF8(PageCharset),
    CodeVersion::Int32, IsLink, IsDownload, IsNotBounce, FUniqID::Int64, toValidUTF8(OriginalURL), HID::Int32, IsOldCounter, IsEvent,
    IsParameter, DontCountHits, WithHash, toValidUTF8(HitColor::String), LocalEventTime, Age, Sex, Income, Interests::Int16, Robotness, RemoteIP::Int32,
    WindowName, OpenerName, HistoryLength, toValidUTF8(BrowserLanguage::String), toValidUTF8(BrowserCountry::String),
    toValidUTF8(SocialNetwork), toValidUTF8(SocialAction),
    HTTPError, SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming, SocialSourceNetworkID,
    toValidUTF8(SocialSourcePage), ParamPrice, toValidUTF8(ParamOrderID), toValidUTF8(ParamCurrency::String),
    ParamCurrencyID::Int16, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID,
    UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash::Int64, URLHash::Int64, CLID::Int32
FROM hits_100m_obfuscated
INTO OUTFILE 'dump.csv'
FORMAT CSV
```

```
$ sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 1 --copy-options "CSV" -connection 'host=localhost password=12345'
panic: pq: value too long for type character(2)

goroutine 6 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
```

ALTER does not work:

```
tutorial=# ALTER TABLE hits_100m_obfuscated MODIFY COLUMN UserAgentMinor TEXT
tutorial-# ;
ERROR:  syntax error at or near "MODIFY"
LINE 1: ALTER TABLE hits_100m_obfuscated MODIFY COLUMN UserAgentMino...
                                         ^
```

PostgreSQL is using unusual syntax for ALTER:

```
tutorial=# ALTER TABLE hits_100m_obfuscated ALTER COLUMN UserAgentMinor TYPE TEXT
;
ALTER TABLE
tutorial=# \q
```

https://github.com/ClickHouse/ClickHouse/issues/30874

Now something again:

```
$ sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 1 --copy-options "CSV" -connection 'host=localhost password=12345'
panic: pq: value "2149615427" is out of range for type integer

goroutine 6 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
```

```
$ grep -F '2149615427' dump.csv
5607505572457935073,0,"Лазар автоматические пылесосы подробная школы. Когалерея — Курсы на Автория пище Сноудента новые устами",1,"2013-07-15 07:47:45","2013-07-15",38,-1194330980,229,-6649844357037090659,0,2,3,"https://produkty%2Fkategory_id=&auto-nexus.html?blockfesty-i-korroszhego","http://tambov.irr.ua/yandex.ru/saledParam=0&user/auto.ria",1,10282,995,15014,519,1996,1781,23,14,2,"800",0,0,7,"D<>",1,1,0,0,"","",3392210,-1,0,"",0,0,1261,1007,135,"2013-07-15 21:54:13",0,0,0,0,"windows-1251;charset",1601,0,0,0,8184671896482443026,"",451733382,0,0,0,0,0,"5","2013-07-15 15:41:14",31,1,3,60,13,-1855237933,-1,-1,-1,"S0","h1","","",0,0,0,0,2149615427,36,3,0,"",0,"","NH",0,"","","","","","","","","","",0,-1103774879459415602,-2414747266057209563,0
^C
```

Let's recreate the dump:

```
rm dump.csv

SELECT WatchID::Int64, JavaEnable, toValidUTF8(Title), GoodEvent, EventTime, EventDate, CounterID::Int32, ClientIP::Int32, RegionID::Int32,
    UserID::Int64, CounterClass, OS, UserAgent, toValidUTF8(URL), toValidUTF8(Referer), Refresh, RefererCategoryID::Int16, RefererRegionID::Int32,
    URLCategoryID::Int16, URLRegionID::Int32, ResolutionWidth::Int16, ResolutionHeight::Int16, ResolutionDepth, FlashMajor, FlashMinor,
    FlashMinor2, NetMajor, NetMinor, UserAgentMajor::Int16, toValidUTF8(UserAgentMinor::String), CookieEnable, JavascriptEnable, IsMobile, MobilePhone,
    toValidUTF8(MobilePhoneModel), toValidUTF8(Params), IPNetworkID::Int32, TraficSourceID, SearchEngineID::Int16, toValidUTF8(SearchPhrase),
    AdvEngineID, IsArtifical, WindowClientWidth::Int16, WindowClientHeight::Int16, ClientTimeZone, ClientEventTime,
    SilverlightVersion1, SilverlightVersion2, SilverlightVersion3::Int32, SilverlightVersion4::Int16, toValidUTF8(PageCharset),
    CodeVersion::Int32, IsLink, IsDownload, IsNotBounce, FUniqID::Int64, toValidUTF8(OriginalURL), HID::Int32, IsOldCounter, IsEvent,
    IsParameter, DontCountHits, WithHash, toValidUTF8(HitColor::String), LocalEventTime, Age, Sex, Income, Interests::Int16, Robotness, RemoteIP::Int32,
    WindowName, OpenerName, HistoryLength, toValidUTF8(BrowserLanguage::String), toValidUTF8(BrowserCountry::String),
    toValidUTF8(SocialNetwork), toValidUTF8(SocialAction),
    HTTPError, least(SendTiming, 30000), least(DNSTiming, 30000), least(ConnectTiming, 30000), least(ResponseStartTiming, 30000),
    least(ResponseEndTiming, 30000), least(FetchTiming, 30000), SocialSourceNetworkID,
    toValidUTF8(SocialSourcePage), ParamPrice, toValidUTF8(ParamOrderID), toValidUTF8(ParamCurrency::String),
    ParamCurrencyID::Int16, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID,
    UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash::Int64, URLHash::Int64, CLID::Int32
FROM hits_100m_obfuscated
INTO OUTFILE 'dump.csv'
FORMAT CSV
```

PostgreSQL does not support USE database.
But I remember, that I can write `\c` instead. I guess `\c` means "change" (the database). Or it is called "schema" or "catalog".

```
milovidov@mtlog-perftest03j:~$ sudo -u postgres psql
psql (13.4 (Ubuntu 13.4-4.pgdg18.04+1))
Type "help" for help.

postgres=# SELECT count(*) FROM hits_100m_obfuscated;
ERROR:  relation "hits_100m_obfuscated" does not exist
LINE 1: SELECT count(*) FROM hits_100m_obfuscated;
                             ^
postgres=# USE tutorial;
ERROR:  syntax error at or near "USE"
LINE 1: USE tutorial;
        ^
postgres=# \c tutorial
You are now connected to database "tutorial" as user "postgres".
tutorial=# SELECT count(*) FROM hits_100m_obfuscated;
 count
-------
 69996
(1 row)
```

And parallel loader already loaded some part of data into my table (it is not transactional).
Let's truncate table:

```
tutorial=# TRUNCATE TABLE hits_100m_obfuscated;
TRUNCATE TABLE
```

Surprisingly, it works!

Now it started loading data:
```
$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password=12345'
```

But the loading is not using 16 CPU cores and it is not bottlenecked by IO.

WTF:

```
$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password=12345'
panic: pq: could not extend file "base/16384/31264.1": wrote only 4096 of 8192 bytes at block 145407

goroutine 6 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    3m31.328s
user    0m35.016s
sys     0m6.964s
```

Looks like there is no space:

```
milovidov@mtlog-perftest03j:~$ df -h /var/lib/postgresql/13/main
Filesystem      Size  Used Avail Use% Mounted on
/dev/md1         35G   33G  1.4G  97% /
```

https://github.com/ClickHouse/ClickHouse/issues/30883

Let's move to another device.

```
milovidov@mtlog-perftest03j:~$ sudo mkdir /opt/postgresql
milovidov@mtlog-perftest03j:~$ sudo ls -l /var/lib/postgresql/13/main
total 88
drwx------ 6 postgres postgres  4096 Oct 30 00:06 base
drwx------ 2 postgres postgres  4096 Oct 30 02:07 global
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_commit_ts
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_dynshmem
drwx------ 4 postgres postgres  4096 Oct 30 02:10 pg_logical
drwx------ 4 postgres postgres  4096 Oct 29 23:27 pg_multixact
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_notify
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_replslot
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_serial
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_snapshots
drwx------ 2 postgres postgres  4096 Oct 30 02:10 pg_stat
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_stat_tmp
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_subtrans
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_tblspc
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_twophase
-rw------- 1 postgres postgres     3 Oct 29 23:27 PG_VERSION
drwx------ 3 postgres postgres 12288 Oct 30 02:10 pg_wal
drwx------ 2 postgres postgres  4096 Oct 29 23:27 pg_xact
-rw------- 1 postgres postgres    88 Oct 29 23:27 postgresql.auto.conf
-rw------- 1 postgres postgres   130 Oct 30 00:03 postmaster.opts
milovidov@mtlog-perftest03j:~$ sudo chown postgres:postgres /opt/postgresql
milovidov@mtlog-perftest03j:~$ sudo mv /var/lib/postgresql/13/main/* /opt/postgresql
mv: cannot stat '/var/lib/postgresql/13/main/*': No such file or directory
milovidov@mtlog-perftest03j:~$ sudo bash -c 'mv /var/lib/postgresql/13/main/* /opt/postgresql'
sudo ln milovidov@mtlog-perftest03j:~$ #sudo ln -s /opt/postgresql /var/lib/postgresql/13/main
milovidov@mtlog-perftest03j:~$ sudo rm /var/lib/postgresql/13/main
rm: cannot remove '/var/lib/postgresql/13/main': Is a directory
milovidov@mtlog-perftest03j:~$ sudo rm -rf /var/lib/postgresql/13/main
milovidov@mtlog-perftest03j:~$ sudo ln -s /opt/postgresql /var/lib/postgresql/13/main
milovidov@mtlog-perftest03j:~$ sudo ls -l /var/lib/postgresql/13/main
lrwxrwxrwx 1 root root 15 Oct 30 02:12 /var/lib/postgresql/13/main -> /opt/postgresql
milovidov@mtlog-perftest03j:~$ sudo ls -l /opt/postgresql/
total 80
drwx------ 6 postgres postgres 4096 Oct 30 00:06 base
drwx------ 2 postgres postgres 4096 Oct 30 02:07 global
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_commit_ts
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_dynshmem
drwx------ 4 postgres postgres 4096 Oct 30 02:10 pg_logical
drwx------ 4 postgres postgres 4096 Oct 29 23:27 pg_multixact
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_notify
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_replslot
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_serial
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_snapshots
drwx------ 2 postgres postgres 4096 Oct 30 02:10 pg_stat
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_subtrans
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_tblspc
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_twophase
-rw------- 1 postgres postgres    3 Oct 29 23:27 PG_VERSION
drwx------ 3 postgres postgres 4096 Oct 30 02:10 pg_wal
drwx------ 2 postgres postgres 4096 Oct 29 23:27 pg_xact
-rw------- 1 postgres postgres   88 Oct 29 23:27 postgresql.auto.conf
-rw------- 1 postgres postgres  130 Oct 30 00:03 postmaster.opts

sudo service postgresql start

sudo less /var/log/postgresql/postgresql-13-main.log

2021-10-30 02:13:41.284 MSK [791362] FATAL:  data directory "/var/lib/postgresql/13/main" has invalid permissions
2021-10-30 02:13:41.284 MSK [791362] DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
pg_ctl: could not start server
Examine the log output.

sudo chmod 0700 /var/lib/postgresql/13/main /opt/postgresql
sudo service postgresql start

postgres=# \c tutorial
You are now connected to database "tutorial" as user "postgres".
tutorial=# TRUNCATE TABLE hits_100m_obfuscated;
TRUNCATE TABLE
```

```
$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password=12345'
```

No success:

```
$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password=12345'
panic: pq: invalid byte sequence for encoding "UTF8": 0x00

goroutine 29 [running]:
main.processBatches(0xc000132350, 0xc000136660)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    11m47.879s
user    3m10.980s
sys     0m45.256s
```

The error message is false, because UTF-8 **does** support 0x00. It is just some PostgreSQL quirk.

Let's recreate the dump:

```
rm dump.csv

SELECT WatchID::Int64, JavaEnable, replaceAll(toValidUTF8(Title), '\0', ''), GoodEvent, EventTime, EventDate, CounterID::Int32, ClientIP::Int32, RegionID::Int32,
    UserID::Int64, CounterClass, OS, UserAgent, replaceAll(toValidUTF8(URL), '\0', ''), replaceAll(toValidUTF8(Referer), '\0', ''), Refresh, RefererCategoryID::Int16, RefererRegionID::Int32,
    URLCategoryID::Int16, URLRegionID::Int32, ResolutionWidth::Int16, ResolutionHeight::Int16, ResolutionDepth, FlashMajor, FlashMinor,
    FlashMinor2, NetMajor, NetMinor, UserAgentMajor::Int16, replaceAll(toValidUTF8(UserAgentMinor::String), '\0', ''), CookieEnable, JavascriptEnable, IsMobile, MobilePhone,
    replaceAll(toValidUTF8(MobilePhoneModel), '\0', ''), replaceAll(toValidUTF8(Params), '\0', ''), IPNetworkID::Int32, TraficSourceID, SearchEngineID::Int16, replaceAll(toValidUTF8(SearchPhrase), '\0', ''),
    AdvEngineID, IsArtifical, WindowClientWidth::Int16, WindowClientHeight::Int16, ClientTimeZone, ClientEventTime,
    SilverlightVersion1, SilverlightVersion2, SilverlightVersion3::Int32, SilverlightVersion4::Int16, replaceAll(toValidUTF8(PageCharset), '\0', ''),
    CodeVersion::Int32, IsLink, IsDownload, IsNotBounce, FUniqID::Int64, replaceAll(toValidUTF8(OriginalURL), '\0', ''), HID::Int32, IsOldCounter, IsEvent,
    IsParameter, DontCountHits, WithHash, replaceAll(toValidUTF8(HitColor::String), '\0', ''), LocalEventTime, Age, Sex, Income, Interests::Int16, Robotness, RemoteIP::Int32,
    WindowName, OpenerName, HistoryLength, replaceAll(toValidUTF8(BrowserLanguage::String), '\0', ''), replaceAll(toValidUTF8(BrowserCountry::String), '\0', ''),
    replaceAll(toValidUTF8(SocialNetwork), '\0', ''), replaceAll(toValidUTF8(SocialAction), '\0', ''),
    HTTPError, least(SendTiming, 30000), least(DNSTiming, 30000), least(ConnectTiming, 30000), least(ResponseStartTiming, 30000),
    least(ResponseEndTiming, 30000), least(FetchTiming, 30000), SocialSourceNetworkID,
    replaceAll(toValidUTF8(SocialSourcePage), '\0', ''), ParamPrice, replaceAll(toValidUTF8(ParamOrderID), '\0', ''), replaceAll(toValidUTF8(ParamCurrency::String), '\0', ''),
    ParamCurrencyID::Int16, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID,
    UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash::Int64, URLHash::Int64, CLID::Int32
FROM hits_100m_obfuscated
INTO OUTFILE 'dump.csv'
FORMAT CSV
```

WTF:

```
tutorial=# SELECT count(*) FROM hits_100m_obfuscated;
ERROR:  could not load library "/usr/lib/postgresql/13/lib/llvmjit.so": libLLVM-6.0.so.1: cannot open shared object file: No such file or directory
```

Maybe just install LLVM?

```
sudo apt install llvm
```

It does not help:

```
milovidov@mtlog-perftest03j:~$ sudo -u postgres psql
psql (13.4 (Ubuntu 13.4-4.pgdg18.04+1))
Type "help" for help.

postgres=# \c tutorial
You are now connected to database "tutorial" as user "postgres".
tutorial=# SELECT count(*) FROM hits_100m_obfuscated;
ERROR:  could not load library "/usr/lib/postgresql/13/lib/llvmjit.so": libLLVM-6.0.so.1: cannot open shared object file: No such file or directory
tutorial=#
```

Dependency on system libraries is harmful.

```
milovidov@mtlog-perftest03j:~$ ls -l /usr/lib/x86_64-linux-gnu/libLLVM-6.0.so
lrwxrwxrwx 1 root root 16 Apr  6  2018 /usr/lib/x86_64-linux-gnu/libLLVM-6.0.so -> libLLVM-6.0.so.1
milovidov@mtlog-perftest03j:~$ ls -l /usr/lib/x86_64-linux-gnu/libLLVM-6.0.so.1
ls: cannot access '/usr/lib/x86_64-linux-gnu/libLLVM-6.0.so.1': No such file or directory
```

https://askubuntu.com/questions/481/how-do-i-find-the-package-that-provides-a-file

```
milovidov@mtlog-perftest03j:~$ dpkg -S libLLVM-6.0.so.1
llvm-6.0-dev: /usr/lib/llvm-6.0/lib/libLLVM-6.0.so.1
libllvm6.0:amd64: /usr/lib/x86_64-linux-gnu/libLLVM-6.0.so.1
```

Wow, it's absolutely broken:

```
milovidov@mtlog-perftest03j:~$ sudo apt remove llvm-6.0-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libcgal13 libgmpxx4ldbl liblldb-11 libprotobuf-c1 libsfcgal1 mysql-server-core-5.7
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  liblld-6.0-dev lld lld-6.0 llvm-6.0-dev
0 upgraded, 0 newly installed, 4 to remove and 293 not upgraded.
After this operation, 163 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 268641 files and directories currently installed.)
Removing liblld-6.0-dev (1:6.0-1ubuntu2) ...
Removing lld (1:6.0-41~exp5~ubuntu1) ...
Removing lld-6.0 (1:6.0-1ubuntu2) ...
Removing llvm-6.0-dev (1:6.0-1ubuntu2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
milovidov@mtlog-perftest03j:~$ sudo apt install llvm-6.0-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libcgal13 libgmpxx4ldbl liblldb-11 libprotobuf-c1 libsfcgal1 mysql-server-core-5.7
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  llvm-6.0-dev
0 upgraded, 1 newly installed, 0 to remove and 293 not upgraded.
Need to get 23.0 MB of archives.
After this operation, 160 MB of additional disk space will be used.
Get:1 http://mirror.yandex.ru/ubuntu bionic/main amd64 llvm-6.0-dev amd64 1:6.0-1ubuntu2 [23.0 MB]
Fetched 23.0 MB in 1s (42.5 MB/s)
Selecting previously unselected package llvm-6.0-dev.
(Reading database ... 267150 files and directories currently installed.)
Preparing to unpack .../llvm-6.0-dev_1%3a6.0-1ubuntu2_amd64.deb ...
Unpacking llvm-6.0-dev (1:6.0-1ubuntu2) ...
Setting up llvm-6.0-dev (1:6.0-1ubuntu2) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
milovidov@mtlog-perftest03j:~$ ls -l /usr/lib/x86_64-linux-gnu/libLLVM-6.0.so
lrwxrwxrwx 1 root root 16 Apr  6  2018 /usr/lib/x86_64-linux-gnu/libLLVM-6.0.so -> libLLVM-6.0.so.1
milovidov@mtlog-perftest03j:~$ ls -l /usr/lib/x86_64-linux-gnu/libLLVM-6.0.so.1
ls: cannot access '/usr/lib/x86_64-linux-gnu/libLLVM-6.0.so.1': No such file or directory
```

Let's remove just in case:

```
sudo apt remove llvm-6.0-dev
```

https://dba.stackexchange.com/questions/264955/handling-performance-problems-with-jit-in-postgres-12

JIT can be disabled by `set jit = off;`

```
tutorial=# set jit = off;
SET
tutorial=#
tutorial=# SELECT count(*) FROM hits_100m_obfuscated;
```

But now this SELECT query started and hanged for multiple minutes without any result.
And I see something strange in `top`:

```
 792553 postgres  20   0 32.418g 0.031t 0.031t D   2.4 25.3   3:43.84 postgres: 13/main: checkpointer
 814659 postgres  20   0 32.432g 0.023t 0.023t D   0.0 18.8   0:14.53 postgres: 13/main: parallel worker for PID 813980
 813980 postgres  20   0 32.433g 0.023t 0.023t D   0.0 18.4   0:14.47 postgres: 13/main: postgres tutorial [local] SELECT
 814657 postgres  20   0 32.432g 0.016t 0.016t D   0.0 12.6   0:09.83 postgres: 13/main: parallel worker for PID 813980
 814658 postgres  20   0 32.432g 0.015t 0.015t D   2.4 12.6   0:09.45 postgres: 13/main: parallel worker for PID 813980
 814656 postgres  20   0 32.432g 0.015t 0.015t D   0.0 12.0   0:07.36 postgres: 13/main: parallel worker for PID 813980
 792554 postgres  20   0 32.417g 5.394g 5.392g D   0.0  4.3   0:04.78 postgres: 13/main: background writer
```

The query did not finish in 30 minutes. How it can be so enormously slow?


Loading failed, again:

```
$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password=12345'
panic: pq: extra data after last expected column

goroutine 14 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    20m57.936s
user    4m14.444s
sys     1m11.412s
```

Most likely PostgreSQL cannot recognize proper CSV escaping of quotes like `"Hello "" world"`.
Let's simply remove all double quotes from String values.

```
rm dump.csv

SELECT WatchID::Int64, JavaEnable, replaceAll(replaceAll(toValidUTF8(Title), '\0', ''), '"', ''), GoodEvent, EventTime, EventDate, CounterID::Int32, ClientIP::Int32, RegionID::Int32,
    UserID::Int64, CounterClass, OS, UserAgent, replaceAll(replaceAll(toValidUTF8(URL), '\0', ''), '"', ''), replaceAll(replaceAll(toValidUTF8(Referer), '\0', ''), '"', ''), Refresh, RefererCategoryID::Int16, RefererRegionID::Int32,
    URLCategoryID::Int16, URLRegionID::Int32, ResolutionWidth::Int16, ResolutionHeight::Int16, ResolutionDepth, FlashMajor, FlashMinor,
    FlashMinor2, NetMajor, NetMinor, UserAgentMajor::Int16, replaceAll(replaceAll(toValidUTF8(UserAgentMinor::String), '\0', ''), '"', ''), CookieEnable, JavascriptEnable, IsMobile, MobilePhone,
    replaceAll(replaceAll(toValidUTF8(MobilePhoneModel), '\0', ''), '"', ''), replaceAll(replaceAll(toValidUTF8(Params), '\0', ''), '"', ''), IPNetworkID::Int32, TraficSourceID, SearchEngineID::Int16, replaceAll(replaceAll(toValidUTF8(SearchPhrase), '\0', ''), '"', ''),
    AdvEngineID, IsArtifical, WindowClientWidth::Int16, WindowClientHeight::Int16, ClientTimeZone, ClientEventTime,
    SilverlightVersion1, SilverlightVersion2, SilverlightVersion3::Int32, SilverlightVersion4::Int16, replaceAll(replaceAll(toValidUTF8(PageCharset), '\0', ''), '"', ''),
    CodeVersion::Int32, IsLink, IsDownload, IsNotBounce, FUniqID::Int64, replaceAll(replaceAll(toValidUTF8(OriginalURL), '\0', ''), '"', ''), HID::Int32, IsOldCounter, IsEvent,
    IsParameter, DontCountHits, WithHash, replaceAll(replaceAll(toValidUTF8(HitColor::String), '\0', ''), '"', ''), LocalEventTime, Age, Sex, Income, Interests::Int16, Robotness, RemoteIP::Int32,
    WindowName, OpenerName, HistoryLength, replaceAll(replaceAll(toValidUTF8(BrowserLanguage::String), '\0', ''), '"', ''), replaceAll(replaceAll(toValidUTF8(BrowserCountry::String), '\0', ''), '"', ''),
    replaceAll(replaceAll(toValidUTF8(SocialNetwork), '\0', ''), '"', ''), replaceAll(replaceAll(toValidUTF8(SocialAction), '\0', ''), '"', ''),
    HTTPError, least(SendTiming, 30000), least(DNSTiming, 30000), least(ConnectTiming, 30000), least(ResponseStartTiming, 30000),
    least(ResponseEndTiming, 30000), least(FetchTiming, 30000), SocialSourceNetworkID,
    replaceAll(replaceAll(toValidUTF8(SocialSourcePage), '\0', ''), '"', ''), ParamPrice, replaceAll(replaceAll(toValidUTF8(ParamOrderID), '\0', ''), '"', ''), replaceAll(replaceAll(toValidUTF8(ParamCurrency::String), '\0', ''), '"', ''),
    ParamCurrencyID::Int16, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID,
    UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash::Int64, URLHash::Int64, CLID::Int32
FROM hits_100m_obfuscated
INTO OUTFILE 'dump.csv'
FORMAT CSV
```

Oops, another trouble:

```
$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password=12345'
panic: pq: unterminated CSV quoted field

goroutine 19 [running]:
main.processBatches(0xc000132350, 0xc000136660)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m38.278s
user    0m13.544s
sys     0m3.552s
```

I have hypothesis, maybe it is interpreting both backslashes and quotes in CSV?
We need to check, what is CSV, exactly, from TimescaleDB's standpoint.

https://www.postgresql.org/docs/9.2/sql-copy.html

Yes, PostgreSQL is using "fake CSV":

> This format option is used for importing and exporting the Comma Separated Value (CSV) file format used by many other programs, such as spreadsheets. Instead of the escaping rules used by PostgreSQL's standard text format, it produces and recognizes the common CSV escaping mechanism.

> The values in each record are separated by the DELIMITER character. If the value contains the delimiter character, the QUOTE character, the NULL string, a carriage return, or line feed character, then the whole value is prefixed and suffixed by the QUOTE character, and any occurrence within the value of a QUOTE character or the ESCAPE character is preceded by the escape character.

So, it looks like CSV but is using C-style backslash escapes inside values.
Let's remove both backslash and quote from our strings to make PostgreSQL happy.

```
rm dump.csv

SELECT WatchID::Int64, JavaEnable, replaceAll(replaceAll(replaceAll(toValidUTF8(Title), '\0', ''), '"', ''), '\\', ''), GoodEvent, EventTime, EventDate, CounterID::Int32, ClientIP::Int32, RegionID::Int32,
    UserID::Int64, CounterClass, OS, UserAgent, replaceAll(replaceAll(replaceAll(toValidUTF8(URL), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(Referer), '\0', ''), '"', ''), '\\', ''), Refresh, RefererCategoryID::Int16, RefererRegionID::Int32,
    URLCategoryID::Int16, URLRegionID::Int32, ResolutionWidth::Int16, ResolutionHeight::Int16, ResolutionDepth, FlashMajor, FlashMinor,
    FlashMinor2, NetMajor, NetMinor, UserAgentMajor::Int16, replaceAll(replaceAll(replaceAll(toValidUTF8(UserAgentMinor::String), '\0', ''), '"', ''), '\\', ''), CookieEnable, JavascriptEnable, IsMobile, MobilePhone,
    replaceAll(replaceAll(replaceAll(toValidUTF8(MobilePhoneModel), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(Params), '\0', ''), '"', ''), '\\', ''), IPNetworkID::Int32, TraficSourceID, SearchEngineID::Int16, replaceAll(replaceAll(replaceAll(toValidUTF8(SearchPhrase), '\0', ''), '"', ''), '\\', ''),
    AdvEngineID, IsArtifical, WindowClientWidth::Int16, WindowClientHeight::Int16, ClientTimeZone, ClientEventTime,
    SilverlightVersion1, SilverlightVersion2, SilverlightVersion3::Int32, SilverlightVersion4::Int16, replaceAll(replaceAll(replaceAll(toValidUTF8(PageCharset), '\0', ''), '"', ''), '\\', ''),
    CodeVersion::Int32, IsLink, IsDownload, IsNotBounce, FUniqID::Int64, replaceAll(replaceAll(replaceAll(toValidUTF8(OriginalURL), '\0', ''), '"', ''), '\\', ''), HID::Int32, IsOldCounter, IsEvent,
    IsParameter, DontCountHits, WithHash, replaceAll(replaceAll(replaceAll(toValidUTF8(HitColor::String), '\0', ''), '"', ''), '\\', ''), LocalEventTime, Age, Sex, Income, Interests::Int16, Robotness, RemoteIP::Int32,
    WindowName, OpenerName, HistoryLength, replaceAll(replaceAll(replaceAll(toValidUTF8(BrowserLanguage::String), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(BrowserCountry::String), '\0', ''), '"', ''), '\\', ''),
    replaceAll(replaceAll(replaceAll(toValidUTF8(SocialNetwork), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(SocialAction), '\0', ''), '"', ''), '\\', ''),
    HTTPError, least(SendTiming, 30000), least(DNSTiming, 30000), least(ConnectTiming, 30000), least(ResponseStartTiming, 30000),
    least(ResponseEndTiming, 30000), least(FetchTiming, 30000), SocialSourceNetworkID,
    replaceAll(replaceAll(replaceAll(toValidUTF8(SocialSourcePage), '\0', ''), '"', ''), '\\', ''), ParamPrice, replaceAll(replaceAll(replaceAll(toValidUTF8(ParamOrderID), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(ParamCurrency::String), '\0', ''), '"', ''), '\\', ''),
    ParamCurrencyID::Int16, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID,
    UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash::Int64, URLHash::Int64, CLID::Int32
FROM hits_100m_obfuscated
INTO OUTFILE 'dump.csv'
FORMAT CSV
```

It does not work at all:

```
$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.csv --workers 16 --copy-options "CSV" -connection 'host=localhost password=12345'
panic: pq: invalid input syntax for type bigint: "       ПЕСНЮ  ПРЕСТИВАРКЕ ДОЛЖНО ЛИ,1,306,31432,304,22796,1011,879,37,15,5,700.224,2,7,13,D<>,1,1,0,0,",",3039109,-1,0,",0,0,779,292,135,2013-07-31 09:37:12,0,0,0,0,windows,1,0,0,0,6888403766694734958,http%3A//maps&sort_order_Kurzarm_DOB&sr=http%3A%2F%3Fpage=/ok.html?1=1&cid=577&oki=1&op_seo_entry=&op_uid=13225;IC"

goroutine 20 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    1m47.915s
user    0m33.676s
sys     0m8.028s
```

Maybe let's switch from CSV to TSV that PostgreSQL seems to understand better.

```
SELECT WatchID::Int64, JavaEnable, replaceAll(replaceAll(replaceAll(toValidUTF8(Title), '\0', ''), '"', ''), '\\', ''), GoodEvent, EventTime, EventDate, CounterID::Int32, ClientIP::Int32, RegionID::Int32,
    UserID::Int64, CounterClass, OS, UserAgent, replaceAll(replaceAll(replaceAll(toValidUTF8(URL), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(Referer), '\0', ''), '"', ''), '\\', ''), Refresh, RefererCategoryID::Int16, RefererRegionID::Int32,
    URLCategoryID::Int16, URLRegionID::Int32, ResolutionWidth::Int16, ResolutionHeight::Int16, ResolutionDepth, FlashMajor, FlashMinor,
    FlashMinor2, NetMajor, NetMinor, UserAgentMajor::Int16, replaceAll(replaceAll(replaceAll(toValidUTF8(UserAgentMinor::String), '\0', ''), '"', ''), '\\', ''), CookieEnable, JavascriptEnable, IsMobile, MobilePhone,
    replaceAll(replaceAll(replaceAll(toValidUTF8(MobilePhoneModel), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(Params), '\0', ''), '"', ''), '\\', ''), IPNetworkID::Int32, TraficSourceID, SearchEngineID::Int16, replaceAll(replaceAll(replaceAll(toValidUTF8(SearchPhrase), '\0', ''), '"', ''), '\\', ''),
    AdvEngineID, IsArtifical, WindowClientWidth::Int16, WindowClientHeight::Int16, ClientTimeZone, ClientEventTime,
    SilverlightVersion1, SilverlightVersion2, SilverlightVersion3::Int32, SilverlightVersion4::Int16, replaceAll(replaceAll(replaceAll(toValidUTF8(PageCharset), '\0', ''), '"', ''), '\\', ''),
    CodeVersion::Int32, IsLink, IsDownload, IsNotBounce, FUniqID::Int64, replaceAll(replaceAll(replaceAll(toValidUTF8(OriginalURL), '\0', ''), '"', ''), '\\', ''), HID::Int32, IsOldCounter, IsEvent,
    IsParameter, DontCountHits, WithHash, replaceAll(replaceAll(replaceAll(toValidUTF8(HitColor::String), '\0', ''), '"', ''), '\\', ''), LocalEventTime, Age, Sex, Income, Interests::Int16, Robotness, RemoteIP::Int32,
    WindowName, OpenerName, HistoryLength, replaceAll(replaceAll(replaceAll(toValidUTF8(BrowserLanguage::String), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(BrowserCountry::String), '\0', ''), '"', ''), '\\', ''),
    replaceAll(replaceAll(replaceAll(toValidUTF8(SocialNetwork), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(SocialAction), '\0', ''), '"', ''), '\\', ''),
    HTTPError, least(SendTiming, 30000), least(DNSTiming, 30000), least(ConnectTiming, 30000), least(ResponseStartTiming, 30000),
    least(ResponseEndTiming, 30000), least(FetchTiming, 30000), SocialSourceNetworkID,
    replaceAll(replaceAll(replaceAll(toValidUTF8(SocialSourcePage), '\0', ''), '"', ''), '\\', ''), ParamPrice, replaceAll(replaceAll(replaceAll(toValidUTF8(ParamOrderID), '\0', ''), '"', ''), '\\', ''), replaceAll(replaceAll(replaceAll(toValidUTF8(ParamCurrency::String), '\0', ''), '"', ''), '\\', ''),
    ParamCurrencyID::Int16, OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID,
    UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID, RefererHash::Int64, URLHash::Int64, CLID::Int32
FROM hits_100m_obfuscated
INTO OUTFILE 'dump.tsv'
FORMAT TSV
```

But how to pass TSV to `timescaledb-parallel-copy` tool?

```
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --workers 16 -connection 'host=localhost password=12345'           panic: pq: invalid input syntax for type bigint: "9076997425961590393\t0\tКино\t1\t2013-07-06 17:47:29\t2013-07-06\t225510\t-1056921538\t229\t3467937489264290637\t0\t2\t3\thttp://liver.ru/belgorod/page/1006.jки/доп_приборы\thttp://video.yandex.ru/1.561.540.000703/?order_Kurzarm_alia\t0\t16124\t20\t14328\t22\t1638\t1658\t23\t15\t7\t700\t0\t0\t17\tD<74>\t1\t1\t0\t0\t\t\t2095433\t-1\t0\t\t0\t1\t1369\t713\t135\t2013-07-06 16:25:42\t0\t0\t0\t0\twindows\t1601\t0\t0\t0\t5566829288329160346\t\t940752990\t0\t0\t0\t0\t0\t5\t2013-07-06 01:32:13\t55\t2\t3\t0\t2\t-1352932082\t-1\t-1\t-1\tS0\t<>\\f\t\t\t0\t0\t0\t0\t0\t0\t0\t0\t\t0\t\tNH\t0\t\t\t\t\t\t\t\t\t\t\t0\t6811023348165660452\t7011450103338277684\t0"

goroutine 20 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.304s
user    0m0.044s
sys     0m0.044s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --copy-options "TEXT" --workers 16 -connection 'host=localhost password=12345'
panic: pq: syntax error at or near "TEXT"

goroutine 18 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.044s
user    0m0.048s
sys     0m0.036s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --copy-options "text" --workers 16 -connection 'host=localhost password=12345'
panic: pq: syntax error at or near "text"

goroutine 18 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
panic: pq: syntax error at or near "text"

goroutine 19 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.057s
user    0m0.060s
sys     0m0.028s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --copy-options "Text" --workers 16 -connection 'host=localhost password=12345'
panic: pq: syntax error at or near "Text"

goroutine 11 [running]:
main.processBatches(0xc0000183d0, 0xc0000a66c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.041s
user    0m0.052s
sys     0m0.032s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --copy-options "FORMAT text" --workers 16 -connection 'host=localhost password=12345'
panic: pq: syntax error at or near "FORMAT"

goroutine 21 [running]:
main.processBatches(0xc00019a350, 0xc00019e660)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.045s
user    0m0.052s
sys     0m0.028s
```

Nothing works:

```
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --help
Usage of timescaledb-parallel-copy:
  -batch-size int
        Number of rows per insert (default 5000)
  -columns string
        Comma-separated columns present in CSV
  -connection string
        PostgreSQL connection url (default "host=localhost user=postgres sslmode=disable")
  -copy-options string
        Additional options to pass to COPY (e.g., NULL 'NULL') (default "CSV")
  -db-name string
        Database where the destination table exists
  -file string
        File to read from rather than stdin
  -header-line-count int
        Number of header lines (default 1)
  -limit int
        Number of rows to insert overall; 0 means to insert all
  -log-batches
        Whether to time individual batches.
  -reporting-period duration
        Period to report insert stats; if 0s, intermediate results will not be reported
  -schema string
        Destination table's schema (default "public")
  -skip-header
        Skip the first line of the input
  -split string
        Character to split by (default ",")
  -table string
        Destination table for insertions (default "test_table")
  -token-size int
        Maximum size to use for tokens. By default, this is 64KB, so any value less than that will be ignored (default 65536)
  -truncate
        Truncate the destination table before insert
  -verbose
        Print more information about copying statistics
  -version
        Show the version of this tool
  -workers int
        Number of parallel requests to make (default 1)

real    0m0.009s
user    0m0.004s
sys     0m0.000s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --truncate --copy-options "" --workers 16 -connection 'host=localhost password=12345'
panic: pq: invalid input syntax for type bigint: "9076997425961590393   0       Кино    1       2013-07-06 17:47:29     2013-07-06      225510  -1056921538     229     3467937489264290637     0       2       3http://liver.ru/belgorod/page/1006.jки/доп_приборы       http://video.yandex.ru/1.561.540.000703/?order_Kurzarm_alia     0       16124   20      14328   22      1638    1658    23      15      7       700     0017      D<>      1       1       0       0                       2095433 -1      0               0       1       1369    713     135     2013-07-06 16:25:42     0       0       0       0       windows 1601    000       5566829288329160346             940752990       0       0       0       0       0       5       2013-07-06 01:32:13     55      2       3       0       2       -1352932082     -1      -1      -1      S0<53>\f                     0       0       0       0       0       0       0       0               0               NH      0                                                                                       06811023348165660452      7011450103338277684     0"

goroutine 13 [running]:
main.processBatches(0xc000019140, 0xc0001eb080)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.191s
user    0m0.036s
sys     0m0.040s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --truncate --copy-options "NULL AS '\N'" --workers 16 -connection 'host=localhost password=12345'
panic: pq: invalid input syntax for type bigint: "9076997425961590393   0       Кино    1       2013-07-06 17:47:29     2013-07-06      225510  -1056921538     229     3467937489264290637     0       2       3http://liver.ru/belgorod/page/1006.jки/доп_приборы       http://video.yandex.ru/1.561.540.000703/?order_Kurzarm_alia     0       16124   20      14328   22      1638    1658    23      15      7       700     0017      D<>      1       1       0       0                       2095433 -1      0               0       1       1369    713     135     2013-07-06 16:25:42     0       0       0       0       windows 1601    000       5566829288329160346             940752990       0       0       0       0       0       5       2013-07-06 01:32:13     55      2       3       0       2       -1352932082     -1      -1      -1      S0<53>\f                     0       0       0       0       0       0       0       0               0               NH      0                                                                                       06811023348165660452      7011450103338277684     0"

goroutine 11 [running]:
main.processBatches(0xc000018900, 0xc0002886c0)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.187s
user    0m0.020s
sys     0m0.048s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --truncate --copy-options "DELIMITER AS '\t'" --workers 16 -connection 'host=localhost password=12345'
panic: pq: conflicting or redundant options

goroutine 13 [running]:
main.processBatches(0xc000019140, 0xc0001e9080)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.196s
user    0m0.048s
sys     0m0.020s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --truncate --copy-options "TEXT DELIMITER AS '\t'" --workers 16 -connection 'host=localhost password=12345'
panic: pq: syntax error at or near "TEXT"

goroutine 22 [running]:
main.processBatches(0xc000019140, 0xc0001e9080)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb
panic: pq: syntax error at or near "TEXT"

goroutine 11 [running]:
main.processBatches(0xc000019140, 0xc0001e9080)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.191s
user    0m0.032s
sys     0m0.036s
milovidov@mtlog-perftest03j:~$ time sudo -u postgres timescaledb-parallel-copy --db-name tutorial --table hits_100m_obfuscated --file dump.tsv --truncate --copy-options "DELIMITER AS e'\t'" --workers 16 -connection 'host=localhost password=12345'
panic: pq: conflicting or redundant options

goroutine 26 [running]:
main.processBatches(0xc0001330d0, 0xc0001e3020)
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:262 +0x879
created by main.main
        /home/builder/go/src/github.com/timescale/timescaledb-parallel-copy/cmd/timescaledb-parallel-copy/main.go:148 +0x1bb

real    0m0.169s
user    0m0.056s
sys     0m0.016s
```

I will try to avoid `timescaledb-parallel-copy` and use `psql` instead.

```
milovidov@mtlog-perftest03j:~$ sudo -u postgres psql
psql (13.4 (Ubuntu 13.4-4.pgdg18.04+1))
Type "help" for help.

postgres=# \c tutorial
You are now connected to database "tutorial" as user "postgres".
tutorial=# timing
tutorial-# COPY hits_100m_obfuscated FROM 'dump.tsv'
tutorial-# ;
ERROR:  syntax error at or near "timing"
LINE 1: timing
        ^
tutorial=# \timing
Timing is on.
tutorial=# COPY hits_100m_obfuscated FROM 'dump.tsv';
ERROR:  could not open file "dump.tsv" for reading: No such file or directory
HINT:  COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
Time: 4.348 ms
tutorial=# \copy hits_100m_obfuscated FROM 'dump.tsv';
```

It started to do something... fairly slow with using less than one CPU core.

Folks from TimescaleDB always recommend to enable compression, which is not by default.
Let's read about it:

https://docs.timescale.com/timescaledb/latest/how-to-guides/compression/

> We strongly recommend that you understand how compression works before you start enabling it on your hypertables.

The amount of hackery to overcome PostgreSQL limitations is overwhelming:

> When compression is enabled, TimescaleDB converts data stored in many rows into an array. This means that instead of using lots of rows to store the data, it stores the same data in a single row.

In the meantime, copy finished in "just" 1.5 hours, 19 245 rows/second. This is extremely slow, even for single core.

```
tutorial=# \copy hits_100m_obfuscated FROM 'dump.tsv';
COPY 100000000
Time: 5195909.154 ms (01:26:35.909)
```

## Running Benchmark

Let's prepare for benchmark...
What is needed to execute single query in batch mode?

`man psql`

```
sudo -u postgres psql tutorial -t -c '\timing' -c 'SELECT 1' | grep 'Time'
```

Now we are ready to run our benchmark.

PostgreSQL does not have `SHOW PROCESSLIST`.
It has `select * from pg_stat_activity;` instead.

https://ma.ttias.be/show-full-processlist-equivalent-of-mysql-for-postgresql/

But it does not show query progress.
The first query `SELECT count(*) FROM hits_100m_obfuscated` just hanged. It reads something from disk...

Let's check the data volume:

```
$ sudo du -hcs /opt/postgresql/
68G     /opt/postgresql/
```

Looks consistent for uncompressed data.

```
./benchmark.sh

grep -oP 'Time: \d+' log | grep -oP '\d+' | awk '{ if (n % 3 == 0) { printf("[") }; ++n; printf("%g", $1 / 1000); if (n % 3 == 0) { printf("],\n") } else { printf(", ") } }'
```

Now let's enable compression.

```
ALTER TABLE hits_100m_obfuscated SET (timescaledb.compress);
SELECT add_compression_policy('hits_100m_obfuscated', INTERVAL '0 seconds');
```

```
milovidov@mtlog-perftest03j:~ClickHouse/benchmark/timescaledb$ sudo -u postgres psql tutorial
psql (13.4 (Ubuntu 13.4-4.pgdg18.04+1))
Type "help" for help.

tutorial=# ALTER TABLE hits_100m_obfuscated SET (timescaledb.compress);
ALTER TABLE
tutorial=# SELECT add_compression_policy('hits_100m_obfuscated', INTERVAL '0 seconds');
 add_compression_policy
------------------------
                   1000
(1 row)
```

Ok, in `top` I see that it started compression with using single CPU core.

```
300464 postgres  20   0 32.456g 932044 911452 D  48.0  0.7   1:08.11 postgres: 13/main: Compression Policy [1000]
```

Let's also define better order of data:

```
ALTER TABLE hits_100m_obfuscated
  SET (timescaledb.compress,
       timescaledb.compress_orderby = 'counterid, userid, event_time');
```

The query hanged. Maybe it's waiting for finish of previous compression?

After several minutes it answered:

```
ERROR:  cannot change configuration on already compressed chunks
DETAIL:  There are compressed chunks that prevent changing the existing compression configuration.
```

Ok, at least some of the chunks will have the proper order.

After a few hours looks like the compression finished.

```
sudo ncdu /var/lib/postgresql/13/main/

28.9 GiB [##########] /base
```

Yes, looks like it's compressed. About two times - not too much.

Let's rerun the benchmark.

Ok, it's slightly faster.