Commit Graph

664 Commits

Author SHA1 Message Date
Alexey Milovidov
599ec4c577 Merge branch 'veloman-yunkan-jsoneachrow_import_nested' 2018-09-17 23:36:01 +03:00
Alexey Milovidov
8852660b24 Merge branch 'jsoneachrow_import_nested' of https://github.com/veloman-yunkan/ClickHouse into veloman-yunkan-jsoneachrow_import_nested 2018-09-17 23:26:43 +03:00
alexey-milovidov
006119b81d
Merge pull request #3155 from yandex/fix-float-parsing-with-custom-csv-delimiter
Fix float parsing with custom csv delimiter
2018-09-17 22:39:45 +03:00
Alexey Milovidov
c00147f37d Fixed parsing of floats in CSV file with custom delimiter #3142 2018-09-17 21:17:30 +03:00
chertus
7db097a49a support e-notation for decimal input [issue-3135] 2018-09-17 20:07:38 +03:00
proller
d752d5d62d Prepare to new poco, build fixes (#3072)
* WIp

* fix

* fix

* test

* fix poco netssl find

* clean

* fixes

* wip

* wip

* new poco

* fixes

* Fixes

* wip

*  clean

* clean

*     clean

* clean

* Update HTTPHandlerFactory.h
2018-09-14 22:48:51 +03:00
Veloman Yunkan
47eb0e28b3 Nested JSON data is mapped to nested table data 2018-09-14 12:15:32 +00:00
chertus
dd5c55df2c decimal sum [CLICKHOUSE-3765] 2018-09-11 21:42:06 +03:00
proller
db13ef96d6 Fix ssl init (#3056)
* Try to fix ../libs/libglibc-compatibility/glibc-compatibility.c:151: multiple definition of `explicit_bzero'
contrib/ssl/crypto/libcrypto.a(explicit_bzero.c.o):../contrib/ssl/crypto/compat/explicit_bzero.c:16: first defined here

* Fix ssl init

* fix

* noncopyable

* clean
2018-09-06 21:05:33 +03:00
Alexey Milovidov
2332bf1a50 Code consistency [#CLICKHOUSE-3943] 2018-09-04 22:34:34 +03:00
Alexey Milovidov
d10f9200b9 Consistency of FileOpen event [#CLICKHOUSE-3943] 2018-09-04 22:24:45 +03:00
alexey-milovidov
72c799ad76
Merge pull request #3008 from 4ertus2/decimal
even more Decimal
2018-09-03 07:19:33 +03:00
Alexey Milovidov
538edbb8e2 Addition to prev. revision [#CLICKHOUSE-2] 2018-09-02 06:33:48 +03:00
Alexey Milovidov
fc27be314b Avoid temporary nullptr references [#CLICKHOUSE-2] 2018-09-02 06:00:04 +03:00
Alexey Milovidov
f7107f12b4 Merge branch 'master' into extra-warnings 2018-09-02 03:13:36 +03:00
Alexey Milovidov
8eb746b216 Documentation for ProfileEvents (draft) #3010 2018-09-02 02:13:17 +03:00
chertus
fe1e3300b7 remove dirty code [CLICKHOUSE-3765] 2018-08-31 20:36:27 +03:00
alexey-milovidov
a05cdfbb45
Merge branch 'master' into extra-warnings 2018-08-31 03:03:44 +03:00
Alexey Milovidov
5df5df8971 utils/check-style/check-style -n 2018-08-31 00:13:34 +03:00
Alexey Milovidov
06053d9c1f Added comments #2980 2018-08-30 23:12:15 +03:00
alexey-milovidov
e0b1b5ff78
Merge pull request #2980 from amosbird/memcpySmall
Padding for IO buffers.
2018-08-30 22:53:47 +03:00
chertus
7bac9698f6 fix build 2018-08-29 16:24:33 +03:00
chertus
a69c936b44 improve field visitors for UUID CLICKHOUSE-3249 2018-08-29 14:58:40 +03:00
Amos Bird
8851fbcab2 Padding for IO buffers.
Testing data

```
select 'aaaaaaaa','bbbbbbbb','cccccccc','dddddddd','eeeeeeee','ffffffff','gggg','hhh' from numbers(3000000) into outfile '/tmp/test.tsv'
```

Testing command
```
echo "select count() from file('/tmp/test.tsv', CSV, 'a String, b String, c String, d String, e String, f String, g String, h String') where not ignore(e)" | clickhouse-benchmark
```

TSV parser has less overhead than CSV, using it would better unveil the benefits of memcpySmall.

Before
```
QPS: 1.662, RPS: 4985463.906, MiB/s: 603.823, result RPS: 1.662, result MiB/s: 0.000.
0.000%  0.559 sec.
10.000% 0.564 sec.
20.000% 0.568 sec.
30.000% 0.572 sec.
40.000% 0.575 sec.
50.000% 0.581 sec.
60.000% 0.592 sec.
70.000% 0.624 sec.
80.000% 0.639 sec.
90.000% 0.664 sec.
95.000% 0.686 sec.
99.000% 0.711 sec.
99.900% 0.715 sec.
99.990% 0.716 sec.
```

After
```
QPS: 1.861, RPS: 5582303.107, MiB/s: 676.110, result RPS: 1.861, result MiB/s: 0.000.
0.000%  0.510 sec.
10.000% 0.514 sec.
20.000% 0.517 sec.
30.000% 0.521 sec.
40.000% 0.523 sec.
50.000% 0.527 sec.
60.000% 0.530 sec.
70.000% 0.539 sec.
80.000% 0.558 sec.
90.000% 0.584 sec.
95.000% 0.589 sec.
99.000% 0.608 sec.
99.900% 0.655 sec.
99.990% 0.663 sec.
```
2018-08-29 09:39:44 +08:00
alexey-milovidov
a5d7097c08
Merge pull request #2977 from amosbird/sse2csv
Enable sse2 for CSV parsing.
2018-08-29 03:04:17 +03:00
chertus
6a02b18b93 field visitors for UUID CLICKHOUSE-3249 2018-08-28 21:36:41 +03:00
Amos Bird
8d6a26b8df Enable sse2 for CSV parsing.
Testing data

```
select 'aaaaaaaa,bbbbbbbb,cccccccc,dddddddd,eeeeeeee,ffffffff,gggg,hhh' from numbers(3000000) into outfile '/tmp/test.csv'
```

Testing command
```
echo "select count() from file('/tmp/test.csv', CSV, 'a String, b String, c String, d String, e String, f String, g String, h String') where not ignore(e)" | clickhouse-benchmark

```

Before
```
QPS: 1.317, RPS: 3949749.687, MiB/s: 478.380, result RPS: 1.317, result MiB/s: 0.000.
0.000%  0.704 sec.
10.000% 0.712 sec.
20.000% 0.718 sec.
30.000% 0.726 sec.
40.000% 0.739 sec.
50.000% 0.754 sec.
60.000% 0.770 sec.
70.000% 0.788 sec.
80.000% 0.798 sec.
90.000% 0.815 sec.
95.000% 0.826 sec.
99.000% 0.850 sec.
99.900% 0.857 sec.
99.990% 0.858 sec.
```

After
```
QPS: 1.533, RPS: 4598308.336, MiB/s: 556.932, result RPS: 1.533, result MiB/s: 0.000.
0.000%  0.626 sec.
10.000% 0.635 sec.
20.000% 0.639 sec.
30.000% 0.642 sec.
40.000% 0.643 sec.
50.000% 0.645 sec.
60.000% 0.649 sec.
70.000% 0.652 sec.
80.000% 0.658 sec.
90.000% 0.682 sec.
95.000% 0.710 sec.
99.000% 0.727 sec.
99.900% 0.733 sec.
99.990% 0.734 sec.
```
2018-08-28 19:46:11 +08:00
Alexey Milovidov
24bd803c07 Whitespaces #2758 2018-08-27 22:06:32 +03:00
Alexey Milovidov
d4ea8bdb88 Merge remote-tracking branch 'origin/master' into extra-warnings 2018-08-27 20:18:39 +03:00
alexey-milovidov
98148d914c
Merge pull request #2958 from blahgeek/skip-json-object
Skip object fields on json input
2018-08-27 20:07:32 +03:00
Alexey Milovidov
486d87ebcd Merge remote-tracking branch 'origin/master' into extra-warnings 2018-08-27 20:01:33 +03:00
chertus
ee873c8e9f Merge branch 'master' into decimal 2018-08-27 19:18:48 +03:00
alexey-milovidov
7de0549879
Update ReadHelpers.h 2018-08-27 18:00:08 +03:00
alexey-milovidov
01f01b1ee1
Update ReadHelpers.cpp 2018-08-27 17:54:58 +03:00
BlahGeek
739119d355 skip object fields on json input 2018-08-27 12:01:47 +08:00
Alexey Milovidov
eed804ba94 Enabling extra warnings [#CLICKHOUSE-2] 2018-08-26 15:38:45 +03:00
Alexey Milovidov
281faa1686 Enabling -Wshadow [#CLICKHOUSE-2] 2018-08-26 05:13:41 +03:00
Alexey Milovidov
8a0239b6ad Enabling -Wshadow [#CLICKHOUSE-2] 2018-08-26 05:08:35 +03:00
Alexey Milovidov
a1a70d4a65 Enabling -Wzero-as-null-pointer-constant [#CLICKHOUSE-2] 2018-08-26 04:56:34 +03:00
Alexey Milovidov
44e848856e Enabling -Wunused-exception-parameter [#CLICKHOUSE-2] 2018-08-26 04:33:20 +03:00
Alexey Milovidov
2ce783e016 Merge branch 'read' of https://github.com/amosbird/ClickHouse into amosbird-read 2018-08-26 02:55:22 +03:00
alexey-milovidov
00404a911c
Update VarInt.h 2018-08-26 02:46:18 +03:00
Amos Bird
a589230249 Faster path for varint read.
clickhouse-benchmark <<< 'select count() from s where not ignore(s);'

before this patch:

```

QPS: 0.732, RPS: 2346562049.608, MiB/s: 22378.560, result RPS: 0.732, result MiB/s: 0.000.
0.000%  1.310 sec.
10.000% 1.321 sec.
20.000% 1.327 sec.
30.000% 1.337 sec.
40.000% 1.343 sec.
50.000% 1.359 sec.
60.000% 1.366 sec.
70.000% 1.381 sec.
80.000% 1.400 sec.
90.000% 1.434 sec.
95.000% 1.448 sec.
99.000% 1.489 sec.
99.900% 1.499 sec.
99.990% 1.500 sec.

```

after this patch:

```
QPS: 0.787, RPS: 2524560389.064, MiB/s: 24076.084, result RPS: 0.787, result MiB/s: 0.000.
0.000%  1.228 sec.
10.000% 1.232 sec.
20.000% 1.235 sec.
30.000% 1.241 sec.
40.000% 1.246 sec.
50.000% 1.256 sec.
60.000% 1.265 sec.
70.000% 1.278 sec.
80.000% 1.296 sec.
90.000% 1.321 sec.
95.000% 1.354 sec.
99.000% 1.421 sec.
99.900% 1.453 sec.
99.990% 1.456 sec.
```

I also tried a SSE2 implementation and it's much slower (50%)
2018-08-26 02:23:26 +08:00
Amos Bird
9623e59870 Simplify read helpers.
Use similar APIs like https://en.cppreference.com/w/cpp/string/byte/strstr
2018-08-26 01:27:28 +08:00
Amos Bird
5f03082af9 Make sure BufferBase is fully swapped. 2018-08-25 19:58:16 +08:00
chertus
54a3c028c5 Merge branch 'master' into decimal 2018-08-24 14:19:41 +03:00
Alexey Milovidov
080b0f7f1f Speed up fuzz tests, part 2 #2924 2018-08-24 10:35:53 +03:00
Alexey Milovidov
a2fefa1617 Speed up fuzz tests, part 2 #2924 2018-08-24 10:30:53 +03:00
Alexey Milovidov
1437e5cbe3 Fix style [#CLICKHOUSE-2] 2018-08-24 03:07:25 +03:00
chertus
1467ca5610 decimal field visitors CLICKHOUSE-3765 2018-08-23 22:11:31 +03:00