Commit Graph

1043 Commits

Author SHA1 Message Date
Vladimir Chebotarev
531460396d POST S3 requests. 2019-09-04 13:39:53 +00:00
Vladimir Chebotarev
caeacafb76 Fixed GET redirects. 2019-09-04 13:39:53 +00:00
Vladimir Chebotarev
78f57c5f2a First attempt to make redirects. 2019-09-04 13:39:53 +00:00
Alexey Milovidov
9cf0c780fb Better code 2019-09-03 21:23:13 +03:00
alexey-milovidov
845a612b95
Merge pull request #6773 from infinivision/fix_issue_6575
fix DataTypeAggregateFunction deserialization
2019-09-03 21:19:41 +03:00
Zhichang Yu
9e587fb139 fix DataTypeAggregateFunction deserialization 2019-09-02 20:09:40 +08:00
Alexander Tokmakov
e2928481a3 Merge branch 'master' into merging_template_format 2019-09-02 12:41:34 +03:00
Alexander Tokmakov
cd5d733f75 rewrite unit test with gtest 2019-08-30 22:08:23 +03:00
Alexander Tokmakov
4c744af113 minor style fixes 2019-08-30 20:25:35 +03:00
alexey-milovidov
ce73e17ff6
Merge pull request #6730 from yandex/fallback_o_direct
Fallback from O_DIRECT.
2019-08-30 16:08:19 +03:00
Alexander Tokmakov
d95d53b4e4 better error messages 2019-08-29 22:35:59 +03:00
Alexey Milovidov
a894288fa0 Fallback from O_DIRECT. 2019-08-29 18:48:00 +03:00
Alexander Tokmakov
03c83169bd Merge branch 'master' into merging_template_format 2019-08-29 16:34:25 +03:00
Alexander Tokmakov
d6ca9fee63 unit test for PeekableReadBuffer 2019-08-29 16:30:43 +03:00
Alexey Milovidov
39d50b5144 Remove mimalloc 2019-08-28 22:01:52 +03:00
Weiqing Xu
b7fdfcc797 fix HDFS HA can't work on DEBUG mode
The Describe the bug
when the using HDFS HA nameserivce as the uri, the port will be 0.
hdfsBuilderSetNameNodePort will be called to set the port.
hdfsBuilderSetNameNodePort call asset to check if the port is greater
than 0.
So in Release mode, it works OK. In the Debug mode, the asset will fail.

How to reproduce
when compiler the Clickhouse, use DEBUG mode, it will throw error when
using HDFS HA nameservice url
2019-08-24 17:27:36 +08:00
Alexander Tokmakov
734a0f23a9 Merge branch 'master' into merging_template_format 2019-08-23 15:32:35 +03:00
alex-zaitsev
1e6437e4ee WIP: Fix MacOS build (#6429)
* Fix broken MacOS build

* Fix MacOS build

* Update ReadBufferFromFile.cpp

* Update WriteBufferFromFile.cpp

* Update Elf.h

* Update Elf.h
2019-08-21 03:35:41 +03:00
Nikolai Kochetov
aef03fd9bf
Merge pull request #6425 from yandex/vzakaznikov-liveview
Merge LIVE VIEW
2019-08-20 17:13:42 +03:00
Ivan
a502424c33
Implement support for insertion into Kafka tables (#6012)
* Add write-callback on each row for RowOutputStream
* Fix build of new rdkafka library
* Poll messages if Kafka outgoing queue is full
* Add test
* Add test producer-consumer
* Truncate delimiter from last row in message
2019-08-20 14:17:57 +03:00
Nikolai Kochetov
cdf1ce3171 Merge branch 'master' into vzakaznikov-liveview 2019-08-20 10:41:31 +03:00
stavrolia
6055c61e7f wip hdfs 2019-08-19 21:12:50 +03:00
alexey-milovidov
f36d20915b
Merge pull request #6529 from yandex/better-signals
Better debug-ability of fault signals.
2019-08-18 16:44:54 +03:00
proller
d09c5c871b Build fixes (#6491)
* Fix build

* cmake: fix cpuinfo

* Fix includes after processors merge

Conflicts:
	dbms/src/Processors/Formats/Impl/CapnProtoRowInputFormat.cpp
	dbms/src/Processors/Formats/Impl/ParquetBlockOutputFormat.cpp
	dbms/src/Processors/Formats/Impl/ProtobufRowInputFormat.cpp
	dbms/src/Processors/Formats/Impl/ProtobufRowOutputFormat.cpp

* Fix build in gcc8

* fix test link

* fix test link

* Fix test link

* link fix

* Fix includes after processors merge 2

Conflicts:
	dbms/src/Processors/Formats/Impl/ParquetBlockInputFormat.cpp

* Fix includes after processors merge 3

* link fix

* Fix likely/unlikely conflict with cython

* Fix conflict with protobuf/stubs/atomicops.h

* remove unlikely.h

* Fix macos build (do not use timer_t)
2019-08-18 01:53:46 +03:00
Alexey Milovidov
2cd669fa1d Minor improvement 2019-08-17 23:33:50 +03:00
Nikolai Kochetov
631bbde420 Merged with master. 2019-08-12 13:19:13 +03:00
kreuzerkrieg
112fc71276 adding -Wshadow for GCC 2019-08-09 23:58:16 +03:00
Nikolai Kochetov
2ae3db7920 Merged with master. 2019-08-09 18:32:44 +03:00
Alexey Milovidov
40a3f77917 Removed duplicate include 2019-08-08 19:01:42 +03:00
alexey-milovidov
99fad0904b
Merge pull request #6314 from infinivision/add_bitmapRange
added bitmapRange function
2019-08-08 04:51:48 +03:00
alexey-milovidov
355b1e5594
Merge pull request #6352 from yandex/no_space_left_on_device_message
Print extra info in exception message for "no space left on device"
2019-08-07 22:28:38 +03:00
Alexander Tokmakov
5cf183cbe3 fixes after review 2019-08-07 15:55:27 +03:00
Alexander Tokmakov
aefc664819 refactor throwFromErrno 2019-08-06 21:54:06 +03:00
Nikolai Kochetov
599f3ddf6e Merged with master. 2019-08-04 15:35:21 +03:00
proller
77bd2624ea Build fixes (#6016)
* Fix building without submodules

* Fix more gcc9 warnings

* was wrong!
../dbms/src/IO/WriteBufferAIO.cpp:277:54: error: result of comparison 'ssize_t' (aka 'long') > 9223372036854775807 is always false [-Werror,-Wtautological-type-limit-compare]
    if ((static_cast<ssize_t>(flush_buffer.offset()) > std::numeric_limits<off_t>::max()) ||
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* mimalloc off MI_OVERRIDE

* Fix freebsd build

* Fix shared build

* Fix build in gcc9

* Fix split build

* fix

* fix

* fix

* fix

* clean

* zstd

* Update CMakeLists.txt

* Update Defines.h

* Try fix -Wsign-compare

* Freebsd fix

* Add missing libs

* Fix double-conversion include for copy-headers

* Fix zlib link

* Fix includes for arcadia

* Fix includes for arcadia

* Fix includes for arcadia

* Freebsd fix

* Arcadia fixes

* Update QueryProfiler.cpp

* Freebsd fix

* clean

* 19.11: Fixes for arcadia porting (#6223)

* fix glibc-compatibility

* M_LIBRARY fix

* wip

* Fix

* Fix arm build

* unwind fix

* Update CMakeLists.txt
2019-08-04 03:19:03 +03:00
Zhichang Yu
4f146eaa7e added bitmapRange function 2019-08-03 23:07:46 +08:00
alexey-milovidov
e9c2077c71
Merge pull request #6208 from vitlibar/write-buffer-call-next-impl-even-if-zero-offset
Fix formula for new_size in WriteBufferFromVector(AppendModeTag).
2019-08-03 06:27:31 +03:00
Nikolai Kochetov
ab914010e1 Remove TSKVRowInputStream. 2019-08-02 19:56:16 +03:00
Vitaly Baranov
d7891ec9cf Fix formula for new_size in WriteBufferFromVector(AppendModeTag). 2019-08-01 15:27:32 +03:00
Alexey Milovidov
f3921ce3e0 Changed boost::filesystem to std::filesystem 2019-08-01 01:37:41 +03:00
Alexey Milovidov
f03df37b57 Fixed error #6195 2019-07-28 20:16:48 +03:00
Yuriy
383fa25c51 resolved conflict 2019-07-28 02:00:15 +03:00
alexey-milovidov
7219e167d2
Merge pull request #6026 from bopohaa/fix-kafka-unclean-stream
Clear Kafka's buffer if an invalid message is found.
2019-07-25 04:30:37 +03:00
Alexander Tokmakov
7e9c3f2022 PeekableReadBufferCheckpoint 2019-07-22 03:12:11 +03:00
Alexander Tokmakov
456a8c4b17 Fix buffer padding 2019-07-22 03:10:13 +03:00
Alexander Tokmakov
23fddc8688 Fixes 2019-07-22 03:09:35 +03:00
Alexander Tokmakov
4d0a6bad5e PeekableReadBuffer 2019-07-22 03:07:55 +03:00
Alexander Tokmakov
f43f7c43a5 Fixes 2019-07-22 02:59:31 +03:00
Alexander Tokmakov
c92a0f3ac5 Fixes 2019-07-22 02:58:39 +03:00
Alexander Tokmakov
892b67492d Checking for suffix, not for eof 2019-07-22 01:03:29 +03:00
Yuriy
fa2dfcd71d better 2019-07-19 21:43:52 +03:00
Weiqing Xu
ec3c5a369d support reading from HA mode HDSF
the driver libhdfs3 can support HA mode. When the uri doesn't contain
port num, the libhdfs3 will handle it in HA mode, treat the uri as
nameservice name and read the real host and port from configuration file
as the origin Java client.
the default configuration file is hdfs-client.xml in the working
directory, it also can be set in env variable "LIBHDFS3_CONF".
the format of the configuration file is same with hdfs-site.xml.
2019-07-19 07:32:59 +08:00
Nik
4a30eba32f Fix style check 2019-07-16 22:34:47 +04:00
Nik
922c3eb22e Clear Kafka's buffer if an invalid message is found. 2019-07-16 19:27:42 +04:00
Yuriy
1d6e31be4d merged master 2019-07-16 10:33:25 +03:00
alexey-milovidov
5215d13437
Merge pull request #5998 from Enmk/double_delta_fix_2
Fixed DoubleDelta encoding cases for random Int32 and Int64.
2019-07-16 00:54:22 +03:00
Vasily Nemkov
0f4075478e Fixed DoubleDelta encoding cases for random Int32 and Int64. 2019-07-15 08:17:14 +03:00
Yuriy
baa62f2aba style check 2019-07-15 01:53:30 +03:00
Yuriy
0ed77453d3 reading packets in parts 2019-07-15 01:13:56 +03:00
Yuriy
a6bd5bccff tiny fix 2019-07-14 11:27:58 +03:00
Nikolai Kochetov
e9f3d4d6b8
Merge pull request #4914 from yandex/processors
Processors
2019-07-11 19:22:02 +03:00
Alexey Milovidov
499d798b34 Remove "clickhouse_compression" library because it is not a library (cyclic dependency via IDataType) 2019-07-10 00:37:43 +03:00
alexey-milovidov
f6df911307
Update HDFSCommon.cpp 2019-07-09 20:35:47 +03:00
akonyaev
54d890b984 add user parsing in HDFS URI (rewrite split boost->std::string) 2019-07-09 17:09:56 +03:00
akonyaev
c0dc8fc9d3 add user parsing in HDFS URI (fixes after review, style fixes) 2019-07-09 16:16:04 +03:00
akonyaev
74b2440f66 add user parsing in HDFS URI 2019-07-09 13:26:06 +03:00
Alexey Milovidov
e07235e294 Addition to prev. revision 2019-07-08 21:55:06 +03:00
Nikolai Kochetov
209793c981 Merged with master. 2019-07-08 16:00:54 +03:00
Alexey Milovidov
ac20c515ab Better code in unit tests 2019-07-08 04:58:31 +03:00
Alexey Milovidov
03712aabec Add two more warnings from -Weverything 2019-07-08 03:16:39 +03:00
Vitaliy Zakaznikov
a363189dfc Merge branch 'master' of https://github.com/vzakaznikov/ClickHouse into liveview 2019-07-01 18:05:14 -04:00
Yuriy
450b20ca89 max_memory_usage support in MySQL wire protocol 2019-07-01 08:58:31 +03:00
alexey-milovidov
6e77fc4abe
Merge pull request #5775 from yandex/lfalloc2mimalloc
Use mimalloc instead of lfallocator
2019-06-28 14:48:18 +03:00
fredchenbj
e3b3ca8d0e IO: fix endian portable on macos x
Signed-off-by: fredchenbj <cfworking@163.com>
2019-06-28 15:15:56 +08:00
Danila Kutenin
e67431e455 Use mimalloc instead of lfallocator 2019-06-27 18:13:19 +03:00
proller
e9a66d427c Fix freebsd build 2019-06-26 23:32:28 +03:00
alexey-milovidov
dca6d2947c
Merge pull request #5600 from Enmk/gorilla_column_encoding
Gorilla column encoding
2019-06-26 12:53:12 +03:00
Nikolai Kochetov
f9b29bfdd2 Merged with master. 2019-06-25 20:00:54 +03:00
Alexey Milovidov
8f77d898f3 Renamed functions for consistency #5704 2019-06-24 17:51:37 +03:00
alexey-milovidov
8e41d89126
Merge pull request #5704 from zhang2014/fix/ISSUES-5697
ISSUES-5697 fix insert and select query with mysql style identifier
2019-06-24 17:40:28 +03:00
alexey-milovidov
dacbe9a410
Update WriteHelpers.h 2019-06-24 17:38:58 +03:00
proller
1ac09fa721
Split Common/config.h by libs (dbms) (#5715) 2019-06-24 14:17:15 +03:00
zhang2014
029e2ccf1f some review fixes 2019-06-22 19:10:00 +08:00
zhang2014
b9cb1feb4c some review fixes 2019-06-22 12:40:41 +08:00
zhang2014
1786407621 ISSUES-5697 fix insert and select query with mysql style identifier 2019-06-21 13:51:27 +08:00
eejoin
f3e0c60b89 Flush buffer when WriteBufferFromHDFS's destructor is called 2019-06-19 19:50:34 +08:00
Vasily Nemkov
e79c3d0e22 Post-PR fixes stage 3
maskLowBits()
building for clang
more test cases for codecs.
2019-06-19 13:38:15 +03:00
Vasily Nemkov
81f9055d81 Post-PR fixes #2
Fixed style issues and build for clang-7;
64-bit buffer for BitReader and BitWriter;
Fixed overflow and writing more bytes on flushing;
Added maskLowBits() and tests for it.
2019-06-17 21:49:01 +03:00
Vasily Nemkov
0ebb145d32 Post-PR fixes:
* BitHelpers.cpp was removed, corresponding code was moved to the header
* BitIO test as GTest-based test binary
* gtest-based unit test for DoubleDelta and Gorilla codecs
* getLeadingZeroBits from SFINAE to consexpr if
* removed couple of unneeded if's
* Fixed sql-test to use one table
2019-06-17 21:39:45 +03:00
Vasily Nemkov
dfd8ff7675 Gorilla column encoding
Added Gorilla column encoding for any fixed-width type;
Added tests for Float32 and Float64 values.
2019-06-17 21:39:42 +03:00
Vasily Nemkov
4ae63072d0 DoubleDelta column encoding.
Added DoubleDelta column encoding + test cases;
Added BitWriter and BitReader that allow to read data from ReadBuffer
and write data to WriteBuffer bit by bit, up to 64 bits at once;
Added test for BitReader and BitWriter.
2019-06-17 21:38:27 +03:00
Vitaliy Zakaznikov
51bf8a1baa Merge branch 'master' of https://github.com/vzakaznikov/ClickHouse into liveview 2019-06-17 12:42:31 -04:00
Alexey Milovidov
413b780b65 Various fixes for PVS-Studio 2019-06-16 21:13:39 +03:00
Vitaliy Zakaznikov
117f2d191e Merge branch 'master' of https://github.com/vzakaznikov/ClickHouse into liveview 2019-06-13 15:23:20 -04:00
Alexey Milovidov
8b87511e25 Namespace comments are unneeded according to the code style.
find dbms -name '*.h' -or -name '*.cpp' | xargs grep -l -P '}\s*//+\s*namespace\s*' | xargs sed -i -r -e 's/}\s*\/\/+\s*namespace\s*\w+/}/'
2019-06-13 13:37:13 +03:00
Vitaliy Zakaznikov
b65ff910b6 Merge branch 'master' of https://github.com/vzakaznikov/ClickHouse into liveview 2019-06-07 08:02:59 -04:00
alesapin
a0e7d183f7
Merge pull request #5534 from yandex/debug_dns_cache
Fix bug in pooled sessions and host ip change
2019-06-05 22:37:17 +03:00
Orivej Desh
5ec179377a DEVTOOLS-5170 Delete Y_IGNORE markers (#5533)
They have been superseded by a new include resolution configuration that lives
outside clickhouse source tree.
2019-06-05 14:52:39 +03:00
Vitaliy Zakaznikov
5f3be8134d * Disallowing writes into LIVE VIEW tables
* Adding support for watching live views via HTTPHandler
* Small fix to WriteBufferValidUTF8.cpp to propagate next() call
* Updated copyData.cpp to treat block with no rows as
  "flush block"
* Updated PushingToViewsBlockOutputStream.cpp to directly use
  LIVE VIEW output stream instead of calling its write method
2019-06-05 07:30:29 -04:00
alesapin
07ee60b546 Fix bug with pooled HTTP sessions and host ip change 2019-06-05 12:23:41 +03:00
Anton Yuzhaninov
091b591681 Fix build without Brotli (#5513) (#5521) 2019-06-03 23:27:53 +03:00
alexey-milovidov
461c4919d3
Merge pull request #5116 from PerformanceVision/send_header
Send a X-ClickHouse summary on the header for HTTP client with number of rows inserted
2019-05-25 20:20:35 +03:00
Guillaume Tassery
77b7158cd8 Progress rename for total_rows and write_[rows|bytes] 2019-05-20 18:37:41 +07:00
alexey-milovidov
24b16da9fd
Merge pull request #5235 from vitlibar/improve-new-json-functions
Improve new json functions
2019-05-18 12:51:41 +03:00
alexey-milovidov
c2132880f6
Update WriteBufferFromVector.h 2019-05-18 12:43:43 +03:00
Danila Kutenin
0d3422fa8a toValidUTF8 function added 2019-05-17 15:55:21 +03:00
chertus
21cf553f16 extract readDecimalText.h from readFloatText.h 2019-05-16 19:41:10 +03:00
Vitaly Baranov
c93bd31695 Simplify template implementation. 2019-05-16 14:09:22 +03:00
chertus
2739a3f6d0 Merge branch 'master' into nulls 2019-05-16 14:02:12 +03:00
chertus
5f19c0981d add toDecimalOrZero and toDecimalOrNull fuctions 2019-05-15 21:50:35 +03:00
chertus
720f8667e4 fix decimal quoted csv input 2019-05-15 16:51:17 +03:00
Ivan
4a5832b18a
Check the time limit every (flush_interval / poll_timeout) number of rows from Kafka (#5249) 2019-05-14 18:52:03 +03:00
Nikolai Kochetov
808d0f3b08 Merged with master. 2019-05-14 13:31:19 +03:00
Guillaume Tassery
9c295fa524 Don't ask client to send write info byte if the version cannot handle it 2019-05-13 14:18:02 +07:00
Guillaume Tassery
1ae8948177 Merge branch 'master' of https://github.com/yandex/ClickHouse into send_header 2019-05-13 13:35:12 +07:00
Guillaume Tassery
a1ed3f8f6b Add checks for header progress 2019-05-13 12:55:53 +07:00
Guillaume Tassery
2bd3895e40 Rename HTTP header on total_rows to rows_in_set 2019-05-13 11:56:08 +07:00
Guillaume Tassery
1b30f91c42 Return to old implementation of read and write values for progress 2019-05-13 11:48:09 +07:00
Guillaume Tassery
a4c891e529 Typo for progress and summary header 2019-05-10 13:50:48 +07:00
proller
b9385f9620 Disable test in macos 2019-05-08 13:24:50 +03:00
Guillaume Tassery
6dcaadc620 Merge remote-tracking branch 'upstream/master' into send_header 2019-05-06 17:18:41 +07:00
Guillaume Tassery
7d578edc85 Check null values for when we send summary header 2019-05-06 16:09:45 +07:00
Guillaume Tassery
55842dce10 code style 2019-05-06 13:57:48 +07:00
Guillaume Tassery
e0bd16c1f4 Backward compatibility for write progress with old version of server and client 2019-05-06 13:14:37 +07:00
Guillaume Tassery
7b1be6efc4 Clode cleaning and add test for X-ClickHouse-Summary header 2019-04-26 10:46:52 +07:00
Guillaume Tassery
3707f26edf Write number of rows written in a summary instead of progress header 2019-04-26 10:38:39 +07:00
Nikolai Kochetov
d613716dc5 More bytes for DoubleConverter buffer. 2019-04-25 17:32:39 +03:00
Nikolai Kochetov
e9a63de607 More bytes for DoubleConverter buffer. 2019-04-25 17:00:07 +03:00
Guillaume Tassery
ced71ac9fd Send how many rows and bytes were written during the query 2019-04-25 18:25:49 +07:00
Alexey Milovidov
58d7d1e451 Resolved some of PVS-Studio warnings 2019-04-22 19:07:09 +03:00
Guillaume Tassery
7a75003d26 Send X-ClickHouse-Progress header in the end of a query execution 2019-04-19 19:14:10 +07:00
Danila Kutenin
622b2c8472 Enable LFAlloc for MarkCache and UncompressedCache 2019-04-13 15:07:43 +03:00
Alexey Milovidov
36d016d331 Added [[noreturn]] attributes for functions that throw unconditionally 2019-04-10 23:27:14 +03:00
Alexey Milovidov
f37d66bb2e Brotli: additions to #4388 2019-04-10 19:12:31 +03:00
Mikhail Fandyushin
e566d10eee HTTP Accept-Encoding: br; Revert using unbundled brotli library. FCixed using not initialized http respones ostream pointer 2019-04-10 00:59:44 +03:00
Mikhail Fandyushin
7fc9b8c6fb Merge branch 'master' into brotli 2019-04-08 21:07:58 +03:00
alexey-milovidov
27f7f0908f
Merge pull request #4913 from yandex/fix_additional_seek
Fix bug with additional file processing in cache compressed buffer
2019-04-07 01:24:50 +03:00
Danila Kutenin
1c030e4ed1 Fix allocations for a while 2019-04-06 23:53:25 +03:00
Danila Kutenin
37d70ac558 MarkCache and UncompressedCache use different allocator to catch segfaults more reliable 2019-04-06 18:27:39 +03:00
alesapin
7e1975b0c8 Fix bug with additional seek in cache compressed buffer 2019-04-05 12:42:44 +03:00
alesapin
5a6c4bf526 Better comment 2019-04-03 17:34:46 +03:00
alesapin
804a7612c2 Add comment 2019-04-03 17:28:39 +03:00
alesapin
bee8d61e26 Fix seek backwards after eof in ReadBufferAIO 2019-04-03 17:24:55 +03:00
Vitaly Baranov
37b904b8f5
Merge pull request #4808 from vitlibar/kafka-with-protobuf-format
Kafka with protobuf format
2019-03-30 19:00:57 +03:00
alesapin
2764401d0f Fix exception 'No message received' 2019-03-29 21:10:03 +03:00
Vitaly Baranov
a78e067c70 Allow empty row_delimiter when using Kafka. 2019-03-29 17:38:52 +03:00
proller
e4d45b7e50 Fix test include 2019-03-28 16:10:42 +03:00
Mikhail Fandyushin
c53b303bc2 HTTP Accept-Encoding: br; Added brotli errors codes 2019-03-27 00:32:29 +03:00
Mikhail Fandyushin
4b7ac9e9b4 HTTP Accept-Encoding: br; Fixed review notes 2019-03-26 23:27:34 +03:00
proller
fce9db12b9 Move Interpreters/Settings* -> Core/Settings* 2019-03-22 15:08:30 +03:00