thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-25 17:12:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master'

Browse Source
This commit is contained in:
BayoNet 2017-11-09 09:32:19 +03:00
commit f2395516e1
330 changed files with 6659 additions and 8204 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -35,6 +35,7 @@ cmake_install.cmake
CTestTestfile.cmake
*.a
*.o
cmake-build-*
# Python cache
*.pyc

3
.gitmodules vendored
View File

@ -13,3 +13,6 @@
[submodule "contrib/librdkafka"]
path = contrib/librdkafka
url = https://github.com/edenhill/librdkafka.git
[submodule "contrib/cctz"]
path = contrib/cctz
url = https://github.com/google/cctz.git

54
CHANGELOG.md
View File

@ -1,3 +1,57 @@
# ClickHouse release 1.1.54310
## New features:
* Custom partitioning key for the MergeTree family of table engines.
* [Kafka](https://clickhouse.yandex/docs/en/single/index.html#document-table_engines/kafka) table engine.
* Added support for loading [CatBoost](https://catboost.yandex/) models and applying them to data stored in ClickHouse.
* Added support for time zones with non-integer offsets from UTC.
* Added support for arithmetic operations with time intervals.
* The range of values for the Date and DateTime types is extended to the year 2105.
* Added the `CREATE MATERIALIZED VIEW x TO y` query (specifies an existing table for storing the data of a materialized view).
* Added the `ATTACH TABLE` query without arguments.
* The processing logic for Nested columns with names ending in -Map in a SummingMergeTree table was extracted to the sumMap aggregate function. You can now specify such columns explicitly.
* Max size of the IP trie dictionary is increased to 128M entries.
* Added the `getSizeOfEnumType` function.
* Added the `sumWithOverflow` aggregate function.
* Added support for the Cap'n Proto input format.
* You can now customize compression level when using the zstd algorithm.
## Backwards incompatible changes:
* Creation of temporary tables with an engine other than Memory is forbidden.
* Explicit creation of tables with the View or MaterializedView engine is forbidden.
* During table creation, a new check verifies that the sampling key expression is included in the primary key.
## Bug fixes:
* Fixed hangups when synchronously inserting into a Distributed table.
* Fixed nonatomic adding and removing of parts in Replicated tables.
* Data inserted into a materialized view is not subjected to unnecessary deduplication.
* Executing a query to a Distributed table for which the local replica is lagging and remote replicas are unavailable does not result in an error anymore.
* Users don't need access permissions to the `default` database to create temporary tables anymore.
* Fixed crashing when specifying the Array type without arguments.
* Fixed hangups when the disk volume containing server logs is full.
* Fixed an overflow in the `toRelativeWeekNum` function for the first week of the Unix epoch.
## Build improvements:
* Several third-party libraries (notably Poco) were updated and converted to git submodules.
# ClickHouse release 1.1.54304
## New features:
* TLS support in the native protocol (to enable, set `tcp_ssl_port` in `config.xml`)
## Bug fixes:
* `ALTER` for replicated tables now tries to start running as soon as possible
* Fixed crashing when reading data with the setting `preferred_block_size_bytes=0`
* Fixed crashes of `clickhouse-client` when `Page Down` is pressed
* Correct interpretation of certain complex queries with `GLOBAL IN` and `UNION ALL`
* `FREEZE PARTITION` always works atomically now
* Empty POST requests now return a response with code 411
* Fixed interpretation errors for expressions like `CAST(1 AS Nullable(UInt8))`
* Fixed an error when reading columns like `Array(Nullable(String))` from `MergeTree` tables
* Fixed crashing when parsing queries like `SELECT dummy AS dummy, dummy AS b`
* Users are updated correctly when `users.xml` is invalid
* Correct handling when an executable dictionary returns a non-zero response code
# ClickHouse release 1.1.54292
## New features:

36
CHANGELOG_RU.md
View File

@ -1,3 +1,39 @@
# Релиз ClickHouse 1.1.54310
## Новые возможности:
* Произвольный ключ партиционирования для таблиц семейства MergeTree.
* Движок таблиц [Kafka](https://clickhouse.yandex/docs/en/single/index.html#document-table_engines/kafka).
* Возможность загружать модели [CatBoost](https://catboost.yandex/) и применять их к данным, хранящимся в ClickHouse.
* Поддержка часовых поясов с нецелым смещением от UTC.
* Поддержка операций с временными интервалами.
* Диапазон значений типов Date и DateTime расширен до 2105 года.
* Запрос `CREATE MATERIALIZED VIEW x TO y` (позволяет указать существующую таблицу для хранения данных материализованного представления).
* Запрос `ATTACH TABLE` без аргументов.
* Логика обработки Nested-столбцов в SummingMergeTree, заканчивающихся на -Map, вынесена в агрегатную функцию sumMap. Такие столбцы теперь можно задавать явно.
* Максимальный размер IP trie-словаря увеличен до 128М записей.
* Функция getSizeOfEnumType.
* Агрегатная функция sumWithOverflow.
* Поддержка входного формата Cap’n Proto.
* Возможность задавать уровень сжатия при использовании алгоритма zstd.
## Обратно несовместимые изменения:
* Запрещено создание временных таблиц с движком, отличным от Memory.
* Запрещено явное создание таблиц с движком View и MaterializedView.
* При создании таблицы теперь проверяется, что ключ сэмплирования входит в первичный ключ.
## Исправления ошибок:
* Исправлено зависание при синхронной вставке в Distributed таблицу.
* Исправлена неатомарность при добавлении/удалении кусков в реплицированных таблицах.
* Данные, вставляемые в материализованное представление, теперь не подвергаются излишней дедупликации.
* Запрос в Distributed таблицу, для которого локальная реплика отстаёт, а удалённые недоступны, теперь не падает.
* Для создания временных таблиц теперь не требуется прав доступа к БД `default`.
* Исправлено падение при указании типа Array без аргументов.
* Исправлено зависание при недостатке места на диске в разделе с логами.
* Исправлено переполнение в функции toRelativeWeekNum для первой недели Unix-эпохи.
## Улучшения сборки:
* Несколько сторонних библиотек (в частности, Poco) обновлены и переведены на git submodules.
# Релиз ClickHouse 1.1.54304
## Новые возможности:
* Добавлена поддержка TLS в нативном протоколе (включается заданием `tcp_ssl_port` в `config.xml`)

14
CMakeLists.txt
View File

@ -24,9 +24,6 @@ if (CMAKE_SYSTEM MATCHES "FreeBSD")
set (PLATFORM_EXTRA_CXX_FLAG "-DCLOCK_MONOTONIC_COARSE=CLOCK_MONOTONIC_FAST")
endif ()
cmake_policy (SET CMP0014 OLD) # Ignore warning about CMakeLists.txt in each directory
cmake_policy (SET CMP0012 NEW) # Don't dereference TRUE and FALSE
# Write compile_commands.json
set(CMAKE_EXPORT_COMPILE_COMMANDS 1)
@ -117,8 +114,16 @@ endif ()
# set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -Wno-unused-command-line-argument -mllvm -inline-threshold=10000")
#endif ()
if (CMAKE_VERSION VERSION_LESS "3.8.0")
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++1z")
else ()
set (CMAKE_CXX_STANDARD 17)
set (CMAKE_CXX_STANDARD_REQUIRED ON)
set (CXX_FLAGS_INTERNAL_COMPILER "-std=gnu++1z")
endif ()
set (CMAKE_BUILD_COLOR_MAKEFILE ON)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${COMPILER_FLAGS} -std=gnu++1z ${PLATFORM_EXTRA_CXX_FLAG} -fno-omit-frame-pointer ${COMMON_WARNING_FLAGS} ${CXX_WARNING_FLAGS} ${GLIBC_COMPATIBILITY_COMPILE_FLAGS}")
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${COMPILER_FLAGS} ${PLATFORM_EXTRA_CXX_FLAG} -fno-omit-frame-pointer ${COMMON_WARNING_FLAGS} ${CXX_WARNING_FLAGS} ${GLIBC_COMPATIBILITY_COMPILE_FLAGS}")
#set (CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE}")
set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -O3")
set (CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -O0 -g3 -ggdb3 -fno-inline")
@ -230,6 +235,7 @@ include (cmake/find_readline_edit.cmake)
include (cmake/find_zookeeper.cmake)
include (cmake/find_re2.cmake)
include (cmake/find_rdkafka.cmake)
include (cmake/find_capnp.cmake)
include (cmake/find_contrib_lib.cmake)
find_contrib_lib(cityhash)

2
README.md
View File

@ -1,5 +1,3 @@
ClickHouse is an open-source column-oriented database management system that allows generating analytical data reports in real time.
[Read more...](https://clickhouse.yandex/)
[ClickHouse Community Meetup in Palo Alto on October 25, 2017](http://bit.ly/clickhouse-meetup-palo-alto-october-2017)

22
cmake/find_capnp.cmake Normal file
View File

@ -0,0 +1,22 @@
option (ENABLE_CAPNP "Enable Cap'n Proto" ON)
if (ENABLE_CAPNP)
set (CAPNP_PATHS "/usr/local/opt/capnp/lib")
set (CAPNP_INCLUDE_PATHS "/usr/local/opt/capnp/include")
find_library (CAPNP capnp PATHS ${CAPNP_PATHS})
find_library (CAPNPC capnpc PATHS ${CAPNP_PATHS})
find_library (KJ kj PATHS ${CAPNP_PATHS})
set (CAPNP_LIBS ${CAPNP} ${CAPNPC} ${KJ})
find_path (CAPNP_INCLUDE_DIR NAMES capnp/schema-parser.h PATHS ${CAPNP_INCLUDE_PATHS})
if (CAPNP_INCLUDE_DIR AND CAPNP_LIBS)
include_directories (${CAPNP_INCLUDE_DIR})
set(USE_CAPNP 1)
endif ()
endif ()
if (USE_CAPNP)
message (STATUS "Using capnp=${USE_CAPNP}: ${CAPNP_INCLUDE_DIR} : ${CAPNP_LIBS}")
else ()
message (STATUS "Build without capnp (support for Cap'n Proto format will be disabled)")
endif ()

2
cmake/find_lz4.cmake
View File

@ -1,6 +1,6 @@
option (USE_INTERNAL_LZ4_LIBRARY "Set to FALSE to use system lz4 library instead of bundled" ${NOT_UNBUNDLED})
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/lz4/lib/lz4.h")
if (USE_INTERNAL_LZ4_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/lz4/lib/lz4.h")
message (WARNING "submodule contrib/lz4 is missing. to fix try run: \n git submodule update --init --recursive")
set (USE_INTERNAL_LZ4_LIBRARY 0)
endif ()

5
cmake/find_openssl.cmake
View File

@ -5,11 +5,12 @@ endif ()
find_package (OpenSSL)
if (NOT OPENSSL_FOUND)
# Try to find manually.
set (OPENSSL_INCLUDE_DIR "/usr/local/opt/openssl/include")
set (OPENSSL_INCLUDE_PATHS "/usr/local/opt/openssl/include")
set (OPENSSL_PATHS "/usr/local/opt/openssl/lib")
find_path (OPENSSL_INCLUDE_DIR NAMES openssl/ssl.h PATHS ${OPENSSL_INCLUDE_PATHS})
find_library (OPENSSL_SSL_LIBRARY ssl PATHS ${OPENSSL_PATHS})
find_library (OPENSSL_CRYPTO_LIBRARY crypto PATHS ${OPENSSL_PATHS})
if (OPENSSL_SSL_LIBRARY AND OPENSSL_CRYPTO_LIBRARY)
if (OPENSSL_SSL_LIBRARY AND OPENSSL_CRYPTO_LIBRARY AND OPENSSL_INCLUDE_DIR)
set (OPENSSL_LIBRARIES ${OPENSSL_SSL_LIBRARY} ${OPENSSL_CRYPTO_LIBRARY})
set (OPENSSL_FOUND 1)
endif ()

11
cmake/find_poco.cmake
View File

@ -1,6 +1,6 @@
option (USE_INTERNAL_POCO_LIBRARY "Set to FALSE to use system poco library instead of bundled" ${NOT_UNBUNDLED})
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/poco/CMakeLists.txt")
if (USE_INTERNAL_POCO_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/poco/CMakeLists.txt")
message (WARNING "submodule contrib/poco is missing. to fix try run: \n git submodule update --init --recursive")
set (USE_INTERNAL_POCO_LIBRARY 0)
endif ()
@ -46,9 +46,7 @@ else ()
if (ODBC_FOUND)
set (Poco_DataODBC_FOUND 1)
set (Poco_DataODBC_LIBRARY PocoDataODBC)
if (USE_STATIC_LIBRARIES)
list (APPEND Poco_DataODBC_LIBRARY ${LTDL_LIB})
endif ()
list (APPEND Poco_DataODBC_LIBRARY ${LTDL_LIB})
set (Poco_DataODBC_INCLUDE_DIRS "${ClickHouse_SOURCE_DIR}/contrib/poco/Data/ODBC/include/")
endif ()
@ -87,10 +85,11 @@ message(STATUS "Using Poco: ${Poco_INCLUDE_DIRS} : ${Poco_Foundation_LIBRARY},${
# and merge:
# ClickHouse-Extras/clickhouse_unbundled
# ClickHouse-Extras/clickhouse_unbundled_zlib
# ClickHouse-Extras/clickhouse_task # uses c++11, can't push to poco
# ClickHouse-Extras/clickhouse_task
# ClickHouse-Extras/clickhouse_misc
# ClickHouse-Extras/clickhouse_anl
# ClickHouse-Extras/clickhouse_http_header https://github.com/pocoproject/poco/pull/1574
# ClickHouse-Extras/clickhouse_socket
# ClickHouse-Extras/clickhouse_warning
# ClickHouse-Extras/clickhouse-purge-logs-on-no-space
# ClickHouse-Extras/clickhouse_freebsd

2
cmake/find_rdkafka.cmake
View File

@ -4,7 +4,7 @@ if (ENABLE_RDKAFKA)
option (USE_INTERNAL_RDKAFKA_LIBRARY "Set to FALSE to use system librdkafka instead of the bundled" ${NOT_UNBUNDLED})
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/librdkafka/CMakeLists.txt")
if (USE_INTERNAL_RDKAFKA_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/librdkafka/CMakeLists.txt")
message (WARNING "submodule contrib/librdkafka is missing. to fix try run: \n git submodule update --init --recursive")
set (USE_INTERNAL_RDKAFKA_LIBRARY 0)
set (MISSING_INTERNAL_RDKAFKA_LIBRARY 1)

2
cmake/find_readline_edit.cmake
View File

@ -24,7 +24,7 @@ if (READLINE_LIB AND TERMCAP_LIB)
set (LINE_EDITING_LIBS ${READLINE_LIB} ${TERMCAP_LIB})
message (STATUS "Using line editing libraries (readline): ${READLINE_INCLUDE_DIR} : ${LINE_EDITING_LIBS}")
endif ()
elseif (EDIT_LIB)
elseif (EDIT_LIB AND TERMCAP_LIB)
find_library (CURSES_LIB NAMES curses)
find_path (READLINE_INCLUDE_DIR NAMES editline/readline.h PATHS ${READLINE_INCLUDE_PATHS})
if (CURSES_LIB AND READLINE_INCLUDE_DIR)

2
cmake/find_zookeeper.cmake
View File

@ -1,6 +1,6 @@
option (USE_INTERNAL_ZOOKEEPER_LIBRARY "Set to FALSE to use system zookeeper library instead of bundled" ${NOT_UNBUNDLED})
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/zookeeper/src/c/CMakeLists.txt")
if (USE_INTERNAL_ZOOKEEPER_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/zookeeper/src/c/CMakeLists.txt")
message (WARNING "submodule contrib/zookeeper is missing. to fix try run: \n git submodule update --init --recursive")
set (USE_INTERNAL_ZOOKEEPER_LIBRARY 0)
endif ()

2
cmake/find_zstd.cmake
View File

@ -1,6 +1,6 @@
option (USE_INTERNAL_ZSTD_LIBRARY "Set to FALSE to use system zstd library instead of bundled" ${NOT_UNBUNDLED})
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/zstd/lib/zstd.h")
if (USE_INTERNAL_ZSTD_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/zstd/lib/zstd.h")
message (WARNING "submodule contrib/zstd is missing. to fix try run: \n git submodule update --init --recursive")
set (USE_INTERNAL_ZSTD_LIBRARY 0)
endif ()

2
contrib/CMakeLists.txt vendored
View File

@ -49,7 +49,7 @@ if (USE_INTERNAL_ZLIB_LIBRARY)
endif ()
if (USE_INTERNAL_CCTZ_LIBRARY)
add_subdirectory (libcctz)
add_subdirectory (cctz-cmake)
endif ()
if (ENABLE_LIBTCMALLOC AND USE_INTERNAL_GPERFTOOLS_LIBRARY)

1
contrib/cctz vendored Submodule

@ -0,0 +1 @@
Subproject commit 4f9776a310f4952454636363def82c2bf6641d5f

31
contrib/cctz-cmake/CMakeLists.txt Normal file
View File

@ -0,0 +1,31 @@
SET(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/cctz)
add_library(cctz
${LIBRARY_DIR}/src/civil_time_detail.cc
${LIBRARY_DIR}/src/time_zone_fixed.cc
${LIBRARY_DIR}/src/time_zone_format.cc
${LIBRARY_DIR}/src/time_zone_if.cc
${LIBRARY_DIR}/src/time_zone_impl.cc
${LIBRARY_DIR}/src/time_zone_info.cc
${LIBRARY_DIR}/src/time_zone_libc.cc
${LIBRARY_DIR}/src/time_zone_lookup.cc
${LIBRARY_DIR}/src/time_zone_posix.cc
${LIBRARY_DIR}/src/zone_info_source.cc
${LIBRARY_DIR}/src/time_zone_libc.h
${LIBRARY_DIR}/src/time_zone_if.h
${LIBRARY_DIR}/src/tzfile.h
${LIBRARY_DIR}/src/time_zone_impl.h
${LIBRARY_DIR}/src/time_zone_posix.h
${LIBRARY_DIR}/src/time_zone_info.h
${LIBRARY_DIR}/include/cctz/time_zone.h
${LIBRARY_DIR}/include/cctz/civil_time_detail.h
${LIBRARY_DIR}/include/cctz/civil_time.h)
if (CMAKE_SYSTEM MATCHES "FreeBSD")
# yes, need linux, because bsd check inside linux in time_zone_libc.cc:24
target_compile_definitions (cctz PRIVATE __USE_BSD linux _XOPEN_SOURCE=600)
endif ()
target_include_directories (cctz PUBLIC ${LIBRARY_DIR}/include)

27
contrib/libcctz/CMakeLists.txt
View File

@ -1,27 +0,0 @@
add_library(cctz
src/time_zone_libc.cc
src/time_zone_posix.cc
src/time_zone_lookup.cc
src/time_zone_info.cc
src/time_zone_if.cc
src/time_zone_format.cc
src/time_zone_impl.cc
src/time_zone_libc.h
src/time_zone_if.h
src/tzfile.h
src/time_zone_impl.h
src/time_zone_posix.h
src/time_zone_info.h
include/time_zone.h
include/civil_time_detail.h
include/civil_time.h)
if (CMAKE_SYSTEM MATCHES "FreeBSD")
# yes, need linux, because bsd check inside linux in time_zone_libc.cc:24
target_compile_definitions (cctz PRIVATE __USE_BSD linux _XOPEN_SOURCE=600)
endif ()
target_include_directories (cctz PUBLIC include)

202
contrib/libcctz/LICENSE.txt
View File

@ -1,202 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

192
contrib/libcctz/README.md
View File

@ -1,192 +0,0 @@
This is not an official Google product.
# Overview
CCTZ contains two libraries that cooperate with `<chrono>` to give C++
programmers all the necessary tools for computing with dates, times, and time
zones in a simple and correct manner. The libraries in CCTZ are:
* **The Civil-Time Library** &mdash; This is a header-only library that supports
computing with human-scale time, such as dates (which are represented by the
`cctz::civil_day` class). This library is declared in [`include/civil_time.h`]
(https://github.com/google/cctz/blob/master/include/civil_time.h).
* **The Time-Zone Library** &mdash; This library uses the IANA time zone
database that is installed on the system to convert between *absolute time*
and *civil time*. This library is declared in [`include/time_zone.h`](https://github.com/google/cctz/blob/master/include/time_zone.h).
These libraries are currently known to work on **Linux** and **Mac OS X**. We
are actively interested in help getting them working on Windows. Please contact
us if you're interested in contributing.
# Getting Started
CCTZ is best built and tested using the [Bazel](http://bazel.io) build system
and the [Google Test](https://github.com/google/googletest) framework. (There
is also a simple [`Makefile`](https://github.com/google/cctz/blob/master/Makefile)
that should work if you're unable to use Bazel.)
1. Download/install Bazel http://bazel.io/docs/install.html
2. Get the cctz source: `git clone https://github.com/google/cctz.git` then `cd
cctz`
3. Build cctz and run the tests: `bazel test :all`
Note: When using CCTZ in your own project, you might find it easiest to compile
the sources using your existing build system.
Next Steps:
1. See the documentation for the libraries in CCTZ:
* Civil Time: [`include/civil_time.h`](https://github.com/google/cctz/blob/master/include/civil_time.h)
* Time Zone: [`include/time_zone.h`](https://github.com/google/cctz/blob/master/include/time_zone.h)
2. Look at the examples in https://github.com/google/cctz/tree/master/examples
3. Join our mailing list to ask questions and keep informed of changes:
* https://groups.google.com/forum/#!forum/cctz
# Fundamental Concepts
*[The concepts presented here describe general truths about the problem domain
and are library and language agnostic. An understanding of these concepts helps
the programmer correctly reason about even the most-complicated time-programming
challenges and produce the simplest possible solutions.]*
There are two main ways to think about time in a computer program: as *absolute
time*, and as *civil time*. Both have their uses and it is important to
understand when each is appropriate. Absolute and civil times may be converted
back and forth using a *time zone* &mdash; this is the only way to correctly
convert between them. Let us now look more deeply at the three main concepts of
time programming: Absolute Time, Civil Time, and Time Zone.
*Absolute time* uniquely and universally represents a specific instant in time.
It has no notion of calendars, or dates, or times of day. Instead, it is a
measure of the passage of real time, typically as a simple count of ticks since
some epoch. Absolute times are independent of all time zones and do not suffer
from human-imposed complexities such as daylight-saving time (DST). Many C++
types exist to represent absolute times, classically `time_t` and more recently
`std::chrono::time_point`.
*Civil time* is the legally recognized representation of time for ordinary
affairs (cf. http://www.merriam-webster.com/dictionary/civil). It is a
human-scale representation of time that consists of the six fields &mdash;
year, month, day, hour, minute, and second (sometimes shortened to "YMDHMS")
&mdash; and it follows the rules of the Proleptic Gregorian Calendar, with
24-hour days divided into 60-minute hours and 60-second minutes. Like absolute
times, civil times are also independent of all time zones and their related
complexities (e.g., DST). While `std::tm` contains the six civil-time fields
(YMDHMS), plus a few more, it does not have behavior to enforce the rules of
civil time.
*Time zones* are geo-political regions within which human-defined rules are
shared to convert between the absolute-time and civil-time domains. A time
zone's rules include things like the region's offset from the UTC time standard,
daylight-saving adjustments, and short abbreviation strings. Time zones often
have a history of disparate rules that apply only for certain periods, because
the rules may change at the whim of a region's local government. For this
reason, time-zone rules are usually compiled into data snapshots that are used
at runtime to perform conversions between absolute and civil times. There is
currently no C++ standard library supporting arbitrary time zones.
In order for programmers to reason about and program applications that correctly
deal with these concepts, they must have a library that correctly implements the
above concepts. CCTZ adds to the existing C++11 `<chrono>` library to fully
implement the above concepts.
* Absolute time &mdash; This is implemented by the existing C++11
[`<chrono>`](http://en.cppreference.com/w/cpp/chrono)
library without modification. For example, an absolute point in time is
represented by a `std::chrono::time_point`.
* Civil time &mdash; This is implemented by the [`include/civil_time.h`](https://github.com/google/cctz/blob/master/include/civil_time.h) library
that is provided as part of CCTZ. For example, a "date" is represented by a
`cctz::civil_day`.
* Time zone &mdash; This is implemented by the [`include/time_zone.h`](https://github.com/google/cctz/blob/master/include/time_zone.h) library
that is provided as part of CCTZ. For example, a time zone is represented by
an instance of the class `cctz::time_zone`.
# Examples
## Hello February 2016
This "hello world" example uses a for-loop to iterate the days from the first of
February until the month of March. Each day is streamed to output, and if the
day happens to be the 29th, we also output the day of the week.
```
#include <iostream>
#include "civil_time.h"
int main() {
for (cctz::civil_day d(2016, 2, 1); d < cctz::civil_month(2016, 3); ++d) {
std::cout << "Hello " << d;
if (d.day() == 29) {
std::cout << " <- leap day is a " << cctz::get_weekday(d);
}
std::cout << "\n";
}
}
```
The output of the above program is
```
Hello 2016-02-01
Hello 2016-02-02
Hello 2016-02-03
[...]
Hello 2016-02-27
Hello 2016-02-28
Hello 2016-02-29 <- leap day is a Monday
```
## One giant leap
This example shows how to use all three libraries (`<chrono>`, civil time, and
time zone) together. In this example, we know that viewers in New York watched
Neil Armstrong first walk on the moon on July 20, 1969 at 10:56 PM. But we'd
like to see what time it was for our friend watching in Sydney Australia.
```
#include <iostream>
#include "civil_time.h"
#include "time_zone.h"
int main() {
cctz::time_zone nyc;
cctz::load_time_zone("America/New_York", &nyc);
// Converts the input civil time in NYC to an absolute time.
const auto moon_walk =
cctz::convert(cctz::civil_second(1969, 7, 20, 22, 56, 0), nyc);
std::cout << "Moon walk in NYC: "
<< cctz::format("%Y-%m-%d %H:%M:%S %Ez\n", moon_walk, nyc);
cctz::time_zone syd;
if (!cctz::load_time_zone("Australia/Sydney", &syd)) return -1;
std::cout << "Moon walk in SYD: "
<< cctz::format("%Y-%m-%d %H:%M:%S %Ez\n", moon_walk, syd);
}
```
The output of the above program is
```
Moon walk in NYC: 1969-07-20 22:56:00 -04:00
Moon walk in SYD: 1969-07-21 12:56:00 +10:00
```
This example shows that the absolute time (the `std::chrono::time_point`) of the
first walk on the moon is the same no matter the time zone of the viewer (the
same time point is used in both calls to `format()`). The only difference is the
time zone in which the `moon_walk` time point is rendered. And in this case we
can see that our friend in Sydney was probably eating lunch while watching that
historic event.
# References
* CCTZ [FAQ](https://github.com/google/cctz/wiki/FAQ)
* See also the [Time Programming Fundamentals](https://youtu.be/2rnIHsqABfM)
talk from CppCon 2015 ([slides available here](http://goo.gl/ofof4N)). This
talk mostly describes the older CCTZ v1 API, but the *concepts* are the same.
* ISO C++ proposal to standardize the Civil-Time Library:
https://github.com/devjgm/papers/blob/master/d0215r1.md
* ISO C++ proposal to standardize the Time-Zone Library:
https://github.com/devjgm/papers/blob/master/d0216r1.md

325
contrib/libcctz/include/civil_time.h
View File

@ -1,325 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef CCTZ_CIVIL_TIME_H_
#define CCTZ_CIVIL_TIME_H_
#include "civil_time_detail.h"
namespace cctz {
// The term "civil time" refers to the legally recognized human-scale time
// that is represented by the six fields YYYY-MM-DD hh:mm:ss. Modern-day civil
// time follows the Gregorian Calendar and is a time-zone-independent concept.
// A "date" is perhaps the most common example of a civil time (represented in
// this library as cctz::civil_day). This library provides six classes and a
// handful of functions that help with rounding, iterating, and arithmetic on
// civil times while avoiding complications like daylight-saving time (DST).
//
// The following six classes form the core of this civil-time library:
//
// * civil_second
// * civil_minute
// * civil_hour
// * civil_day
// * civil_month
// * civil_year
//
// Each class is a simple value type with the same interface for construction
// and the same six accessors for each of the civil fields (year, month, day,
// hour, minute, and second, aka YMDHMS). These classes differ only in their
// alignment, which is indicated by the type name and specifies the field on
// which arithmetic operates.
//
// Each class can be constructed by passing up to six optional integer
// arguments representing the YMDHMS fields (in that order) to the
// constructor. Omitted fields are assigned their minimum valid value. Hours,
// minutes, and seconds will be set to 0, month and day will be set to 1, and
// since there is no minimum valid year, it will be set to 1970. So, a
// default-constructed civil-time object will have YMDHMS fields representing
// "1970-01-01 00:00:00". Fields that are out-of-range are normalized (e.g.,
// October 32 -> November 1) so that all civil-time objects represent valid
// values.
//
// Each civil-time class is aligned to the civil-time field indicated in the
// class's name after normalization. Alignment is performed by setting all the
// inferior fields to their minimum valid value (as described above). The
// following are examples of how each of the six types would align the fields
// representing November 22, 2015 at 12:34:56 in the afternoon. (Note: the
// string format used here is not important; it's just a shorthand way of
// showing the six YMDHMS fields.)
//
// civil_second 2015-11-22 12:34:56
// civil_minute 2015-11-22 12:34:00
// civil_hour 2015-11-22 12:00:00
// civil_day 2015-11-22 00:00:00
// civil_month 2015-11-01 00:00:00
// civil_year 2015-01-01 00:00:00
//
// Each civil-time type performs arithmetic on the field to which it is
// aligned. This means that adding 1 to a civil_day increments the day field
// (normalizing as necessary), and subtracting 7 from a civil_month operates
// on the month field (normalizing as necessary). All arithmetic produces a
// valid civil time. Difference requires two similarly aligned civil-time
// objects and returns the scalar answer in units of the objects' alignment.
// For example, the difference between two civil_hour objects will give an
// answer in units of civil hours.
//
// In addition to the six civil-time types just described, there are
// a handful of helper functions and algorithms for performing common
// calculations. These are described below.
//
// Note: In C++14 and later, this library is usable in a constexpr context.
//
// CONSTRUCTION:
//
// Each of the civil-time types can be constructed in two ways: by directly
// passing to the constructor up to six (optional) integers representing the
// YMDHMS fields, or by copying the YMDHMS fields from a differently aligned
// civil-time type.
//
// civil_day default_value; // 1970-01-01 00:00:00
//
// civil_day a(2015, 2, 3); // 2015-02-03 00:00:00
// civil_day b(2015, 2, 3, 4, 5, 6); // 2015-02-03 00:00:00
// civil_day c(2015); // 2015-01-01 00:00:00
//
// civil_second ss(2015, 2, 3, 4, 5, 6); // 2015-02-03 04:05:06
// civil_minute mm(ss); // 2015-02-03 04:05:00
// civil_hour hh(mm); // 2015-02-03 04:00:00
// civil_day d(hh); // 2015-02-03 00:00:00
// civil_month m(d); // 2015-02-01 00:00:00
// civil_year y(m); // 2015-01-01 00:00:00
//
// m = civil_month(y); // 2015-01-01 00:00:00
// d = civil_day(m); // 2015-01-01 00:00:00
// hh = civil_hour(d); // 2015-01-01 00:00:00
// mm = civil_minute(hh); // 2015-01-01 00:00:00
// ss = civil_second(mm); // 2015-01-01 00:00:00
//
// ALIGNMENT CONVERSION:
//
// The alignment of a civil-time object cannot change, but the object may be
// used to construct a new object with a different alignment. This is referred
// to as "realigning". When realigning to a type with the same or more
// precision (e.g., civil_day -> civil_second), the conversion may be
// performed implicitly since no information is lost. However, if information
// could be discarded (e.g., civil_second -> civil_day), the conversion must
// be explicit at the call site.
//
// void fun(const civil_day& day);
//
// civil_second cs;
// fun(cs); // Won't compile because data may be discarded
// fun(civil_day(cs)); // OK: explicit conversion
//
// civil_day cd;
// fun(cd); // OK: no conversion needed
//
// civil_month cm;
// fun(cm); // OK: implicit conversion to civil_day
//
// NORMALIZATION:
//
// Integer arguments passed to the constructor may be out-of-range, in which
// case they are normalized to produce a valid civil-time object. This enables
// natural arithmetic on constructor arguments without worrying about the
// field's range. Normalization guarantees that there are no invalid
// civil-time objects.
//
// civil_day d(2016, 10, 32); // Out-of-range day; normalized to 2016-11-01
//
// Note: If normalization is undesired, you can signal an error by comparing
// the constructor arguments to the normalized values returned by the YMDHMS
// properties.
//
// PROPERTIES:
//
// All civil-time types have accessors for all six of the civil-time fields:
// year, month, day, hour, minute, and second. Recall that fields inferior to
// the type's aligment will be set to their minimum valid value.
//
// civil_day d(2015, 6, 28);
// // d.year() == 2015
// // d.month() == 6
// // d.day() == 28
// // d.hour() == 0
// // d.minute() == 0
// // d.second() == 0
//
// COMPARISON:
//
// Comparison always considers all six YMDHMS fields, regardless of the type's
// alignment. Comparison between differently aligned civil-time types is
// allowed.
//
// civil_day feb_3(2015, 2, 3); // 2015-02-03 00:00:00
// civil_day mar_4(2015, 3, 4); // 2015-03-04 00:00:00
// // feb_3 < mar_4
// // civil_year(feb_3) == civil_year(mar_4)
//
// civil_second feb_3_noon(2015, 2, 3, 12, 0, 0); // 2015-02-03 12:00:00
// // feb_3 < feb_3_noon
// // feb_3 == civil_day(feb_3_noon)
//
// // Iterates all the days of February 2015.
// for (civil_day d(2015, 2, 1); d < civil_month(2015, 3); ++d) {
// // ...
// }
//
// STREAMING:
//
// Each civil-time type may be sent to an output stream using operator<<().
// The output format follows the pattern "YYYY-MM-DDThh:mm:ss" where fields
// inferior to the type's alignment are omitted.
//
// civil_second cs(2015, 2, 3, 4, 5, 6);
// std::cout << cs << "\n"; // Outputs: 2015-02-03T04:05:06
//
// civil_day cd(cs);
// std::cout << cd << "\n"; // Outputs: 2015-02-03
//
// civil_year cy(cs);
// std::cout << cy << "\n"; // Outputs: 2015
//
// ARITHMETIC:
//
// Civil-time types support natural arithmetic operators such as addition,
// subtraction, and difference. Arithmetic operates on the civil-time field
// indicated in the type's name. Difference requires arguments with the same
// alignment and returns the answer in units of the alignment.
//
// civil_day a(2015, 2, 3);
// ++a; // 2015-02-04 00:00:00
// --a; // 2015-02-03 00:00:00
// civil_day b = a + 1; // 2015-02-04 00:00:00
// civil_day c = 1 + b; // 2015-02-05 00:00:00
// int n = c - a; // n = 2 (civil days)
// int m = c - civil_month(c); // Won't compile: different types.
//
// EXAMPLE: Adding a month to January 31.
//
// One of the classic questions that arises when considering a civil-time
// library (or a date library or a date/time library) is this: "What happens
// when you add a month to January 31?" This is an interesting question
// because there could be a number of possible answers:
//
// 1. March 3 (or 2 if a leap year). This may make sense if the operation
// wants the equivalent of February 31.
// 2. February 28 (or 29 if a leap year). This may make sense if the operation
// wants the last day of January to go to the last day of February.
// 3. Error. The caller may get some error, an exception, an invalid date
// object, or maybe false is returned. This may make sense because there is
// no single unambiguously correct answer to the question.
//
// Practically speaking, any answer that is not what the programmer intended
// is the wrong answer.
//
// This civil-time library avoids the problem by making it impossible to ask
// ambiguous questions. All civil-time objects are aligned to a particular
// civil-field boundary (such as aligned to a year, month, day, hour, minute,
// or second), and arithmetic operates on the field to which the object is
// aligned. This means that in order to "add a month" the object must first be
// aligned to a month boundary, which is equivalent to the first day of that
// month.
//
// Of course, there are ways to compute an answer the question at hand using
// this civil-time library, but they require the programmer to be explicit
// about the answer they expect. To illustrate, let's see how to compute all
// three of the above possible answers to the question of "Jan 31 plus 1
// month":
//
// const civil_day d(2015, 1, 31);
//
// // Answer 1:
// // Add 1 to the month field in the constructor, and rely on normalization.
// const auto ans_normalized = civil_day(d.year(), d.month() + 1, d.day());
// // ans_normalized == 2015-03-03 (aka Feb 31)
//
// // Answer 2:
// // Add 1 to month field, capping to the end of next month.
// const auto next_month = civil_month(d) + 1;
// const auto last_day_of_next_month = civil_day(next_month + 1) - 1;
// const auto ans_capped = std::min(ans_normalized, last_day_of_next_month);
// // ans_capped == 2015-02-28
//
// // Answer 3:
// // Signal an error if the normalized answer is not in next month.
// if (civil_month(ans_normalized) != next_month) {
// // error, month overflow
// }
//
using civil_year = detail::civil_year;
using civil_month = detail::civil_month;
using civil_day = detail::civil_day;
using civil_hour = detail::civil_hour;
using civil_minute = detail::civil_minute;
using civil_second = detail::civil_second;
// An enum class with members monday, tuesday, wednesday, thursday, friday,
// saturday, and sunday. These enum values may be sent to an output stream
// using operator<<(). The result is the full weekday name in English with a
// leading capital letter.
//
// weekday wd = weekday::thursday;
// std::cout << wd << "\n"; // Outputs: Thursday
//
using detail::weekday;
// Returns the weekday for the given civil_day.
//
// civil_day a(2015, 8, 13);
// weekday wd = get_weekday(a); // wd == weekday::thursday
//
using detail::get_weekday;
// Returns the civil_day that strictly follows or precedes the given
// civil_day, and that falls on the given weekday.
//
// For example, given:
//
// August 2015
// Su Mo Tu We Th Fr Sa
// 1
// 2 3 4 5 6 7 8
// 9 10 11 12 13 14 15
// 16 17 18 19 20 21 22
// 23 24 25 26 27 28 29
// 30 31
//
// civil_day a(2015, 8, 13); // get_weekday(a) == weekday::thursday
// civil_day b = next_weekday(a, weekday::thursday); // b = 2015-08-20
// civil_day c = prev_weekday(a, weekday::thursday); // c = 2015-08-06
//
// civil_day d = ...
// // Gets the following Thursday if d is not already Thursday
// civil_day thurs1 = prev_weekday(d, weekday::thursday) + 7;
// // Gets the previous Thursday if d is not already Thursday
// civil_day thurs2 = next_weekday(d, weekday::thursday) - 7;
//
using detail::next_weekday;
using detail::prev_weekday;
// Returns the day-of-year for the given civil_day.
//
// civil_day a(2015, 1, 1);
// int yd_jan_1 = get_yearday(a); // yd_jan_1 = 1
// civil_day b(2015, 12, 31);
// int yd_dec_31 = get_yearday(b); // yd_dec_31 = 365
//
using detail::get_yearday;
} // namespace cctz
#endif // CCTZ_CIVIL_TIME_H_

567
contrib/libcctz/include/civil_time_detail.h
View File

@ -1,567 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <cstdint>
#include <iomanip>
#include <limits>
#include <ostream>
#include <sstream>
#include <type_traits>
// Disable constexpr support unless we are using clang in C++14 mode.
#if __clang__ && __cpp_constexpr >= 201304
#define CONSTEXPR_D constexpr // data
#define CONSTEXPR_F constexpr // function
#define CONSTEXPR_M constexpr // member
#define CONSTEXPR_T constexpr // template
#else
#define CONSTEXPR_D const
#define CONSTEXPR_F inline
#define CONSTEXPR_M
#define CONSTEXPR_T
#endif
namespace cctz {
// Support years that at least span the range of 64-bit time_t values.
using year_t = std::int_fast64_t;
// Type alias that indicates an argument is not normalized (e.g., the
// constructor parameters and operands/results of addition/subtraction).
using diff_t = std::int_fast64_t;
namespace detail {
// Type aliases that indicate normalized argument values.
using month_t = std::int_fast8_t; // [1:12]
using day_t = std::int_fast8_t; // [1:31]
using hour_t = std::int_fast8_t; // [0:23]
using minute_t = std::int_fast8_t; // [0:59]
using second_t = std::int_fast8_t; // [0:59]
// Normalized civil-time fields: Y-M-D HH:MM:SS.
struct fields {
CONSTEXPR_M fields(year_t year, month_t month, day_t day,
hour_t hour, minute_t minute, second_t second)
: y(year), m(month), d(day), hh(hour), mm(minute), ss(second) {}
std::int_least64_t y;
std::int_least8_t m;
std::int_least8_t d;
std::int_least8_t hh;
std::int_least8_t mm;
std::int_least8_t ss;
};
struct second_tag {};
struct minute_tag : second_tag {};
struct hour_tag : minute_tag {};
struct day_tag : hour_tag {};
struct month_tag : day_tag {};
struct year_tag : month_tag {};
////////////////////////////////////////////////////////////////////////
// Field normalization (without avoidable overflow).
namespace impl {
CONSTEXPR_F bool is_leap_year(year_t y) noexcept {
return y % 4 == 0 && (y % 100 != 0 || y % 400 == 0);
}
CONSTEXPR_F int year_index(year_t y, month_t m) noexcept {
return (((y + (m > 2)) % 400) + 400) % 400;
}
CONSTEXPR_F int days_per_century(year_t y, month_t m) noexcept {
const int yi = year_index(y, m);
return 36524 + (yi == 0 || yi > 300);
}
CONSTEXPR_F int days_per_4years(year_t y, month_t m) noexcept {
const int yi = year_index(y, m);
return 1460 + (yi == 0 || yi > 300 || (yi - 1) % 100 < 96);
}
CONSTEXPR_F int days_per_year(year_t y, month_t m) noexcept {
return is_leap_year(y + (m > 2)) ? 366 : 365;
}
CONSTEXPR_F int days_per_month(year_t y, month_t m) noexcept {
CONSTEXPR_D signed char k_days_per_month[12] = {
31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 // non leap year
};
return k_days_per_month[m - 1] + (m == 2 && is_leap_year(y));
}
CONSTEXPR_F fields n_day(year_t y, month_t m, diff_t d, diff_t cd,
hour_t hh, minute_t mm, second_t ss) noexcept {
y += (cd / 146097) * 400;
cd %= 146097;
if (cd < 0) {
y -= 400;
cd += 146097;
}
y += (d / 146097) * 400;
d = d % 146097 + cd;
if (d <= 0) {
y -= 400;
d += 146097;
} else if (d > 146097) {
y += 400;
d -= 146097;
}
if (d > 365) {
for (int n = days_per_century(y, m); d > n; n = days_per_century(y, m)) {
d -= n;
y += 100;
}
for (int n = days_per_4years(y, m); d > n; n = days_per_4years(y, m)) {
d -= n;
y += 4;
}
for (int n = days_per_year(y, m); d > n; n = days_per_year(y, m)) {
d -= n;
++y;
}
}
if (d > 28) {
for (int n = days_per_month(y, m); d > n; n = days_per_month(y, m)) {
d -= n;
if (++m > 12) {
++y;
m = 1;
}
}
}
return fields(y, m, d, hh, mm, ss);
}
CONSTEXPR_F fields n_mon(year_t y, diff_t m, diff_t d, diff_t cd,
hour_t hh, minute_t mm, second_t ss) noexcept {
if (m != 12) {
y += m / 12;
m %= 12;
if (m <= 0) {
y -= 1;
m += 12;
}
}
return n_day(y, m, d, cd, hh, mm, ss);
}
CONSTEXPR_F fields n_hour(year_t y, diff_t m, diff_t d, diff_t cd,
diff_t hh, minute_t mm, second_t ss) noexcept {
cd += hh / 24;
hh %= 24;
if (hh < 0) {
cd -= 1;
hh += 24;
}
return n_mon(y, m, d, cd, hh, mm, ss);
}
CONSTEXPR_F fields n_min(year_t y, diff_t m, diff_t d, diff_t hh, diff_t ch,
diff_t mm, second_t ss) noexcept {
ch += mm / 60;
mm %= 60;
if (mm < 0) {
ch -= 1;
mm += 60;
}
return n_hour(y, m, d, hh / 24 + ch / 24, hh % 24 + ch % 24, mm, ss);
}
CONSTEXPR_F fields n_sec(year_t y, diff_t m, diff_t d, diff_t hh, diff_t mm,
diff_t ss) noexcept {
diff_t cm = ss / 60;
ss %= 60;
if (ss < 0) {
cm -= 1;
ss += 60;
}
return n_min(y, m, d, hh, mm / 60 + cm / 60, mm % 60 + cm % 60, ss);
}
} // namespace impl
////////////////////////////////////////////////////////////////////////
// Increments the indicated (normalized) field by "n".
CONSTEXPR_F fields step(second_tag, fields f, diff_t n) noexcept {
return impl::n_sec(f.y, f.m, f.d, f.hh, f.mm + n / 60, f.ss + n % 60);
}
CONSTEXPR_F fields step(minute_tag, fields f, diff_t n) noexcept {
return impl::n_min(f.y, f.m, f.d, f.hh + n / 60, 0, f.mm + n % 60, f.ss);
}
CONSTEXPR_F fields step(hour_tag, fields f, diff_t n) noexcept {
return impl::n_hour(f.y, f.m, f.d + n / 24, 0, f.hh + n % 24, f.mm, f.ss);
}
CONSTEXPR_F fields step(day_tag, fields f, diff_t n) noexcept {
return impl::n_day(f.y, f.m, f.d, n, f.hh, f.mm, f.ss);
}
CONSTEXPR_F fields step(month_tag, fields f, diff_t n) noexcept {
return impl::n_mon(f.y + n / 12, f.m + n % 12, f.d, 0, f.hh, f.mm, f.ss);
}
CONSTEXPR_F fields step(year_tag, fields f, diff_t n) noexcept {
return fields(f.y + n, f.m, f.d, f.hh, f.mm, f.ss);
}
////////////////////////////////////////////////////////////////////////
namespace impl {
// Returns (v * f + a) but avoiding intermediate overflow when possible.
CONSTEXPR_F diff_t scale_add(diff_t v, diff_t f, diff_t a) noexcept {
return (v < 0) ? ((v + 1) * f + a) - f : ((v - 1) * f + a) + f;
}
// Map a (normalized) Y/M/D to the number of days before/after 1970-01-01.
// Probably overflows for years outside [-292277022656:292277026595].
CONSTEXPR_F diff_t ymd_ord(year_t y, month_t m, day_t d) noexcept {
const diff_t eyear = (m <= 2) ? y - 1 : y;
const diff_t era = (eyear >= 0 ? eyear : eyear - 399) / 400;
const diff_t yoe = eyear - era * 400;
const diff_t doy = (153 * (m + (m > 2 ? -3 : 9)) + 2) / 5 + d - 1;
const diff_t doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
return era * 146097 + doe - 719468;
}
// Returns the difference in days between two normalized Y-M-D tuples.
// ymd_ord() will encounter integer overflow given extreme year values,
// yet the difference between two such extreme values may actually be
// small, so we take a little care to avoid overflow when possible by
// exploiting the 146097-day cycle.
CONSTEXPR_F diff_t day_difference(year_t y1, month_t m1, day_t d1,
year_t y2, month_t m2, day_t d2) noexcept {
const diff_t a_c4_off = y1 % 400;
const diff_t b_c4_off = y2 % 400;
diff_t c4_diff = (y1 - a_c4_off) - (y2 - b_c4_off);
diff_t delta = ymd_ord(a_c4_off, m1, d1) - ymd_ord(b_c4_off, m2, d2);
if (c4_diff > 0 && delta < 0) {
delta += 2 * 146097;
c4_diff -= 2 * 400;
} else if (c4_diff < 0 && delta > 0) {
delta -= 2 * 146097;
c4_diff += 2 * 400;
}
return (c4_diff / 400 * 146097) + delta;
}
} // namespace impl
// Returns the difference between fields structs using the indicated unit.
CONSTEXPR_F diff_t difference(year_tag, fields f1, fields f2) noexcept {
return f1.y - f2.y;
}
CONSTEXPR_F diff_t difference(month_tag, fields f1, fields f2) noexcept {
return impl::scale_add(difference(year_tag{}, f1, f2), 12, (f1.m - f2.m));
}
CONSTEXPR_F diff_t difference(day_tag, fields f1, fields f2) noexcept {
return impl::day_difference(f1.y, f1.m, f1.d, f2.y, f2.m, f2.d);
}
CONSTEXPR_F diff_t difference(hour_tag, fields f1, fields f2) noexcept {
return impl::scale_add(difference(day_tag{}, f1, f2), 24, (f1.hh - f2.hh));
}
CONSTEXPR_F diff_t difference(minute_tag, fields f1, fields f2) noexcept {
return impl::scale_add(difference(hour_tag{}, f1, f2), 60, (f1.mm - f2.mm));
}
CONSTEXPR_F diff_t difference(second_tag, fields f1, fields f2) noexcept {
return impl::scale_add(difference(minute_tag{}, f1, f2), 60, f1.ss - f2.ss);
}
////////////////////////////////////////////////////////////////////////
// Aligns the (normalized) fields struct to the indicated field.
CONSTEXPR_F fields align(second_tag, fields f) noexcept {
return f;
}
CONSTEXPR_F fields align(minute_tag, fields f) noexcept {
return fields{f.y, f.m, f.d, f.hh, f.mm, 0};
}
CONSTEXPR_F fields align(hour_tag, fields f) noexcept {
return fields{f.y, f.m, f.d, f.hh, 0, 0};
}
CONSTEXPR_F fields align(day_tag, fields f) noexcept {
return fields{f.y, f.m, f.d, 0, 0, 0};
}
CONSTEXPR_F fields align(month_tag, fields f) noexcept {
return fields{f.y, f.m, 1, 0, 0, 0};
}
CONSTEXPR_F fields align(year_tag, fields f) noexcept {
return fields{f.y, 1, 1, 0, 0, 0};
}
////////////////////////////////////////////////////////////////////////
template <typename T>
class civil_time {
public:
explicit CONSTEXPR_M civil_time(year_t y, diff_t m = 1, diff_t d = 1,
diff_t hh = 0, diff_t mm = 0,
diff_t ss = 0) noexcept
: civil_time(impl::n_sec(y, m, d, hh, mm, ss)) {}
CONSTEXPR_M civil_time() noexcept : f_{1970, 1, 1, 0, 0, 0} {}
civil_time(const civil_time&) = default;
civil_time& operator=(const civil_time&) = default;
// Conversion between civil times of different alignment. Conversion to
// a more precise alignment is allowed implicitly (e.g., day -> hour),
// but conversion where information is discarded must be explicit
// (e.g., second -> minute).
template <typename U, typename S>
using preserves_data =
typename std::enable_if<std::is_base_of<U, S>::value>::type;
template <typename U>
CONSTEXPR_M civil_time(const civil_time<U>& ct,
preserves_data<T, U>* = nullptr) noexcept
: civil_time(ct.f_) {}
template <typename U>
explicit CONSTEXPR_M civil_time(const civil_time<U>& ct,
preserves_data<U, T>* = nullptr) noexcept
: civil_time(ct.f_) {}
// Factories for the maximum/minimum representable civil_time.
static civil_time max() {
const auto max_year = std::numeric_limits<std::int_least64_t>::max();
return civil_time(max_year, 12, 31, 23, 59, 59);
}
static civil_time min() {
const auto min_year = std::numeric_limits<std::int_least64_t>::min();
return civil_time(min_year, 1, 1, 0, 0, 0);
}
// Field accessors. Note: All but year() return an int.
CONSTEXPR_M year_t year() const noexcept { return f_.y; }
CONSTEXPR_M int month() const noexcept { return f_.m; }
CONSTEXPR_M int day() const noexcept { return f_.d; }
CONSTEXPR_M int hour() const noexcept { return f_.hh; }
CONSTEXPR_M int minute() const noexcept { return f_.mm; }
CONSTEXPR_M int second() const noexcept { return f_.ss; }
// Assigning arithmetic.
CONSTEXPR_M civil_time& operator+=(diff_t n) noexcept {
f_ = step(T{}, f_, n);
return *this;
}
CONSTEXPR_M civil_time& operator-=(diff_t n) noexcept {
if (n != std::numeric_limits<diff_t>::min()) {
f_ = step(T{}, f_, -n);
} else {
f_ = step(T{}, step(T{}, f_, -(n + 1)), 1);
}
return *this;
}
CONSTEXPR_M civil_time& operator++() noexcept {
return *this += 1;
}
CONSTEXPR_M civil_time operator++(int) noexcept {
const civil_time a = *this;
++*this;
return a;
}
CONSTEXPR_M civil_time& operator--() noexcept {
return *this -= 1;
}
CONSTEXPR_M civil_time operator--(int) noexcept {
const civil_time a = *this;
--*this;
return a;
}
// Binary arithmetic operators.
inline friend CONSTEXPR_M civil_time operator+(civil_time a,
diff_t n) noexcept {
return a += n;
}
inline friend CONSTEXPR_M civil_time operator+(diff_t n,
civil_time a) noexcept {
return a += n;
}
inline friend CONSTEXPR_M civil_time operator-(civil_time a,
diff_t n) noexcept {
return a -= n;
}
inline friend CONSTEXPR_M diff_t operator-(const civil_time& lhs,
const civil_time& rhs) noexcept {
return difference(T{}, lhs.f_, rhs.f_);
}
private:
// All instantiations of this template are allowed to call the following
// private constructor and access the private fields member.
template <typename U>
friend class civil_time;
// The designated constructor that all others eventually call.
explicit CONSTEXPR_M civil_time(fields f) noexcept : f_(align(T{}, f)) {}
fields f_;
};
// Disallows difference between differently aligned types.
// auto n = civil_day(...) - civil_hour(...); // would be confusing.
template <typename Tag1, typename Tag2>
CONSTEXPR_F diff_t operator-(civil_time<Tag1>, civil_time<Tag2>) = delete;
using civil_year = civil_time<year_tag>;
using civil_month = civil_time<month_tag>;
using civil_day = civil_time<day_tag>;
using civil_hour = civil_time<hour_tag>;
using civil_minute = civil_time<minute_tag>;
using civil_second = civil_time<second_tag>;
////////////////////////////////////////////////////////////////////////
// Relational operators that work with differently aligned objects.
// Always compares all six fields.
template <typename T1, typename T2>
CONSTEXPR_T bool operator<(const civil_time<T1>& lhs,
const civil_time<T2>& rhs) noexcept {
return (lhs.year() < rhs.year() ||
(lhs.year() == rhs.year() &&
(lhs.month() < rhs.month() ||
(lhs.month() == rhs.month() &&
(lhs.day() < rhs.day() ||
(lhs.day() == rhs.day() &&
(lhs.hour() < rhs.hour() ||
(lhs.hour() == rhs.hour() &&
(lhs.minute() < rhs.minute() ||
(lhs.minute() == rhs.minute() &&
(lhs.second() < rhs.second())))))))))));
}
template <typename T1, typename T2>
CONSTEXPR_T bool operator<=(const civil_time<T1>& lhs,
const civil_time<T2>& rhs) noexcept {
return !(rhs < lhs);
}
template <typename T1, typename T2>
CONSTEXPR_T bool operator>=(const civil_time<T1>& lhs,
const civil_time<T2>& rhs) noexcept {
return !(lhs < rhs);
}
template <typename T1, typename T2>
CONSTEXPR_T bool operator>(const civil_time<T1>& lhs,
const civil_time<T2>& rhs) noexcept {
return rhs < lhs;
}
template <typename T1, typename T2>
CONSTEXPR_T bool operator==(const civil_time<T1>& lhs,
const civil_time<T2>& rhs) noexcept {
return lhs.year() == rhs.year() && lhs.month() == rhs.month() &&
lhs.day() == rhs.day() && lhs.hour() == rhs.hour() &&
lhs.minute() == rhs.minute() && lhs.second() == rhs.second();
}
template <typename T1, typename T2>
CONSTEXPR_T bool operator!=(const civil_time<T1>& lhs,
const civil_time<T2>& rhs) noexcept {
return !(lhs == rhs);
}
////////////////////////////////////////////////////////////////////////
// Output stream operators output a format matching YYYY-MM-DDThh:mm:ss,
// while omitting fields inferior to the type's alignment. For example,
// civil_day is formatted only as YYYY-MM-DD.
inline std::ostream& operator<<(std::ostream& os, const civil_year& y) {
std::stringstream ss;
ss << y.year(); // No padding.
return os << ss.str();
}
inline std::ostream& operator<<(std::ostream& os, const civil_month& m) {
std::stringstream ss;
ss << civil_year(m) << '-';
ss << std::setfill('0') << std::setw(2) << m.month();
return os << ss.str();
}
inline std::ostream& operator<<(std::ostream& os, const civil_day& d) {
std::stringstream ss;
ss << civil_month(d) << '-';
ss << std::setfill('0') << std::setw(2) << d.day();
return os << ss.str();
}
inline std::ostream& operator<<(std::ostream& os, const civil_hour& h) {
std::stringstream ss;
ss << civil_day(h) << 'T';
ss << std::setfill('0') << std::setw(2) << h.hour();
return os << ss.str();
}
inline std::ostream& operator<<(std::ostream& os, const civil_minute& m) {
std::stringstream ss;
ss << civil_hour(m) << ':';
ss << std::setfill('0') << std::setw(2) << m.minute();
return os << ss.str();
}
inline std::ostream& operator<<(std::ostream& os, const civil_second& s) {
std::stringstream ss;
ss << civil_minute(s) << ':';
ss << std::setfill('0') << std::setw(2) << s.second();
return os << ss.str();
}
////////////////////////////////////////////////////////////////////////
enum class weekday {
monday,
tuesday,
wednesday,
thursday,
friday,
saturday,
sunday,
};
inline std::ostream& operator<<(std::ostream& os, weekday wd) {
switch (wd) {
case weekday::monday:
return os << "Monday";
case weekday::tuesday:
return os << "Tuesday";
case weekday::wednesday:
return os << "Wednesday";
case weekday::thursday:
return os << "Thursday";
case weekday::friday:
return os << "Friday";
case weekday::saturday:
return os << "Saturday";
case weekday::sunday:
return os << "Sunday";
}
}
CONSTEXPR_F weekday get_weekday(const civil_day& cd) noexcept {
CONSTEXPR_D weekday k_weekday_by_thu_off[] = {
weekday::thursday, weekday::friday, weekday::saturday,
weekday::sunday, weekday::monday, weekday::tuesday,
weekday::wednesday,
};
return k_weekday_by_thu_off[((cd - civil_day()) % 7 + 7) % 7];
}
////////////////////////////////////////////////////////////////////////
CONSTEXPR_F civil_day next_weekday(civil_day cd, weekday wd) noexcept {
do { cd += 1; } while (get_weekday(cd) != wd);
return cd;
}
CONSTEXPR_F civil_day prev_weekday(civil_day cd, weekday wd) noexcept {
do { cd -= 1; } while (get_weekday(cd) != wd);
return cd;
}
CONSTEXPR_F int get_yearday(const civil_day& cd) noexcept {
return cd - civil_day(civil_year(cd)) + 1;
}
} // namespace detail
} // namespace cctz
#undef CONSTEXPR_T
#undef CONSTEXPR_M
#undef CONSTEXPR_F
#undef CONSTEXPR_D

297
contrib/libcctz/include/time_zone.h
View File

@ -1,297 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// A library for translating between absolute times (represented by
// std::chrono::time_points of the std::chrono::system_clock) and civil
// times (represented by cctz::civil_second) using the rules defined by
// a time zone (cctz::time_zone).
#ifndef CCTZ_TIME_ZONE_H_
#define CCTZ_TIME_ZONE_H_
#include <chrono>
#include <cstdint>
#include <string>
#include "civil_time.h"
namespace cctz {
// Convenience aliases. Not intended as public API points.
template <typename D>
using time_point = std::chrono::time_point<std::chrono::system_clock, D>;
using sys_seconds = std::chrono::duration<std::int_fast64_t>;
// cctz::time_zone is an opaque, small, value-type class representing a
// geo-political region within which particular rules are used for mapping
// between absolute and civil times. Time zones are named using the TZ
// identifiers from the IANA Time Zone Database, such as "America/Los_Angeles"
// or "Australia/Sydney". Time zones are created from factory functions such
// as load_time_zone(). Note: strings like "PST" and "EDT" are not valid TZ
// identifiers.
//
// Example:
// cctz::time_zone utc = cctz::utc_time_zone();
// cctz::time_zone loc = cctz::local_time_zone();
// cctz::time_zone lax;
// if (!cctz::load_time_zone("America/Los_Angeles", &lax)) { ... }
//
// See also:
// - http://www.iana.org/time-zones
// - http://en.wikipedia.org/wiki/Zoneinfo
class time_zone {
public:
time_zone() = default; // Equivalent to UTC
time_zone(const time_zone&) = default;
time_zone& operator=(const time_zone&) = default;
std::string name() const;
// An absolute_lookup represents the civil time (cctz::civil_second) within
// this time_zone at the given absolute time (time_point). There are
// additionally a few other fields that may be useful when working with
// older APIs, such as std::tm.
//
// Example:
// const cctz::time_zone tz = ...
// const auto tp = std::chrono::system_clock::now();
// const cctz::time_zone::absolute_lookup al = tz.lookup(tp);
struct absolute_lookup {
civil_second cs;
// Note: The following fields exist for backward compatibility with older
// APIs. Accessing these fields directly is a sign of imprudent logic in
// the calling code. Modern time-related code should only access this data
// indirectly by way of cctz::format().
int offset; // civil seconds east of UTC
bool is_dst; // is offset non-standard?
std::string abbr; // time-zone abbreviation (e.g., "PST")
};
absolute_lookup lookup(const time_point<sys_seconds>& tp) const;
template <typename D>
absolute_lookup lookup(const time_point<D>& tp) const {
return lookup(std::chrono::time_point_cast<sys_seconds>(tp));
}
// A civil_lookup represents the absolute time(s) (time_point) that
// correspond to the given civil time (cctz::civil_second) within this
// time_zone. Usually the given civil time represents a unique instant in
// time, in which case the conversion is unambiguous and correct. However,
// within this time zone, the given civil time may be skipped (e.g., during
// a positive UTC offset shift), or repeated (e.g., during a negative UTC
// offset shift). To account for these possibilities, civil_lookup is richer
// than just a single output time_point.
//
// In all cases the civil_lookup::kind enum will indicate the nature of the
// given civil-time argument, and the pre, trans, and post, members will
// give the absolute time answers using the pre-transition offset, the
// transition point itself, and the post-transition offset, respectively
// (these are all equal if kind == UNIQUE).
//
// Example:
// cctz::time_zone lax;
// if (!cctz::load_time_zone("America/Los_Angeles", &lax)) { ... }
//
// // A unique civil time.
// auto jan01 = lax.lookup(cctz::civil_second(2011, 1, 1, 0, 0, 0));
// // jan01.kind == cctz::time_zone::civil_lookup::UNIQUE
// // jan01.pre is 2011/01/01 00:00:00 -0800
// // jan01.trans is 2011/01/01 00:00:00 -0800
// // jan01.post is 2011/01/01 00:00:00 -0800
//
// // A Spring DST transition, when there is a gap in civil time.
// auto mar13 = lax.lookup(cctz::civil_second(2011, 3, 13, 2, 15, 0));
// // mar13.kind == cctz::time_zone::civil_lookup::SKIPPED
// // mar13.pre is 2011/03/13 03:15:00 -0700
// // mar13.trans is 2011/03/13 03:00:00 -0700
// // mar13.post is 2011/03/13 01:15:00 -0800
//
// // A Fall DST transition, when civil times are repeated.
// auto nov06 = lax.lookup(cctz::civil_second(2011, 11, 6, 1, 15, 0));
// // nov06.kind == cctz::time_zone::civil_lookup::REPEATED
// // nov06.pre is 2011/11/06 01:15:00 -0700
// // nov06.trans is 2011/11/06 01:00:00 -0800
// // nov06.post is 2011/11/06 01:15:00 -0800
struct civil_lookup {
enum civil_kind {
UNIQUE, // the civil time was singular (pre == trans == post)
SKIPPED, // the civil time did not exist
REPEATED, // the civil time was ambiguous
} kind;
time_point<sys_seconds> pre; // Uses the pre-transition offset
time_point<sys_seconds> trans; // Instant of civil-offset change
time_point<sys_seconds> post; // Uses the post-transition offset
};
civil_lookup lookup(const civil_second& cs) const;
class Impl;
private:
explicit time_zone(const Impl* impl) : impl_(impl) {}
const Impl* impl_ = nullptr;
};
// Relational operators.
bool operator==(time_zone lhs, time_zone rhs);
inline bool operator!=(time_zone lhs, time_zone rhs) { return !(lhs == rhs); }
// Loads the named time zone. May perform I/O on the initial load.
// If the name is invalid, or some other kind of error occurs, returns
// false and "*tz" is set to the UTC time zone.
bool load_time_zone(const std::string& name, time_zone* tz);
// Returns a time_zone representing UTC. Cannot fail.
time_zone utc_time_zone();
// Returns a time zone representing the local time zone. Falls back to UTC.
time_zone local_time_zone();
// Returns the civil time (cctz::civil_second) within the given time zone at
// the given absolute time (time_point). Since the additional fields provided
// by the time_zone::absolute_lookup struct should rarely be needed in modern
// code, this convert() function is simpler and should be preferred.
template <typename D>
inline civil_second convert(const time_point<D>& tp, const time_zone& tz) {
return tz.lookup(tp).cs;
}
// Returns the absolute time (time_point) that corresponds to the given civil
// time within the given time zone. If the civil time is not unique (i.e., if
// it was either repeated or non-existent), then the returned time_point is
// the best estimate that preserves relative order. That is, this function
// guarantees that if cs1 < cs2, then convert(cs1, tz) <= convert(cs2, tz).
inline time_point<sys_seconds> convert(const civil_second& cs,
const time_zone& tz) {
const time_zone::civil_lookup cl = tz.lookup(cs);
if (cl.kind == time_zone::civil_lookup::SKIPPED) return cl.trans;
return cl.pre;
}
namespace detail {
template <typename D>
inline std::pair<time_point<sys_seconds>, D>
split_seconds(const time_point<D>& tp) {
auto sec = std::chrono::time_point_cast<sys_seconds>(tp);
auto sub = tp - sec;
if (sub.count() < 0) {
sec -= sys_seconds(1);
sub += sys_seconds(1);
}
return {sec, std::chrono::duration_cast<D>(sub)};
}
inline std::pair<time_point<sys_seconds>, sys_seconds>
split_seconds(const time_point<sys_seconds>& tp) {
return {tp, sys_seconds(0)};
}
using femtoseconds = std::chrono::duration<std::int_fast64_t, std::femto>;
std::string format(const std::string&, const time_point<sys_seconds>&,
const femtoseconds&, const time_zone&);
bool parse(const std::string&, const std::string&, const time_zone&,
time_point<sys_seconds>*, femtoseconds*);
} // namespace detail
// Formats the given time_point in the given cctz::time_zone according to
// the provided format string. Uses strftime()-like formatting options,
// with the following extensions:
//
// - %Ez - RFC3339-compatible numeric time zone (+hh:mm or -hh:mm)
// - %E#S - Seconds with # digits of fractional precision
// - %E*S - Seconds with full fractional precision (a literal '*')
// - %E#f - Fractional seconds with # digits of precision
// - %E*f - Fractional seconds with full precision (a literal '*')
// - %E4Y - Four-character years (-999 ... -001, 0000, 0001 ... 9999)
//
// Note that %E0S behaves like %S, and %E0f produces no characters. In
// contrast %E*f always produces at least one digit, which may be '0'.
//
// Note that %Y produces as many characters as it takes to fully render the
// year. A year outside of [-999:9999] when formatted with %E4Y will produce
// more than four characters, just like %Y.
//
// Tip: Format strings should include the UTC offset (e.g., %z or %Ez) so that
// the resultng string uniquely identifies an absolute time.
//
// Example:
// cctz::time_zone lax;
// if (!cctz::load_time_zone("America/Los_Angeles", &lax)) { ... }
// auto tp = cctz::convert(cctz::civil_second(2013, 1, 2, 3, 4, 5), lax);
// std::string f = cctz::format("%H:%M:%S", tp, lax); // "03:04:05"
// f = cctz::format("%H:%M:%E3S", tp, lax); // "03:04:05.000"
template <typename D>
inline std::string format(const std::string& fmt, const time_point<D>& tp,
const time_zone& tz) {
const auto p = detail::split_seconds(tp);
const auto n = std::chrono::duration_cast<detail::femtoseconds>(p.second);
return detail::format(fmt, p.first, n, tz);
}
// Parses an input string according to the provided format string and
// returns the corresponding time_point. Uses strftime()-like formatting
// options, with the same extensions as cctz::format(), but with the
// exceptions that %E#S is interpreted as %E*S, and %E#f as %E*f.
//
// %Y consumes as many numeric characters as it can, so the matching data
// should always be terminated with a non-numeric. %E4Y always consumes
// exactly four characters, including any sign.
//
// Unspecified fields are taken from the default date and time of ...
//
// "1970-01-01 00:00:00.0 +0000"
//
// For example, parsing a string of "15:45" (%H:%M) will return a time_point
// that represents "1970-01-01 15:45:00.0 +0000".
//
// Note that parse() returns time instants, so it makes most sense to parse
// fully-specified date/time strings that include a UTC offset (%z or %Ez).
//
// Note also that parse() only heeds the fields year, month, day, hour,
// minute, (fractional) second, and UTC offset. Other fields, like weekday (%a
// or %A), while parsed for syntactic validity, are ignored in the conversion.
//
// Date and time fields that are out-of-range will be treated as errors rather
// than normalizing them like cctz::civil_second() would do. For example, it
// is an error to parse the date "Oct 32, 2013" because 32 is out of range.
//
// A second of ":60" is normalized to ":00" of the following minute with
// fractional seconds discarded. The following table shows how the given
// seconds and subseconds will be parsed:
//
// "59.x" -> 59.x // exact
// "60.x" -> 00.0 // normalized
// "00.x" -> 00.x // exact
//
// Errors are indicated by returning false.
//
// Example:
// const cctz::time_zone tz = ...
// std::chrono::system_clock::time_point tp;
// if (cctz::parse("%Y-%m-%d", "2015-10-09", tz, &tp)) {
// ...
// }
template <typename D>
inline bool parse(const std::string& fmt, const std::string& input,
const time_zone& tz, time_point<D>* tpp) {
time_point<sys_seconds> sec;
detail::femtoseconds fs;
const bool b = detail::parse(fmt, input, tz, &sec, &fs);
if (b) {
// TODO: Return false if unrepresentable as a time_point<D>.
*tpp = std::chrono::time_point_cast<D>(sec);
*tpp += std::chrono::duration_cast<D>(fs);
}
return b;
}
} // namespace cctz
#endif // CCTZ_TIME_ZONE_H_

260
contrib/libcctz/src/time_tool.cc
View File

@ -1,260 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// A command-line tool for exercising the CCTZ library.
#include <getopt.h>
#include <algorithm>
#include <cctype>
#include <chrono>
#include <cstring>
#include <ctime>
#include <iomanip>
#include <iostream>
#include <sstream>
#include <string>
#include "civil_time.h"
#include "time_zone.h"
// Pulls in the aliases from cctz for brevity.
template <typename D>
using time_point = cctz::time_point<D>;
using sys_seconds = cctz::sys_seconds;
// parse() specifiers for command-line time arguments.
const char* const kFormats[] = {
"%Y %m %d %H %M %E*S",
"%Y - %m - %d T %H : %M : %E*S",
"%Y - %m - %d %H : %M : %E*S",
"%Y - %m - %d T %H : %M",
"%Y - %m - %d %H : %M",
"%Y - %m - %d",
"%a %b %d %H : %M : %E*S %Z %Y",
"%a %e %b %Y %H : %M : %E*S",
"%a %b %e %Y %H : %M : %E*S",
"%e %b %Y %H : %M : %E*S",
"%b %e %Y %H : %M : %E*S",
"%a %e %b %Y %H : %M",
"%a %b %e %Y %H : %M",
"%e %b %Y %H : %M",
"%b %e %Y %H : %M",
"%a %e %b %Y",
"%a %b %e %Y",
"%e %b %Y",
"%b %e %Y",
nullptr
};
bool ParseTimeSpec(const std::string& args, cctz::time_zone zone,
time_point<sys_seconds>* when) {
for (const char* const* fmt = kFormats; *fmt != NULL; ++fmt) {
const std::string format = std::string(*fmt) + " %Ez";
time_point<sys_seconds> tp;
if (cctz::parse(format, args, zone, &tp)) {
*when = tp;
return true;
}
}
return false;
}
bool ParseBreakdownSpec(const std::string& args, cctz::civil_second* when) {
const cctz::time_zone utc = cctz::utc_time_zone();
for (const char* const* fmt = kFormats; *fmt != NULL; ++fmt) {
time_point<sys_seconds> tp;
if (cctz::parse(*fmt, args, utc, &tp)) {
*when = cctz::convert(tp, utc);
return true;
}
}
return false;
}
// The FormatTime() specifier for output.
const char* const kFormat = "%Y-%m-%d %H:%M:%S %Ez (%Z)";
const char* WeekDayName(cctz::weekday wd) {
switch (wd) {
case cctz::weekday::monday: return "Mon";
case cctz::weekday::tuesday: return "Tue";
case cctz::weekday::wednesday: return "Wed";
case cctz::weekday::thursday: return "Thu";
case cctz::weekday::friday: return "Fri";
case cctz::weekday::saturday: return "Sat";
case cctz::weekday::sunday: return "Sun";
}
return "XXX";
}
std::string FormatTimeInZone(time_point<sys_seconds> when,
cctz::time_zone zone) {
std::ostringstream oss;
oss << std::setw(33) << std::left << cctz::format(kFormat, when, zone);
cctz::time_zone::absolute_lookup al = zone.lookup(when);
cctz::civil_day cd(al.cs);
oss << " [wd=" << WeekDayName(cctz::get_weekday(cd))
<< " yd=" << std::setw(3) << std::setfill('0')
<< std::right << cctz::get_yearday(cd)
<< " dst=" << (al.is_dst ? 'T' : 'F')
<< " off=" << std::showpos << al.offset << std::noshowpos << "]";
return oss.str();
}
void InstantInfo(const std::string& label, time_point<sys_seconds> when,
cctz::time_zone zone) {
const cctz::time_zone utc = cctz::utc_time_zone(); // might == zone
const std::string time_label = "time_t";
const std::string utc_label = "UTC";
const std::string zone_label = "in-tz"; // perhaps zone.name()?
std::size_t width =
2 + std::max(std::max(time_label.size(), utc_label.size()),
zone_label.size());
std::cout << label << " {\n";
std::cout << std::setw(width) << std::right << time_label << ": ";
std::cout << std::setw(10) << format("%s", when, utc);
std::cout << "\n";
std::cout << std::setw(width) << std::right << utc_label << ": ";
std::cout << FormatTimeInZone(when, utc) << "\n";
std::cout << std::setw(width) << std::right << zone_label << ": ";
std::cout << FormatTimeInZone(when, zone) << "\n";
std::cout << "}\n";
}
// Report everything we know about a cctz::civil_second (YMDHMS).
int BreakdownInfo(const cctz::civil_second& cs, cctz::time_zone zone) {
std::cout << "tz: " << zone.name() << "\n";
cctz::time_zone::civil_lookup cl = zone.lookup(cs);
switch (cl.kind) {
case cctz::time_zone::civil_lookup::UNIQUE: {
std::cout << "kind: UNIQUE\n";
InstantInfo("when", cl.pre, zone);
break;
}
case cctz::time_zone::civil_lookup::SKIPPED: {
std::cout << "kind: SKIPPED\n";
InstantInfo("post", cl.post, zone); // might == trans-1
InstantInfo("trans-1", cl.trans - std::chrono::seconds(1), zone);
InstantInfo("trans", cl.trans, zone);
InstantInfo("pre", cl.pre, zone); // might == trans
break;
}
case cctz::time_zone::civil_lookup::REPEATED: {
std::cout << "kind: REPEATED\n";
InstantInfo("pre", cl.pre, zone); // might == trans-1
InstantInfo("trans-1", cl.trans - std::chrono::seconds(1), zone);
InstantInfo("trans", cl.trans, zone);
InstantInfo("post", cl.post, zone); // might == trans
break;
}
}
return 0;
}
// Report everything we know about a time_point<sys_seconds>.
int TimeInfo(time_point<sys_seconds> when, cctz::time_zone zone) {
std::cout << "tz: " << zone.name() << "\n";
std::cout << "kind: UNIQUE\n";
InstantInfo("when", when, zone);
return 0;
}
const char* Basename(const char* p) {
if (const char* b = strrchr(p, '/')) return ++b;
return p;
}
// std::regex doesn't work before gcc 4.9.
bool LooksLikeNegOffset(const char* s) {
if (s[0] == '-' && std::isdigit(s[1]) && std::isdigit(s[2])) {
int i = (s[3] == ':') ? 4 : 3;
if (std::isdigit(s[i]) && std::isdigit(s[i + 1])) {
return s[i + 2] == '\0';
}
}
return false;
}
int main(int argc, char** argv) {
const std::string prog = argv[0] ? Basename(argv[0]) : "time_tool";
// Escape arguments that look like negative offsets so that they
// don't look like flags.
for (int i = 1; i < argc; ++i) {
if (strcmp(argv[i], "--") == 0) break;
if (LooksLikeNegOffset(argv[i])) {
char* buf = new char[strlen(argv[i] + 2)];
buf[0] = ' '; // will later be ignorned
strcpy(buf + 1, argv[i]);
argv[i] = buf;
}
}
// Determine the time zone.
cctz::time_zone zone = cctz::local_time_zone();
for (;;) {
static option opts[] = {
{"tz", required_argument, nullptr, 'z'},
{nullptr, 0, nullptr, 0},
};
int c = getopt_long(argc, argv, "z:", opts, nullptr);
if (c == -1) break;
switch (c) {
case 'z':
if (!cctz::load_time_zone(optarg, &zone)) {
std::cerr << optarg << ": Unrecognized time zone\n";
return 1;
}
break;
default:
std::cerr << "Usage: " << prog << " [--tz=<zone>] [<time-spec>]\n";
return 1;
}
}
// Determine the time point.
time_point<sys_seconds> tp = std::chrono::time_point_cast<sys_seconds>(
std::chrono::system_clock::now());
std::string args;
for (int i = optind; i < argc; ++i) {
if (i != optind) args += " ";
args += argv[i];
}
std::replace(args.begin(), args.end(), ',', ' ');
std::replace(args.begin(), args.end(), '/', '-');
bool have_time = ParseTimeSpec(args, zone, &tp);
if (!have_time && !args.empty()) {
std::string spec = args.substr((args[0] == '@') ? 1 : 0);
if ((spec.size() > 0 && std::isdigit(spec[0])) ||
(spec.size() > 1 && spec[0] == '-' && std::isdigit(spec[1]))) {
std::size_t end;
const time_t t = std::stoll(spec, &end);
if (end == spec.size()) {
tp = std::chrono::time_point_cast<cctz::sys_seconds>(
std::chrono::system_clock::from_time_t(0)) +
sys_seconds(t);
have_time = true;
}
}
}
cctz::civil_second when = cctz::convert(tp, zone);
bool have_break_down = !have_time && ParseBreakdownSpec(args, &when);
// Show results.
if (have_break_down) return BreakdownInfo(when, zone);
if (have_time || args.empty()) return TimeInfo(tp, zone);
std::cerr << args << ": Malformed time spec\n";
return 1;
}

773
contrib/libcctz/src/time_zone_format.cc
View File

@ -1,773 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#if !defined(HAS_STRPTIME)
# if !defined(_MSC_VER)
# define HAS_STRPTIME 1 // assume everyone has strptime() except windows
# endif
#endif
#include "time_zone.h"
#include "time_zone_if.h"
#include <cctype>
#include <chrono>
#include <cstdint>
#include <cstring>
#include <ctime>
#include <limits>
#include <vector>
#if !HAS_STRPTIME
#include <iomanip>
#include <sstream>
#endif
namespace cctz {
namespace detail {
namespace {
#if !HAS_STRPTIME
// Build a strptime() using C++11's std::get_time().
char* strptime(const char* s, const char* fmt, std::tm* tm) {
std::istringstream input(s);
input >> std::get_time(tm, fmt);
if (input.fail()) return nullptr;
return const_cast<char*>(s) + input.tellg();
}
#endif
std::tm ToTM(const time_zone::absolute_lookup& al) {
std::tm tm{};
tm.tm_sec = al.cs.second();
tm.tm_min = al.cs.minute();
tm.tm_hour = al.cs.hour();
tm.tm_mday = al.cs.day();
tm.tm_mon = al.cs.month() - 1;
// Saturate tm.tm_year is cases of over/underflow.
if (al.cs.year() < std::numeric_limits<int>::min() + 1900) {
tm.tm_year = std::numeric_limits<int>::min();
} else if (al.cs.year() - 1900 > std::numeric_limits<int>::max()) {
tm.tm_year = std::numeric_limits<int>::max();
} else {
tm.tm_year = static_cast<int>(al.cs.year() - 1900);
}
switch (get_weekday(civil_day(al.cs))) {
case weekday::sunday:
tm.tm_wday = 0;
break;
case weekday::monday:
tm.tm_wday = 1;
break;
case weekday::tuesday:
tm.tm_wday = 2;
break;
case weekday::wednesday:
tm.tm_wday = 3;
break;
case weekday::thursday:
tm.tm_wday = 4;
break;
case weekday::friday:
tm.tm_wday = 5;
break;
case weekday::saturday:
tm.tm_wday = 6;
break;
}
tm.tm_yday = get_yearday(civil_day(al.cs)) - 1;
tm.tm_isdst = al.is_dst ? 1 : 0;
return tm;
}
const char kDigits[] = "0123456789";
// Formats a 64-bit integer in the given field width. Note that it is up
// to the caller of Format64() [and Format02d()/FormatOffset()] to ensure
// that there is sufficient space before ep to hold the conversion.
char* Format64(char* ep, int width, std::int_fast64_t v) {
bool neg = false;
if (v < 0) {
--width;
neg = true;
if (v == INT64_MIN) {
// Avoid negating INT64_MIN.
int last_digit = -(v % 10);
v /= 10;
if (last_digit < 0) {
++v;
last_digit += 10;
}
--width;
*--ep = kDigits[last_digit];
}
v = -v;
}
do {
--width;
*--ep = kDigits[v % 10];
} while (v /= 10);
while (--width >= 0) *--ep = '0'; // zero pad
if (neg) *--ep = '-';
return ep;
}
// Formats [0 .. 99] as %02d.
char* Format02d(char* ep, int v) {
*--ep = kDigits[v % 10];
*--ep = kDigits[(v / 10) % 10];
return ep;
}
// Formats a UTC offset, like +00:00.
char* FormatOffset(char* ep, int minutes, char sep) {
char sign = '+';
if (minutes < 0) {
minutes = -minutes;
sign = '-';
}
ep = Format02d(ep, minutes % 60);
if (sep != '\0') *--ep = sep;
ep = Format02d(ep, minutes / 60);
*--ep = sign;
return ep;
}
// Formats a std::tm using strftime(3).
void FormatTM(std::string* out, const std::string& fmt, const std::tm& tm) {
// strftime(3) returns the number of characters placed in the output
// array (which may be 0 characters). It also returns 0 to indicate
// an error, like the array wasn't large enough. To accomodate this,
// the following code grows the buffer size from 2x the format string
// length up to 32x.
for (int i = 2; i != 32; i *= 2) {
std::size_t buf_size = fmt.size() * i;
std::vector<char> buf(buf_size);
if (std::size_t len = strftime(&buf[0], buf_size, fmt.c_str(), &tm)) {
out->append(&buf[0], len);
return;
}
}
}
// Used for %E#S/%E#f specifiers and for data values in parse().
template <typename T>
const char* ParseInt(const char* dp, int width, T min, T max, T* vp) {
if (dp != nullptr) {
const T kmin = std::numeric_limits<T>::min();
bool erange = false;
bool neg = false;
T value = 0;
if (*dp == '-') {
neg = true;
if (width <= 0 || --width != 0) {
++dp;
} else {
dp = nullptr; // width was 1
}
}
if (const char* const bp = dp) {
while (const char* cp = strchr(kDigits, *dp)) {
int d = static_cast<int>(cp - kDigits);
if (d >= 10) break;
if (value < kmin / 10) {
erange = true;
break;
}
value *= 10;
if (value < kmin + d) {
erange = true;
break;
}
value -= d;
dp += 1;
if (width > 0 && --width == 0) break;
}
if (dp != bp && !erange && (neg || value != kmin)) {
if (!neg || value != 0) {
if (!neg) value = -value; // make positive
if (min <= value && value <= max) {
*vp = value;
} else {
dp = nullptr;
}
} else {
dp = nullptr;
}
} else {
dp = nullptr;
}
}
}
return dp;
}
// The number of base-10 digits that can be represented by a signed 64-bit
// integer. That is, 10^kDigits10_64 <= 2^63 - 1 < 10^(kDigits10_64 + 1).
const int kDigits10_64 = 18;
// 10^n for everything that can be represented by a signed 64-bit integer.
const std::int_fast64_t kExp10[kDigits10_64 + 1] = {
1,
10,
100,
1000,
10000,
100000,
1000000,
10000000,
100000000,
1000000000,
10000000000,
100000000000,
1000000000000,
10000000000000,
100000000000000,
1000000000000000,
10000000000000000,
100000000000000000,
1000000000000000000,
};
} // namespace
// Uses strftime(3) to format the given Time. The following extended format
// specifiers are also supported:
//
// - %Ez - RFC3339-compatible numeric timezone (+hh:mm or -hh:mm)
// - %E#S - Seconds with # digits of fractional precision
// - %E*S - Seconds with full fractional precision (a literal '*')
// - %E4Y - Four-character years (-999 ... -001, 0000, 0001 ... 9999)
//
// The standard specifiers from RFC3339_* (%Y, %m, %d, %H, %M, and %S) are
// handled internally for performance reasons. strftime(3) is slow due to
// a POSIX requirement to respect changes to ${TZ}.
//
// The TZ/GNU %s extension is handled internally because strftime() has
// to use mktime() to generate it, and that assumes the local time zone.
//
// We also handle the %z and %Z specifiers to accommodate platforms that do
// not support the tm_gmtoff and tm_zone extensions to std::tm.
//
// Requires that zero() <= fs < seconds(1).
std::string format(const std::string& format, const time_point<sys_seconds>& tp,
const detail::femtoseconds& fs, const time_zone& tz) {
std::string result;
const time_zone::absolute_lookup al = tz.lookup(tp);
const std::tm tm = ToTM(al);
// Scratch buffer for internal conversions.
char buf[3 + kDigits10_64]; // enough for longest conversion
char* const ep = buf + sizeof(buf);
char* bp; // works back from ep
// Maintain three, disjoint subsequences that span format.
// [format.begin() ... pending) : already formatted into result
// [pending ... cur) : formatting pending, but no special cases
// [cur ... format.end()) : unexamined
// Initially, everything is in the unexamined part.
const char* pending = format.c_str(); // NUL terminated
const char* cur = pending;
const char* end = pending + format.length();
while (cur != end) { // while something is unexamined
// Moves cur to the next percent sign.
const char* start = cur;
while (cur != end && *cur != '%') ++cur;
// If the new pending text is all ordinary, copy it out.
if (cur != start && pending == start) {
result.append(pending, cur - pending);
pending = start = cur;
}
// Span the sequential percent signs.
const char* percent = cur;
while (cur != end && *cur == '%') ++cur;
// If the new pending text is all percents, copy out one
// percent for every matched pair, then skip those pairs.
if (cur != start && pending == start) {
std::size_t escaped = (cur - pending) / 2;
result.append(pending, escaped);
pending += escaped * 2;
// Also copy out a single trailing percent.
if (pending != cur && cur == end) {
result.push_back(*pending++);
}
}
// Loop unless we have an unescaped percent.
if (cur == end || (cur - percent) % 2 == 0) continue;
// Simple specifiers that we handle ourselves.
if (strchr("YmdeHMSzZs", *cur)) {
if (cur - 1 != pending) {
FormatTM(&result, std::string(pending, cur - 1), tm);
}
switch (*cur) {
case 'Y':
// This avoids the tm.tm_year overflow problem for %Y, however
// tm.tm_year will still be used by other specifiers like %D.
bp = Format64(ep, 0, al.cs.year());
result.append(bp, ep - bp);
break;
case 'm':
bp = Format02d(ep, al.cs.month());
result.append(bp, ep - bp);
break;
case 'd':
case 'e':
bp = Format02d(ep, al.cs.day());
if (*cur == 'e' && *bp == '0') *bp = ' '; // for Windows
result.append(bp, ep - bp);
break;
case 'H':
bp = Format02d(ep, al.cs.hour());
result.append(bp, ep - bp);
break;
case 'M':
bp = Format02d(ep, al.cs.minute());
result.append(bp, ep - bp);
break;
case 'S':
bp = Format02d(ep, al.cs.second());
result.append(bp, ep - bp);
break;
case 'z':
bp = FormatOffset(ep, al.offset / 60, '\0');
result.append(bp, ep - bp);
break;
case 'Z':
result.append(al.abbr);
break;
case 's':
bp = Format64(ep, 0, ToUnixSeconds(tp));
result.append(bp, ep - bp);
break;
}
pending = ++cur;
continue;
}
// Loop if there is no E modifier.
if (*cur != 'E' || ++cur == end) continue;
// Format our extensions.
if (*cur == 'z') {
// Formats %Ez.
if (cur - 2 != pending) {
FormatTM(&result, std::string(pending, cur - 2), tm);
}
bp = FormatOffset(ep, al.offset / 60, ':');
result.append(bp, ep - bp);
pending = ++cur;
} else if (*cur == '*' && cur + 1 != end &&
(*(cur + 1) == 'S' || *(cur + 1) == 'f')) {
// Formats %E*S or %E*F.
if (cur - 2 != pending) {
FormatTM(&result, std::string(pending, cur - 2), tm);
}
char* cp = ep;
bp = Format64(cp, 15, fs.count());
while (cp != bp && cp[-1] == '0') --cp;
switch (*(cur + 1)) {
case 'S':
if (cp != bp) *--bp = '.';
bp = Format02d(bp, al.cs.second());
break;
case 'f':
if (cp == bp) *--bp = '0';
break;
}
result.append(bp, cp - bp);
pending = cur += 2;
} else if (*cur == '4' && cur + 1 != end && *(cur + 1) == 'Y') {
// Formats %E4Y.
if (cur - 2 != pending) {
FormatTM(&result, std::string(pending, cur - 2), tm);
}
bp = Format64(ep, 4, al.cs.year());
result.append(bp, ep - bp);
pending = cur += 2;
} else if (std::isdigit(*cur)) {
// Possibly found %E#S or %E#f.
int n = 0;
if (const char* np = ParseInt(cur, 0, 0, 1024, &n)) {
if (*np == 'S' || *np == 'f') {
// Formats %E#S or %E#f.
if (cur - 2 != pending) {
FormatTM(&result, std::string(pending, cur - 2), tm);
}
bp = ep;
if (n > 0) {
if (n > kDigits10_64) n = kDigits10_64;
bp = Format64(bp, n, (n > 15) ? fs.count() * kExp10[n - 15]
: fs.count() / kExp10[15 - n]);
if (*np == 'S') *--bp = '.';
}
if (*np == 'S') bp = Format02d(bp, al.cs.second());
result.append(bp, ep - bp);
pending = cur = ++np;
}
}
}
}
// Formats any remaining data.
if (end != pending) {
FormatTM(&result, std::string(pending, end), tm);
}
return result;
}
namespace {
const char* ParseOffset(const char* dp, char sep, int* offset) {
if (dp != nullptr) {
const char sign = *dp++;
if (sign == '+' || sign == '-') {
int hours = 0;
const char* ap = ParseInt(dp, 2, 0, 23, &hours);
if (ap != nullptr && ap - dp == 2) {
dp = ap;
if (sep != '\0' && *ap == sep) ++ap;
int minutes = 0;
const char* bp = ParseInt(ap, 2, 0, 59, &minutes);
if (bp != nullptr && bp - ap == 2) dp = bp;
*offset = (hours * 60 + minutes) * 60;
if (sign == '-') *offset = -*offset;
} else {
dp = nullptr;
}
} else {
dp = nullptr;
}
}
return dp;
}
const char* ParseZone(const char* dp, std::string* zone) {
zone->clear();
if (dp != nullptr) {
while (*dp != '\0' && !std::isspace(*dp)) zone->push_back(*dp++);
if (zone->empty()) dp = nullptr;
}
return dp;
}
const char* ParseSubSeconds(const char* dp, detail::femtoseconds* subseconds) {
if (dp != nullptr) {
std::int_fast64_t v = 0;
std::int_fast64_t exp = 0;
const char* const bp = dp;
while (const char* cp = strchr(kDigits, *dp)) {
int d = static_cast<int>(cp - kDigits);
if (d >= 10) break;
if (exp < 15) {
exp += 1;
v *= 10;
v += d;
}
++dp;
}
if (dp != bp) {
v *= kExp10[15 - exp];
*subseconds = detail::femtoseconds(v);
} else {
dp = nullptr;
}
}
return dp;
}
// Parses a string into a std::tm using strptime(3).
const char* ParseTM(const char* dp, const char* fmt, std::tm* tm) {
if (dp != nullptr) {
dp = strptime(dp, fmt, tm);
}
return dp;
}
} // namespace
// Uses strptime(3) to parse the given input. Supports the same extended
// format specifiers as format(), although %E#S and %E*S are treated
// identically (and similarly for %E#f and %E*f).
//
// The standard specifiers from RFC3339_* (%Y, %m, %d, %H, %M, and %S) are
// handled internally so that we can normally avoid strptime() altogether
// (which is particularly helpful when the native implementation is broken).
//
// The TZ/GNU %s extension is handled internally because strptime() has to
// use localtime_r() to generate it, and that assumes the local time zone.
//
// We also handle the %z specifier to accommodate platforms that do not
// support the tm_gmtoff extension to std::tm. %Z is parsed but ignored.
bool parse(const std::string& format, const std::string& input,
const time_zone& tz, time_point<sys_seconds>* sec,
detail::femtoseconds* fs) {
// The unparsed input.
const char* data = input.c_str(); // NUL terminated
// Skips leading whitespace.
while (std::isspace(*data)) ++data;
const year_t kyearmax = std::numeric_limits<year_t>::max();
const year_t kyearmin = std::numeric_limits<year_t>::min();
// Sets default values for unspecified fields.
bool saw_year = false;
year_t year = 1970;
std::tm tm{};
tm.tm_year = 1970 - 1900;
tm.tm_mon = 1 - 1; // Jan
tm.tm_mday = 1;
tm.tm_hour = 0;
tm.tm_min = 0;
tm.tm_sec = 0;
tm.tm_wday = 4; // Thu
tm.tm_yday = 0;
tm.tm_isdst = 0;
auto subseconds = detail::femtoseconds::zero();
bool saw_offset = false;
int offset = 0; // No offset from passed tz.
std::string zone = "UTC";
const char* fmt = format.c_str(); // NUL terminated
bool twelve_hour = false;
bool afternoon = false;
bool saw_percent_s = false;
std::int_fast64_t percent_s = 0;
// Steps through format, one specifier at a time.
while (data != nullptr && *fmt != '\0') {
if (std::isspace(*fmt)) {
while (std::isspace(*data)) ++data;
while (std::isspace(*++fmt)) continue;
continue;
}
if (*fmt != '%') {
if (*data == *fmt) {
++data;
++fmt;
} else {
data = nullptr;
}
continue;
}
const char* percent = fmt;
if (*++fmt == '\0') {
data = nullptr;
continue;
}
switch (*fmt++) {
case 'Y':
// Symmetrically with FormatTime(), directly handing %Y avoids the
// tm.tm_year overflow problem. However, tm.tm_year will still be
// used by other specifiers like %D.
data = ParseInt(data, 0, kyearmin, kyearmax, &year);
if (data != nullptr) saw_year = true;
continue;
case 'm':
data = ParseInt(data, 2, 1, 12, &tm.tm_mon);
if (data != nullptr) tm.tm_mon -= 1;
continue;
case 'd':
data = ParseInt(data, 2, 1, 31, &tm.tm_mday);
continue;
case 'H':
data = ParseInt(data, 2, 0, 23, &tm.tm_hour);
twelve_hour = false;
continue;
case 'M':
data = ParseInt(data, 2, 0, 59, &tm.tm_min);
continue;
case 'S':
data = ParseInt(data, 2, 0, 60, &tm.tm_sec);
continue;
case 'I':
case 'l':
case 'r': // probably uses %I
twelve_hour = true;
break;
case 'R': // uses %H
case 'T': // uses %H
case 'c': // probably uses %H
case 'X': // probably uses %H
twelve_hour = false;
break;
case 'z':
data = ParseOffset(data, '\0', &offset);
if (data != nullptr) saw_offset = true;
continue;
case 'Z': // ignored; zone abbreviations are ambiguous
data = ParseZone(data, &zone);
continue;
case 's':
data = ParseInt(data, 0, INT_FAST64_MIN, INT_FAST64_MAX, &percent_s);
if (data != nullptr) saw_percent_s = true;
continue;
case 'E':
if (*fmt == 'z') {
if (data != nullptr && *data == 'Z') { // Zulu
offset = 0;
data += 1;
} else {
data = ParseOffset(data, ':', &offset);
}
if (data != nullptr) saw_offset = true;
fmt += 1;
continue;
}
if (*fmt == '*' && *(fmt + 1) == 'S') {
data = ParseInt(data, 2, 0, 60, &tm.tm_sec);
if (data != nullptr && *data == '.') {
data = ParseSubSeconds(data + 1, &subseconds);
}
fmt += 2;
continue;
}
if (*fmt == '*' && *(fmt + 1) == 'f') {
if (data != nullptr && std::isdigit(*data)) {
data = ParseSubSeconds(data, &subseconds);
}
fmt += 2;
continue;
}
if (*fmt == '4' && *(fmt + 1) == 'Y') {
const char* bp = data;
data = ParseInt(data, 4, year_t{-999}, year_t{9999}, &year);
if (data != nullptr) {
if (data - bp == 4) {
saw_year = true;
} else {
data = nullptr; // stopped too soon
}
}
fmt += 2;
continue;
}
if (std::isdigit(*fmt)) {
int n = 0; // value ignored
if (const char* np = ParseInt(fmt, 0, 0, 1024, &n)) {
if (*np == 'S') {
data = ParseInt(data, 2, 0, 60, &tm.tm_sec);
if (data != nullptr && *data == '.') {
data = ParseSubSeconds(data + 1, &subseconds);
}
fmt = ++np;
continue;
}
if (*np == 'f') {
if (data != nullptr && std::isdigit(*data)) {
data = ParseSubSeconds(data, &subseconds);
}
fmt = ++np;
continue;
}
}
}
if (*fmt == 'c') twelve_hour = false; // probably uses %H
if (*fmt == 'X') twelve_hour = false; // probably uses %H
if (*fmt != '\0') ++fmt;
break;
case 'O':
if (*fmt == 'H') twelve_hour = false;
if (*fmt == 'I') twelve_hour = true;
if (*fmt != '\0') ++fmt;
break;
}
// Parses the current specifier.
const char* orig_data = data;
std::string spec(percent, fmt - percent);
data = ParseTM(data, spec.c_str(), &tm);
// If we successfully parsed %p we need to remember whether the result
// was AM or PM so that we can adjust tm_hour before ConvertDateTime().
// So reparse the input with a known AM hour, and check if it is shifted
// to a PM hour.
if (spec == "%p" && data != nullptr) {
std::string test_input = "1" + std::string(orig_data, data - orig_data);
const char* test_data = test_input.c_str();
std::tm tmp{};
ParseTM(test_data, "%I%p", &tmp);
afternoon = (tmp.tm_hour == 13);
}
}
// Adjust a 12-hour tm_hour value if it should be in the afternoon.
if (twelve_hour && afternoon && tm.tm_hour < 12) {
tm.tm_hour += 12;
}
if (data == nullptr) return false;
// Skip any remaining whitespace.
while (std::isspace(*data)) ++data;
// parse() must consume the entire input string.
if (*data != '\0') return false;
// If we saw %s then we ignore anything else and return that time.
if (saw_percent_s) {
*sec = FromUnixSeconds(percent_s);
*fs = detail::femtoseconds::zero();
return true;
}
// If we saw %z or %Ez then we want to interpret the parsed fields in
// UTC and then shift by that offset. Otherwise we want to interpret
// the fields directly in the passed time_zone.
time_zone ptz = saw_offset ? utc_time_zone() : tz;
// Allows a leap second of 60 to normalize forward to the following ":00".
if (tm.tm_sec == 60) {
tm.tm_sec -= 1;
offset -= 1;
subseconds = detail::femtoseconds::zero();
}
if (!saw_year) {
year = year_t{tm.tm_year};
if (year > kyearmax - 1900) return false;
year += 1900;
}
// TODO: Eliminate extra normalization.
const civil_second cs(year, tm.tm_mon + 1, tm.tm_mday,
tm.tm_hour, tm.tm_min, tm.tm_sec);
// parse() fails if any normalization was done. That is,
// parsing "Sep 31" will not produce the equivalent of "Oct 1".
if (cs.year() != year || cs.month() != tm.tm_mon + 1 ||
cs.day() != tm.tm_mday || cs.hour() != tm.tm_hour ||
cs.minute() != tm.tm_min || cs.second() != tm.tm_sec) {
return false;
}
*sec = ptz.lookup(cs).pre - sys_seconds(offset);
*fs = subseconds;
return true;
}
} // namespace detail
} // namespace cctz

34
contrib/libcctz/src/time_zone_if.cc
View File

@ -1,34 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "time_zone_if.h"
#include "time_zone_info.h"
#include "time_zone_libc.h"
namespace cctz {
std::unique_ptr<TimeZoneIf> TimeZoneIf::Load(const std::string& name) {
// Support "libc:localtime" and "libc:*" to access the legacy
// localtime and UTC support respectively from the C library.
if (name.compare(0, 5, "libc:") == 0) {
return std::unique_ptr<TimeZoneIf>(new TimeZoneLibC(name.substr(5)));
}
// Otherwise use the "zoneinfo" implementation by default.
std::unique_ptr<TimeZoneInfo> tz(new TimeZoneInfo);
if (!tz->Load(name)) tz.reset();
return std::unique_ptr<TimeZoneIf>(tz.release());
}
} // namespace cctz

62
contrib/libcctz/src/time_zone_if.h
View File

@ -1,62 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef CCTZ_TIME_ZONE_IF_H_
#define CCTZ_TIME_ZONE_IF_H_
#include <cstdint>
#include <memory>
#include <string>
#include "civil_time.h"
#include "time_zone.h"
namespace cctz {
// A simple interface used to hide time-zone complexities from time_zone::Impl.
// Subclasses implement the functions for civil-time conversions in the zone.
class TimeZoneIf {
public:
// A factory function for TimeZoneIf implementations.
static std::unique_ptr<TimeZoneIf> Load(const std::string& name);
virtual ~TimeZoneIf() {}
virtual time_zone::absolute_lookup BreakTime(
const time_point<sys_seconds>& tp) const = 0;
virtual time_zone::civil_lookup MakeTime(
const civil_second& cs) const = 0;
protected:
TimeZoneIf() {}
};
// Converts tp to a count of seconds since the Unix epoch.
inline std::int_fast64_t ToUnixSeconds(const time_point<sys_seconds>& tp) {
return (tp - std::chrono::time_point_cast<sys_seconds>(
std::chrono::system_clock::from_time_t(0)))
.count();
}
// Converts a count of seconds since the Unix epoch to a
// time_point<sys_seconds>.
inline time_point<sys_seconds> FromUnixSeconds(std::int_fast64_t t) {
return std::chrono::time_point_cast<sys_seconds>(
std::chrono::system_clock::from_time_t(0)) +
sys_seconds(t);
}
} // namespace cctz
#endif // CCTZ_TIME_ZONE_IF_H_

99
contrib/libcctz/src/time_zone_impl.cc
View File

@ -1,99 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "time_zone_impl.h"
#include <mutex>
#include <unordered_map>
namespace cctz {
namespace {
// time_zone::Impls are linked into a map to support fast lookup by name.
using TimeZoneImplByName =
std::unordered_map<std::string, const time_zone::Impl*>;
TimeZoneImplByName* time_zone_map = nullptr;
// Mutual exclusion for time_zone_map.
std::mutex time_zone_mutex;
} // namespace
time_zone time_zone::Impl::UTC() {
return time_zone(UTCImpl());
}
bool time_zone::Impl::LoadTimeZone(const std::string& name, time_zone* tz) {
// First check for UTC.
if (name.compare("UTC") == 0) {
*tz = time_zone(UTCImpl());
return true;
}
// Then check, under a shared lock, whether the time zone has already
// been loaded. This is the common path. TODO: Move to shared_mutex.
{
std::lock_guard<std::mutex> lock(time_zone_mutex);
if (time_zone_map != nullptr) {
TimeZoneImplByName::const_iterator itr = time_zone_map->find(name);
if (itr != time_zone_map->end()) {
*tz = time_zone(itr->second);
return itr->second != UTCImpl();
}
}
}
// Now check again, under an exclusive lock.
std::lock_guard<std::mutex> lock(time_zone_mutex);
if (time_zone_map == nullptr) time_zone_map = new TimeZoneImplByName;
const Impl*& impl = (*time_zone_map)[name];
bool fallback_utc = false;
if (impl == nullptr) {
// The first thread in loads the new time zone.
Impl* new_impl = new Impl(name);
new_impl->zone_ = TimeZoneIf::Load(new_impl->name_);
if (new_impl->zone_ == nullptr) {
delete new_impl; // free the nascent Impl
impl = UTCImpl(); // and fallback to UTC
fallback_utc = true;
} else {
impl = new_impl; // install new time zone
}
}
*tz = time_zone(impl);
return !fallback_utc;
}
const time_zone::Impl& time_zone::Impl::get(const time_zone& tz) {
if (tz.impl_ == nullptr) {
// Dereferencing an implicit-UTC time_zone is expected to be
// rare, so we don't mind paying a small synchronization cost.
return *UTCImpl();
}
return *tz.impl_;
}
time_zone::Impl::Impl(const std::string& name) : name_(name) {}
const time_zone::Impl* time_zone::Impl::UTCImpl() {
static Impl* utc_impl = [] {
Impl* impl = new Impl("UTC");
impl->zone_ = TimeZoneIf::Load(impl->name_); // never fails
return impl;
}();
return utc_impl;
}
} // namespace cctz

65
contrib/libcctz/src/time_zone_impl.h
View File

@ -1,65 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef CCTZ_TIME_ZONE_IMPL_H_
#define CCTZ_TIME_ZONE_IMPL_H_
#include <memory>
#include <string>
#include "time_zone.h"
#include "time_zone_info.h"
namespace cctz {
// time_zone::Impl is the internal object referenced by a cctz::time_zone.
class time_zone::Impl {
public:
// The UTC time zone. Also used for other time zones that fail to load.
static time_zone UTC();
// Load a named time zone. Returns false if the name is invalid, or if
// some other kind of error occurs. Note that loading "UTC" never fails.
static bool LoadTimeZone(const std::string& name, time_zone* tz);
// Dereferences the time_zone to obtain its Impl.
static const time_zone::Impl& get(const time_zone& tz);
// The primary key is the time-zone ID (e.g., "America/New_York").
const std::string& name() const { return name_; }
// Breaks a time_point down to civil-time components in this time zone.
time_zone::absolute_lookup BreakTime(
const time_point<sys_seconds>& tp) const {
return zone_->BreakTime(tp);
}
// Converts the civil-time components in this time zone into a time_point.
// That is, the opposite of BreakTime(). The requested civil time may be
// ambiguous or illegal due to a change of UTC offset.
time_zone::civil_lookup MakeTime(const civil_second& cs) const {
return zone_->MakeTime(cs);
}
private:
explicit Impl(const std::string& name);
static const Impl* UTCImpl();
const std::string name_;
std::unique_ptr<TimeZoneIf> zone_;
};
} // namespace cctz
#endif // CCTZ_TIME_ZONE_IMPL_H_

738
contrib/libcctz/src/time_zone_info.cc
View File

@ -1,738 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// This file implements the TimeZoneIf interface using the "zoneinfo"
// data provided by the IANA Time Zone Database (i.e., the only real game
// in town).
//
// TimeZoneInfo represents the history of UTC-offset changes within a time
// zone. Most changes are due to daylight-saving rules, but occasionally
// shifts are made to the time-zone's base offset. The database only attempts
// to be definitive for times since 1970, so be wary of local-time conversions
// before that. Also, rule and zone-boundary changes are made at the whim
// of governments, so the conversion of future times needs to be taken with
// a grain of salt.
//
// For more information see tzfile(5), http://www.iana.org/time-zones, or
// http://en.wikipedia.org/wiki/Zoneinfo.
//
// Note that we assume the proleptic Gregorian calendar and 60-second
// minutes throughout.
#include "time_zone_info.h"
#include <algorithm>
#include <cerrno>
#include <chrono>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <iostream>
#include <limits>
#include "time_zone_posix.h"
namespace cctz {
namespace {
// Convert errnum to a message, using buf[buflen] if necessary.
// buf must be non-null, and buflen non-zero.
char* errmsg(int errnum, char* buf, std::size_t buflen) {
#if defined(_MSC_VER)
strerror_s(buf, buflen, errnum);
return buf;
#elif defined(__APPLE__)
strerror_r(errnum, buf, buflen);
return buf;
#elif (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && !_GNU_SOURCE
strerror_r(errnum, buf, buflen);
return buf;
#else
return strerror_r(errnum, buf, buflen);
#endif
}
// Wrap the tzfile.h isleap() macro with an inline function, which will
// then have normal argument-passing semantics (i.e., single evaluation).
inline bool IsLeap(cctz::year_t year) { return isleap(year); }
// The day offsets of the beginning of each (1-based) month in non-leap
// and leap years respectively. That is, sigma[1:n]:kDaysPerMonth[][i].
// For example, in a leap year there are 335 days before December.
const std::int_least16_t kMonthOffsets[2][1 + MONSPERYEAR + 1] = {
{-1, 0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365},
{-1, 0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366},
};
// 400-year chunks always have 146097 days (20871 weeks).
const std::int_least64_t kSecPer400Years = 146097LL * SECSPERDAY;
// The number of seconds in non-leap and leap years respectively.
const std::int_least32_t kSecPerYear[2] = {
DAYSPERNYEAR * SECSPERDAY,
DAYSPERLYEAR * SECSPERDAY,
};
// Like kSecPerYear[] but scaled down by a factor of SECSPERDAY.
const std::int_least32_t kDaysPerYear[2] = {DAYSPERNYEAR, DAYSPERLYEAR};
// January 1st at 00:00:00 in the epoch year.
const civil_second unix_epoch(EPOCH_YEAR, 1, 1, 0, 0, 0);
// Single-byte, unsigned numeric values are encoded directly.
inline std::uint_fast8_t Decode8(const char* cp) {
return static_cast<std::uint_fast8_t>(*cp) & 0xff;
}
// Multi-byte, numeric values are encoded using a MSB first,
// twos-complement representation. These helpers decode, from
// the given address, 4-byte and 8-byte values respectively.
std::int_fast32_t Decode32(const char* cp) {
std::uint_fast32_t v = 0;
for (int i = 0; i != (32 / 8); ++i) v = (v << 8) | Decode8(cp++);
if (v <= INT32_MAX) return static_cast<std::int_fast32_t>(v);
return static_cast<std::int_fast32_t>(v - INT32_MAX - 1) + INT32_MIN;
}
std::int_fast64_t Decode64(const char* cp) {
std::uint_fast64_t v = 0;
for (int i = 0; i != (64 / 8); ++i) v = (v << 8) | Decode8(cp++);
if (v <= INT64_MAX) return static_cast<std::int_fast64_t>(v);
return static_cast<std::int_fast64_t>(v - INT64_MAX - 1) + INT64_MIN;
}
// Generate a year-relative offset for a PosixTransition.
std::int_fast64_t TransOffset(bool leap_year, int jan1_weekday,
const PosixTransition& pt) {
std::int_fast64_t days = 0;
switch (pt.date.fmt) {
case PosixTransition::J: {
days = pt.date.j.day;
if (!leap_year || days < kMonthOffsets[1][TM_MARCH + 1]) days -= 1;
break;
}
case PosixTransition::N: {
days = pt.date.n.day;
break;
}
case PosixTransition::M: {
const bool last_week = (pt.date.m.week == 5);
days = kMonthOffsets[leap_year][pt.date.m.month + last_week];
const int weekday = (jan1_weekday + days) % DAYSPERWEEK;
if (last_week) {
days -=
(weekday + DAYSPERWEEK - 1 - pt.date.m.weekday) % DAYSPERWEEK + 1;
} else {
days += (pt.date.m.weekday + DAYSPERWEEK - weekday) % DAYSPERWEEK;
days += (pt.date.m.week - 1) * DAYSPERWEEK;
}
break;
}
}
return (days * SECSPERDAY) + pt.time.offset;
}
inline time_zone::civil_lookup MakeUnique(std::int_fast64_t unix_time) {
time_zone::civil_lookup cl;
cl.pre = cl.trans = cl.post = FromUnixSeconds(unix_time);
cl.kind = time_zone::civil_lookup::UNIQUE;
return cl;
}
inline time_zone::civil_lookup MakeSkipped(const Transition& tr,
const DateTime& dt) {
time_zone::civil_lookup cl;
cl.pre = FromUnixSeconds(tr.unix_time - 1 + (dt - tr.prev_date_time));
cl.trans = FromUnixSeconds(tr.unix_time);
cl.post = FromUnixSeconds(tr.unix_time - (tr.date_time - dt));
cl.kind = time_zone::civil_lookup::SKIPPED;
return cl;
}
inline time_zone::civil_lookup MakeRepeated(const Transition& tr,
const DateTime& dt) {
time_zone::civil_lookup cl;
cl.pre = FromUnixSeconds(tr.unix_time - 1 - (tr.prev_date_time - dt));
cl.trans = FromUnixSeconds(tr.unix_time);
cl.post = FromUnixSeconds(tr.unix_time + (dt - tr.date_time));
cl.kind = time_zone::civil_lookup::REPEATED;
return cl;
}
civil_second YearShift(const civil_second& cs, cctz::year_t year_shift) {
// TODO: How do we do this while avoiding any normalization tests?
return civil_second(cs.year() + year_shift, cs.month(), cs.day(),
cs.hour(), cs.minute(), cs.second());
}
} // namespace
// Assign from a civil_second, created using a TimeZoneInfo timestamp.
void DateTime::Assign(const civil_second& cs) {
offset = cs - unix_epoch;
}
// What (no leap-seconds) UTC+seconds zoneinfo would look like.
bool TimeZoneInfo::ResetToBuiltinUTC(std::int_fast32_t seconds) {
transition_types_.resize(1);
TransitionType& tt(transition_types_.back());
tt.utc_offset = seconds;
tt.is_dst = false;
tt.abbr_index = 0;
transitions_.clear();
transitions_.reserve(2);
for (const std::int_fast64_t unix_time : {-(1LL << 59), 2147483647LL}) {
Transition& tr(*transitions_.emplace(transitions_.end()));
tr.unix_time = unix_time;
tr.type_index = 0;
tr.date_time.Assign(LocalTime(tr.unix_time, tt).cs);
tr.prev_date_time = tr.date_time;
tr.prev_date_time.offset -= 1;
}
default_transition_type_ = 0;
abbreviations_ = "UTC"; // TODO: Handle non-zero offset.
abbreviations_.append(1, '\0'); // add NUL
future_spec_.clear(); // never needed for a fixed-offset zone
extended_ = false;
transitions_.shrink_to_fit();
return true;
}
// Builds the in-memory header using the raw bytes from the file.
void TimeZoneInfo::Header::Build(const tzhead& tzh) {
timecnt = Decode32(tzh.tzh_timecnt);
typecnt = Decode32(tzh.tzh_typecnt);
charcnt = Decode32(tzh.tzh_charcnt);
leapcnt = Decode32(tzh.tzh_leapcnt);
ttisstdcnt = Decode32(tzh.tzh_ttisstdcnt);
ttisgmtcnt = Decode32(tzh.tzh_ttisgmtcnt);
}
// How many bytes of data are associated with this header. The result
// depends upon whether this is a section with 4-byte or 8-byte times.
std::size_t TimeZoneInfo::Header::DataLength(std::size_t time_len) const {
std::size_t len = 0;
len += (time_len + 1) * timecnt; // unix_time + type_index
len += (4 + 1 + 1) * typecnt; // utc_offset + is_dst + abbr_index
len += 1 * charcnt; // abbreviations
len += (time_len + 4) * leapcnt; // leap-time + TAI-UTC
len += 1 * ttisstdcnt; // UTC/local indicators
len += 1 * ttisgmtcnt; // standard/wall indicators
return len;
}
// Check that the TransitionType has the expected offset/is_dst/abbreviation.
void TimeZoneInfo::CheckTransition(const std::string& name,
const TransitionType& tt,
std::int_fast32_t offset, bool is_dst,
const std::string& abbr) const {
if (tt.utc_offset != offset || tt.is_dst != is_dst ||
&abbreviations_[tt.abbr_index] != abbr) {
std::clog << name << ": Transition"
<< " offset=" << tt.utc_offset << "/"
<< (tt.is_dst ? "DST" : "STD")
<< "/abbr=" << &abbreviations_[tt.abbr_index]
<< " does not match POSIX spec '" << future_spec_ << "'\n";
}
}
// zic(8) can generate no-op transitions when a zone changes rules at an
// instant when there is actually no discontinuity. So we check whether
// two transitions have equivalent types (same offset/is_dst/abbr).
bool TimeZoneInfo::EquivTransitions(std::uint_fast8_t tt1_index,
std::uint_fast8_t tt2_index) const {
if (tt1_index == tt2_index) return true;
const TransitionType& tt1(transition_types_[tt1_index]);
const TransitionType& tt2(transition_types_[tt2_index]);
if (tt1.is_dst != tt2.is_dst) return false;
if (tt1.utc_offset != tt2.utc_offset) return false;
if (tt1.abbr_index != tt2.abbr_index) return false;
return true;
}
// Use the POSIX-TZ-environment-variable-style string to handle times
// in years after the last transition stored in the zoneinfo data.
void TimeZoneInfo::ExtendTransitions(const std::string& name,
const Header& hdr) {
extended_ = false;
bool extending = !future_spec_.empty();
PosixTimeZone posix;
if (extending && !ParsePosixSpec(future_spec_, &posix)) {
std::clog << name << ": Failed to parse '" << future_spec_ << "'\n";
extending = false;
}
if (extending && posix.dst_abbr.empty()) { // std only
// The future specification should match the last/default transition,
// and that means that handling the future will fall out naturally.
std::uint_fast8_t index = default_transition_type_;
if (hdr.timecnt != 0) index = transitions_[hdr.timecnt - 1].type_index;
const TransitionType& tt(transition_types_[index]);
CheckTransition(name, tt, posix.std_offset, false, posix.std_abbr);
extending = false;
}
if (extending && hdr.timecnt < 2) {
std::clog << name << ": Too few transitions for POSIX spec\n";
extending = false;
}
if (!extending) {
// Ensure that there is always a transition in the second half of the
// time line (the BIG_BANG transition is in the first half) so that the
// signed difference between a DateTime and the DateTime of its previous
// transition is always representable, without overflow.
const Transition& last(transitions_.back());
if (last.unix_time < 0) {
const std::uint_fast8_t type_index = last.type_index;
Transition& tr(*transitions_.emplace(transitions_.end()));
tr.unix_time = 2147483647; // 2038-01-19T03:14:07+00:00
tr.type_index = type_index;
}
return; // last transition wins
}
// Extend the transitions for an additional 400 years using the
// future specification. Years beyond those can be handled by
// mapping back to a cycle-equivalent year within that range.
// zic(8) should probably do this so that we don't have to.
transitions_.resize(hdr.timecnt + 400 * 2);
extended_ = true;
// The future specification should match the last two transitions,
// and those transitions should have different is_dst flags but be
// in the same year.
// TODO: Investigate the actual guarantees made by zic.
const Transition& tr0(transitions_[hdr.timecnt - 1]);
const Transition& tr1(transitions_[hdr.timecnt - 2]);
const TransitionType& tt0(transition_types_[tr0.type_index]);
const TransitionType& tt1(transition_types_[tr1.type_index]);
const TransitionType& spring(tt0.is_dst ? tt0 : tt1);
const TransitionType& autumn(tt0.is_dst ? tt1 : tt0);
CheckTransition(name, spring, posix.dst_offset, true, posix.dst_abbr);
CheckTransition(name, autumn, posix.std_offset, false, posix.std_abbr);
last_year_ = LocalTime(tr0.unix_time, tt0).cs.year();
if (LocalTime(tr1.unix_time, tt1).cs.year() != last_year_) {
std::clog << name << ": Final transitions not in same year\n";
}
// Add the transitions to tr1 and back to tr0 for each extra year.
const PosixTransition& pt1(tt0.is_dst ? posix.dst_end : posix.dst_start);
const PosixTransition& pt0(tt0.is_dst ? posix.dst_start : posix.dst_end);
Transition* tr = &transitions_[hdr.timecnt]; // next trans to fill
const civil_day jan1(last_year_, 1, 1);
std::int_fast64_t jan1_time = civil_second(jan1) - unix_epoch;
int jan1_weekday = (static_cast<int>(get_weekday(jan1)) + 1) % DAYSPERWEEK;
bool leap_year = IsLeap(last_year_);
for (const cctz::year_t limit = last_year_ + 400; last_year_ < limit;) {
last_year_ += 1; // an additional year of generated transitions
jan1_time += kSecPerYear[leap_year];
jan1_weekday = (jan1_weekday + kDaysPerYear[leap_year]) % DAYSPERWEEK;
leap_year = !leap_year && IsLeap(last_year_);
tr->unix_time =
jan1_time + TransOffset(leap_year, jan1_weekday, pt1) - tt0.utc_offset;
tr++->type_index = tr1.type_index;
tr->unix_time =
jan1_time + TransOffset(leap_year, jan1_weekday, pt0) - tt1.utc_offset;
tr++->type_index = tr0.type_index;
}
}
bool TimeZoneInfo::Load(const std::string& name, FILE* fp) {
// Read and validate the header.
tzhead tzh;
if (fread(&tzh, 1, sizeof tzh, fp) != sizeof tzh)
return false;
if (strncmp(tzh.tzh_magic, TZ_MAGIC, sizeof(tzh.tzh_magic)) != 0)
return false;
Header hdr;
hdr.Build(tzh);
std::size_t time_len = 4;
if (tzh.tzh_version[0] != '\0') {
// Skip the 4-byte data.
if (fseek(fp, static_cast<long>(hdr.DataLength(time_len)), SEEK_CUR) != 0)
return false;
// Read and validate the header for the 8-byte data.
if (fread(&tzh, 1, sizeof tzh, fp) != sizeof tzh)
return false;
if (strncmp(tzh.tzh_magic, TZ_MAGIC, sizeof(tzh.tzh_magic)) != 0)
return false;
if (tzh.tzh_version[0] == '\0')
return false;
hdr.Build(tzh);
time_len = 8;
}
if (hdr.timecnt < 0 || hdr.typecnt <= 0)
return false;
if (hdr.leapcnt != 0) {
// This code assumes 60-second minutes so we do not want
// the leap-second encoded zoneinfo. We could reverse the
// compensation, but it's never in a Google zoneinfo anyway,
// so currently we simply reject such data.
return false;
}
if (hdr.ttisstdcnt != 0 && hdr.ttisstdcnt != hdr.typecnt)
return false;
if (hdr.ttisgmtcnt != 0 && hdr.ttisgmtcnt != hdr.typecnt)
return false;
// Read the data into a local buffer.
std::size_t len = hdr.DataLength(time_len);
std::vector<char> tbuf(len);
if (fread(tbuf.data(), 1, len, fp) != len)
return false;
const char* bp = tbuf.data();
// Decode and validate the transitions.
transitions_.reserve(hdr.timecnt + 2); // We might add a couple.
transitions_.resize(hdr.timecnt);
for (std::int_fast32_t i = 0; i != hdr.timecnt; ++i) {
transitions_[i].unix_time = (time_len == 4) ? Decode32(bp) : Decode64(bp);
bp += time_len;
if (i != 0) {
// Check that the transitions are ordered by time (as zic guarantees).
if (!Transition::ByUnixTime()(transitions_[i - 1], transitions_[i]))
return false; // out of order
}
}
bool seen_type_0 = false;
for (std::int_fast32_t i = 0; i != hdr.timecnt; ++i) {
transitions_[i].type_index = Decode8(bp++);
if (transitions_[i].type_index >= hdr.typecnt)
return false;
if (transitions_[i].type_index == 0)
seen_type_0 = true;
}
// Decode and validate the transition types.
transition_types_.resize(hdr.typecnt);
for (std::int_fast32_t i = 0; i != hdr.typecnt; ++i) {
transition_types_[i].utc_offset = Decode32(bp);
if (transition_types_[i].utc_offset >= SECSPERDAY ||
transition_types_[i].utc_offset <= -SECSPERDAY)
return false;
bp += 4;
transition_types_[i].is_dst = (Decode8(bp++) != 0);
transition_types_[i].abbr_index = Decode8(bp++);
if (transition_types_[i].abbr_index >= hdr.charcnt)
return false;
}
// Determine the before-first-transition type.
default_transition_type_ = 0;
if (seen_type_0 && hdr.timecnt != 0) {
std::int_fast8_t index = 0;
if (transition_types_[0].is_dst) {
index = transitions_[0].type_index;
while (index != 0 && transition_types_[index].is_dst)
--index;
}
while (index != hdr.typecnt && transition_types_[index].is_dst)
++index;
if (index != hdr.typecnt)
default_transition_type_ = index;
}
// Copy all the abbreviations.
abbreviations_.assign(bp, hdr.charcnt);
bp += hdr.charcnt;
// Skip the unused portions. We've already dispensed with leap-second
// encoded zoneinfo. The ttisstd/ttisgmt indicators only apply when
// interpreting a POSIX spec that does not include start/end rules, and
// that isn't the case here (see "zic -p").
bp += (8 + 4) * hdr.leapcnt; // leap-time + TAI-UTC
bp += 1 * hdr.ttisstdcnt; // UTC/local indicators
bp += 1 * hdr.ttisgmtcnt; // standard/wall indicators
future_spec_.clear();
if (tzh.tzh_version[0] != '\0') {
// Snarf up the NL-enclosed future POSIX spec. Note
// that version '3' files utilize an extended format.
if (fgetc(fp) != '\n')
return false;
for (int c = fgetc(fp); c != '\n'; c = fgetc(fp)) {
if (c == EOF)
return false;
future_spec_.push_back(static_cast<char>(c));
}
}
// We don't check for EOF so that we're forwards compatible.
// Trim redundant transitions. zic may have added these to work around
// differences between the glibc and reference implementations (see
// zic.c:dontmerge) and the Qt library (see zic.c:WORK_AROUND_QTBUG_53071).
// For us, they just get in the way when we do future_spec_ extension.
while (hdr.timecnt > 1) {
if (!EquivTransitions(transitions_[hdr.timecnt - 1].type_index,
transitions_[hdr.timecnt - 2].type_index)) {
break;
}
hdr.timecnt -= 1;
}
transitions_.resize(hdr.timecnt);
// Ensure that there is always a transition in the first half of the
// time line (the second half is handled in ExtendTransitions()) so
// that the signed difference between a DateTime and the DateTime of
// its previous transition is always representable, without overflow.
// A contemporary zic will usually have already done this for us.
if (transitions_.empty() || transitions_.front().unix_time >= 0) {
Transition& tr(*transitions_.emplace(transitions_.begin()));
tr.unix_time = -(1LL << 59); // see tz/zic.c "BIG_BANG"
tr.type_index = default_transition_type_;
hdr.timecnt += 1;
}
// Extend the transitions using the future specification.
ExtendTransitions(name, hdr);
// Compute the local civil time for each transition and the preceeding
// second. These will be used for reverse conversions in MakeTime().
const TransitionType* ttp = &transition_types_[default_transition_type_];
for (std::size_t i = 0; i != transitions_.size(); ++i) {
Transition& tr(transitions_[i]);
tr.prev_date_time.Assign(LocalTime(tr.unix_time, *ttp).cs);
tr.prev_date_time.offset -= 1;
ttp = &transition_types_[tr.type_index];
tr.date_time.Assign(LocalTime(tr.unix_time, *ttp).cs);
if (i != 0) {
// Check that the transitions are ordered by date/time. Essentially
// this means that an offset change cannot cross another such change.
// No one does this in practice, and we depend on it in MakeTime().
if (!Transition::ByDateTime()(transitions_[i - 1], tr))
return false; // out of order
}
}
// We remember the transitions found during the last BreakTime() and
// MakeTime() calls. If the next request is for the same transition we
// will avoid re-searching.
local_time_hint_ = 0;
time_local_hint_ = 0;
transitions_.shrink_to_fit();
return true;
}
bool TimeZoneInfo::Load(const std::string& name) {
// We can ensure that the loading of UTC or any other fixed-offset
// zone never fails because the simple, fixed-offset state can be
// internally generated. Note that this depends on our choice to not
// accept leap-second encoded ("right") zoneinfo.
if (name == "UTC") return ResetToBuiltinUTC(0);
// Map time-zone name to its machine-specific path.
std::string path;
if (name == "localtime") {
#if defined(_MSC_VER)
char* localtime = nullptr;
_dupenv_s(&localtime, nullptr, "LOCALTIME");
path = localtime ? localtime : "/etc/localtime";
free(localtime);
#else
const char* localtime = std::getenv("LOCALTIME");
path = localtime ? localtime : "/etc/localtime";
#endif
} else if (!name.empty() && name[0] == '/') {
path = name;
} else {
#if defined(_MSC_VER)
char* tzdir = nullptr;
_dupenv_s(&tzdir, nullptr, "TZDIR");
path = tzdir ? tzdir : "/usr/share/zoneinfo";
free(tzdir);
#else
const char* tzdir = std::getenv("TZDIR");
path = tzdir ? tzdir : "/usr/share/zoneinfo";
#endif
path += '/';
path += name;
}
// Load the time-zone data.
bool loaded = false;
#if defined(_MSC_VER)
FILE* fp;
if (fopen_s(&fp, path.c_str(), "rb") != 0) fp = nullptr;
#else
FILE* fp = fopen(path.c_str(), "rb");
#endif
if (fp != nullptr) {
loaded = Load(name, fp);
fclose(fp);
} else {
char ebuf[64];
std::clog << path << ": " << errmsg(errno, ebuf, sizeof ebuf) << "\n";
}
return loaded;
}
// BreakTime() translation for a particular transition type.
time_zone::absolute_lookup TimeZoneInfo::LocalTime(
std::int_fast64_t unix_time, const TransitionType& tt) const {
time_zone::absolute_lookup al;
// A civil time in "+offset" looks like (time+offset) in UTC.
// Note: We perform two additions in the civil_second domain to
// sidestep the chance of overflow in (unix_time + tt.utc_offset).
al.cs = unix_epoch + unix_time;
al.cs += tt.utc_offset;
// Handle offset, is_dst, and abbreviation.
al.offset = tt.utc_offset;
al.is_dst = tt.is_dst;
al.abbr = &abbreviations_[tt.abbr_index];
return al;
}
// MakeTime() translation with a conversion-preserving offset.
time_zone::civil_lookup TimeZoneInfo::TimeLocal(
const civil_second& cs, std::int_fast64_t offset) const {
time_zone::civil_lookup cl = MakeTime(cs);
cl.pre += sys_seconds(offset);
cl.trans += sys_seconds(offset);
cl.post += sys_seconds(offset);
return cl;
}
time_zone::absolute_lookup TimeZoneInfo::BreakTime(
const time_point<sys_seconds>& tp) const {
std::int_fast64_t unix_time = ToUnixSeconds(tp);
const std::size_t timecnt = transitions_.size();
if (timecnt == 0 || unix_time < transitions_[0].unix_time) {
const std::uint_fast8_t type_index = default_transition_type_;
return LocalTime(unix_time, transition_types_[type_index]);
}
if (unix_time >= transitions_[timecnt - 1].unix_time) {
// After the last transition. If we extended the transitions using
// future_spec_, shift back to a supported year using the 400-year
// cycle of calendaric equivalence and then compensate accordingly.
if (extended_) {
const std::int_fast64_t diff =
unix_time - transitions_[timecnt - 1].unix_time;
const cctz::year_t shift = diff / kSecPer400Years + 1;
const auto d = sys_seconds(shift * kSecPer400Years);
time_zone::absolute_lookup al = BreakTime(tp - d);
al.cs = YearShift(al.cs, shift * 400);
return al;
}
const std::uint_fast8_t type_index = transitions_[timecnt - 1].type_index;
return LocalTime(unix_time, transition_types_[type_index]);
}
const std::size_t hint = local_time_hint_.load(std::memory_order_relaxed);
if (0 < hint && hint < timecnt) {
if (unix_time < transitions_[hint].unix_time) {
if (!(unix_time < transitions_[hint - 1].unix_time)) {
const std::uint_fast8_t type_index = transitions_[hint - 1].type_index;
return LocalTime(unix_time, transition_types_[type_index]);
}
}
}
const Transition target = {unix_time, 0, {0}, {0}};
const Transition* begin = &transitions_[0];
const Transition* tr = std::upper_bound(begin, begin + timecnt, target,
Transition::ByUnixTime());
local_time_hint_.store(tr - begin, std::memory_order_relaxed);
const std::uint_fast8_t type_index = (--tr)->type_index;
return LocalTime(unix_time, transition_types_[type_index]);
}
time_zone::civil_lookup TimeZoneInfo::MakeTime(const civil_second& cs) const {
Transition target;
DateTime& dt(target.date_time);
dt.Assign(cs);
const std::size_t timecnt = transitions_.size();
if (timecnt == 0) {
// Use the default offset.
const std::int_fast32_t default_offset =
transition_types_[default_transition_type_].utc_offset;
return MakeUnique((dt - DateTime{0}) - default_offset);
}
// Find the first transition after our target date/time.
const Transition* tr = nullptr;
const Transition* begin = &transitions_[0];
const Transition* end = begin + timecnt;
if (dt < begin->date_time) {
tr = begin;
} else if (!(dt < transitions_[timecnt - 1].date_time)) {
tr = end;
} else {
const std::size_t hint = time_local_hint_.load(std::memory_order_relaxed);
if (0 < hint && hint < timecnt) {
if (dt < transitions_[hint].date_time) {
if (!(dt < transitions_[hint - 1].date_time)) {
tr = begin + hint;
}
}
}
if (tr == nullptr) {
tr = std::upper_bound(begin, end, target, Transition::ByDateTime());
time_local_hint_.store(tr - begin, std::memory_order_relaxed);
}
}
if (tr == begin) {
if (!(tr->prev_date_time < dt)) {
// Before first transition, so use the default offset.
const std::int_fast32_t default_offset =
transition_types_[default_transition_type_].utc_offset;
return MakeUnique((dt - DateTime{0}) - default_offset);
}
// tr->prev_date_time < dt < tr->date_time
return MakeSkipped(*tr, dt);
}
if (tr == end) {
if ((--tr)->prev_date_time < dt) {
// After the last transition. If we extended the transitions using
// future_spec_, shift back to a supported year using the 400-year
// cycle of calendaric equivalence and then compensate accordingly.
if (extended_ && cs.year() > last_year_) {
const cctz::year_t shift = (cs.year() - last_year_) / 400 + 1;
return TimeLocal(YearShift(cs, shift * -400), shift * kSecPer400Years);
}
return MakeUnique(tr->unix_time + (dt - tr->date_time));
}
// tr->date_time <= dt <= tr->prev_date_time
return MakeRepeated(*tr, dt);
}
if (tr->prev_date_time < dt) {
// tr->prev_date_time < dt < tr->date_time
return MakeSkipped(*tr, dt);
}
if (!((--tr)->prev_date_time < dt)) {
// tr->date_time <= dt <= tr->prev_date_time
return MakeRepeated(*tr, dt);
}
// In between transitions.
return MakeUnique(tr->unix_time + (dt - tr->date_time));
}
} // namespace cctz

145
contrib/libcctz/src/time_zone_info.h
View File

@ -1,145 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef CCTZ_TIME_ZONE_INFO_H_
#define CCTZ_TIME_ZONE_INFO_H_
#include <atomic>
#include <cstdint>
#include <cstdio>
#include <string>
#include <vector>
#include "civil_time.h"
#include "time_zone_if.h"
#include "tzfile.h"
namespace cctz {
// A zone-independent date/time. A DateTime represents a "Y/M/D H:M:S"
// as an offset in seconds from some epoch DateTime, without taking into
// account the value of, or changes in any time_zone's UTC offset (i.e., as
// if the date/time was in UTC). This allows "Y/M/D H:M:S" values to be
// quickly ordered by offset (although this may not be the same ordering as
// their corresponding times in a time_zone). But, if two DateTimes are not
// separated by a UTC-offset change in some time_zone, then the number of
// seconds between them can be computed as a simple difference of offsets.
//
// Note: Because the DateTime epoch does not correspond to the time_point
// epoch (even if only because of the unknown UTC offset) there can be valid
// times that will not be representable as DateTimes when DateTime only has
// the same number of "seconds" bits. We accept this at the moment (so as
// to avoid extended arithmetic) and lose a little range as a result.
struct DateTime {
std::int_least64_t offset; // seconds from some epoch DateTime
void Assign(const civil_second& cs);
};
inline bool operator<(const DateTime& lhs, const DateTime& rhs) {
return lhs.offset < rhs.offset;
}
// The difference between two DateTimes in seconds. Requires that all
// intervening DateTimes share the same UTC offset (i.e., no transitions).
inline std::int_fast64_t operator-(const DateTime& lhs, const DateTime& rhs) {
return lhs.offset - rhs.offset;
}
// A transition to a new UTC offset.
struct Transition {
std::int_least64_t unix_time; // the instant of this transition
std::uint_least8_t type_index; // index of the transition type
DateTime date_time; // local date/time of transition
DateTime prev_date_time; // local date/time one second earlier
struct ByUnixTime {
inline bool operator()(const Transition& lhs, const Transition& rhs) const {
return lhs.unix_time < rhs.unix_time;
}
};
struct ByDateTime {
inline bool operator()(const Transition& lhs, const Transition& rhs) const {
return lhs.date_time < rhs.date_time;
}
};
};
// The characteristics of a particular transition.
struct TransitionType {
std::int_least32_t utc_offset; // the new prevailing UTC offset
bool is_dst; // did we move into daylight-saving time
std::uint_least8_t abbr_index; // index of the new abbreviation
};
// A time zone backed by the IANA Time Zone Database (zoneinfo).
class TimeZoneInfo : public TimeZoneIf {
public:
TimeZoneInfo() = default;
TimeZoneInfo(const TimeZoneInfo&) = delete;
TimeZoneInfo& operator=(const TimeZoneInfo&) = delete;
// Loads the zoneinfo for the given name, returning true if successful.
bool Load(const std::string& name);
// TimeZoneIf implementations.
time_zone::absolute_lookup BreakTime(
const time_point<sys_seconds>& tp) const override;
time_zone::civil_lookup MakeTime(
const civil_second& cs) const override;
private:
struct Header { // counts of:
std::int_fast32_t timecnt; // transition times
std::int_fast32_t typecnt; // transition types
std::int_fast32_t charcnt; // zone abbreviation characters
std::int_fast32_t leapcnt; // leap seconds (we expect none)
std::int_fast32_t ttisstdcnt; // UTC/local indicators (unused)
std::int_fast32_t ttisgmtcnt; // standard/wall indicators (unused)
void Build(const tzhead& tzh);
std::size_t DataLength(std::size_t time_len) const;
};
void CheckTransition(const std::string& name, const TransitionType& tt,
std::int_fast32_t offset, bool is_dst,
const std::string& abbr) const;
bool EquivTransitions(std::uint_fast8_t tt1_index,
std::uint_fast8_t tt2_index) const;
void ExtendTransitions(const std::string& name, const Header& hdr);
bool ResetToBuiltinUTC(std::int_fast32_t seconds);
bool Load(const std::string& name, FILE* fp);
// Helpers for BreakTime() and MakeTime() respectively.
time_zone::absolute_lookup LocalTime(std::int_fast64_t unix_time,
const TransitionType& tt) const;
time_zone::civil_lookup TimeLocal(const civil_second& cs,
std::int_fast64_t offset) const;
std::vector<Transition> transitions_; // ordered by unix_time and date_time
std::vector<TransitionType> transition_types_; // distinct transition types
std::uint_fast8_t default_transition_type_; // for before first transition
std::string abbreviations_; // all the NUL-terminated abbreviations
std::string future_spec_; // for after the last zic transition
bool extended_; // future_spec_ was used to generate transitions
cctz::year_t last_year_; // the final year of the generated transitions
mutable std::atomic<std::size_t> local_time_hint_ = {}; // BreakTime() hint
mutable std::atomic<std::size_t> time_local_hint_ = {}; // MakeTime() hint
};
} // namespace cctz
#endif // CCTZ_TIME_ZONE_INFO_H_

118
contrib/libcctz/src/time_zone_libc.cc
View File

@ -1,118 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "time_zone_libc.h"
#include <chrono>
#include <cstdint>
#include <ctime>
// Define OFFSET(tm) and ABBR(tm) for your platform to return the UTC
// offset and zone abbreviation after a call to localtime_r().
#if defined(linux)
# if defined(__USE_BSD) || defined(__USE_MISC)
# define OFFSET(tm) ((tm).tm_gmtoff)
# define ABBR(tm) ((tm).tm_zone)
# else
# define OFFSET(tm) ((tm).__tm_gmtoff)
# define ABBR(tm) ((tm).__tm_zone)
# endif
#elif defined(__APPLE__)
# define OFFSET(tm) ((tm).tm_gmtoff)
# define ABBR(tm) ((tm).tm_zone)
#elif defined(__sun)
# define OFFSET(tm) ((tm).tm_isdst > 0 ? altzone : timezone)
# define ABBR(tm) (tzname[(tm).tm_isdst > 0])
#elif defined(_WIN32) || defined(_WIN64)
static long get_timezone() {
long seconds;
_get_timezone(&seconds);
return seconds;
}
static std::string get_tzname(int index) {
char time_zone_name[32] = {0};
std::size_t size_in_bytes = sizeof time_zone_name;
_get_tzname(&size_in_bytes, time_zone_name, size_in_bytes, index);
return time_zone_name;
}
# define OFFSET(tm) (get_timezone() + ((tm).tm_isdst > 0 ? 60 * 60 : 0))
# define ABBR(tm) (get_tzname((tm).tm_isdst > 0))
#else
# define OFFSET(tm) (timezone + ((tm).tm_isdst > 0 ? 60 * 60 : 0))
# define ABBR(tm) (tzname[(tm).tm_isdst > 0])
#endif
namespace cctz {
TimeZoneLibC::TimeZoneLibC(const std::string& name) {
local_ = (name == "localtime");
if (!local_) {
// TODO: Support "UTC-05:00", for example.
offset_ = 0;
abbr_ = "UTC";
}
}
time_zone::absolute_lookup TimeZoneLibC::BreakTime(
const time_point<sys_seconds>& tp) const {
time_zone::absolute_lookup al;
std::time_t t = ToUnixSeconds(tp);
std::tm tm;
if (local_) {
#if defined(_WIN32) || defined(_WIN64)
localtime_s(&tm, &t);
#else
localtime_r(&t, &tm);
#endif
al.offset = OFFSET(tm);
al.abbr = ABBR(tm);
} else {
#if defined(_WIN32) || defined(_WIN64)
gmtime_s(&tm, &t);
#else
gmtime_r(&t, &tm);
#endif
al.offset = offset_;
al.abbr = abbr_;
}
// TODO: Eliminate redundant normalization.
al.cs = civil_second(tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,
tm.tm_hour, tm.tm_min, tm.tm_sec);
al.is_dst = tm.tm_isdst > 0;
return al;
}
time_zone::civil_lookup TimeZoneLibC::MakeTime(const civil_second& cs) const {
time_zone::civil_lookup cl;
std::time_t t;
if (local_) {
// Does not handle SKIPPED/AMBIGUOUS or huge years.
std::tm tm;
tm.tm_year = static_cast<int>(cs.year() - 1900);
tm.tm_mon = cs.month() - 1;
tm.tm_mday = cs.day();
tm.tm_hour = cs.hour();
tm.tm_min = cs.minute();
tm.tm_sec = cs.second();
tm.tm_isdst = -1;
t = std::mktime(&tm);
} else {
t = cs - civil_second();
}
cl.kind = time_zone::civil_lookup::UNIQUE;
cl.pre = cl.trans = cl.post = FromUnixSeconds(t);
return cl;
}
} // namespace cctz

45
contrib/libcctz/src/time_zone_libc.h
View File

@ -1,45 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef CCTZ_TIME_ZONE_LIBC_H_
#define CCTZ_TIME_ZONE_LIBC_H_
#include <cstdint>
#include <string>
#include "time_zone_if.h"
namespace cctz {
// A time zone backed by gmtime_r(3), localtime_r(3), and mktime(3), and
// which therefore only supports "localtime" and fixed offsets from UTC.
class TimeZoneLibC : public TimeZoneIf {
public:
explicit TimeZoneLibC(const std::string& name);
// TimeZoneIf implementations.
time_zone::absolute_lookup BreakTime(
const time_point<sys_seconds>& tp) const override;
time_zone::civil_lookup MakeTime(
const civil_second& cs) const override;
private:
bool local_; // localtime or UTC
int offset_; // UTC offset when !local_
std::string abbr_; // abbreviation when !local_
};
} // namespace cctz
#endif // CCTZ_TIME_ZONE_LIBC_H_

69
contrib/libcctz/src/time_zone_lookup.cc
View File

@ -1,69 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "time_zone.h"
#include <cstdlib>
#include "time_zone_impl.h"
namespace cctz {
std::string time_zone::name() const {
return time_zone::Impl::get(*this).name();
}
time_zone::absolute_lookup time_zone::lookup(
const time_point<sys_seconds>& tp) const {
return time_zone::Impl::get(*this).BreakTime(tp);
}
time_zone::civil_lookup time_zone::lookup(const civil_second& cs) const {
return time_zone::Impl::get(*this).MakeTime(cs);
}
bool operator==(time_zone lhs, time_zone rhs) {
return &time_zone::Impl::get(lhs) == &time_zone::Impl::get(rhs);
}
bool load_time_zone(const std::string& name, time_zone* tz) {
return time_zone::Impl::LoadTimeZone(name, tz);
}
time_zone utc_time_zone() {
return time_zone::Impl::UTC();
}
time_zone local_time_zone() {
#if defined(_MSC_VER)
char* tz_env = nullptr;
_dupenv_s(&tz_env, nullptr, "TZ");
const char* zone = tz_env;
#else
const char* zone = std::getenv("TZ");
#endif
if (zone != nullptr) {
if (*zone == ':') ++zone;
} else {
zone = "localtime";
}
time_zone tz;
load_time_zone(zone, &tz);
#if defined(_MSC_VER)
free(tz_env);
#endif
return tz;
}
} // namespace cctz

150
contrib/libcctz/src/time_zone_posix.cc
View File

@ -1,150 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "time_zone_posix.h"
#include <cstring>
#include <limits>
#include <string>
namespace cctz {
namespace {
const char kDigits[] = "0123456789";
const char* ParseInt(const char* p, int min, int max, int* vp) {
int value = 0;
const char* op = p;
const int kMaxInt = std::numeric_limits<int>::max();
for (; const char* dp = strchr(kDigits, *p); ++p) {
int d = static_cast<int>(dp - kDigits);
if (d >= 10) break; // '\0'
if (value > kMaxInt / 10) return nullptr;
value *= 10;
if (value > kMaxInt - d) return nullptr;
value += d;
}
if (p == op || value < min || value > max) return nullptr;
*vp = value;
return p;
}
// abbr = <.*?> | [^-+,\d]{3,}
const char* ParseAbbr(const char* p, std::string* abbr) {
const char* op = p;
if (*p == '<') { // special zoneinfo <...> form
while (*++p != '>') {
if (*p == '\0') return nullptr;
}
abbr->assign(op + 1, p - op - 1);
return ++p;
}
while (*p != '\0') {
if (strchr("-+,", *p)) break;
if (strchr(kDigits, *p)) break;
++p;
}
if (p - op < 3) return nullptr;
abbr->assign(op, p - op);
return p;
}
// offset = [+|-]hh[:mm[:ss]] (aggregated into single seconds value)
const char* ParseOffset(const char* p, int min_hour, int max_hour, int sign,
std::int_fast32_t* offset) {
if (p == nullptr) return nullptr;
if (*p == '+' || *p == '-') {
if (*p++ == '-') sign = -sign;
}
int hours = 0;
int minutes = 0;
int seconds = 0;
p = ParseInt(p, min_hour, max_hour, &hours);
if (p == nullptr) return nullptr;
if (*p == ':') {
p = ParseInt(p + 1, 0, 59, &minutes);
if (p == nullptr) return nullptr;
if (*p == ':') {
p = ParseInt(p + 1, 0, 59, &seconds);
if (p == nullptr) return nullptr;
}
}
*offset = sign * ((((hours * 60) + minutes) * 60) + seconds);
return p;
}
// datetime = ( Jn | n | Mm.w.d ) [ / offset ]
const char* ParseDateTime(const char* p, PosixTransition* res) {
if (p != nullptr && *p == ',') {
if (*++p == 'M') {
int month = 0;
if ((p = ParseInt(p + 1, 1, 12, &month)) != nullptr && *p == '.') {
int week = 0;
if ((p = ParseInt(p + 1, 1, 5, &week)) != nullptr && *p == '.') {
int weekday = 0;
if ((p = ParseInt(p + 1, 0, 6, &weekday)) != nullptr) {
res->date.fmt = PosixTransition::M;
res->date.m.month = static_cast<int_fast8_t>(month);
res->date.m.week = static_cast<int_fast8_t>(week);
res->date.m.weekday = static_cast<int_fast8_t>(weekday);
}
}
}
} else if (*p == 'J') {
int day = 0;
if ((p = ParseInt(p + 1, 1, 365, &day)) != nullptr) {
res->date.fmt = PosixTransition::J;
res->date.j.day = static_cast<int_fast16_t>(day);
}
} else {
int day = 0;
if ((p = ParseInt(p, 0, 365, &day)) != nullptr) {
res->date.fmt = PosixTransition::N;
res->date.j.day = static_cast<int_fast16_t>(day);
}
}
}
if (p != nullptr) {
res->time.offset = 2 * 60 * 60; // default offset is 02:00:00
if (*p == '/') p = ParseOffset(p + 1, -167, 167, 1, &res->time.offset);
}
return p;
}
} // namespace
// spec = std offset [ dst [ offset ] , datetime , datetime ]
bool ParsePosixSpec(const std::string& spec, PosixTimeZone* res) {
const char* p = spec.c_str();
if (*p == ':') return false;
p = ParseAbbr(p, &res->std_abbr);
p = ParseOffset(p, 0, 24, -1, &res->std_offset);
if (p == nullptr) return false;
if (*p == '\0') return true;
p = ParseAbbr(p, &res->dst_abbr);
if (p == nullptr) return false;
res->dst_offset = res->std_offset + (60 * 60); // default
if (*p != ',') p = ParseOffset(p, 0, 24, -1, &res->dst_offset);
p = ParseDateTime(p, &res->dst_start);
p = ParseDateTime(p, &res->dst_end);
return p != nullptr && *p == '\0';
}
} // namespace cctz

114
contrib/libcctz/src/time_zone_posix.h
View File

@ -1,114 +0,0 @@
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Parsing of a POSIX zone spec as described in the TZ part of section 8.3 in
// http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html.
//
// The current POSIX spec for America/Los_Angeles is "PST8PDT,M3.2.0,M11.1.0",
// which would be broken down as ...
//
// PosixTimeZone {
// std_abbr = "PST"
// std_offset = -28800
// dst_abbr = "PDT"
// dst_offset = -25200
// dst_start = PosixTransition {
// date {
// m {
// month = 3
// week = 2
// weekday = 0
// }
// }
// time {
// offset = 7200
// }
// }
// dst_end = PosixTransition {
// date {
// m {
// month = 11
// week = 1
// weekday = 0
// }
// }
// time {
// offset = 7200
// }
// }
// }
#ifndef CCTZ_TIME_ZONE_POSIX_H_
#define CCTZ_TIME_ZONE_POSIX_H_
#include <cstdint>
#include <string>
namespace cctz {
// The date/time of the transition. The date is specified as either:
// (J) the Nth day of the year (1 <= N <= 365), excluding leap days, or
// (N) the Nth day of the year (0 <= N <= 365), including leap days, or
// (M) the Nth weekday of a month (e.g., the 2nd Sunday in March).
// The time, specified as a day offset, identifies the particular moment
// of the transition, and may be negative or >= 24h, and in which case
// it would take us to another day, and perhaps week, or even month.
struct PosixTransition {
enum DateFormat { J, N, M };
struct {
DateFormat fmt;
union {
struct {
std::int_fast16_t day; // day of non-leap year [1:365]
} j;
struct {
std::int_fast16_t day; // day of year [0:365]
} n;
struct {
std::int_fast8_t month; // month of year [1:12]
std::int_fast8_t week; // week of month [1:5] (5==last)
std::int_fast8_t weekday; // 0==Sun, ..., 6=Sat
} m;
};
} date;
struct {
std::int_fast32_t offset; // seconds before/after 00:00:00
} time;
};
// The entirety of a POSIX-string specified time-zone rule. The standard
// abbreviation and offset are always given. If the time zone includes
// daylight saving, then the daylight abbrevation is non-empty and the
// remaining fields are also valid. Note that the start/end transitions
// are not ordered---in the southern hemisphere the transition to end
// daylight time occurs first in any particular year.
struct PosixTimeZone {
std::string std_abbr;
std::int_fast32_t std_offset;
std::string dst_abbr;
std::int_fast32_t dst_offset;
PosixTransition dst_start;
PosixTransition dst_end;
};
// Breaks down a POSIX time-zone specification into its constituent pieces,
// filling in any missing values (DST offset, or start/end transition times)
// with the standard-defined defaults. Returns false if the specification
// could not be parsed (although some fields of *res may have been altered).
bool ParsePosixSpec(const std::string& spec, PosixTimeZone* res);
} // namespace cctz
#endif // CCTZ_TIME_ZONE_POSIX_H_

169
contrib/libcctz/src/tzfile.h
View File

@ -1,169 +0,0 @@
#ifndef TZFILE_H
#define TZFILE_H
/*
** This file is in the public domain, so clarified as of
** 1996-06-05 by Arthur David Olson.
*/
/*
** This header is for use ONLY with the time conversion code.
** There is no guarantee that it will remain unchanged,
** or that it will remain at all.
** Do NOT copy it to any system include directory.
** Thank you!
*/
/*
** Information about time zone files.
*/
#ifndef TZDIR
#define TZDIR "/usr/local/etc/zoneinfo" /* Time zone object file directory */
#endif /* !defined TZDIR */
#ifndef TZDEFAULT
#define TZDEFAULT "localtime"
#endif /* !defined TZDEFAULT */
#ifndef TZDEFRULES
#define TZDEFRULES "posixrules"
#endif /* !defined TZDEFRULES */
/*
** Each file begins with. . .
*/
#define TZ_MAGIC "TZif"
struct tzhead {
char tzh_magic[4]; /* TZ_MAGIC */
char tzh_version[1]; /* '\0' or '2' or '3' as of 2013 */
char tzh_reserved[15]; /* reserved; must be zero */
char tzh_ttisgmtcnt[4]; /* coded number of trans. time flags */
char tzh_ttisstdcnt[4]; /* coded number of trans. time flags */
char tzh_leapcnt[4]; /* coded number of leap seconds */
char tzh_timecnt[4]; /* coded number of transition times */
char tzh_typecnt[4]; /* coded number of local time types */
char tzh_charcnt[4]; /* coded number of abbr. chars */
};
/*
** . . .followed by. . .
**
** tzh_timecnt (char [4])s coded transition times a la time(2)
** tzh_timecnt (unsigned char)s types of local time starting at above
** tzh_typecnt repetitions of
** one (char [4]) coded UT offset in seconds
** one (unsigned char) used to set tm_isdst
** one (unsigned char) that's an abbreviation list index
** tzh_charcnt (char)s '\0'-terminated zone abbreviations
** tzh_leapcnt repetitions of
** one (char [4]) coded leap second transition times
** one (char [4]) total correction after above
** tzh_ttisstdcnt (char)s indexed by type; if 1, transition
** time is standard time, if 0,
** transition time is wall clock time
** if absent, transition times are
** assumed to be wall clock time
** tzh_ttisgmtcnt (char)s indexed by type; if 1, transition
** time is UT, if 0,
** transition time is local time
** if absent, transition times are
** assumed to be local time
*/
/*
** If tzh_version is '2' or greater, the above is followed by a second instance
** of tzhead and a second instance of the data in which each coded transition
** time uses 8 rather than 4 chars,
** then a POSIX-TZ-environment-variable-style string for use in handling
** instants after the last transition time stored in the file
** (with nothing between the newlines if there is no POSIX representation for
** such instants).
**
** If tz_version is '3' or greater, the above is extended as follows.
** First, the POSIX TZ string's hour offset may range from -167
** through 167 as compared to the POSIX-required 0 through 24.
** Second, its DST start time may be January 1 at 00:00 and its stop
** time December 31 at 24:00 plus the difference between DST and
** standard time, indicating DST all year.
*/
/*
** In the current implementation, "tzset()" refuses to deal with files that
** exceed any of the limits below.
*/
#ifndef TZ_MAX_TIMES
#define TZ_MAX_TIMES 2000
#endif /* !defined TZ_MAX_TIMES */
#ifndef TZ_MAX_TYPES
/* This must be at least 17 for Europe/Samara and Europe/Vilnius. */
#define TZ_MAX_TYPES 256 /* Limited by what (unsigned char)'s can hold */
#endif /* !defined TZ_MAX_TYPES */
#ifndef TZ_MAX_CHARS
#define TZ_MAX_CHARS 50 /* Maximum number of abbreviation characters */
/* (limited by what unsigned chars can hold) */
#endif /* !defined TZ_MAX_CHARS */
#ifndef TZ_MAX_LEAPS
#define TZ_MAX_LEAPS 50 /* Maximum number of leap second corrections */
#endif /* !defined TZ_MAX_LEAPS */
#define SECSPERMIN 60
#define MINSPERHOUR 60
#define HOURSPERDAY 24
#define DAYSPERWEEK 7
#define DAYSPERNYEAR 365
#define DAYSPERLYEAR 366
#define SECSPERHOUR (SECSPERMIN * MINSPERHOUR)
#define SECSPERDAY ((int_fast32_t) SECSPERHOUR * HOURSPERDAY)
#define MONSPERYEAR 12
#define TM_SUNDAY 0
#define TM_MONDAY 1
#define TM_TUESDAY 2
#define TM_WEDNESDAY 3
#define TM_THURSDAY 4
#define TM_FRIDAY 5
#define TM_SATURDAY 6
#define TM_JANUARY 0
#define TM_FEBRUARY 1
#define TM_MARCH 2
#define TM_APRIL 3
#define TM_MAY 4
#define TM_JUNE 5
#define TM_JULY 6
#define TM_AUGUST 7
#define TM_SEPTEMBER 8
#define TM_OCTOBER 9
#define TM_NOVEMBER 10
#define TM_DECEMBER 11
#define TM_YEAR_BASE 1900
#define EPOCH_YEAR 1970
#define EPOCH_WDAY TM_THURSDAY
#define isleap(y) (((y) % 4) == 0 && (((y) % 100) != 0 || ((y) % 400) == 0))
/*
** Since everything in isleap is modulo 400 (or a factor of 400), we know that
** isleap(y) == isleap(y % 400)
** and so
** isleap(a + b) == isleap((a + b) % 400)
** or
** isleap(a + b) == isleap(a % 400 + b % 400)
** This is true even if % means modulo rather than Fortran remainder
** (which is allowed by C89 but not C99).
** We use this to avoid addition overflow problems.
*/
#define isleap_sum(a, b) isleap((a) % 400 + (b) % 400)
#endif /* !defined TZFILE_H */

2
contrib/libcityhash/CMakeLists.txt
View File

@ -5,5 +5,5 @@ add_library(cityhash
include/city.h
src/config.h)
target_include_directories (cityhash PUBLIC include)
target_include_directories (cityhash BEFORE PUBLIC include)
target_include_directories (cityhash PRIVATE src)

2
contrib/poco vendored

@ -1 +1 @@
Subproject commit ad1643c6698a8c890b68186d5c9d72e496c27af2
Subproject commit e30352c2c24eebecbd82d41f7054d908ac7fdc37

2
contrib/zookeeper vendored

@ -1 +1 @@
Subproject commit 7652f34ddec8e6aef5e0beb4a0361667e8bbb402
Subproject commit d2f05a6946820a6e1a3ba326da446f0a8f9546ed

2
contrib/zstd vendored

@ -1 +1 @@
Subproject commit aecf3b479c45affa9fd8ead068e9160253a8ec5c
Subproject commit f4340f46b2387bc8de7d5320c0b83bb1499933ad

11
contrib/zstd-cmake/CMakeLists.txt
View File

@ -60,7 +60,12 @@ SET(Sources
${LIBRARY_DIR}/compress/fse_compress.c
${LIBRARY_DIR}/compress/huf_compress.c
${LIBRARY_DIR}/compress/zstd_compress.c
${LIBRARY_DIR}/compress/zstd_double_fast.c
${LIBRARY_DIR}/compress/zstd_fast.c
${LIBRARY_DIR}/compress/zstd_lazy.c
${LIBRARY_DIR}/compress/zstd_ldm.c
${LIBRARY_DIR}/compress/zstdmt_compress.c
${LIBRARY_DIR}/compress/zstd_opt.c
${LIBRARY_DIR}/decompress/huf_decompress.c
${LIBRARY_DIR}/decompress/zstd_decompress.c
${LIBRARY_DIR}/deprecated/zbuff_common.c
@ -81,8 +86,14 @@ SET(Headers
${LIBRARY_DIR}/common/xxhash.h
${LIBRARY_DIR}/common/zstd_errors.h
${LIBRARY_DIR}/common/zstd_internal.h
${LIBRARY_DIR}/compress/zstd_compress.h
${LIBRARY_DIR}/compress/zstd_double_fast.h
${LIBRARY_DIR}/compress/zstd_fast.h
${LIBRARY_DIR}/compress/zstd_lazy.h
${LIBRARY_DIR}/compress/zstd_ldm.h
${LIBRARY_DIR}/compress/zstdmt_compress.h
${LIBRARY_DIR}/compress/zstd_opt.h
${LIBRARY_DIR}/compress/zstd_ldm.h
${LIBRARY_DIR}/deprecated/zbuff.h
${LIBRARY_DIR}/dictBuilder/divsufsort.h
${LIBRARY_DIR}/dictBuilder/zdict.h

9
dbms/CMakeLists.txt
View File

@ -42,7 +42,6 @@ add_headers_and_sources(dbms src/Core)
add_headers_and_sources(dbms src/DataStreams)
add_headers_and_sources(dbms src/DataTypes)
add_headers_and_sources(dbms src/Databases)
add_headers_and_sources(dbms src/DataBases/Distributed)
add_headers_and_sources(dbms src/Dictionaries)
add_headers_and_sources(dbms src/Dictionaries/Embedded)
add_headers_and_sources(dbms src/Interpreters)
@ -93,8 +92,6 @@ list (APPEND dbms_headers src/TableFunctions/ITableFunction.h src/TableFunctions
list(REMOVE_ITEM dbms_sources
src/Storages/StorageCloud.cpp
src/Databases/DatabaseCloud.cpp
src/Common/StringUtils.cpp)
if (APPLE OR CMAKE_SYSTEM MATCHES "FreeBSD")
@ -116,7 +113,7 @@ else ()
endif ()
if (NOT CMAKE_BUILD_TYPE_UC STREQUAL "DEBUG")
if (CMAKE_BUILD_TYPE_UC STREQUAL "RELEASE" OR CMAKE_BUILD_TYPE_UC STREQUAL "RELWITHDEBINFO" OR CMAKE_BUILD_TYPE_UC STREQUAL "MINSIZEREL")
# Won't generate debug info for files with heavy template instantiation to achieve faster linking and lower size.
set_source_files_properties(
src/Dictionaries/FlatDictionary.cpp
@ -185,6 +182,10 @@ if (USE_ICU)
target_link_libraries (dbms ${ICU_LIBS})
endif ()
if (USE_CAPNP)
target_link_libraries (dbms ${CAPNP_LIBS})
endif ()
target_link_libraries (dbms
${PLATFORM_LIBS}
${CMAKE_DL_LIBS}

2
dbms/benchmark/greenplum/benchmark.sh
View File

@ -7,7 +7,7 @@ orca=${4-on}
host1=somehost
host2=somehost
mem='15GB'
cat $filename | sed "s/{table}/$table/g" | while read query ;
cat $filename | sed "s/{table}/$table/g" | while read query ;
do
ssh -n $host1 'echo 3 | tee /proc/sys/vm/drop_caches; sync' > /dev/null
ssh -n $host2 'echo 3 | tee /proc/sys/vm/drop_caches; sync' > /dev/null

2
dbms/benchmark/hive/run_hive.sh
View File

@ -1,2 +1,2 @@
cd /home/kartavyy/benchmark
./benchmark.sh -c hive/conf.sh -n $1 > hive/log/log_$1
./benchmark.sh -c hive/conf.sh -n $1 > hive/log/log_$1

4
dbms/cmake/version.cmake
View File

@ -1,6 +1,6 @@
# This strings autochanged from release_lib.sh:
set(VERSION_DESCRIBE v1.1.54300-testing)
set(VERSION_REVISION 54300)
set(VERSION_DESCRIBE v1.1.54310-testing)
set(VERSION_REVISION 54310)
# end of autochange
set (VERSION_MAJOR 1)

2
dbms/src/AggregateFunctions/AggregateFunctionForEach.h
View File

@ -92,7 +92,7 @@ private:
}
public:
AggregateFunctionForEach(AggregateFunctionPtr nested_)
explicit AggregateFunctionForEach(AggregateFunctionPtr nested_)
: nested_func_owner(nested_), nested_func(nested_func_owner.get())
{
}

14
dbms/src/AggregateFunctions/AggregateFunctionSum.cpp
View File

@ -9,6 +9,19 @@ namespace
{
AggregateFunctionPtr createAggregateFunctionSum(const std::string & name, const DataTypes & argument_types, const Array & parameters)
{
if (argument_types.size() != 1)
throw Exception("Incorrect number of arguments for aggregate function " + name, ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
AggregateFunctionPtr res(createWithNumericTypeNearest<AggregateFunctionSum>(*argument_types[0]));
if (!res)
throw Exception("Illegal type " + argument_types[0]->getName() + " of argument for aggregate function " + name, ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
return res;
}
AggregateFunctionPtr createAggregateFunctionSumWithOverflow(const std::string & name, const DataTypes & argument_types, const Array & parameters)
{
if (argument_types.size() != 1)
throw Exception("Incorrect number of arguments for aggregate function " + name, ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
@ -26,6 +39,7 @@ AggregateFunctionPtr createAggregateFunctionSum(const std::string & name, const
void registerAggregateFunctionSum(AggregateFunctionFactory & factory)
{
factory.registerFunction("sum", createAggregateFunctionSum, AggregateFunctionFactory::CaseInsensitive);
factory.registerFunction("sumWithOverflow", createAggregateFunctionSumWithOverflow);
}
}

8
dbms/src/AggregateFunctions/AggregateFunctionSum.h
View File

@ -20,15 +20,15 @@ struct AggregateFunctionSumData
/// Counts the sum of the numbers.
template <typename T>
class AggregateFunctionSum final : public IUnaryAggregateFunction<AggregateFunctionSumData<typename NearestFieldType<T>::Type>, AggregateFunctionSum<T>>
template <typename T, typename TResult = T>
class AggregateFunctionSum final : public IUnaryAggregateFunction<AggregateFunctionSumData<TResult>, AggregateFunctionSum<T, TResult>>
{
public:
String getName() const override { return "sum"; }
DataTypePtr getReturnType() const override
{
return std::make_shared<DataTypeNumber<typename NearestFieldType<T>::Type>>();
return std::make_shared<DataTypeNumber<TResult>>();
}
void setArgument(const DataTypePtr & argument)
@ -61,7 +61,7 @@ public:
void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override
{
static_cast<ColumnVector<typename NearestFieldType<T>::Type> &>(to).getData().push_back(this->data(place).sum);
static_cast<ColumnVector<TResult> &>(to).getData().push_back(this->data(place).sum);
}
const char * getHeaderFilePath() const override { return __FILE__; }

22
dbms/src/AggregateFunctions/AggregateFunctionSumMap.h
View File

@ -198,7 +198,27 @@ public:
void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override
{
const auto & merged_maps = this->data(place).merged_maps;
// Final step does compaction of keys that have zero values, this mutates the state
auto & merged_maps = this->data(const_cast<AggregateDataPtr>(place)).merged_maps;
for (auto it = merged_maps.cbegin(); it != merged_maps.cend();)
{
// Key is not compacted if it has at least one non-zero value
bool erase = true;
for (size_t col = 0; col < values_types.size(); ++col)
{
if (it->second[col] != values_types[col]->getDefault())
{
erase = false;
break;
}
}
if (erase)
it = merged_maps.erase(it);
else
++it;
}
size_t size = merged_maps.size();
auto & to_cols = static_cast<ColumnTuple &>(to).getColumns();

22
dbms/src/AggregateFunctions/Helpers.h
View File

@ -15,7 +15,7 @@ namespace DB
/** Create an aggregate function with a numeric type in the template parameter, depending on the type of the argument.
*/
template <template <typename> class AggregateFunctionTemplate>
template <template <typename, typename ... TArgs> class AggregateFunctionTemplate>
static IAggregateFunction * createWithNumericType(const IDataType & argument_type)
{
if (typeid_cast<const DataTypeUInt8 *>(&argument_type)) return new AggregateFunctionTemplate<UInt8>;
@ -92,6 +92,26 @@ static IAggregateFunction * createWithNumericType(const IDataType & argument_typ
}
template <template <typename, typename> class AggregateFunctionTemplate>
static IAggregateFunction * createWithNumericTypeNearest(const IDataType & argument_type)
{
if (typeid_cast<const DataTypeUInt8 *>(&argument_type)) return new AggregateFunctionTemplate<UInt8, NearestFieldType<UInt8>::Type>;
else if (typeid_cast<const DataTypeUInt16 *>(&argument_type)) return new AggregateFunctionTemplate<UInt16, NearestFieldType<UInt16>::Type>;
else if (typeid_cast<const DataTypeUInt32 *>(&argument_type)) return new AggregateFunctionTemplate<UInt32, NearestFieldType<UInt32>::Type>;
else if (typeid_cast<const DataTypeUInt64 *>(&argument_type)) return new AggregateFunctionTemplate<UInt64, NearestFieldType<UInt64>::Type>;
else if (typeid_cast<const DataTypeInt8 *>(&argument_type)) return new AggregateFunctionTemplate<Int8, NearestFieldType<Int8>::Type>;
else if (typeid_cast<const DataTypeInt16 *>(&argument_type)) return new AggregateFunctionTemplate<Int16, NearestFieldType<Int16>::Type>;
else if (typeid_cast<const DataTypeInt32 *>(&argument_type)) return new AggregateFunctionTemplate<Int32, NearestFieldType<Int32>::Type>;
else if (typeid_cast<const DataTypeInt64 *>(&argument_type)) return new AggregateFunctionTemplate<Int64, NearestFieldType<Int64>::Type>;
else if (typeid_cast<const DataTypeFloat32 *>(&argument_type)) return new AggregateFunctionTemplate<Float32, NearestFieldType<Float32>::Type>;
else if (typeid_cast<const DataTypeFloat64 *>(&argument_type)) return new AggregateFunctionTemplate<Float64, NearestFieldType<Float64>::Type>;
else if (typeid_cast<const DataTypeEnum8 *>(&argument_type)) return new AggregateFunctionTemplate<UInt8, NearestFieldType<UInt8>::Type>;
else if (typeid_cast<const DataTypeEnum16 *>(&argument_type)) return new AggregateFunctionTemplate<UInt16, NearestFieldType<UInt16>::Type>;
else
return nullptr;
}
template <template <typename, typename> class AggregateFunctionTemplate, template <typename> class Data>
static IAggregateFunction * createWithUnsignedIntegerType(const IDataType & argument_type)
{

31
dbms/src/Client/Connection.cpp
View File

@ -1,13 +1,7 @@
#include <iomanip>
#include <Poco/Net/NetException.h>
#include <Poco/Net/SecureStreamSocket.h>
#include <Common/ClickHouseRevision.h>
#include <Core/Defines.h>
#include <Common/Exception.h>
#include <IO/CompressedReadBuffer.h>
#include <IO/CompressedWriteBuffer.h>
#include <IO/ReadBufferFromPocoSocket.h>
@ -15,17 +9,20 @@
#include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h>
#include <IO/copyData.h>
#include <DataStreams/NativeBlockInputStream.h>
#include <DataStreams/NativeBlockOutputStream.h>
#include <Client/Connection.h>
#include <Common/ClickHouseRevision.h>
#include <Common/Exception.h>
#include <Common/NetException.h>
#include <Common/CurrentMetrics.h>
#include <Interpreters/ClientInfo.h>
#include <Common/config.h>
#if Poco_NetSSL_FOUND
#include <Poco/Net/SecureStreamSocket.h>
#endif
namespace CurrentMetrics
{
@ -42,6 +39,7 @@ namespace ErrorCodes
extern const int SERVER_REVISION_IS_TOO_OLD;
extern const int UNEXPECTED_PACKET_FROM_SERVER;
extern const int UNKNOWN_PACKET_FROM_SERVER;
extern const int SUPPORT_IS_DISABLED;
}
@ -54,7 +52,18 @@ void Connection::connect()
LOG_TRACE(log_wrapper.get(), "Connecting. Database: " << (default_database.empty() ? "(not specified)" : default_database) << ". User: " << user);
socket = static_cast<bool>(encryption) ? std::make_unique<Poco::Net::SecureStreamSocket>() : std::make_unique<Poco::Net::StreamSocket>();
if (static_cast<bool>(encryption))
{
#if Poco_NetSSL_FOUND
socket = std::make_unique<Poco::Net::SecureStreamSocket>();
#else
throw Exception{"tcp_ssl protocol is disabled because poco library built without NetSSL support.", ErrorCodes::SUPPORT_IS_DISABLED};
#endif
}
else
{
socket = std::make_unique<Poco::Net::StreamSocket>();
}
socket->connect(resolved_address, connect_timeout);
socket->setReceiveTimeout(receive_timeout);
socket->setSendTimeout(send_timeout);

0
dbms/tests/integration/test_sync_insert_into_distributed/__init__.py → dbms/src/Columns/CMakeLists.txt
View File

23
dbms/src/Common/Allocator.cpp
View File

@ -1,3 +1,5 @@
#include <Common/Allocator.h>
#if !defined(__APPLE__) && !defined(__FreeBSD__)
#include <malloc.h>
#endif
@ -7,10 +9,10 @@
#include <Common/MemoryTracker.h>
#include <Common/Exception.h>
#include <Common/Allocator.h>
#include <Common/formatReadable.h>
#include <IO/WriteHelpers.h>
/// Required for older Darwin builds, that lack definition of MAP_ANONYMOUS
#ifndef MAP_ANONYMOUS
#define MAP_ANONYMOUS MAP_ANON
@ -54,11 +56,12 @@ void * Allocator<clear_memory_>::alloc(size_t size, size_t alignment)
if (size >= MMAP_THRESHOLD)
{
if (alignment > MMAP_MIN_ALIGNMENT)
throw DB::Exception("Too large alignment: more than page size.", DB::ErrorCodes::BAD_ARGUMENTS);
throw DB::Exception("Too large alignment " + formatReadableSizeWithBinarySuffix(alignment) + ": more than page size when allocating "
+ formatReadableSizeWithBinarySuffix(size) + ".", DB::ErrorCodes::BAD_ARGUMENTS);
buf = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (MAP_FAILED == buf)
DB::throwFromErrno("Allocator: Cannot mmap.", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
DB::throwFromErrno("Allocator: Cannot mmap " + formatReadableSizeWithBinarySuffix(size) + ".", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
/// No need for zero-fill, because mmap guarantees it.
}
@ -72,7 +75,7 @@ void * Allocator<clear_memory_>::alloc(size_t size, size_t alignment)
buf = ::malloc(size);
if (nullptr == buf)
DB::throwFromErrno("Allocator: Cannot malloc.", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
DB::throwFromErrno("Allocator: Cannot malloc " + formatReadableSizeWithBinarySuffix(size) + ".", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
}
else
{
@ -80,7 +83,7 @@ void * Allocator<clear_memory_>::alloc(size_t size, size_t alignment)
int res = posix_memalign(&buf, alignment, size);
if (0 != res)
DB::throwFromErrno("Cannot allocate memory (posix_memalign)", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY, res);
DB::throwFromErrno("Cannot allocate memory (posix_memalign) " + formatReadableSizeWithBinarySuffix(size) + ".", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY, res);
if (clear_memory)
memset(buf, 0, size);
@ -97,7 +100,7 @@ void Allocator<clear_memory_>::free(void * buf, size_t size)
if (size >= MMAP_THRESHOLD)
{
if (0 != munmap(buf, size))
DB::throwFromErrno("Allocator: Cannot munmap.", DB::ErrorCodes::CANNOT_MUNMAP);
DB::throwFromErrno("Allocator: Cannot munmap " + formatReadableSizeWithBinarySuffix(size) + ".", DB::ErrorCodes::CANNOT_MUNMAP);
}
else
{
@ -119,7 +122,7 @@ void * Allocator<clear_memory_>::realloc(void * buf, size_t old_size, size_t new
buf = ::realloc(buf, new_size);
if (nullptr == buf)
DB::throwFromErrno("Allocator: Cannot realloc.", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
DB::throwFromErrno("Allocator: Cannot realloc from " + formatReadableSizeWithBinarySuffix(old_size) + " to " + formatReadableSizeWithBinarySuffix(new_size) + ".", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
if (clear_memory)
memset(reinterpret_cast<char *>(buf) + old_size, 0, new_size - old_size);
@ -130,7 +133,7 @@ void * Allocator<clear_memory_>::realloc(void * buf, size_t old_size, size_t new
buf = mremap(buf, old_size, new_size, MREMAP_MAYMOVE);
if (MAP_FAILED == buf)
DB::throwFromErrno("Allocator: Cannot mremap memory chunk from " + DB::toString(old_size) + " to " + DB::toString(new_size) + " bytes.", DB::ErrorCodes::CANNOT_MREMAP);
DB::throwFromErrno("Allocator: Cannot mremap memory chunk from " + formatReadableSizeWithBinarySuffix(old_size) + " to " + formatReadableSizeWithBinarySuffix(new_size) + ".", DB::ErrorCodes::CANNOT_MREMAP);
/// No need for zero-fill, because mmap guarantees it.
}
@ -144,7 +147,7 @@ void * Allocator<clear_memory_>::realloc(void * buf, size_t old_size, size_t new
buf = ::realloc(buf, new_size);
if (nullptr == buf)
DB::throwFromErrno("Allocator: Cannot realloc.", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
DB::throwFromErrno("Allocator: Cannot realloc from " + formatReadableSizeWithBinarySuffix(old_size) + " to " + formatReadableSizeWithBinarySuffix(new_size) + ".", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
if (clear_memory)
memset(reinterpret_cast<char *>(buf) + old_size, 0, new_size - old_size);

5
dbms/src/Common/ArrayCache.h
View File

@ -15,6 +15,7 @@
#include <Common/Exception.h>
#include <Common/randomSeed.h>
#include <Common/formatReadable.h>
/// Required for older Darwin builds, that lack definition of MAP_ANONYMOUS
#ifndef MAP_ANONYMOUS
@ -172,13 +173,13 @@ private:
{
ptr = mmap(address_hint, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (MAP_FAILED == ptr)
DB::throwFromErrno("Allocator: Cannot mmap.", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
DB::throwFromErrno("Allocator: Cannot mmap " + formatReadableSizeWithBinarySuffix(size) + ".", DB::ErrorCodes::CANNOT_ALLOCATE_MEMORY);
}
~Chunk()
{
if (ptr && 0 != munmap(ptr, size))
DB::throwFromErrno("Allocator: Cannot munmap.", DB::ErrorCodes::CANNOT_MUNMAP);
DB::throwFromErrno("Allocator: Cannot munmap " + formatReadableSizeWithBinarySuffix(size) + ".", DB::ErrorCodes::CANNOT_MUNMAP);
}
Chunk(Chunk && other) : ptr(other.ptr), size(other.size)

2
dbms/src/Common/ExternalTable.h
View File

@ -197,7 +197,7 @@ public:
/// Create table
NamesAndTypesListPtr columns = std::make_shared<NamesAndTypesList>(sample_block.getColumnsList());
StoragePtr storage = StorageMemory::create(data.second, columns);
StoragePtr storage = StorageMemory::create(data.second, columns, NamesAndTypesList{}, NamesAndTypesList{}, ColumnDefaults{});
storage->startup();
context.addExternalTable(data.second, storage);
BlockOutputStreamPtr output = storage->write(ASTPtr(), context.getSettingsRef());

2
dbms/src/Common/LRUCache.h
View File

@ -165,7 +165,7 @@ private:
/// Represents pending insertion attempt.
struct InsertToken
{
InsertToken(LRUCache & cache_) : cache(cache_) {}
explicit InsertToken(LRUCache & cache_) : cache(cache_) {}
std::mutex mutex;
bool cleaned_up = false; /// Protected by the token mutex

11
dbms/src/Common/MemoryTracker.cpp
View File

@ -20,7 +20,16 @@ namespace DB
MemoryTracker::~MemoryTracker()
{
if (peak)
logPeakMemoryUsage();
{
try
{
logPeakMemoryUsage();
}
catch (...)
{
/// Exception in Logger, intentionally swallow.
}
}
/** This is needed for next memory tracker to be consistent with sum of all referring memory trackers.
*

2
dbms/src/Common/OptimizedRegularExpression.inl.h
View File

@ -228,7 +228,7 @@ void OptimizedRegularExpressionImpl<thread_safe>::analyze(
}
}
}
else
else if (!trivial_substrings.empty())
{
required_substring = trivial_substrings.front().first;
required_substring_is_prefix = trivial_substrings.front().second == 0;

52
dbms/src/Common/ZooKeeper/KeeperException.h
View File

@ -21,6 +21,27 @@ namespace ProfileEvents
namespace zkutil
{
/// You should reinitialize ZooKeeper session in case of these errors
inline bool isUnrecoverableErrorCode(int32_t zk_return_code)
{
return zk_return_code == ZINVALIDSTATE || zk_return_code == ZSESSIONEXPIRED || zk_return_code == ZSESSIONMOVED;
}
/// Errors related with temporary network problems
inline bool isTemporaryErrorCode(int32_t zk_return_code)
{
return zk_return_code == ZCONNECTIONLOSS || zk_return_code == ZOPERATIONTIMEOUT;
}
/// Any error related with network or master election
/// In case of these errors you should retry the query or reinitialize ZooKeeper session (see isUnrecoverable())
inline bool isHardwareErrorCode(int32_t zk_return_code)
{
return isUnrecoverableErrorCode(zk_return_code) || isTemporaryErrorCode(zk_return_code);
}
class KeeperException : public DB::Exception
{
private:
@ -29,35 +50,36 @@ private:
: DB::Exception(msg, DB::ErrorCodes::KEEPER_EXCEPTION), code(code) { incrementEventCounter(); }
public:
KeeperException(const std::string & msg) : KeeperException(msg, ZOK, 0) {}
explicit KeeperException(const std::string & msg) : KeeperException(msg, ZOK, 0) {}
KeeperException(const std::string & msg, const int32_t code)
: KeeperException(msg + " (" + zerror(code) + ")", code, 0) {}
KeeperException(const int32_t code) : KeeperException(zerror(code), code, 0) {}
explicit KeeperException(const int32_t code) : KeeperException(zerror(code), code, 0) {}
KeeperException(const int32_t code, const std::string & path)
: KeeperException(std::string{zerror(code)} + ", path: " + path, code, 0) {}
KeeperException(const KeeperException & exc) : DB::Exception(exc), code(exc.code) { incrementEventCounter(); }
const char * name() const throw() { return "zkutil::KeeperException"; }
const char * className() const throw() { return "zkutil::KeeperException"; }
KeeperException * clone() const { return new KeeperException(*this); }
const char * name() const throw() override { return "zkutil::KeeperException"; }
const char * className() const throw() override { return "zkutil::KeeperException"; }
KeeperException * clone() const override { return new KeeperException(*this); }
/// при этих ошибках надо переинициализировать сессию с zookeeper
/// You should reinitialize ZooKeeper session in case of these errors
bool isUnrecoverable() const
{
return code == ZINVALIDSTATE || code == ZSESSIONEXPIRED || code == ZSESSIONMOVED;
}
/// любая ошибка связанная с работой сети, перевыбором мастера
/// при этих ошибках надо либо повторить запрос повторно, либо переинициализировать сессию (см. isUnrecoverable())
bool isHardwareError() const
{
return isUnrecoverable() || code == ZCONNECTIONLOSS || code == ZOPERATIONTIMEOUT;
return isUnrecoverableErrorCode(code);
}
/// Errors related with temporary network problems
bool isTemporaryError() const
{
return code == ZCONNECTIONLOSS || code == ZOPERATIONTIMEOUT;
return isTemporaryErrorCode(code);
}
/// Any error related with network or master election
/// In case of these errors you should retry the query or reinitialize ZooKeeper session (see isUnrecoverable())
bool isHardwareError() const
{
return isHardwareErrorCode(code);
}
const int32_t code;

21
dbms/src/Common/ZooKeeper/ZooKeeper.cpp
View File

@ -71,7 +71,8 @@ void ZooKeeper::processCallback(zhandle_t * zh, int type, int state, const char
destroyContext(context);
}
void ZooKeeper::init(const std::string & hosts_, const std::string & identity_, int32_t session_timeout_ms_)
void ZooKeeper::init(const std::string & hosts_, const std::string & identity_,
int32_t session_timeout_ms_, bool check_root_exists)
{
log = &Logger::get("ZooKeeper");
zoo_set_debug_level(ZOO_LOG_LEVEL_ERROR);
@ -87,7 +88,7 @@ void ZooKeeper::init(const std::string & hosts_, const std::string & identity_,
if (!identity.empty())
{
auto code = zoo_add_auth(impl, "digest", identity.c_str(), static_cast<int>(identity.size()), 0, 0);
auto code = zoo_add_auth(impl, "digest", identity.c_str(), static_cast<int>(identity.size()), nullptr, nullptr);
if (code != ZOK)
throw KeeperException("Zookeeper authentication failed. Hosts are " + hosts, code);
@ -97,11 +98,15 @@ void ZooKeeper::init(const std::string & hosts_, const std::string & identity_,
default_acl = &ZOO_OPEN_ACL_UNSAFE;
LOG_TRACE(log, "initialized, hosts: " << hosts);
if (check_root_exists && !exists("/"))
throw KeeperException("Zookeeper root doesn't exist. You should create root node before start.");
}
ZooKeeper::ZooKeeper(const std::string & hosts, const std::string & identity, int32_t session_timeout_ms)
ZooKeeper::ZooKeeper(const std::string & hosts, const std::string & identity,
int32_t session_timeout_ms, bool check_root_exists)
{
init(hosts, identity, session_timeout_ms);
init(hosts, identity, session_timeout_ms, check_root_exists);
}
struct ZooKeeperArgs
@ -115,6 +120,7 @@ struct ZooKeeperArgs
std::string root;
session_timeout_ms = DEFAULT_SESSION_TIMEOUT;
has_chroot = false;
for (const auto & key : keys)
{
if (startsWith(key, "node"))
@ -154,19 +160,24 @@ struct ZooKeeperArgs
{
if (root.front() != '/')
throw KeeperException(std::string("Root path in config file should start with '/', but got ") + root);
if (root.back() == '/')
root.pop_back();
hosts += root;
has_chroot = true;
}
}
std::string hosts;
std::string identity;
int session_timeout_ms;
bool has_chroot;
};
ZooKeeper::ZooKeeper(const Poco::Util::AbstractConfiguration & config, const std::string & config_name)
{
ZooKeeperArgs args(config, config_name);
init(args.hosts, args.identity, args.session_timeout_ms);
init(args.hosts, args.identity, args.session_timeout_ms, args.has_chroot);
}
WatchCallback ZooKeeper::callbackForEvent(const EventPtr & event)

6
dbms/src/Common/ZooKeeper/ZooKeeper.h
View File

@ -54,7 +54,8 @@ class ZooKeeper
public:
using Ptr = std::shared_ptr<ZooKeeper>;
ZooKeeper(const std::string & hosts, const std::string & identity = "", int32_t session_timeout_ms = DEFAULT_SESSION_TIMEOUT);
ZooKeeper(const std::string & hosts, const std::string & identity = "",
int32_t session_timeout_ms = DEFAULT_SESSION_TIMEOUT, bool check_root_exists = false);
/** Config of the form:
<zookeeper>
@ -357,7 +358,8 @@ private:
friend struct WatchContext;
friend class EphemeralNodeHolder;
void init(const std::string & hosts, const std::string & identity, int32_t session_timeout_ms);
void init(const std::string & hosts, const std::string & identity,
int32_t session_timeout_ms, bool check_root_exists);
void removeChildrenRecursive(const std::string & path);
void tryRemoveChildrenRecursive(const std::string & path);

1
dbms/src/Common/config.h.in
View File

@ -7,6 +7,7 @@
#cmakedefine01 USE_RE2_ST
#cmakedefine01 USE_VECTORCLASS
#cmakedefine01 USE_RDKAFKA
#cmakedefine01 USE_CAPNP
#cmakedefine01 Poco_DataODBC_FOUND
#cmakedefine01 Poco_MongoDB_FOUND
#cmakedefine01 Poco_NetSSL_FOUND

2
dbms/src/Common/tests/Random.h
View File

@ -20,7 +20,7 @@ struct Rand
reseed(static_cast<uint32_t>(0));
}
Rand( uint32_t seed )
explicit Rand( uint32_t seed )
{
reseed(seed);
}

4
dbms/src/Common/tests/gtest_rw_lock_fifo.cpp.cpp
View File

@ -67,7 +67,7 @@ TEST(Common, RWLockFIFO_1)
Stopwatch watch(CLOCK_MONOTONIC_COARSE);
std::list<std::thread> threads;
for (int thread = 0; thread < pool_size; ++thread)
for (size_t thread = 0; thread < pool_size; ++thread)
threads.emplace_back([=] () { func(pool_size, round); });
for (auto & thread : threads)
@ -139,7 +139,7 @@ TEST(Common, RWLockFIFO_PerfTest_Readers)
};
std::list<std::thread> threads;
for (int thread = 0; thread < pool_size; ++thread)
for (size_t thread = 0; thread < pool_size; ++thread)
threads.emplace_back(func);
for (auto & thread : threads)

2
dbms/src/Core/ErrorCodes.cpp
View File

@ -384,6 +384,8 @@ namespace ErrorCodes
extern const int UNKNOWN_STATUS_OF_DISTRIBUTED_DDL_TASK = 379;
extern const int CANNOT_KILL = 380;
extern const int HTTP_LENGTH_REQUIRED = 381;
extern const int CANNOT_LOAD_CATBOOST_MODEL = 382;
extern const int CANNOT_APPLY_CATBOOST_MODEL = 383;
extern const int KEEPER_EXCEPTION = 999;
extern const int POCO_EXCEPTION = 1000;

197
dbms/src/DataStreams/CapnProtoRowInputStream.cpp Normal file
View File

@ -0,0 +1,197 @@
#if USE_CAPNP
#include <Core/Block.h>
#include <IO/ReadBuffer.h>
#include <DataStreams/CapnProtoRowInputStream.h>
#include <capnp/serialize.h>
#include <capnp/dynamic.h>
#include <boost/algorithm/string.hpp>
#include <boost/range/join.hpp>
#include <common/logger_useful.h>
namespace DB
{
CapnProtoRowInputStream::NestedField split(const Block & sample, size_t i)
{
CapnProtoRowInputStream::NestedField field = {{}, i};
// Remove leading dot in field definition, e.g. ".msg" -> "msg"
String name(sample.safeGetByPosition(i).name);
if (name.size() > 0 && name[0] == '.')
name.erase(0, 1);
boost::split(field.tokens, name, boost::is_any_of("."));
return field;
}
Field convertNodeToField(capnp::DynamicValue::Reader value)
{
switch (value.getType()) {
case capnp::DynamicValue::UNKNOWN:
throw Exception("Unknown field type");
case capnp::DynamicValue::VOID:
return Field();
case capnp::DynamicValue::BOOL:
return UInt64(value.as<bool>() ? 1 : 0);
case capnp::DynamicValue::INT:
return Int64((value.as<int64_t>()));
case capnp::DynamicValue::UINT:
return UInt64(value.as<uint64_t>());
case capnp::DynamicValue::FLOAT:
return Float64(value.as<double>());
case capnp::DynamicValue::TEXT:
{
auto arr = value.as<capnp::Text>();
return String(arr.begin(), arr.size());
}
case capnp::DynamicValue::DATA:
{
auto arr = value.as<capnp::Data>().asChars();
return String(arr.begin(), arr.size());
}
case capnp::DynamicValue::LIST:
{
auto listValue = value.as<capnp::DynamicList>();
Array res(listValue.size());
for (auto i : kj::indices(listValue))
res[i] = convertNodeToField(listValue[i]);
return res;
}
case capnp::DynamicValue::ENUM:
return UInt64(value.as<capnp::DynamicEnum>().getRaw());
case capnp::DynamicValue::STRUCT:
throw Exception("STRUCT type not supported, read individual fields instead");
case capnp::DynamicValue::CAPABILITY:
throw Exception("CAPABILITY type not supported");
case capnp::DynamicValue::ANY_POINTER:
throw Exception("ANY_POINTER type not supported");
}
}
capnp::StructSchema::Field getFieldOrThrow(capnp::StructSchema node, const std::string & field)
{
KJ_IF_MAYBE(child, node.findFieldByName(field))
return *child;
else
throw Exception("Field " + field + " doesn't exist in schema.");
}
void CapnProtoRowInputStream::createActions(const NestedFieldList & sortedFields, capnp::StructSchema reader)
{
String last;
size_t level = 0;
capnp::StructSchema::Field parent;
for (const auto & field : sortedFields)
{
// Move to a different field in the same structure, keep parent
if (level > 0 && field.tokens[level - 1] != last)
{
auto child = getFieldOrThrow(parent.getContainingStruct(), field.tokens[level - 1]);
reader = child.getType().asStruct();
actions.push_back({Action::POP});
actions.push_back({Action::PUSH, child});
}
// Descend to a nested structure
for (; level < field.tokens.size() - 1; ++level)
{
last = field.tokens[level];
parent = getFieldOrThrow(reader, last);
reader = parent.getType().asStruct();
actions.push_back({Action::PUSH, parent});
}
// Read field from the structure
actions.push_back({Action::READ, getFieldOrThrow(reader, field.tokens[level]), field.pos});
}
}
CapnProtoRowInputStream::CapnProtoRowInputStream(ReadBuffer & istr_, const Block & sample_, const String & schema_file, const String & root_object)
: istr(istr_), sample(sample_), parser(std::make_shared<SchemaParser>())
{
// Parse the schema and fetch the root object
auto schema = parser->impl.parseDiskFile(schema_file, schema_file, {});
root = schema.getNested(root_object).asStruct();
/**
* The schema typically consists of fields in various nested structures.
* Here we gather the list of fields and sort them in a way so that fields in the same structur are adjacent,
* and the nesting level doesn't decrease to make traversal easier.
*/
NestedFieldList list;
size_t columns = sample.columns();
for (size_t i = 0; i < columns; ++i)
list.push_back(split(sample, i));
// Reorder list to make sure we don't have to backtrack
std::sort(list.begin(), list.end(), [](const NestedField & a, const NestedField & b)
{
if (a.tokens.size() == b.tokens.size())
return a.tokens < b.tokens;
return a.tokens.size() < b.tokens.size();
});
createActions(list, root);
}
bool CapnProtoRowInputStream::read(Block & block)
{
if (istr.eof())
return false;
// Read from underlying buffer directly
auto buf = istr.buffer();
auto base = reinterpret_cast<const capnp::word *>(istr.position());
// Check if there's enough bytes in the buffer to read the full message
kj::Array<capnp::word> heap_array;
auto array = kj::arrayPtr(base, buf.size() - istr.offset());
auto expected_words = capnp::expectedSizeInWordsFromPrefix(array);
if (expected_words * sizeof(capnp::word) > array.size())
{
// We'll need to reassemble the message in a contiguous buffer
heap_array = kj::heapArray<capnp::word>(expected_words);
istr.readStrict(heap_array.asChars().begin(), heap_array.asChars().size());
array = heap_array.asPtr();
}
capnp::FlatArrayMessageReader msg(array);
std::vector<capnp::DynamicStruct::Reader> stack;
stack.push_back(msg.getRoot<capnp::DynamicStruct>(root));
for (auto action : actions)
{
switch (action.type) {
case Action::READ: {
auto & col = block.getByPosition(action.column);
Field value = convertNodeToField(stack.back().get(action.field));
col.column->insert(value);
break;
}
case Action::POP:
stack.pop_back();
break;
case Action::PUSH:
stack.push_back(stack.back().get(action.field).as<capnp::DynamicStruct>());
break;
}
}
// Advance buffer position if used directly
if (heap_array.size() == 0)
{
auto parsed = (msg.getEnd() - base) * sizeof(capnp::word);
istr.position() += parsed;
}
return true;
}
}
#endif

68
dbms/src/DataStreams/CapnProtoRowInputStream.h Normal file
View File

@ -0,0 +1,68 @@
#pragma once
#include <Core/Block.h>
#include <DataStreams/IRowInputStream.h>
#include <capnp/schema-parser.h>
namespace DB
{
class ReadBuffer;
/** A stream for reading messages in Cap'n Proto format in given schema.
* Like Protocol Buffers and Thrift (but unlike JSON or MessagePack),
* Cap'n Proto messages are strongly-typed and not self-describing.
* The schema in this case cannot be compiled in, so it uses a runtime schema parser.
* See https://capnproto.org/cxx.html
*/
class CapnProtoRowInputStream : public IRowInputStream
{
public:
struct NestedField
{
std::vector<std::string> tokens;
size_t pos;
};
using NestedFieldList = std::vector<NestedField>;
/** schema_file - location of the capnproto schema, e.g. "schema.canpn"
* root_object - name to the root object, e.g. "Message"
*/
CapnProtoRowInputStream(ReadBuffer & istr_, const Block & sample_, const String & schema_file, const String & root_object);
bool read(Block & block) override;
private:
// Build a traversal plan from a sorted list of fields
void createActions(const NestedFieldList & sortedFields, capnp::StructSchema reader);
/* Action for state machine for traversing nested structures. */
struct Action
{
enum Type { POP, PUSH, READ };
Type type;
capnp::StructSchema::Field field;
size_t column;
};
// Wrapper for classes that could throw in destructor
// https://github.com/capnproto/capnproto/issues/553
template <typename T>
struct DestructorCatcher
{
T impl;
template <typename ... Arg>
DestructorCatcher(Arg && ... args) : impl(kj::fwd<Arg>(args)...) {}
~DestructorCatcher() noexcept try { } catch (...) { }
};
using SchemaParser = DestructorCatcher<capnp::SchemaParser>;
ReadBuffer & istr;
const Block sample;
std::shared_ptr<SchemaParser> parser;
capnp::StructSchema root;
std::vector<Action> actions;
};
}

18
dbms/src/DataStreams/FormatFactory.cpp
View File

@ -1,3 +1,4 @@
#include <Common/config.h>
#include <Interpreters/Context.h>
#include <DataStreams/NativeBlockInputStream.h>
#include <DataStreams/NativeBlockOutputStream.h>
@ -30,6 +31,11 @@
#include <DataStreams/FormatFactory.h>
#include <DataStreams/SquashingBlockOutputStream.h>
#include <DataTypes/FormatSettingsJSON.h>
#if USE_CAPNP
#include <DataStreams/CapnProtoRowInputStream.h>
#endif
#include <boost/algorithm/string.hpp>
namespace DB
{
@ -92,6 +98,18 @@ BlockInputStreamPtr FormatFactory::getInput(const String & name, ReadBuffer & bu
{
return wrap_row_stream(std::make_shared<JSONEachRowRowInputStream>(buf, sample, settings.input_format_skip_unknown_fields));
}
#if USE_CAPNP
else if (name == "CapnProto")
{
std::vector<String> tokens;
auto schema_and_root = settings.format_schema.toString();
boost::split(tokens, schema_and_root, boost::is_any_of(":"));
if (tokens.size() != 2)
throw Exception("Format CapnProto requires 'format_schema' setting to have schema_file:root_object format, e.g. 'schema.capnp:Message'");
return wrap_row_stream(std::make_shared<CapnProtoRowInputStream>(buf, sample, tokens[0], tokens[1]));
}
#endif
else if (name == "TabSeparatedRaw"
|| name == "TSVRaw"
|| name == "BlockTabSeparated"

30
dbms/src/DataStreams/MaterializingBlockInputStream.cpp
View File

@ -1,6 +1,6 @@
#include <DataStreams/MaterializingBlockInputStream.h>
#include <DataTypes/DataTypeNullable.h>
#include <DataTypes/DataTypesNumber.h>
#include <DataStreams/materializeBlock.h>
namespace DB
{
@ -24,31 +24,7 @@ String MaterializingBlockInputStream::getID() const
Block MaterializingBlockInputStream::readImpl()
{
Block res = children.back()->read();
if (!res)
return res;
size_t columns = res.columns();
for (size_t i = 0; i < columns; ++i)
{
auto & element = res.safeGetByPosition(i);
auto & src = element.column;
ColumnPtr converted = src->convertToFullColumnIfConst();
if (converted)
{
src = converted;
auto & type = element.type;
if (type->isNull())
{
/// A ColumnNull that is converted to a full column
/// has the type DataTypeNullable(DataTypeUInt8).
type = std::make_shared<DataTypeNullable>(std::make_shared<DataTypeUInt8>());
}
}
}
return res;
return materializeBlock(children.back()->read());
}
}

26
dbms/src/DataStreams/MaterializingBlockOutputStream.h
View File

@ -1,6 +1,6 @@
#pragma once
#include <Core/Block.h>
#include <DataStreams/materializeBlock.h>
#include <DataStreams/IBlockOutputStream.h>
@ -15,24 +15,18 @@ public:
MaterializingBlockOutputStream(const BlockOutputStreamPtr & output)
: output{output} {}
void write(const Block & block) override
{
output->write(materialize(block));
}
void flush() override { output->flush(); }
void writePrefix() override { output->writePrefix(); }
void writeSuffix() override { output->writeSuffix(); }
void setRowsBeforeLimit(size_t rows_before_limit) override { output->setRowsBeforeLimit(rows_before_limit); }
void setTotals(const Block & totals) override { output->setTotals(materialize(totals)); }
void setExtremes(const Block & extremes) override { output->setExtremes(materialize(extremes)); }
void onProgress(const Progress & progress) override { output->onProgress(progress); }
String getContentType() const override { return output->getContentType(); }
void write(const Block & block) override { output->write(materializeBlock(block)); }
void flush() override { output->flush(); }
void writePrefix() override { output->writePrefix(); }
void writeSuffix() override { output->writeSuffix(); }
void setRowsBeforeLimit(size_t rows_before_limit) override { output->setRowsBeforeLimit(rows_before_limit); }
void setTotals(const Block & totals) override { output->setTotals(materializeBlock(totals)); }
void setExtremes(const Block & extremes) override { output->setExtremes(materializeBlock(extremes)); }
void onProgress(const Progress & progress) override { output->onProgress(progress); }
String getContentType() const override { return output->getContentType(); }
private:
BlockOutputStreamPtr output;
static Block materialize(const Block & original_block);
};
}

69
dbms/src/DataStreams/PushingToViewsBlockOutputStream.cpp Normal file
View File

@ -0,0 +1,69 @@
#include "PushingToViewsBlockOutputStream.h"
#include <Storages/MergeTree/ReplicatedMergeTreeBlockOutputStream.h>
namespace DB
{
PushingToViewsBlockOutputStream::PushingToViewsBlockOutputStream(String database, String table, const Context & context_,
const ASTPtr & query_ptr_, bool no_destination)
: context(context_), query_ptr(query_ptr_)
{
storage = context.getTable(database, table);
/** TODO This is a very important line. At any insertion into the table one of streams should own lock.
* Although now any insertion into the table is done via PushingToViewsBlockOutputStream,
* but it's clear that here is not the best place for this functionality.
*/
addTableLock(storage->lockStructure(true, __PRETTY_FUNCTION__));
Dependencies dependencies = context.getDependencies(database, table);
/// We need special context for materialized views insertions
if (!dependencies.empty())
{
views_context = std::make_unique<Context>(context);
// Do not deduplicate insertions into MV if the main insertion is Ok
views_context->getSettingsRef().insert_deduplicate = false;
}
for (const auto & database_table : dependencies)
{
auto dependent_table = context.getTable(database_table.first, database_table.second);
auto & materialized_view = dynamic_cast<const StorageMaterializedView &>(*dependent_table);
auto query = materialized_view.getInnerQuery();
auto next = std::make_shared<PushingToViewsBlockOutputStream>(database_table.first, database_table.second, *views_context, ASTPtr());
views.emplace_back(std::move(query), std::move(next));
}
/* Do not push to destination table if the flag is set */
if (!no_destination)
{
output = storage->write(query_ptr, context.getSettingsRef());
replicated_output = dynamic_cast<ReplicatedMergeTreeBlockOutputStream *>(output.get());
}
}
void PushingToViewsBlockOutputStream::write(const Block & block)
{
if (output)
output->write(block);
/// Don't process materialized views if this block is duplicate
if (replicated_output && replicated_output->lastBlockIsDuplicate())
return;
/// Insert data into materialized views only after successful insert into main table
for (auto & view : views)
{
BlockInputStreamPtr from = std::make_shared<OneBlockInputStream>(block);
InterpreterSelectQuery select(view.first, *views_context, QueryProcessingStage::Complete, 0, from);
BlockInputStreamPtr data = std::make_shared<MaterializingBlockInputStream>(select.execute().in);
copyData(*data, *view.second);
}
}
}

43
dbms/src/DataStreams/PushingToViewsBlockOutputStream.h
View File

@ -11,47 +11,17 @@
namespace DB
{
class ReplicatedMergeTreeBlockOutputStream;
/** Writes data to the specified table and to all dependent materialized views.
*/
class PushingToViewsBlockOutputStream : public IBlockOutputStream
{
public:
PushingToViewsBlockOutputStream(String database, String table, const Context & context_, const ASTPtr & query_ptr_, bool no_destination = false)
: context(context_), query_ptr(query_ptr_)
{
storage = context.getTable(database, table);
PushingToViewsBlockOutputStream(String database, String table, const Context & context_, const ASTPtr & query_ptr_, bool no_destination = false);
/** TODO This is a very important line. At any insertion into the table one of streams should own lock.
* Although now any insertion into the table is done via PushingToViewsBlockOutputStream,
* but it's clear that here is not the best place for this functionality.
*/
addTableLock(storage->lockStructure(true, __PRETTY_FUNCTION__));
Dependencies dependencies = context.getDependencies(database, table);
for (const auto & database_table : dependencies)
views.emplace_back(
dynamic_cast<const StorageMaterializedView &>(*context.getTable(database_table.first, database_table.second)).getInnerQuery(),
std::make_shared<PushingToViewsBlockOutputStream>(database_table.first, database_table.second, context, ASTPtr()));
/* Do not push to destination table if the flag is set */
if (!no_destination)
output = storage->write(query_ptr, context.getSettingsRef());
}
void write(const Block & block) override
{
for (auto & view : views)
{
BlockInputStreamPtr from = std::make_shared<OneBlockInputStream>(block);
InterpreterSelectQuery select(view.first, context, QueryProcessingStage::Complete, 0, from);
BlockInputStreamPtr data = std::make_shared<MaterializingBlockInputStream>(select.execute().in);
copyData(*data, *view.second);
}
if (output)
output->write(block);
}
void write(const Block & block) override;
void flush() override
{
@ -72,11 +42,16 @@ public:
}
private:
StoragePtr storage;
BlockOutputStreamPtr output;
ReplicatedMergeTreeBlockOutputStream * replicated_output = nullptr;
const Context & context;
ASTPtr query_ptr;
std::vector<std::pair<ASTPtr, BlockOutputStreamPtr>> views;
std::unique_ptr<Context> views_context;
};

61
dbms/src/DataStreams/SummingSortedBlockInputStream.cpp
View File

@ -1,6 +1,7 @@
#include <DataStreams/SummingSortedBlockInputStream.h>
#include <DataTypes/DataTypesNumber.h>
#include <DataTypes/DataTypeNested.h>
#include <DataTypes/DataTypeTuple.h>
#include <DataTypes/DataTypeArray.h>
#include <Columns/ColumnTuple.h>
#include <Common/StringUtils.h>
@ -10,6 +11,7 @@
#include <AggregateFunctions/AggregateFunctionFactory.h>
#include <Functions/FunctionFactory.h>
#include <Functions/FunctionHelpers.h>
#include <Interpreters/Context.h>
namespace DB
@ -151,12 +153,16 @@ Block SummingSortedBlockInputStream::readImpl()
// Create aggregator to sum this column
auto desc = AggregateDescription{};
desc.column_numbers = {i};
desc.merged_column = column.column;
desc.function = factory.get("sum", {column.type});
desc.function = factory.get("sumWithOverflow", {column.type});
desc.function->setArguments({column.type});
desc.state.resize(desc.function->sizeOfData());
columns_to_aggregate.emplace_back(std::move(desc));
}
else
{
// Column is not going to be summed, use last value
column_numbers_not_to_aggregate.push_back(i);
}
}
}
@ -165,7 +171,11 @@ Block SummingSortedBlockInputStream::readImpl()
{
/// map should contain at least two elements (key -> value)
if (map.second.size() < 2)
{
for (auto col : map.second)
column_numbers_not_to_aggregate.push_back(col);
continue;
}
/// no elements of map could be in primary key
auto column_num_it = map.second.begin();
@ -173,12 +183,13 @@ Block SummingSortedBlockInputStream::readImpl()
if (isInPrimaryKey(description, merged_block.safeGetByPosition(*column_num_it).name, *column_num_it))
break;
if (column_num_it != map.second.end())
{
for (auto col : map.second)
column_numbers_not_to_aggregate.push_back(col);
continue;
}
// Wrap aggregated columns in a tuple to match function signature
DataTypes argument_types = {};
auto tuple = std::make_shared<ColumnTuple>();
auto & tuple_columns = tuple->getColumns();
auto desc = AggregateDescription{};
auto map_desc = MapDescription{};
@ -212,16 +223,18 @@ Block SummingSortedBlockInputStream::readImpl()
// Add column to function arguments
desc.column_numbers.push_back(*column_num_it);
argument_types.push_back(key_col.type);
tuple_columns.push_back(key_col.column);
}
if (column_num_it != map.second.end())
{
for (auto col : map.second)
column_numbers_not_to_aggregate.push_back(col);
continue;
}
if (map_desc.key_col_nums.size() == 1)
{
// Create summation for all value columns in the map
desc.merged_column = static_cast<ColumnPtr>(tuple);
desc.function = factory.get("sumMap", argument_types);
desc.function->setArguments(argument_types);
desc.state.resize(desc.function->sizeOfData());
@ -230,15 +243,30 @@ Block SummingSortedBlockInputStream::readImpl()
else
{
// Fall back to legacy mergeMaps for composite keys
for (auto i : map_desc.key_col_nums)
column_numbers_not_to_aggregate.push_back(i);
for (auto i : map_desc.val_col_nums)
column_numbers_not_to_aggregate.push_back(i);
for (auto col : map.second)
column_numbers_not_to_aggregate.push_back(col);
maps_to_sum.emplace_back(std::move(map_desc));
}
}
}
// Update aggregation result columns for current block
for (auto & desc : columns_to_aggregate)
{
// Wrap aggregated columns in a tuple to match function signature
if (checkDataType<DataTypeTuple>(desc.function->getReturnType().get()))
{
auto tuple = std::make_shared<ColumnTuple>();
auto & tuple_columns = tuple->getColumns();
for (auto i : desc.column_numbers)
tuple_columns.push_back(merged_block.safeGetByPosition(i).column);
desc.merged_column = tuple;
}
else
desc.merged_column = merged_block.safeGetByPosition(desc.column_numbers[0]).column;
}
if (has_collation)
merge(merged_columns, queue_with_collation);
else
@ -290,7 +318,6 @@ void SummingSortedBlockInputStream::merge(ColumnPlainPtrs & merged_columns, std:
current_key.swap(next_key);
setRow(current_row, current);
current_row_is_zero = false;
/// Reset aggregation states for next row
for (auto & desc : columns_to_aggregate)
@ -298,9 +325,14 @@ void SummingSortedBlockInputStream::merge(ColumnPlainPtrs & merged_columns, std:
desc.function->create(desc.state.data());
desc.created = true;
}
// Start aggregations with current row
current_row_is_zero = !addRow(current_row, current);
}
else
{
current_row_is_zero = !addRow(current_row, current);
// Merge maps only for same rows
for (auto & desc : maps_to_sum)
{
@ -309,9 +341,6 @@ void SummingSortedBlockInputStream::merge(ColumnPlainPtrs & merged_columns, std:
}
}
if (addRow(current_row, current))
current_row_is_zero = false;
if (!current->isLast())
{
current->next();
@ -445,6 +474,8 @@ bool SummingSortedBlockInputStream::addRow(Row & row, TSortCursor & cursor)
columns[i] = cursor->all_columns[desc.column_numbers[i]];
desc.function->add(desc.state.data(), columns.data(), cursor->pos, nullptr);
// Note: we can't detect whether the aggregation result is non-empty here yet
res = true;
}
}
}

18
dbms/src/DataStreams/MaterializingBlockOutputStream.cpp → dbms/src/DataStreams/materializeBlock.cpp
View File

@ -1,25 +1,27 @@
#include <DataStreams/MaterializingBlockOutputStream.h>
#include <DataStreams/materializeBlock.h>
#include <DataTypes/DataTypeNullable.h>
#include <DataTypes/DataTypesNumber.h>
#include <ext/range.h>
namespace DB
{
Block MaterializingBlockOutputStream::materialize(const Block & original_block)
Block materializeBlock(const Block & block)
{
/// copy block to get rid of const
auto block = original_block;
if (!block)
return block;
for (const auto i : ext::range(0, block.columns()))
Block res = block;
size_t columns = res.columns();
for (size_t i = 0; i < columns; ++i)
{
auto & element = block.safeGetByPosition(i);
auto & element = res.getByPosition(i);
auto & src = element.column;
ColumnPtr converted = src->convertToFullColumnIfConst();
if (converted)
{
src = converted;
auto & type = element.type;
if (type->isNull())
{
@ -30,7 +32,7 @@ Block MaterializingBlockOutputStream::materialize(const Block & original_block)
}
}
return block;
return res;
}
}

13
dbms/src/DataStreams/materializeBlock.h Normal file
View File

@ -0,0 +1,13 @@
#pragma once
#include <Core/Block.h>
namespace DB
{
/** Converts columns-constants to full columns ("materializes" them).
*/
Block materializeBlock(const Block & block);
}

6
dbms/src/DataStreams/tests/aggregating_stream.cpp
View File

@ -38,7 +38,7 @@ int main(int argc, char ** argv)
{
size_t n = argc > 1 ? atoi(argv[1]) : 10;
bool nested = false, sum = false, composite = false, multivalue = false;
for (size_t i = 2; i < argc; ++i)
for (int i = 2; i < argc; ++i)
{
if (strcmp(argv[i], "nested") == 0)
nested = true;
@ -54,8 +54,8 @@ int main(int argc, char ** argv)
ColumnWithTypeAndName column_x;
column_x.name = "x";
column_x.type = std::make_shared<DataTypeUInt64>();
auto x = std::make_shared<ColumnUInt64>();
column_x.type = std::make_shared<DataTypeUInt32>();
auto x = std::make_shared<ColumnUInt32>();
column_x.column = x;
auto & vec_x = x->getData();

2
dbms/src/DataTypes/DataTypeAggregateFunction.h
View File

@ -26,6 +26,8 @@ private:
Array parameters;
public:
static constexpr bool is_parametric = true;
DataTypeAggregateFunction(const AggregateFunctionPtr & function_, const DataTypes & argument_types_, const Array & parameters_)
: function(function_), argument_types(argument_types_), parameters(parameters_)
{

2
dbms/src/DataTypes/DataTypeArray.h
View File

@ -19,6 +19,8 @@ private:
DataTypePtr offsets;
public:
static constexpr bool is_parametric = true;
DataTypeArray(const DataTypePtr & nested_);
DataTypeArray(const DataTypeTraits::EnrichedDataTypePtr & enriched_nested_);

2
dbms/src/DataTypes/DataTypeEnum.h
View File

@ -36,6 +36,8 @@ public:
using NameToValueMap = HashMap<StringRef, FieldType, StringRefHash>;
using ValueToNameMap = std::unordered_map<FieldType, StringRef>;
static constexpr bool is_parametric = true;
private:
Values values;
NameToValueMap name_to_value_map;

2
dbms/src/DataTypes/DataTypeExpression.h
View File

@ -15,6 +15,8 @@ private:
DataTypePtr return_type;
public:
static constexpr bool is_parametric = true;
/// Some types could be still unknown.
DataTypeExpression(const DataTypes & argument_types_ = DataTypes(), const DataTypePtr & return_type_ = nullptr)
: argument_types(argument_types_), return_type(return_type_) {}

2
dbms/src/DataTypes/DataTypeFactory.cpp
View File

@ -117,6 +117,7 @@ void registerDataTypeNull(DataTypeFactory & factory);
void registerDataTypeUUID(DataTypeFactory & factory);
void registerDataTypeAggregateFunction(DataTypeFactory & factory);
void registerDataTypeNested(DataTypeFactory & factory);
void registerDataTypeInterval(DataTypeFactory & factory);
DataTypeFactory::DataTypeFactory()
@ -134,6 +135,7 @@ DataTypeFactory::DataTypeFactory()
registerDataTypeUUID(*this);
registerDataTypeAggregateFunction(*this);
registerDataTypeNested(*this);
registerDataTypeInterval(*this);
}
}

2
dbms/src/DataTypes/DataTypeFixedString.h
View File

@ -18,6 +18,8 @@ private:
size_t n;
public:
static constexpr bool is_parametric = true;
DataTypeFixedString(size_t n_) : n(n_)
{
if (n == 0)

19
dbms/src/DataTypes/DataTypeInterval.cpp Normal file
View File

@ -0,0 +1,19 @@
#include <DataTypes/DataTypeInterval.h>
#include <DataTypes/DataTypeFactory.h>
namespace DB
{
void registerDataTypeInterval(DataTypeFactory & factory)
{
factory.registerSimpleDataType("IntervalSecond", [] { return DataTypePtr(std::make_shared<DataTypeInterval>(DataTypeInterval::Second)); });
factory.registerSimpleDataType("IntervalMinute", [] { return DataTypePtr(std::make_shared<DataTypeInterval>(DataTypeInterval::Minute)); });
factory.registerSimpleDataType("IntervalHour", [] { return DataTypePtr(std::make_shared<DataTypeInterval>(DataTypeInterval::Hour)); });
factory.registerSimpleDataType("IntervalDay", [] { return DataTypePtr(std::make_shared<DataTypeInterval>(DataTypeInterval::Day)); });
factory.registerSimpleDataType("IntervalWeek", [] { return DataTypePtr(std::make_shared<DataTypeInterval>(DataTypeInterval::Week)); });
factory.registerSimpleDataType("IntervalMonth", [] { return DataTypePtr(std::make_shared<DataTypeInterval>(DataTypeInterval::Month)); });
factory.registerSimpleDataType("IntervalYear", [] { return DataTypePtr(std::make_shared<DataTypeInterval>(DataTypeInterval::Year)); });
}
}

66
dbms/src/DataTypes/DataTypeInterval.h Normal file
View File

@ -0,0 +1,66 @@
#pragma once
#include <DataTypes/DataTypeNumberBase.h>
namespace DB
{
/** Data type to deal with INTERVAL in SQL (arithmetic on time intervals).
*
* Mostly the same as Int64.
* But also tagged with interval kind.
*
* Intended isage is for temporary elements in expressions,
* not for storing values in tables.
*/
class DataTypeInterval final : public DataTypeNumberBase<Int64>
{
public:
enum Kind
{
Second,
Minute,
Hour,
Day,
Week,
Month,
Year
};
private:
Kind kind;
public:
static constexpr bool is_parametric = true;
Kind getKind() const { return kind; }
const char * kindToString() const
{
switch (kind)
{
case Second: return "Second";
case Minute: return "Minute";
case Hour: return "Hour";
case Day: return "Day";
case Week: return "Week";
case Month: return "Month";
case Year: return "Year";
default: __builtin_unreachable();
}
}
DataTypeInterval(Kind kind) : kind(kind) {};
std::string getName() const override { return std::string("Interval") + kindToString(); }
const char * getFamilyName() const override { return "Interval"; }
bool behavesAsNumber() const override { return false; }
bool notForTables() const override { return true; }
DataTypePtr clone() const override { return std::make_shared<DataTypeInterval>(kind); }
};
}

2
dbms/src/DataTypes/DataTypeNested.h
View File

@ -17,6 +17,8 @@ private:
NamesAndTypesListPtr nested;
public:
static constexpr bool is_parametric = true;
DataTypeNested(const NamesAndTypesListPtr & nested_);
std::string getName() const override;

2
dbms/src/DataTypes/DataTypeNull.h
View File

@ -16,6 +16,8 @@ public:
using FieldType = Null;
public:
static constexpr bool is_parametric = false;
String getName() const override
{
return "Null";

2
dbms/src/DataTypes/DataTypeNullable.h
View File

@ -11,6 +11,8 @@ namespace DB
class DataTypeNullable final : public IDataType
{
public:
static constexpr bool is_parametric = true;
DataTypeNullable(const DataTypePtr & nested_data_type_);
std::string getName() const override { return "Nullable(" + nested_data_type->getName() + ")"; }
const char * getFamilyName() const override { return "Nullable"; }

1
dbms/src/DataTypes/DataTypeNumberBase.h
View File

@ -12,6 +12,7 @@ template <typename T>
class DataTypeNumberBase : public IDataType
{
public:
static constexpr bool is_parametric = false;
using FieldType = T;
std::string getName() const override { return TypeName<T>::get(); }

1
dbms/src/DataTypes/DataTypeSet.h
View File

@ -12,6 +12,7 @@ namespace DB
class DataTypeSet final : public IDataTypeDummy
{
public:
static constexpr bool is_parametric = true;
std::string getName() const override { return "Set"; }
const char * getFamilyName() const override { return "Set"; }
DataTypePtr clone() const override { return std::make_shared<DataTypeSet>(); }

1
dbms/src/DataTypes/DataTypeString.h
View File

@ -12,6 +12,7 @@ class DataTypeString final : public IDataType
{
public:
using FieldType = String;
static constexpr bool is_parametric = false;
std::string getName() const override
{

1
dbms/src/DataTypes/DataTypeTuple.h
View File

@ -16,6 +16,7 @@ class DataTypeTuple final : public IDataType
private:
DataTypes elems;
public:
static constexpr bool is_parametric = true;
DataTypeTuple(const DataTypes & elems_) : elems(elems_) {}
std::string getName() const override;

2
dbms/src/DataTypes/DataTypesNumber.h
View File

@ -33,6 +33,7 @@ class DataTypeNumber<void> final : public IDataTypeDummy
{
public:
using FieldType = void;
static constexpr bool is_parametric = false;
std::string getName() const override { return "Void"; }
const char * getFamilyName() const override { return "Void"; }
@ -46,6 +47,7 @@ class DataTypeNumber<Null> final : public IDataTypeDummy
{
public:
using FieldType = Null;
static constexpr bool is_parametric = false;
std::string getName() const override { return "Null"; }
const char * getFamilyName() const override { return "Null"; }

6
dbms/src/DataTypes/IDataType.h
View File

@ -29,6 +29,12 @@ using DataTypes = std::vector<DataTypePtr>;
class IDataType
{
public:
/// Compile time flag. If false, then if C++ types are the same, then SQL types are also the same.
/// Example: DataTypeString is not parametric: thus all instances of DataTypeString are the same SQL type.
/// Example: DataTypeFixedString is parametric: different instances of DataTypeFixedString may be different SQL types.
/// Place it in descendants:
/// static constexpr bool is_parametric = false;
/// Name of data type (examples: UInt64, Array(String)).
virtual String getName() const = 0;

12
dbms/src/DataTypes/NumberTraits.h
View File

@ -60,14 +60,14 @@ struct AddNullability<T, HasNoNull>
template <typename T> struct Next;
template <> struct Next<Bits0> { using Type = Bits0; };
template <> struct Next<Bits8> { using Type = Bits16; };
template <> struct Next<Bits16> { using Type = Bits32; };
template <> struct Next<Bits32> { using Type = Bits64; };
template <> struct Next<Bits64> { using Type = Bits64; };
template <> struct Next<Bits0> { using Type = Bits0; };
template <> struct Next<Bits8> { using Type = Bits16; };
template <> struct Next<Bits16> { using Type = Bits32; };
template <> struct Next<Bits32> { using Type = Bits64; };
template <> struct Next<Bits64> { using Type = Bits64; };
template <typename T> struct ExactNext { using Type = typename Next<T>::Type; };
template <> struct ExactNext<Bits64> { using Type = BitsTooMany; };
template <> struct ExactNext<Bits64> { using Type = BitsTooMany; };
template <typename T> struct Traits;

789
dbms/src/Databases/DatabaseCloud.cpp
View File

@ -1,789 +0,0 @@
#include <Common/ZooKeeper/ZooKeeper.h>
#include <Interpreters/Context.h>
#include <Common/escapeForFileName.h>
#include <Common/SipHash.h>
#include <Common/UInt128.h>
#include <Databases/DatabaseCloud.h>
#include <Databases/DatabasesCommon.h>
#include <IO/CompressedWriteBuffer.h>
#include <IO/CompressedReadBuffer.h>
#include <IO/WriteBufferFromString.h>
#include <IO/ReadBufferFromString.h>
#include <IO/HexWriteBuffer.h>
#include <Parsers/ASTCreateQuery.h>
#include <Parsers/parseQuery.h>
#include <Parsers/ParserCreateQuery.h>
namespace DB
{
namespace ErrorCodes
{
extern const int LOGICAL_ERROR;
extern const int TABLE_ALREADY_EXISTS;
extern const int UNKNOWN_TABLE;
}
namespace
{
constexpr size_t TABLE_TO_NODE_DIVISOR = 4096;
using Hash = DatabaseCloud::Hash;
}
void DatabaseCloud::createZookeeperNodes()
{
zkutil::ZooKeeperPtr zookeeper = context->getZookeeper();
zookeeper->createAncestors(zookeeper_path);
auto acl = zookeeper->getDefaultACL();
zkutil::Ops ops;
ops.emplace_back(std::make_unique<zkutil::Op::Create>(zookeeper_path, "", acl, zkutil::CreateMode::Persistent));
ops.emplace_back(std::make_unique<zkutil::Op::Create>(zookeeper_path + "/table_definitions", "", acl, zkutil::CreateMode::Persistent));
ops.emplace_back(std::make_unique<zkutil::Op::Create>(zookeeper_path + "/tables", "", acl, zkutil::CreateMode::Persistent));
ops.emplace_back(std::make_unique<zkutil::Op::Create>(zookeeper_path + "/local_tables", "", acl, zkutil::CreateMode::Persistent));
ops.emplace_back(std::make_unique<zkutil::Op::Create>(zookeeper_path + "/locality_keys", "", acl, zkutil::CreateMode::Persistent));
ops.emplace_back(std::make_unique<zkutil::Op::Create>(zookeeper_path + "/nodes", "", acl, zkutil::CreateMode::Persistent));
auto code = zookeeper->tryMulti(ops);
if (code == ZOK)
LOG_INFO(log, "Created new cloud.");
else if (code == ZNODEEXISTS)
LOG_INFO(log, "Adding server to existing cloud.");
else
throw zkutil::KeeperException(code);
zookeeper->createIfNotExists(zookeeper_path + "/tables/" + name, "");
zookeeper->createIfNotExists(zookeeper_path + "/local_tables/" + name, "");
zookeeper->createIfNotExists(zookeeper_path + "/nodes/" + hostname, "");
zookeeper->createIfNotExists(zookeeper_path + "/nodes/" + hostname + "/datacenter", datacenter_name);
}
DatabaseCloud::DatabaseCloud(
bool attach,
const String & name_,
const String & zookeeper_path_,
size_t replication_factor_,
const String & datacenter_name_,
Context & context_)
:
name(name_),
zookeeper_path(context_.getMacros().expand(zookeeper_path_)),
replication_factor(replication_factor_),
datacenter_name(context_.getMacros().expand(datacenter_name_)),
log(&Logger::get("DatabaseCloud (" + name + ")")),
context(context_)
{
if (zookeeper_path.empty())
throw Exception("Logical error: empty zookeeper_path passed", ErrorCodes::LOGICAL_ERROR);
if (zookeeper_path.back() == '/')
zookeeper_path.pop_back();
hostname = context.getInterserverIOAddress().first;
data_path = context.getPath() + "/data/" + escapeForFileName(name) + "/";
if (!attach)
createZookeeperNodes();
}
void loadTables(Context & context, ThreadPool * thread_pool, bool has_force_restore_data_flag)
{
/// Do nothing - all tables are loaded lazily.
}
Hash DatabaseCloud::getTableHash(const String & table_name) const
{
SipHash hash;
hash.update(name.data(), name.size() + 1); /// Hashing also a zero byte as a separator.
hash.update(table_name.data(), table_name.size());
Hash res;
hash.get128(reinterpret_cast<char *>(&res));
return res;
}
String DatabaseCloud::getNameOfNodeWithTables(const String & table_name) const
{
Hash hash = getTableHash(table_name);
WriteBufferFromOwnString out;
writeText(hash.first % TABLE_TO_NODE_DIVISOR, out);
return out.str();
}
static String hashToHex(Hash hash)
{
String res;
{
WriteBufferFromString str_out(res);
HexWriteBuffer hex_out(str_out);
writePODBinary(hash, hex_out);
}
return res;
}
String DatabaseCloud::getTableDefinitionFromHash(Hash hash) const
{
zkutil::ZooKeeperPtr zookeeper = context.getZooKeeper();
return zookeeper->get(zookeeper_path + "/table_definitions/" + hashToHex(hash));
}
/** Description of one table in the list of tables in ZooKeeper.
* No table name (right side of map).
*/
struct TableDescription
{
/// Hash of the table structure. The structure itself is stored separately.
Hash definition_hash;
/// The name of the local table to store data. It can be empty if nothing else has been written to the table.
String local_table_name;
/// The list of hosts on which the table data is located. It can be empty if nothing else has been written to the table.
std::vector<String> hosts;
void write(WriteBuffer & buf) const
{
writePODBinary(definition_hash, buf);
writeVectorBinary(hosts, buf);
}
void read(ReadBuffer & buf)
{
readPODBinary(definition_hash, buf);
readVectorBinary(hosts, buf);
}
};
/** Set of tables in ZooKeeper.
* More precisely, its part, referring to one node.
* (The whole set is broken into `TABLE_TO_NODE_DIVISOR` nodes.)
*/
struct TableSet
{
/// Name -> description. In an ordered form, the data will be better compressed.
using Container = std::map<String, TableDescription>;
Container map;
explicit TableSet(const String & data)
{
ReadBufferFromString in(data);
read(in);
}
String toString() const
{
WriteBufferFromOwnString out;
write(out);
return out.str();
}
void write(WriteBuffer & buf) const
{
writeCString("Version 1\n", buf);
CompressedWriteBuffer out(buf); /// NOTE You can reduce size of allocated buffer.
for (const auto & kv : map)
{
writeBinary(kv.first, out);
kv.second.write(out);
}
}
void read(ReadBuffer & buf)
{
assertString("Version 1\n", buf);
CompressedReadBuffer in(buf);
while (!in.eof())
{
Container::value_type kv;
readBinary(kv.first, in);
kv.second.read(in);
map.emplace(std::move(kv));
}
}
};
/** Set of local tables in ZooKeeper.
* More precisely, its part, referring to one node.
* (The whole set is broken into `TABLE_TO_NODE_DIVISOR` nodes.)
*/
struct LocalTableSet
{
/// Hash of name -> hash of structure.
using Container = std::map<Hash, Hash>;
Container map;
TableSet(const String & data)
{
ReadBufferFromString in(data);
read(in);
}
String toString() const
{
WriteBufferFromOwnString out;
write(out);
return out.str();
}
void write(WriteBuffer & buf) const
{
writeCString("Version 1\n", buf);
CompressedWriteBuffer out(buf); /// NOTE You can reduce size of allocated buffer.
for (const auto & kv : map)
{
writePODBinary(kv.first, out);
writePODBinary(kv.second, out);
}
}
void read(ReadBuffer & buf)
{
assertString("Version 1\n", buf);
CompressedReadBuffer in(buf);
while (!in.eof())
{
Container::value_type kv;
readPODBinary(kv.first, in);
readPODBinary(kv.second, in);
map.emplace(std::move(kv));
}
}
};
/** Modify TableSet or LocalTableSet, serialized in ZooKeeper, as a single unit.
* The compare-and-swap is done. `transform` function can be called many times.
* If `transform` returns false, it is considered that there is nothing to modify and the result is not saved in ZK.
*/
template <typename TableSet, typename F>
static void modifyTableSet(zkutil::ZooKeeperPtr & zookeeper, const String & path, F && transform)
{
while (true)
{
zkutil::Stat stat;
String old_value = zookeeper->get(path, &stat);
TableSet tables_info(old_value);
if (!transform(tables_info))
break;
String new_value = tables_info.toString();
auto code = zookeeper->trySet(path, new_value, stat.version);
if (code == ZOK)
break;
else if (code == ZBADVERSION)
continue; /// Node was changed meanwhile - we'll try again.
else
throw zkutil::KeeperException(code, path);
}
}
template <typename TableSet, typename F>
static void modifyTwoTableSets(zkutil::ZooKeeperPtr & zookeeper, const String & path1, const String & path2, F && transform)
{
if (path1 == path2)
{
modifyTableSet<TableSet>(zookeeper, path1, [&] (TableSet & set) { return transform(set, set); });
return;
}
while (true)
{
zkutil::Stat stat1;
zkutil::Stat stat2;
String old_value1 = zookeeper->get(path1, &stat1);
String old_value2 = zookeeper->get(path2, &stat2);
TableSet tables_info1(old_value1);
TableSet tables_info2(old_value2);
if (!transform(tables_info1, tables_info2))
break;
String new_value1 = tables_info1.toString();
String new_value2 = tables_info2.toString();
zkutil::Ops ops;
ops.emplace_back(std::make_unique<zkutil::Op::SetData>(path1, new_value1, stat1.version));
ops.emplace_back(std::make_unique<zkutil::Op::SetData>(path2, new_value2, stat2.version));
auto code = zookeeper->tryMulti(ops);
if (code == ZOK)
break;
else if (code == ZBADVERSION)
continue; /// Node was changed meanwhile - we'll try again.
else
throw zkutil::KeeperException(code, path1 + ", " + path2);
}
}
bool DatabaseCloud::isTableExist(const String & table_name) const
{
/// We are looking for a local table in the local table cache or in the filesystem in `path`.
/// If you do not find it, look for the cloud table in ZooKeeper.
{
std::lock_guard<std::mutex> lock(local_tables_mutex);
if (local_tables_cache.count(table_name))
return true;
}
String table_name_escaped = escapeForFileName(table_name);
if (Poco::File(data_path + table_name_escaped).exists())
return true;
zkutil::ZooKeeperPtr zookeeper = context.getZooKeeper();
String table_set_data;
if (!zookeeper->tryGet(zookeeper_path + "/tables/" + name + "/" + getNameOfNodeWithTables(table_name), table_set_data))
return false;
TableSet table_set(table_set_data);
return table_set.map.count(table_name);
}
StoragePtr DatabaseCloud::tryGetTable(const String & table_name)
{
/// We are looking for a local table.
/// If you do not find it, look for the cloud table in ZooKeeper.
{
std::lock_guard<std::mutex> lock(local_tables_mutex);
auto it = local_tables_cache.find(table_name);
if (it != local_tables_cache.end())
return it->second;
}
zkutil::ZooKeeperPtr zookeeper = context.getZooKeeper();
String table_name_escaped = escapeForFileName(table_name);
if (Poco::File(data_path + table_name_escaped).exists())
{
LocalTableSet local_tables_info(zookeeper->get(
zookeeper_path + "/local_tables/" + name + "/" + getNameOfNodeWithTables(table_name)));
Hash table_hash = getTableHash(table_name);
String definition = getTableDefinitionFromHash(local_tables_info.map.at(table_hash));
/// Initialize local table.
{
std::lock_guard<std::mutex> lock(local_tables_mutex);
/// And if the table has just been created?
auto it = local_tables_cache.find(table_name);
if (it != local_tables_cache.end())
return it->second;
String table_name;
StoragePtr table;
std::tie(table_name, table) = createTableFromDefinition(
definition, name, data_path, context, false,
"in zookeeper node " + zookeeper_path + "/table_definitions/" + hashToHex(table_hash));
table->startup();
local_tables_cache.emplace(table_name, table);
return table;
}
}
else
{
const TableSet tables_info(zookeeper->get(
zookeeper_path + "/tables/" + name + "/" + getNameOfNodeWithTables(table_name)));
const TableDescription & description = tables_info.at(table_name);
String definition = getTableDefinitionFromHash(description.definition_hash);
/// TODO Initialization of `StorageCloud` object
return {};
}
}
/// The list of tables can be inconsistent, as it is obtained not atomically, but in pieces, while iterating.
class DatabaseCloudIterator : public IDatabaseIterator
{
private:
DatabasePtr owned_database;
zkutil::ZooKeeperPtr zookeeper;
const String & zookeeper_path;
bool first = true;
const Strings nodes;
Strings::const_iterator nodes_iterator;
TableSet table_set;
TableSet::Container::iterator table_set_iterator;
DatabaseCloud & parent()
{
return static_cast<DatabaseCloud &>(*owned_database);
}
bool fetchNextTableSet()
{
do
{
if (first)
first = false;
else
++nodes_iterator;
if (nodes_iterator == nodes.end())
return false;
table_set = TableSet(zookeeper->get(zookeeper_path + "/" + *nodes_iterator));
table_set_iterator = table_set.map.begin();
}
while (!table_set.map.empty()); /// Skip empty table sets.
return true;
}
public:
explicit DatabaseCloudIterator(DatabasePtr database)
: owned_database(database),
zookeeper(parent().context.getZooKeeper()),
zookeeper_path(parent().zookeeper_path + "/tables/" + parent().name),
nodes(zookeeper->getChildren(zookeeper_path)),
nodes_iterator(nodes.begin())
{
fetchNextTableSet();
}
void next() override
{
++table_set_iterator;
if (table_set_iterator == table_set.end())
fetchNextTableSet();
}
bool isValid() const override
{
return nodes_iterator != nodes.end();
}
const String & name() const override
{
return table_set_iterator->first;
}
StoragePtr & table() const
{
String definition = parent().getTableDefinitionFromHash(table_set_iterator->second.definition_hash);
/// TODO Initialization of `StorageCloud` object
return {};
}
};
DatabaseIteratorPtr DatabaseCloud::getIterator()
{
return std::make_unique<DatabaseCloudIterator>(shared_from_this());
}
bool DatabaseCloud::empty() const
{
/// There is at least one non-empty node among the list of tables.
zkutil::ZooKeeperPtr zookeeper = context.getZooKeeper();
Strings nodes = zookeeper->getChildren(zookeeper_path + "/tables/" + name);
if (nodes.empty())
return true;
for (const auto & node : nodes)
if (!zookeeper->get(zookeeper_path + "/tables/" + name + "/" + node).empty())
return false;
return true;
}
ASTPtr DatabaseCloud::getCreateQuery(const String & table_name) const
{
zkutil::ZooKeeperPtr zookeeper = context.getZooKeeper();
Hash definition_hash;
String table_name_escaped = escapeForFileName(table_name);
if (Poco::File(data_path + table_name_escaped).exists())
{
LocalTableSet local_tables_info(zookeeper->get(
zookeeper_path + "/local_tables/" + name + "/" + getNameOfNodeWithTables(table_name)));
Hash table_hash = getTableHash(table_name);
definition_hash = local_tables_info.map.at(table_hash);
}
else
{
const TableSet tables_info(zookeeper->get(
zookeeper_path + "/tables/" + name + "/" + getNameOfNodeWithTables(table_name)));
const TableDescription & description = tables_info.at(table_name);
definition_hash = description.definition_hash;
}
String definition = getTableDefinitionFromHash(definition_hash);
ParserCreateQuery parser;
ASTPtr ast = parseQuery(parser, definition.data(), definition.data() + definition.size(),
"in zookeeper node " + zookeeper_path + "/table_definitions/" + hashToHex(definition_hash));
ASTCreateQuery & ast_create_query = typeid_cast<ASTCreateQuery &>(*ast);
ast_create_query.attach = false;
ast_create_query.database = name;
return ast;
}
void DatabaseCloud::attachTable(const String & table_name, const StoragePtr & table)
{
throw Exception("Attaching tables to cloud database is not supported", ErrorCodes::NOT_IMPLEMENTED);
}
StoragePtr DatabaseCloud::detachTable(const String & table_name)
{
throw Exception("Detaching tables from cloud database is not supported", ErrorCodes::NOT_IMPLEMENTED);
}
Hash DatabaseCloud::getHashForTableDefinition(const String & definition) const
{
Hash res;
SipHash hash;
hash.update(definition);
hash.get128(reinterpret_cast<char *>(&res));
return res;
}
void DatabaseCloud::createTable(
const String & table_name, const StoragePtr & table, const ASTPtr & query, const String & engine, const Settings & settings)
{
zkutil::ZooKeeperPtr zookeeper = context.getZooKeeper();
/// Add information about the table structure to ZK.
String definition = getTableDefinitionFromCreateQuery(query);
Hash definition_hash = getHashForTableDefinition(definition);
String zookeeper_definition_path = zookeeper_path + "/table_definitions/" + hashToHex(definition_hash);
String value;
if (zookeeper->tryGet(zookeeper_definition_path, &value))
{
if (value != definition)
throw Exception("Logical error: different table definition with same hash", ErrorCodes::LOGICAL_ERROR);
}
else
{
/// A rarer branch, since there are few unique table definitions.
/// There is a race condition in which the node already exists, but a check for a logical error (see above) will not be performed.
/// It does not matter.
/// By the way, nodes in `table_definitions` are never deleted.
zookeeper->tryCreate(zookeeper_definition_path, definition, zkutil::CreateMode::Persistent);
}
if (engine != "Cloud")
{
/// If the local table.
String table_name_escaped = escapeForFileName(table_name);
Poco::File(data_path + table_name_escaped).createDirectory();
Hash table_hash = getTableHash(table_name);
/// Add information about the local table to ZK.
modifyTableSet<LocalTableSet>(
zookeeper,
zookeeper_path + "/local_tables/" + name + "/" + getNameOfNodeWithTables(table_name),
[&] (LocalTableSet & set)
{
if (!set.map.emplace(table_hash, definition_hash).second)
throw Exception("Table " + table_name + " already exists", ErrorCodes::TABLE_ALREADY_EXISTS);
return true;
});
/// Add the local table to the cache.
{
std::lock_guard<std::mutex> lock(local_tables_mutex);
if (!local_tables_cache.emplace(table_name, table).second)
throw Exception("Table " + table_name + " already exists", ErrorCodes::TABLE_ALREADY_EXISTS);
}
}
else
{
/// When creating an empty cloud table, no local tables are created and no servers are defined for them.
/// Everything is done when you first write to the table.
TableDescription description;
description.definition_hash = definition_hash;
modifyTableSet<TableSet>(
zookeeper,
zookeeper_path + "/tables/" + name + "/" + getNameOfNodeWithTables(table_name),
[&] (TableSet & set)
{
if (!set.map.emplace(table_name, description).second)
throw Exception("Table " + table_name + " already exists", ErrorCodes::TABLE_ALREADY_EXISTS);
return true;
});
}
}
void DatabaseCloud::removeTable(const String & table_name)
{
zkutil::ZooKeeperPtr zookeeper = context.getZooKeeper();
/// Looking for a local table.
/// If you do not find it, look for the cloud table in ZooKeeper.
String table_name_escaped = escapeForFileName(table_name);
if (Poco::File(data_path + table_name_escaped).exists())
{
Hash table_hash = getTableHash(table_name);
/// Delete information about the local table from ZK.
modifyTableSet<LocalTableSet>(
zookeeper,
zookeeper_path + "/local_tables/" + name + "/" + getNameOfNodeWithTables(table_name),
[&] (LocalTableSet & set)
{
auto it = set.map.find(table_hash);
if (it == set.map.end())
return false; /// The table has already been deleted.
set.map.erase(it);
return true;
});
/// Delete the local table from the cache.
{
std::lock_guard<std::mutex> lock(local_tables_mutex);
local_tables_cache.erase(table_name);
}
}
else
{
/// Delete the table from ZK, and also remember the list of servers on which local tables are located.
TableDescription description;
modifyTableSet<TableSet>(
zookeeper,
zookeeper_path + "/tables/" + name + "/" + getNameOfNodeWithTables(table_name),
[&] (TableSet & set) mutable
{
auto it = set.map.find(table_name);
if (it == set.map.end())
return false; /// The table has already been deleted.
description = it->second;
set.map.erase(it);
return true;
});
if (!description.local_table_name.empty() && !description.hosts.empty())
{
/// Deleting local tables. TODO Whether at once here, or in a separate background thread.
}
}
}
void DatabaseCloud::renameTable(
const Context & context, const String & table_name, IDatabase & to_database, const String & to_table_name, const Settings & settings)
{
/// Only cloud tables can be renamed.
/// The transfer between databases is not supported.
if (&to_database != this)
throw Exception("Moving of tables in Cloud database between databases is not supported", ErrorCodes::NOT_IMPLEMENTED);
const String node_from = getNameOfNodeWithTables(table_name);
const String node_to = getNameOfNodeWithTables(to_table_name);
const String table_set_path_from = zookeeper_path + "/tables/" + name + "/" + node_from;
const String table_set_path_to = zookeeper_path + "/tables/" + name + "/" + node_to;
zkutil::ZooKeeperPtr zookeeper = context.getZooKeeper();
modifyTwoTableSets<TableSet>(
zookeeper, table_set_path_from, table_set_path_to, [&] (TableSet & set_from, TableSet & set_to)
{
auto it = set_from.map.find(table_name);
if (it == set_from.map.end())
throw Exception("Table " + table_name + " doesn't exist", ErrorCodes::UNKNOWN_TABLE);
TableDescription description = it->second;
set_from.map.erase(it);
if (!set_to.map.emplace(to_table_name, description).second)
throw Exception("Table " + to_table_name + " already exists", ErrorCodes::TABLE_ALREADY_EXISTS);
return true;
});
}
time_t DatabaseCloud::getTableMetaModTime(const String & table_name)
{
return static_cast<time_t>(0);
}
void DatabaseCloud::shutdown()
{
/// You can not hold a lock during shutdown.
/// Because inside `shutdown` function the tables can work with database, and mutex is not recursive.
Tables local_tables_snapshot;
{
std::lock_guard<std::mutex> lock(local_tables_mutex);
local_tables_snapshot = local_tables_cache;
}
for (auto & name_table : local_tables_snapshot)
name_table.second->shutdown();
{
std::lock_guard<std::mutex> lock(local_tables_mutex);
local_tables_cache.clear();
}
}
std::vector<String> DatabaseCloud::selectHostsForTable(const String & locality_key) const
{
}
void DatabaseCloud::drop()
{
}
}

151
dbms/src/Databases/DatabaseCloud.h
View File

@ -1,151 +0,0 @@
#pragma once
#include <mutex>
#include <Databases/IDatabase.h>
#include <Common/UInt128.h>
#include <Storages/IStorage.h>
namespace Poco { class Logger; }
namespace DB
{
/** Allows you to create "cloud tables".
* A list of such tables is stored in ZooKeeper.
* All servers that refer to the same path in ZooKeeper see the same list of tables.
* CREATE, DROP, RENAME are atomic.
*
* The replication level N is set for the database.
* When writing to a cloud table, N live servers are selected in some way in different datacenters,
* on each of them a local table is created, and data is written to them.
*
* The engine has the parameters: Cloud(zookeeper_path, replication_factor, datacenter_name)
* Example: Cloud('/clickhouse/clouds/production/', 3, 'FIN')
*
* Structure in ZooKeeper:
*
* cloud_path - the path to the "cloud"; There may be several different independent clouds
/table_definitions - set of unique table definitions so you do not write them many times for a large number of tables
/hash128 -> sql - mapping: hash from table definition (identifier) -> table definition itself as CREATE query
/tables - list of tables
/database_name - name of the database
/name_hash_mod -> compressed_table_list
- the list of tables is made two-level to reduce the number of nodes in ZooKeeper if there are a large number of tables
- nodes are created for each remainder of the hash partition from the table name, for example, to 4096
- and each node stores a list of tables (table name, local table name, hash from structure, hosts list) in a compressed form
/local_tables - a list of local tables so that by the name of the local table you can determine its structure
/database_name
/name_hash_mod -> compressed_table_list
- list of pairs (hash from the table name, hash from the structure) in a compressed form
/locality_keys - a serialized list of locality keys in the order in which they appear
- locality key - an arbitrary string
- the database engine defines the servers for the data location in such a way,
that, with the same set of live servers,
one locality key corresponds to one group of N servers for the data location.
/nodes - the list of servers on which cloud databases are registered with the same path in ZK
/hostname - hostname
TODO /alive - an ephemeral node for pre-testing liveliness
/datacenter - the name of the data center
TODO /disk_space
* For one cloud there may be more than one database named differently. For example, DB `hits` and `visits` can belong to the same cloud.
*/
class DatabaseCloud : public IDatabase
{
private:
const String name;
const String data_path;
String zookeeper_path;
const size_t replication_factor;
const String hostname;
const String datacenter_name;
using Logger = Poco::Logger;
Logger * log;
Context & context;
/** Local tables are tables that are located directly on the local server.
* They store the data of the cloud tables: the cloud table is represented by several local tables on different servers.
* These tables are not visible to the user when listing tables, although they are available when referring by name.
* The name of the local tables has a special form, for example, start with _local so that they do not get confused with the cloud tables.
* Local tables are loaded lazily, on first access.
*/
Tables local_tables_cache;
mutable std::mutex local_tables_mutex;
friend class DatabaseCloudIterator;
public:
DatabaseCloud(
bool attach,
const String & name_,
const String & zookeeper_path_,
size_t replication_factor_,
const String & datacenter_name_,
Context & context_);
String getEngineName() const override { return "Cloud"; }
void loadTables(Context & context, ThreadPool * thread_pool, bool has_force_restore_data_flag) override;
bool isTableExist(const String & table_name) const override;
StoragePtr tryGetTable(const String & table_name) override;
DatabaseIteratorPtr getIterator() override;
bool empty() const override;
void createTable(
const String & table_name, const StoragePtr & table, const ASTPtr & query, const String & engine, const Settings & settings) override;
void removeTable(const String & table_name) override;
void attachTable(const String & table_name, const StoragePtr & table) override;
StoragePtr detachTable(const String & table_name) override;
void renameTable(
const Context & context, const String & table_name, IDatabase & to_database, const String & to_table_name, const Settings & settings) override;
time_t getTableMetadataModificationTime(const String & name) override;
ASTPtr getCreateQuery(const String & table_name) const override;
void shutdown() override;
void drop() override;
void alterTable(
const Context & context,
const String & name,
const NamesAndTypesList & columns,
const NamesAndTypesList & materialized_columns,
const NamesAndTypesList & alias_columns,
const ColumnDefaults & column_defaults,
const ASTModifier & engine_modifier) override
{
throw Exception("ALTER TABLE is not supported by database engine " + getEngineName(), ErrorCodes::NOT_IMPLEMENTED);
}
using Hash = UInt128;
private:
void createZookeeperNodes();
/// Get the name of the node in which the part of the list of tables will be stored. (The list of tables is two-level.)
String getNameOfNodeWithTables(const String & table_name) const;
/// Hash the table name along with the database name.
Hash getTableHash(const String & table_name) const;
/// Table definitions are stored indirectly and addressed by its hash. Calculate the hash.
Hash getHashForTableDefinition(const String & definition) const;
/// Go to ZooKeeper and get a table definition by hash.
String getTableDefinitionFromHash(Hash hash) const;
/// Define the servers on which the table data will be stored.
std::vector<String> selectHostsForTable(const String & locality_key) const;
};
}

38
dbms/src/Databases/DatabaseDictionary.cpp
View File

@ -26,18 +26,19 @@ void DatabaseDictionary::loadTables(Context & context, ThreadPool * thread_pool,
Tables DatabaseDictionary::loadTables()
{
const std::lock_guard<std::mutex> lock_dictionaries {external_dictionaries.dictionaries_mutex};
auto objects_map = external_dictionaries.getObjectsMap();
const auto & dictionaries = objects_map.get();
Tables tables;
for (const auto & pair : external_dictionaries.dictionaries)
for (const auto & pair : dictionaries)
{
const std::string & name = pair.first;
if (deleted_tables.count(name))
continue;
auto dict_ptr = pair.second.dict;
auto dict_ptr = std::static_pointer_cast<IDictionaryBase>(pair.second.loadable);
if (dict_ptr)
{
const DictionaryStructure & dictionary_structure = dict_ptr->get()->getStructure();
const DictionaryStructure & dictionary_structure = dict_ptr->getStructure();
auto columns = StorageDictionary::getNamesAndTypes(dictionary_structure);
tables[name] = StorageDictionary::create(name, columns, {}, {}, {}, dictionary_structure, name);
}
@ -50,26 +51,28 @@ bool DatabaseDictionary::isTableExist(
const Context & context,
const String & table_name) const
{
const std::lock_guard<std::mutex> lock_dictionaries {external_dictionaries.dictionaries_mutex};
return external_dictionaries.dictionaries.count(table_name) && !deleted_tables.count(table_name);
auto objects_map = external_dictionaries.getObjectsMap();
const auto & dictionaries = objects_map.get();
return dictionaries.count(table_name) && !deleted_tables.count(table_name);
}
StoragePtr DatabaseDictionary::tryGetTable(
const Context & context,
const String & table_name)
{
const std::lock_guard<std::mutex> lock_dictionaries {external_dictionaries.dictionaries_mutex};
auto objects_map = external_dictionaries.getObjectsMap();
const auto & dictionaries = objects_map.get();
if (deleted_tables.count(table_name))
return {};
{
auto it = external_dictionaries.dictionaries.find(table_name);
if (it != external_dictionaries.dictionaries.end())
auto it = dictionaries.find(table_name);
if (it != dictionaries.end())
{
const auto & dict_ptr = it->second.dict;
const auto & dict_ptr = std::static_pointer_cast<IDictionaryBase>(it->second.loadable);
if (dict_ptr)
{
const DictionaryStructure & dictionary_structure = dict_ptr->get()->getStructure();
const DictionaryStructure & dictionary_structure = dict_ptr->getStructure();
auto columns = StorageDictionary::getNamesAndTypes(dictionary_structure);
return StorageDictionary::create(table_name, columns, {}, {}, {}, dictionary_structure, table_name);
}
@ -86,9 +89,10 @@ DatabaseIteratorPtr DatabaseDictionary::getIterator(const Context & context)
bool DatabaseDictionary::empty(const Context & context) const
{
const std::lock_guard<std::mutex> lock_dictionaries {external_dictionaries.dictionaries_mutex};
for (const auto & pair : external_dictionaries.dictionaries)
if (pair.second.dict && !deleted_tables.count(pair.first))
auto objects_map = external_dictionaries.getObjectsMap();
const auto & dictionaries = objects_map.get();
for (const auto & pair : dictionaries)
if (pair.second.loadable && !deleted_tables.count(pair.first))
return false;
return true;
}
@ -107,8 +111,7 @@ void DatabaseDictionary::createTable(
const Context & context,
const String & table_name,
const StoragePtr & table,
const ASTPtr & query,
const String & engine)
const ASTPtr & query)
{
throw Exception("DatabaseDictionary: createTable() is not supported", ErrorCodes::NOT_IMPLEMENTED);
}
@ -120,7 +123,7 @@ void DatabaseDictionary::removeTable(
if (!isTableExist(context, table_name))
throw Exception("Table " + name + "." + table_name + " doesn't exist.", ErrorCodes::UNKNOWN_TABLE);
const std::lock_guard<std::mutex> lock_dictionaries {external_dictionaries.dictionaries_mutex};
auto objects_map = external_dictionaries.getObjectsMap();
deleted_tables.insert(table_name);
}
@ -157,7 +160,6 @@ ASTPtr DatabaseDictionary::getCreateQuery(
const String & table_name) const
{
throw Exception("DatabaseDictionary: getCreateQuery() is not supported", ErrorCodes::NOT_IMPLEMENTED);
return nullptr;
}
void DatabaseDictionary::shutdown()

3
dbms/src/Databases/DatabaseDictionary.h
View File

@ -61,8 +61,7 @@ public:
const Context & context,
const String & table_name,
const StoragePtr & table,
const ASTPtr & query,
const String & engine) override;
const ASTPtr & query) override;
void removeTable(
const Context & context,

3
dbms/src/Databases/DatabaseMemory.cpp
View File

@ -80,8 +80,7 @@ void DatabaseMemory::createTable(
const Context & context,
const String & table_name,
const StoragePtr & table,
const ASTPtr & query,
const String & engine)
const ASTPtr & query)
{
attachTable(table_name, table);
}

Some files were not shown because too many files have changed in this diff Show More