diff --git a/docs/en/sql-reference/functions/date-time-functions.md b/docs/en/sql-reference/functions/date-time-functions.md index d54028582c5..2c1fcd05bf7 100644 --- a/docs/en/sql-reference/functions/date-time-functions.md +++ b/docs/en/sql-reference/functions/date-time-functions.md @@ -266,8 +266,14 @@ Result: └────────────────┘ ``` -:::note -The return type `toStartOf*` functions described below is `Date` or `DateTime`. Though these functions can take `DateTime64` as an argument, passing them a `DateTime64` that is out of the normal range (years 1900 - 2299) will give an incorrect result. +:::Attention +The return type of `toStartOf*`, `toLastDayOfMonth`, `toMonday` functions described below is `Date` or `DateTime`. +Though these functions can take values of the extended types `Date32` and `DateTime64` as an argument, passing them a time outside the normal range (year 1970 to 2149 for `Date` / 2106 for `DateTime`) will produce wrong results. +In case argument is out of normal range: + * If the argument is smaller than 1970, the result will be calculated from the argument `1970-01-01 (00:00:00)` instead. + * If the return type is `DateTime` and the argument is larger than `2106-02-07 08:28:15`, the result will be calculated from the argument `2106-02-07 08:28:15` instead. + * If the return type is `Date` and the argument is larger than `2149-06-06`, the result will be calculated from the argument `2149-06-06` instead. + * If `toLastDayOfMonth` is called with an argument greater then `2149-05-31`, the result will be calculated from the argument `2149-05-31` instead. ::: ## toStartOfYear @@ -291,20 +297,23 @@ Returns the date. Rounds down a date or date with time to the first day of the month. Returns the date. -:::note -The behavior of parsing incorrect dates is implementation specific. ClickHouse may return zero date, throw an exception or do “natural” overflow. -::: +## toLastDayOfMonth + +Rounds up a date or date with time to the last day of the month. +Returns the date. ## toMonday Rounds down a date or date with time to the nearest Monday. +As a special case, date arguments `1970-01-01`, `1970-01-02`, `1970-01-03` and `1970-01-04` return date `1970-01-01`. Returns the date. ## toStartOfWeek(t\[,mode\]) Rounds down a date or date with time to the nearest Sunday or Monday by mode. Returns the date. -The mode argument works exactly like the mode argument to toWeek(). For the single-argument syntax, a mode value of 0 is used. +As a special case, date arguments `1970-01-01`, `1970-01-02`, `1970-01-03` and `1970-01-04` (and `1970-01-05` if `mode` is `1`) return date `1970-01-01`. +The `mode` argument works exactly like the mode argument to toWeek(). For the single-argument syntax, a mode value of 0 is used. ## toStartOfDay @@ -1039,7 +1048,7 @@ Example: SELECT timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600)); SELECT timeSlots(toDateTime('1980-12-12 21:01:02', 'UTC'), toUInt32(600), 299); SELECT timeSlots(toDateTime64('1980-12-12 21:01:02.1234', 4, 'UTC'), toDecimal64(600.1, 1), toDecimal64(299, 0)); -``` +``` ``` text ┌─timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600))─┐ │ ['2012-01-01 12:00:00','2012-01-01 12:30:00'] │ diff --git a/docs/ru/sql-reference/data-types/date.md b/docs/ru/sql-reference/data-types/date.md index 185fe28d567..7254b82f461 100644 --- a/docs/ru/sql-reference/data-types/date.md +++ b/docs/ru/sql-reference/data-types/date.md @@ -6,7 +6,7 @@ sidebar_label: Date # Date {#data-type-date} -Дата. Хранится в двух байтах в виде (беззнакового) числа дней, прошедших от 1970-01-01. Позволяет хранить значения от чуть больше, чем начала unix-эпохи до верхнего порога, определяющегося константой на этапе компиляции (сейчас - до 2106 года, последний полностью поддерживаемый год - 2105). +Дата. Хранится в двух байтах в виде (беззнакового) числа дней, прошедших от 1970-01-01. Позволяет хранить значения от чуть больше, чем начала unix-эпохи до верхнего порога, определяющегося константой на этапе компиляции (сейчас - до 2149 года, последний полностью поддерживаемый год - 2148). Диапазон значений: \[1970-01-01, 2149-06-06\]. diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index a8dc8d49698..cfffcfb6021 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -267,10 +267,25 @@ SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp; └────────────────┘ ``` -:::note "Attention" - `Date` или `DateTime` это возвращаемый тип функций `toStartOf*`, который описан ниже. Несмотря на то, что эти функции могут принимать `DateTime64` в качестве аргумента, если переданное значение типа `DateTime64` выходит за пределы нормального диапазона (с 1900 по 2299 год), то это даст неверный результат. +:::Attention +Тип возвращаемого описанными далее функциями `toStartOf*`, `toMonday` значения - `Date` или `DateTime`. +Хотя эти функции могут принимать значения типа `Date32` или `DateTime64` в качестве аргумента, при обработке аргумента вне нормального диапазона значений (`1970` - `2148` для `Date` и `1970-01-01 00:00:00`-`2106-02-07 08:28:15` для `DateTime`) будет получен некорректный результат. +Возвращаемые значения для значений вне нормального диапазона: +* `1970-01-01 (00:00:00)` будет возвращён для моментов времени до 1970 года, +* `2106-02-07 08:28:15` будет взят в качестве аргумента, если полученный аргумент превосходит данное значение и возвращаемый тип - `DateTime`, +* `2149-06-06` будет взят в качестве аргумента, если полученный аргумент превосходит данное значение и возвращаемый тип - `Date`, +* `2149-05-31` будет результатом функции `toLastDayOfMonth` при обработке аргумента больше `2149-05-31`. ::: +:::Attention +Тип возвращаемого описанными далее функциями `toStartOf*`, `toLastDayOfMonth`, `toMonday` значения - `Date` или `DateTime`. +Хотя эти функции могут принимать значения типа `Date32` или `DateTime64` в качестве аргумента, при обработке аргумента вне нормального диапазона значений (`1970` - `2148` для `Date` и `1970-01-01 00:00:00`-`2106-02-07 08:28:15` для `DateTime`) будет получен некорректный результат. +Возвращаемые значения для значений вне нормального диапазона: +* `1970-01-01 (00:00:00)` будет возвращён для моментов времени до 1970 года, +* `2106-02-07 08:28:15` будет взят в качестве аргумента, если полученный аргумент превосходит данное значение и возвращаемый тип - `DateTime`, +* `2149-06-06` будет взят в качестве аргумента, если полученный аргумент превосходит данное значение и возвращаемый тип - `Date`. + ::: +* ## toStartOfYear {#tostartofyear} Округляет дату или дату-с-временем вниз до первого дня года. @@ -304,20 +319,23 @@ SELECT toStartOfISOYear(toDate('2017-01-01')) AS ISOYear20170101; Округляет дату или дату-с-временем вниз до первого дня месяца. Возвращается дата. -:::note "Attention" - Возвращаемое значение для некорректных дат зависит от реализации. ClickHouse может вернуть нулевую дату, выбросить исключение, или выполнить «естественное» перетекание дат между месяцами. -::: - +## toLastDayOfMonth + +Округляет дату или дату-с-временем до последнего числа месяца. +Возвращается дата. + ## toMonday {#tomonday} Округляет дату или дату-с-временем вниз до ближайшего понедельника. +Частный случай: для дат `1970-01-01`, `1970-01-02`, `1970-01-03` и `1970-01-04` результатом будет `1970-01-01`. Возвращается дата. ## toStartOfWeek(t[,mode]) {#tostartofweek} Округляет дату или дату со временем до ближайшего воскресенья или понедельника в соответствии с mode. Возвращается дата. -Аргумент mode работает точно так же, как аргумент mode [toWeek()](#toweek). Если аргумент mode опущен, то используется режим 0. +Частный случай: для дат `1970-01-01`, `1970-01-02`, `1970-01-03` и `1970-01-04` (и `1970-01-05`, если `mode` равен `1`) результатом будет `1970-01-01`. +Аргумент `mode` работает точно так же, как аргумент mode [toWeek()](#toweek). Если аргумент mode опущен, то используется режим 0. ## toStartOfDay {#tostartofday} @@ -958,7 +976,7 @@ SELECT now('Europe/Moscow'); ## timeSlots(StartTime, Duration,\[, Size\]) {#timeslotsstarttime-duration-size} Для интервала, начинающегося в `StartTime` и длящегося `Duration` секунд, возвращает массив моментов времени, кратных `Size`. Параметр `Size` указывать необязательно, по умолчанию он равен 1800 секундам (30 минутам) - необязательный параметр. -Данная функция может использоваться, например, для анализа количества просмотров страницы за соответствующую сессию. +Данная функция может использоваться, например, для анализа количества просмотров страницы за соответствующую сессию. Аргумент `StartTime` может иметь тип `DateTime` или `DateTime64`. В случае, если используется `DateTime`, аргументы `Duration` и `Size` должны иметь тип `UInt32`; Для DateTime64 они должны быть типа `Decimal64`. Возвращает массив DateTime/DateTime64 (тип будет совпадать с типом параметра ’StartTime’). Для DateTime64 масштаб(scale) возвращаемой величины может отличаться от масштаба фргумента ’StartTime’ --- результат будет иметь наибольший масштаб среди всех данных аргументов. diff --git a/docs/zh/sql-reference/data-types/date.md b/docs/zh/sql-reference/data-types/date.md index a8874151e75..9b1acdbe939 100644 --- a/docs/zh/sql-reference/data-types/date.md +++ b/docs/zh/sql-reference/data-types/date.md @@ -3,7 +3,7 @@ slug: /zh/sql-reference/data-types/date --- # 日期 {#date} -日期类型，用两个字节存储，表示从 1970-01-01 (无符号) 到当前的日期值。允许存储从 Unix 纪元开始到编译阶段定义的上限阈值常量（目前上限是2106年，但最终完全支持的年份为2105）。最小值输出为1970-01-01。 +日期类型，用两个字节存储，表示从 1970-01-01 (无符号) 到当前的日期值。允许存储从 Unix 纪元开始到编译阶段定义的上限阈值常量（目前上限是2149年，但最终完全支持的年份为2148）。最小值输出为1970-01-01。 值的范围: \[1970-01-01, 2149-06-06\]。 diff --git a/src/Functions/CustomWeekTransforms.h b/src/Functions/CustomWeekTransforms.h index c296c8228b1..3378aec02d5 100644 --- a/src/Functions/CustomWeekTransforms.h +++ b/src/Functions/CustomWeekTransforms.h @@ -62,7 +62,10 @@ struct ToStartOfWeekImpl static inline UInt16 execute(Int64 t, UInt8 week_mode, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfWeek(time_zone.toDayNum(t), week_mode); + if (t < 0) + return 0; + + return time_zone.toFirstDayNumOfWeek(DayNum(std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM))), week_mode); } static inline UInt16 execute(UInt32 t, UInt8 week_mode, const DateLUTImpl & time_zone) { @@ -70,7 +73,10 @@ struct ToStartOfWeekImpl } static inline UInt16 execute(Int32 d, UInt8 week_mode, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfWeek(ExtendedDayNum(d), week_mode); + if (d < 0) + return 0; + + return time_zone.toFirstDayNumOfWeek(DayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM))), week_mode); } static inline UInt16 execute(UInt16 d, UInt8 week_mode, const DateLUTImpl & time_zone) { diff --git a/src/Functions/DateTimeTransforms.h b/src/Functions/DateTimeTransforms.h index 065f08296d0..66d57f2463f 100644 --- a/src/Functions/DateTimeTransforms.h +++ b/src/Functions/DateTimeTransforms.h @@ -61,15 +61,15 @@ struct ToDateImpl static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone) { - return UInt16(time_zone.toDayNum(t)); + return t < 0 ? 0 : std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM)); } static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone) { - return UInt16(time_zone.toDayNum(t)); + return time_zone.toDayNum(t); } - static inline UInt16 execute(Int32, const DateLUTImpl &) + static inline UInt16 execute(Int32 t, const DateLUTImpl &) { - return dateIsNotSupported(name); + return t < 0 ? 0 : std::min(t, Int32(DATE_LUT_MAX_DAY_NUM)); } static inline UInt16 execute(UInt16 d, const DateLUTImpl &) { @@ -111,7 +111,10 @@ struct ToStartOfDayImpl //TODO: right now it is hardcoded to produce DateTime only, needs fixing later. See date_and_time_type_details::ResultDataTypeMap for deduction of result type example. static inline UInt32 execute(const DecimalUtils::DecimalComponents & t, const DateLUTImpl & time_zone) { - return time_zone.toDate(static_cast(t.whole)); + if (t.whole < 0 || (t.whole >= 0 && t.fractional < 0)) + return 0; + + return time_zone.toDate(std::min(t.whole, Int64(0xffffffff))); } static inline UInt32 execute(UInt32 t, const DateLUTImpl & time_zone) { @@ -119,11 +122,19 @@ struct ToStartOfDayImpl } static inline UInt32 execute(Int32 d, const DateLUTImpl & time_zone) { - return time_zone.toDate(ExtendedDayNum(d)); + if (d < 0) + return 0; + + auto date_time = time_zone.fromDayNum(ExtendedDayNum(d)); + if (date_time <= 0xffffffff) + return date_time; + else + return time_zone.toDate(0xffffffff); } static inline UInt32 execute(UInt16 d, const DateLUTImpl & time_zone) { - return time_zone.toDate(DayNum(d)); + auto date_time = time_zone.fromDayNum(ExtendedDayNum(d)); + return date_time < 0xffffffff ? date_time : time_zone.toDate(0xffffffff); } using FactorTransform = ZeroTransform; @@ -135,17 +146,16 @@ struct ToMondayImpl static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone) { - //return time_zone.toFirstDayNumOfWeek(time_zone.toDayNum(t)); - return time_zone.toFirstDayNumOfWeek(t); + return t < 0 ? 0 : time_zone.toFirstDayNumOfWeek(ExtendedDayNum( + std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM)))); } static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone) { - //return time_zone.toFirstDayNumOfWeek(time_zone.toDayNum(t)); return time_zone.toFirstDayNumOfWeek(t); } static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfWeek(ExtendedDayNum(d)); + return d < 0 ? 0 : time_zone.toFirstDayNumOfWeek(ExtendedDayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM)))); } static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone) { @@ -161,15 +171,15 @@ struct ToStartOfMonthImpl static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfMonth(time_zone.toDayNum(t)); + return t < 0 ? 0 : time_zone.toFirstDayNumOfMonth(ExtendedDayNum(std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM)))); } static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfMonth(time_zone.toDayNum(t)); + return time_zone.toFirstDayNumOfMonth(ExtendedDayNum(time_zone.toDayNum(t))); } static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfMonth(ExtendedDayNum(d)); + return d < 0 ? 0 : time_zone.toFirstDayNumOfMonth(ExtendedDayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM)))); } static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone) { @@ -185,7 +195,11 @@ struct ToLastDayOfMonthImpl static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone) { - return time_zone.toLastDayNumOfMonth(time_zone.toDayNum(t)); + if (t < 0) + return 0; + + /// 0xFFF9 is Int value for 2149-05-31 -- the last day where we can actually find LastDayOfMonth. This will also be the return value. + return time_zone.toLastDayNumOfMonth(ExtendedDayNum(std::min(Int32(time_zone.toDayNum(t)), Int32(0xFFF9)))); } static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone) { @@ -193,11 +207,16 @@ struct ToLastDayOfMonthImpl } static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone) { - return time_zone.toLastDayNumOfMonth(ExtendedDayNum(d)); + if (d < 0) + return 0; + + /// 0xFFF9 is Int value for 2149-05-31 -- the last day where we can actually find LastDayOfMonth. This will also be the return value. + return time_zone.toLastDayNumOfMonth(ExtendedDayNum(std::min(d, Int32(0xFFF9)))); } static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone) { - return time_zone.toLastDayNumOfMonth(DayNum(d)); + /// 0xFFF9 is Int value for 2149-05-31 -- the last day where we can actually find LastDayOfMonth. This will also be the return value. + return time_zone.toLastDayNumOfMonth(DayNum(std::min(d, UInt16(0xFFF9)))); } using FactorTransform = ZeroTransform; @@ -209,7 +228,7 @@ struct ToStartOfQuarterImpl static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfQuarter(time_zone.toDayNum(t)); + return t < 0 ? 0 : time_zone.toFirstDayNumOfQuarter(ExtendedDayNum(std::min(Int64(time_zone.toDayNum(t)), Int64(DATE_LUT_MAX_DAY_NUM)))); } static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone) { @@ -217,7 +236,7 @@ struct ToStartOfQuarterImpl } static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfQuarter(ExtendedDayNum(d)); + return d < 0 ? 0 : time_zone.toFirstDayNumOfQuarter(ExtendedDayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM)))); } static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone) { @@ -233,7 +252,7 @@ struct ToStartOfYearImpl static inline UInt16 execute(Int64 t, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfYear(time_zone.toDayNum(t)); + return t < 0 ? 0 : time_zone.toFirstDayNumOfYear(ExtendedDayNum(std::min(Int32(time_zone.toDayNum(t)), Int32(DATE_LUT_MAX_DAY_NUM)))); } static inline UInt16 execute(UInt32 t, const DateLUTImpl & time_zone) { @@ -241,7 +260,7 @@ struct ToStartOfYearImpl } static inline UInt16 execute(Int32 d, const DateLUTImpl & time_zone) { - return time_zone.toFirstDayNumOfYear(ExtendedDayNum(d)); + return d < 0 ? 0 : time_zone.toFirstDayNumOfYear(ExtendedDayNum(std::min(d, Int32(DATE_LUT_MAX_DAY_NUM)))); } static inline UInt16 execute(UInt16 d, const DateLUTImpl & time_zone) { @@ -283,7 +302,10 @@ struct ToStartOfMinuteImpl static inline UInt32 execute(const DecimalUtils::DecimalComponents & t, const DateLUTImpl & time_zone) { - return time_zone.toStartOfMinute(t.whole); + if (t.whole < 0 || (t.whole >= 0 && t.fractional < 0)) + return 0; + + return time_zone.toStartOfMinute(std::min(t.whole, Int64(0xffffffff))); } static inline UInt32 execute(UInt32 t, const DateLUTImpl & time_zone) { @@ -574,7 +596,10 @@ struct ToStartOfHourImpl static inline UInt32 execute(const DecimalUtils::DecimalComponents & t, const DateLUTImpl & time_zone) { - return time_zone.toStartOfHour(t.whole); + if (t.whole < 0 || (t.whole >= 0 && t.fractional < 0)) + return 0; + + return time_zone.toStartOfHour(std::min(t.whole, Int64(0xffffffff))); } static inline UInt32 execute(UInt32 t, const DateLUTImpl & time_zone) diff --git a/src/Functions/FunctionCustomWeekToSomething.h b/src/Functions/FunctionCustomWeekToSomething.h index 6ed751fd889..8a0f474a7e8 100644 --- a/src/Functions/FunctionCustomWeekToSomething.h +++ b/src/Functions/FunctionCustomWeekToSomething.h @@ -41,23 +41,20 @@ public: if (!isDate(arguments[0].type) && !isDate32(arguments[0].type) && !isDateTime(arguments[0].type) && !isDateTime64(arguments[0].type)) throw Exception( "Illegal type " + arguments[0].type->getName() + " of argument of function " + getName() - + ". Should be a date or a date with time", + + ". Must be Date, Date32, DateTime or DateTime64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } else if (arguments.size() == 2) { if (!isDate(arguments[0].type) && !isDate32(arguments[0].type) && !isDateTime(arguments[0].type) && !isDateTime64(arguments[0].type)) throw Exception( - "Illegal type " + arguments[0].type->getName() + " of argument of function " + getName() - + ". Should be a date or a date with time", + "Illegal type " + arguments[0].type->getName() + " of 1st argument of function " + getName() + + ". Must be Date, Date32, DateTime or DateTime64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); if (!isUInt8(arguments[1].type)) throw Exception( - "Function " + getName() - + " supports 1 or 2 or 3 arguments. The 1st argument " - "must be of type Date or DateTime. The 2nd argument (optional) must be " - "a constant UInt8 with week mode. The 3rd argument (optional) must be " - "a constant string with timezone name", + "Illegal type of 2nd (optional) argument of function " + getName() + + ". Must be constant UInt8 (week mode).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } else if (arguments.size() == 3) @@ -65,33 +62,28 @@ public: if (!isDate(arguments[0].type) && !isDate32(arguments[0].type) && !isDateTime(arguments[0].type) && !isDateTime64(arguments[0].type)) throw Exception( "Illegal type " + arguments[0].type->getName() + " of argument of function " + getName() - + ". Should be a date or a date with time", + + ". Must be Date, Date32, DateTime or DateTime64", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); if (!isUInt8(arguments[1].type)) throw Exception( - "Function " + getName() - + " supports 1 or 2 or 3 arguments. The 1st argument " - "must be of type Date or DateTime. The 2nd argument (optional) must be " - "a constant UInt8 with week mode. The 3rd argument (optional) must be " - "a constant string with timezone name", + "Illegal type of 2nd (optional) argument of function " + getName() + + ". Must be constant UInt8 (week mode).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); if (!isString(arguments[2].type)) throw Exception( - "Function " + getName() - + " supports 1 or 2 or 3 arguments. The 1st argument " - "must be of type Date or DateTime. The 2nd argument (optional) must be " - "a constant UInt8 with week mode. The 3rd argument (optional) must be " - "a constant string with timezone name", + "Illegal type of 3rd (optional) argument of function " + getName() + + ". Must be constant string (timezone name).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); - if (isDate(arguments[0].type) && std::is_same_v) + if ((isDate(arguments[0].type) || isDate32(arguments[0].type)) + && (std::is_same_v || std::is_same_v)) throw Exception( - "The timezone argument of function " + getName() + " is allowed only when the 1st argument has the type DateTime", + "The timezone argument of function " + getName() + " is allowed only when the 1st argument is DateTime or DateTime64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } else throw Exception( "Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size()) - + ", should be 1 or 2 or 3", + + ", expected 1, 2 or 3.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); return std::make_shared(); diff --git a/src/Functions/FunctionsConversion.h b/src/Functions/FunctionsConversion.h index 231ec188ade..96c28b21ef0 100644 --- a/src/Functions/FunctionsConversion.h +++ b/src/Functions/FunctionsConversion.h @@ -301,6 +301,11 @@ struct ConvertImpl } }; +/** Conversion of Date32 to Date: check bounds. + */ +template struct ConvertImpl + : DateTimeTransformImpl {}; + /** Conversion of DateTime to Date: throw off time component. */ template struct ConvertImpl @@ -319,12 +324,17 @@ struct ToDateTimeImpl static inline UInt32 execute(UInt16 d, const DateLUTImpl & time_zone) { - return time_zone.fromDayNum(DayNum(d)); + auto date_time = time_zone.fromDayNum(ExtendedDayNum(d)); + return date_time <= 0xffffffff ? UInt32(date_time) : UInt32(0xffffffff); } - static inline Int64 execute(Int32 d, const DateLUTImpl & time_zone) + static inline UInt32 execute(Int32 d, const DateLUTImpl & time_zone) { - return time_zone.fromDayNum(ExtendedDayNum(d)); + if (d < 0) + return 0; + + auto date_time = time_zone.fromDayNum(ExtendedDayNum(d)); + return date_time <= 0xffffffff ? date_time : 0xffffffff; } static inline UInt32 execute(UInt32 dt, const DateLUTImpl & /*time_zone*/) @@ -332,10 +342,21 @@ struct ToDateTimeImpl return dt; } - // TODO: return UInt32 ??? - static inline Int64 execute(Int64 dt64, const DateLUTImpl & /*time_zone*/) + static inline UInt32 execute(Int64 d, const DateLUTImpl & time_zone) { - return dt64; + if (d < 0) + return 0; + + auto date_time = time_zone.toDate(d); + return date_time <= 0xffffffff ? date_time : 0xffffffff; + } + + static inline UInt32 execute(const DecimalUtils::DecimalComponents & t, const DateLUTImpl & /*time_zone*/) + { + if (t.whole < 0 || (t.whole >= 0 && t.fractional < 0)) + return 0; + + return std::min(t.whole, Int64(0xFFFFFFFF)); } }; @@ -355,9 +376,12 @@ struct ToDateTransform32Or64 static inline NO_SANITIZE_UNDEFINED ToType execute(const FromType & from, const DateLUTImpl & time_zone) { // since converting to Date, no need in values outside of default LUT range. + if (from < 0) + return 0; + return (from < DATE_LUT_MAX_DAY_NUM) ? from - : time_zone.toDayNum(std::min(time_t(from), time_t(0xFFFFFFFF))); + : std::min(Int32(time_zone.toDayNum(from)), Int32(DATE_LUT_MAX_DAY_NUM)); } }; @@ -372,9 +396,14 @@ struct ToDateTransform32Or64Signed /// The function should be monotonic (better for query optimizations), so we saturate instead of overflow. if (from < 0) return 0; + + auto day_num = time_zone.toDayNum(ExtendedDayNum(from)); + return day_num < DATE_LUT_MAX_DAY_NUM ? day_num : DATE_LUT_MAX_DAY_NUM; + return (from < DATE_LUT_MAX_DAY_NUM) ? from - : time_zone.toDayNum(std::min(time_t(from), time_t(0xFFFFFFFF))); + : std::min(Int32(time_zone.toDayNum(from)), Int32(0xFFFFFFFF)); + } }; @@ -405,7 +434,7 @@ struct ToDate32Transform32Or64 { return (from < DATE_LUT_MAX_EXTEND_DAY_NUM) ? from - : time_zone.toDayNum(std::min(time_t(from), time_t(0xFFFFFFFF))); + : std::min(Int32(time_zone.toDayNum(from)), Int32(DATE_LUT_MAX_EXTEND_DAY_NUM)); } }; @@ -421,7 +450,7 @@ struct ToDate32Transform32Or64Signed return daynum_min_offset; return (from < DATE_LUT_MAX_EXTEND_DAY_NUM) ? from - : time_zone.toDayNum(std::min(time_t(from), time_t(0xFFFFFFFF))); + : time_zone.toDayNum(std::min(Int64(from), Int64(0xFFFFFFFF))); } }; @@ -447,35 +476,49 @@ struct ToDate32Transform8Or16Signed */ template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; + template struct ConvertImpl : DateTimeTransformImpl> {}; @@ -487,7 +530,7 @@ struct ToDateTimeTransform64 static inline NO_SANITIZE_UNDEFINED ToType execute(const FromType & from, const DateLUTImpl &) { - return std::min(time_t(from), time_t(0xFFFFFFFF)); + return std::min(Int64(from), Int64(0xFFFFFFFF)); } }; @@ -509,11 +552,12 @@ struct ToDateTimeTransform64Signed { static constexpr auto name = "toDateTime"; - static inline NO_SANITIZE_UNDEFINED ToType execute(const FromType & from, const DateLUTImpl &) + static inline NO_SANITIZE_UNDEFINED ToType execute(const FromType & from, const DateLUTImpl & /* time_zone */) { if (from < 0) return 0; - return std::min(time_t(from), time_t(0xFFFFFFFF)); + + return std::min(Int64(from), Int64(0xFFFFFFFF)); } }; @@ -634,8 +678,6 @@ struct FromDateTime64Transform } }; -/** Conversion of DateTime64 to Date or DateTime: discards fractional part. - */ template struct ConvertImpl : DateTimeTransformImpl> {}; template struct ConvertImpl @@ -659,7 +701,7 @@ struct ToDateTime64Transform inline DateTime64::NativeType execute(Int32 d, const DateLUTImpl & time_zone) const { - const auto dt = ToDateTimeImpl::execute(d, time_zone); + const auto dt = time_zone.fromDayNum(ExtendedDayNum(d)); return DecimalUtils::decimalFromComponentsWithMultiplier(dt, 0, scale_multiplier); } @@ -1806,7 +1848,7 @@ private: { /// Account for optional timezone argument. if (arguments.size() != 2 && arguments.size() != 3) - throw Exception{"Function " + getName() + " expects 2 or 3 arguments for DataTypeDateTime64.", + throw Exception{"Function " + getName() + " expects 2 or 3 arguments for DateTime64.", ErrorCodes::TOO_FEW_ARGUMENTS_FOR_FUNCTION}; } else if (arguments.size() != 2) diff --git a/tests/queries/0_stateless/00900_long_parquet.reference b/tests/queries/0_stateless/00900_long_parquet.reference index bbdad7243bd..4dfc726145e 100644 --- a/tests/queries/0_stateless/00900_long_parquet.reference +++ b/tests/queries/0_stateless/00900_long_parquet.reference @@ -44,12 +44,12 @@ converted: diff: dest: 79 81 82 83 84 85 86 87 88 89 str01\0\0\0\0\0\0\0\0\0\0 fstr1\0\0\0\0\0\0\0\0\0\0 2003-03-04 2004-05-06 00:00:00 2004-05-06 07:08:09.012000000 -80 81 82 83 84 85 86 87 88 89 str02 fstr2\0\0\0\0\0\0\0\0\0\0 2005-03-04 2006-08-09 10:11:12 2006-08-09 10:11:12.345000000 +80 81 82 83 84 85 86 87 88 89 str02 fstr2\0\0\0\0\0\0\0\0\0\0 2149-06-06 2006-08-09 10:11:12 2006-08-09 10:11:12.345000000 min: --128 0 0 0 0 0 0 0 -1 -1 string-1\0\0\0\0\0\0\0 fixedstring-1\0\0 2003-04-05 2003-02-03 2003-02-03 04:05:06.789000000 --108 108 8 92 -8 108 -40 -116 -1 -1 string-0\0\0\0\0\0\0\0 fixedstring\0\0\0\0 2001-02-03 2002-02-03 2002-02-03 04:05:06.789000000 +-128 0 0 0 0 0 0 0 -1 -1 string-1\0\0\0\0\0\0\0 fixedstring-1\0\0 2003-04-05 2149-06-06 2003-02-03 04:05:06.789000000 +-108 108 8 92 -8 108 -40 -116 -1 -1 string-0\0\0\0\0\0\0\0 fixedstring\0\0\0\0 2001-02-03 2149-06-06 2002-02-03 04:05:06.789000000 79 81 82 83 84 85 86 87 88 89 str01\0\0\0\0\0\0\0\0\0\0 fstr1\0\0\0\0\0\0\0\0\0\0 2003-03-04 2004-05-06 2004-05-06 07:08:09.012000000 -127 -1 -1 -1 -1 -1 -1 -1 -1 -1 string-2\0\0\0\0\0\0\0 fixedstring-2\0\0 2004-06-07 2004-02-03 2004-02-03 04:05:06.789000000 +127 -1 -1 -1 -1 -1 -1 -1 -1 -1 string-2\0\0\0\0\0\0\0 fixedstring-2\0\0 2004-06-07 2149-06-06 2004-02-03 04:05:06.789000000 max: -128 0 -32768 0 -2147483648 0 -9223372036854775808 0 -1 -1 string-1 fixedstring-1\0\0 2003-04-05 00:00:00 2003-02-03 04:05:06 2003-02-03 04:05:06.789000000 -108 108 -1016 1116 -1032 1132 -1064 1164 -1 -1 string-0 fixedstring\0\0\0\0 2001-02-03 00:00:00 2002-02-03 04:05:06 2002-02-03 04:05:06.789000000 diff --git a/tests/queries/0_stateless/00941_to_custom_week.sql b/tests/queries/0_stateless/00941_to_custom_week.sql index 4dd5d209306..04ff08d4117 100644 --- a/tests/queries/0_stateless/00941_to_custom_week.sql +++ b/tests/queries/0_stateless/00941_to_custom_week.sql @@ -49,4 +49,3 @@ SELECT toStartOfWeek(x, 3) AS w3, toStartOfWeek(x_t, 3) AS wt3 FROM numbers(10); - diff --git a/tests/queries/0_stateless/01440_to_date_monotonicity.reference b/tests/queries/0_stateless/01440_to_date_monotonicity.reference index 2dbec540fbb..dd8545b721d 100644 --- a/tests/queries/0_stateless/01440_to_date_monotonicity.reference +++ b/tests/queries/0_stateless/01440_to_date_monotonicity.reference @@ -1,4 +1,4 @@ 0 -1970-01-01 2106-02-07 1970-04-11 1970-01-01 2149-06-06 +1970-01-01 2120-07-26 1970-04-11 1970-01-01 2149-06-06 1970-01-01 02:00:00 2106-02-07 09:28:15 1970-01-01 02:16:40 2000-01-01 13:12:12 diff --git a/tests/queries/0_stateless/01921_datatype_date32.reference b/tests/queries/0_stateless/01921_datatype_date32.reference index acb0cc4ca59..a33a96ffffb 100644 --- a/tests/queries/0_stateless/01921_datatype_date32.reference +++ b/tests/queries/0_stateless/01921_datatype_date32.reference @@ -43,16 +43,16 @@ -------toMinute--------- -------toSecond--------- -------toStartOfDay--------- -2036-02-07 07:31:20 -2036-02-07 07:31:20 -2027-10-01 11:03:28 -2027-10-17 11:03:28 +1970-01-01 02:00:00 +1970-01-01 02:00:00 +2106-02-07 00:00:00 +2106-02-07 00:00:00 2021-06-22 00:00:00 -------toMonday--------- -2079-06-07 -2079-06-07 -2120-07-06 -2120-07-20 +1970-01-01 +1970-01-01 +2149-06-02 +2149-06-02 2021-06-21 -------toISOWeek--------- 1 @@ -79,28 +79,28 @@ 229953 202125 -------toStartOfWeek--------- -2079-06-06 -2079-06-06 -2120-07-05 -2120-07-26 +1970-01-01 +1970-01-01 +2149-06-01 +2149-06-01 2021-06-20 -------toStartOfMonth--------- -2079-06-07 -2079-06-07 -2120-06-26 -2120-06-26 +1970-01-01 +1970-01-01 +2149-06-01 +2149-06-01 2021-06-01 -------toStartOfQuarter--------- -2079-06-07 -2079-06-07 -2120-04-26 -2120-04-26 +1970-01-01 +1970-01-01 +2149-04-01 +2149-04-01 2021-04-01 -------toStartOfYear--------- -2079-06-07 -2079-06-07 -2119-07-28 -2119-07-28 +1970-01-01 +1970-01-01 +2149-01-01 +2149-01-01 2021-01-01 -------toStartOfSecond--------- -------toStartOfMinute--------- diff --git a/tests/queries/0_stateless/02403_date_time_narrowing.reference b/tests/queries/0_stateless/02403_date_time_narrowing.reference new file mode 100644 index 00000000000..7d6e91c61b8 --- /dev/null +++ b/tests/queries/0_stateless/02403_date_time_narrowing.reference @@ -0,0 +1,20 @@ +1970-01-01 2149-06-06 1970-01-01 2149-06-06 1900-01-01 1970-01-02 1970-01-01 00:00:00 2106-02-07 06:28:15 +1970-01-01 2149-06-06 +1970-01-01 2149-06-06 +1970-01-01 00:00:00 2106-02-07 06:28:15 +1970-01-01 00:00:00 2106-02-07 06:28:15 +2106-02-07 06:28:15 +toStartOfDay +2106-02-07 00:00:00 1970-01-01 00:00:00 2106-02-07 00:00:00 1970-01-01 00:00:00 2106-02-07 00:00:00 +toStartOfWeek +1970-01-01 1970-01-01 1970-01-01 1970-01-01 1970-01-01 2149-06-01 1970-01-01 2149-06-02 +toMonday +1970-01-01 1970-01-01 2149-06-02 1970-01-01 2149-06-02 +toStartOfMonth +1970-01-01 2149-06-01 1970-01-01 2149-06-01 +toLastDayOfMonth +2149-05-31 1970-01-01 2149-05-31 1970-01-01 2149-05-31 +toStartOfQuarter +1970-01-01 2149-04-01 1970-01-01 2149-04-01 +toStartOfYear +1970-01-01 2149-01-01 1970-01-01 2149-01-01 diff --git a/tests/queries/0_stateless/02403_date_time_narrowing.sql b/tests/queries/0_stateless/02403_date_time_narrowing.sql new file mode 100644 index 00000000000..07cbba6f31c --- /dev/null +++ b/tests/queries/0_stateless/02403_date_time_narrowing.sql @@ -0,0 +1,74 @@ +-- check conversion of numbers to date/time -- +SELECT toDate(toInt32(toDate32('1930-01-01', 'UTC')), 'UTC'), + toDate(toInt32(toDate32('2151-01-01', 'UTC')), 'UTC'), + toDate(toInt64(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC')), 'UTC'), + toDate(toInt64(toDateTime64('2151-01-01 12:12:12.123', 3, 'UTC')), 'UTC'), + toDate32(toInt32(toDate32('1900-01-01', 'UTC')) - 1, 'UTC'), + toDate32(toInt32(toDate32('2299-12-31', 'UTC')) + 1, 'UTC'), + toDateTime(toInt64(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC')), 'UTC'), + toDateTime(toInt64(toDateTime64('2151-01-01 12:12:12.123', 3, 'UTC')), 'UTC'); + +-- check conversion of extended range type to normal range type -- +SELECT toDate(toDate32('1930-01-01', 'UTC'), 'UTC'), + toDate(toDate32('2151-01-01', 'UTC'), 'UTC'); + +SELECT toDate(toDateTime64('1930-01-01 12:12:12.12', 3, 'UTC'), 'UTC'), + toDate(toDateTime64('2151-01-01 12:12:12.12', 3, 'UTC'), 'UTC'); + +SELECT toDateTime(toDateTime64('1930-01-01 12:12:12.12', 3, 'UTC'), 'UTC'), + toDateTime(toDateTime64('2151-01-01 12:12:12.12', 3, 'UTC'), 'UTC'); + +SELECT toDateTime(toDate32('1930-01-01', 'UTC'), 'UTC'), + toDateTime(toDate32('2151-01-01', 'UTC'), 'UTC'); + +SELECT toDateTime(toDate('2141-01-01', 'UTC'), 'UTC'); + +-- test DateTimeTransforms -- +SELECT 'toStartOfDay'; +SELECT toStartOfDay(toDate('2141-01-01', 'UTC'), 'UTC'), + toStartOfDay(toDate32('1930-01-01', 'UTC'), 'UTC'), + toStartOfDay(toDate32('2141-01-01', 'UTC'), 'UTC'), + toStartOfDay(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC'), 'UTC'), + toStartOfDay(toDateTime64('2141-01-01 12:12:12.123', 3, 'UTC'), 'UTC'); + +SELECT 'toStartOfWeek'; +SELECT toStartOfWeek(toDate('1970-01-01', 'UTC')), + toStartOfWeek(toDate32('1970-01-01', 'UTC')), + toStartOfWeek(toDateTime('1970-01-01 10:10:10', 'UTC'), 0, 'UTC'), + toStartOfWeek(toDateTime64('1970-01-01 10:10:10.123', 3, 'UTC'), 1, 'UTC'), + toStartOfWeek(toDate32('1930-01-01', 'UTC')), + toStartOfWeek(toDate32('2151-01-01', 'UTC')), + toStartOfWeek(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC'), 2, 'UTC'), + toStartOfWeek(toDateTime64('2151-01-01 12:12:12.123', 3, 'UTC'), 3, 'UTC'); + +SELECT 'toMonday'; +SELECT toMonday(toDate('1970-01-02', 'UTC')), + toMonday(toDate32('1930-01-01', 'UTC')), + toMonday(toDate32('2151-01-01', 'UTC')), + toMonday(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC'), 'UTC'), + toMonday(toDateTime64('2151-01-01 12:12:12.123', 3, 'UTC'), 'UTC'); + +SELECT 'toStartOfMonth'; +SELECT toStartOfMonth(toDate32('1930-01-01', 'UTC')), + toStartOfMonth(toDate32('2151-01-01', 'UTC')), + toStartOfMonth(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC'), 'UTC'), + toStartOfMonth(toDateTime64('2151-01-01 12:12:12.123', 3, 'UTC'), 'UTC'); + +SELECT 'toLastDayOfMonth'; +SELECT toLastDayOfMonth(toDate('2149-06-03', 'UTC')), + toLastDayOfMonth(toDate32('1930-01-01', 'UTC')), + toLastDayOfMonth(toDate32('2151-01-01', 'UTC')), + toLastDayOfMonth(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC'), 'UTC'), + toLastDayOfMonth(toDateTime64('2151-01-01 12:12:12.123', 3, 'UTC'), 'UTC'); + +SELECT 'toStartOfQuarter'; +SELECT toStartOfQuarter(toDate32('1930-01-01', 'UTC')), + toStartOfQuarter(toDate32('2151-01-01', 'UTC')), + toStartOfQuarter(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC'), 'UTC'), + toStartOfQuarter(toDateTime64('2151-01-01 12:12:12.123', 3, 'UTC'), 'UTC'); + +SELECT 'toStartOfYear'; +SELECT toStartOfYear(toDate32('1930-01-01', 'UTC')), + toStartOfYear(toDate32('2151-01-01', 'UTC')), + toStartOfYear(toDateTime64('1930-01-01 12:12:12.123', 3, 'UTC'), 'UTC'), + toStartOfYear(toDateTime64('2151-01-01 12:12:12.123', 3, 'UTC'), 'UTC');