thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-10 09:32:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:ClickHouse/ClickHouse into implement-todo

Browse Source
This commit is contained in:
Alexey Milovidov 2024-02-18 09:24:22 +01:00
commit 73ab7a5556
47 changed files with 245 additions and 231 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
contrib/curl vendored

@ -1 +1 @@
Subproject commit 7161cb17c01dcff1dc5bf89a18437d9d729f1ecd
Subproject commit 5ce164e0e9290c96eb7d502173426c0a135ec008

2
contrib/lz4 vendored

@ -1 +1 @@
Subproject commit 92ebf1870b9acbefc0e7970409a181954a10ff40
Subproject commit ce45a9dbdb059511a3e9576b19db3e7f1a4f172e

14
docs/en/sql-reference/statements/insert-into.md
View File

@ -204,6 +204,20 @@ Result:
└─────┴───────────────────────┘
```
## Inserts into ClickHouse Cloud
By default, services on ClickHouse Cloud provide multiple replicas for high availability. When you connect to a service, a connection is established to one of these replicas.
After an `INSERT` succeeds, data is written to the underlying storage. However, it may take some time for replicas to receive these updates. Therefore, if you use a different connection that executes a `SELECT` query on one of these other replicas, the updated data may not yet be reflected.
It is possible to use the `select_sequential_consistency` to force the replica to receive the latest updates. Here is an example of a SELECT query using this setting:
```sql
SELECT .... SETTINGS select_sequential_consistency = 1;
```
Note that using `select_sequential_consistency` will increase the load on ClickHouse Keeper (used by ClickHouse Cloud internally) and may result in slower performance depending on the load on the service. We recommend against enabling this setting unless necessary. The recommended approach is to execute read/writes in the same session or to use a client driver that uses the native protocol (and thus supports sticky connections).
## Performance Considerations
`INSERT` sorts the input data by primary key and splits them into partitions by a partition key. If you insert data into several partitions at once, it can significantly reduce the performance of the `INSERT` query. To avoid this:

78
programs/client/Client.cpp
View File

@ -845,83 +845,7 @@ bool Client::processWithFuzzing(const String & full_query)
have_error = true;
}
// Check that after the query is formatted, we can parse it back,
// format again and get the same result. Unfortunately, we can't
// compare the ASTs, which would be more sensitive to errors. This
// double formatting check doesn't catch all errors, e.g. we can
// format query incorrectly, but to a valid SQL that we can then
// parse and format into the same SQL.
// There are some complicated cases where we can generate the SQL
// which we can't parse:
// * first argument of lambda() replaced by fuzzer with
// something else, leading to constructs such as
// arrayMap((min(x) + 3) -> x + 1, ....)
// * internals of Enum replaced, leading to:
// Enum(equals(someFunction(y), 3)).
// And there are even the cases when we can parse the query, but
// it's logically incorrect and its formatting is a mess, such as
// when `lambda()` function gets substituted into a wrong place.
// To avoid dealing with these cases, run the check only for the
// queries we were able to successfully execute.
// Another caveat is that sometimes WITH queries are not executed,
// if they are not referenced by the main SELECT, so they can still
// have the aforementioned problems. Disable this check for such
// queries, for lack of a better solution.
// There is also a problem that fuzzer substitutes positive Int64
// literals or Decimal literals, which are then parsed back as
// UInt64, and suddenly duplicate alias substitution starts or stops
// working (ASTWithAlias::formatImpl) or something like that.
// So we compare not even the first and second formatting of the
// query, but second and third.
// If you have to add any more workarounds to this check, just remove
// it altogether, it's not so useful.
if (ast_to_process && !have_error && !queryHasWithClause(*ast_to_process))
{
ASTPtr ast_2;
try
{
const auto * tmp_pos = query_to_execute.c_str();
ast_2 = parseQuery(tmp_pos, tmp_pos + query_to_execute.size(), false /* allow_multi_statements */);
}
catch (Exception & e)
{
if (e.code() != ErrorCodes::SYNTAX_ERROR &&
e.code() != ErrorCodes::TOO_DEEP_RECURSION)
throw;
}
if (ast_2)
{
const auto text_2 = ast_2->formatForErrorMessage();
const auto * tmp_pos = text_2.c_str();
const auto ast_3 = parseQuery(tmp_pos, tmp_pos + text_2.size(),
false /* allow_multi_statements */);
const auto text_3 = ast_3 ? ast_3->formatForErrorMessage() : "";
if (text_3 != text_2)
{
fmt::print(stderr, "Found error: The query formatting is broken.\n");
printChangedSettings();
fmt::print(stderr,
"Got the following (different) text after formatting the fuzzed query and parsing it back:\n'{}'\n, expected:\n'{}'\n",
text_3, text_2);
fmt::print(stderr, "In more detail:\n");
fmt::print(stderr, "AST-1 (generated by fuzzer):\n'{}'\n", ast_to_process->dumpTree());
fmt::print(stderr, "Text-1 (AST-1 formatted):\n'{}'\n", query_to_execute);
fmt::print(stderr, "AST-2 (Text-1 parsed):\n'{}'\n", ast_2->dumpTree());
fmt::print(stderr, "Text-2 (AST-2 formatted):\n'{}'\n", text_2);
fmt::print(stderr, "AST-3 (Text-2 parsed):\n'{}'\n", ast_3 ? ast_3->dumpTree() : "");
fmt::print(stderr, "Text-3 (AST-3 formatted):\n'{}'\n", text_3);
fmt::print(stderr, "Text-3 must be equal to Text-2, but it is not.\n");
_exit(1);
}
}
}
// The server is still alive so we're going to continue fuzzing.
// The server is still alive, so we're going to continue fuzzing.
// Determine what we're going to use as the starting AST.
if (have_error)
{

17
programs/server/Server.cpp
View File

@ -44,6 +44,7 @@
#include <Common/assertProcessUserMatchesDataOwner.h>
#include <Common/makeSocketAddress.h>
#include <Common/FailPoint.h>
#include <Common/CPUID.h>
#include <Server/waitServersToFinish.h>
#include <Interpreters/Cache/FileCacheFactory.h>
#include <Core/ServerUUID.h>
@ -712,6 +713,22 @@ try
getNumberOfPhysicalCPUCores(), // on ARM processors it can show only enabled at current moment cores
std::thread::hardware_concurrency());
#if defined(__x86_64__)
String cpu_info;
#define COLLECT_FLAG(X) \
if (CPU::have##X()) \
{ \
if (!cpu_info.empty()) \
cpu_info += ", "; \
cpu_info += #X; \
}
CPU_ID_ENUMERATE(COLLECT_FLAG)
#undef COLLECT_FLAG
LOG_INFO(log, "Available CPU instruction sets: {}", cpu_info);
#endif
sanityChecks(*this);
// Initialize global thread pool. Do it before we fetch configs from zookeeper

2
src/Client/ConnectionParameters.cpp
View File

@ -125,7 +125,7 @@ ConnectionParameters::ConnectionParameters(const Poco::Util::AbstractConfigurati
Poco::Timespan(config.getInt("send_timeout", DBMS_DEFAULT_SEND_TIMEOUT_SEC), 0))
.withReceiveTimeout(
Poco::Timespan(config.getInt("receive_timeout", DBMS_DEFAULT_RECEIVE_TIMEOUT_SEC), 0))
.withTcpKeepAliveTimeout(
.withTCPKeepAliveTimeout(
Poco::Timespan(config.getInt("tcp_keep_alive_timeout", DEFAULT_TCP_KEEP_ALIVE_TIMEOUT), 0))
.withHandshakeTimeout(
Poco::Timespan(config.getInt("handshake_timeout_ms", DBMS_DEFAULT_RECEIVE_TIMEOUT_SEC * 1000) * 1000))

105
src/Common/CpuId.h → src/Common/CPUID.h
View File

@ -2,7 +2,7 @@
#include <base/types.h>
#if defined(__x86_64__) || defined(__i386__)
#if defined(__x86_64__)
#include <cpuid.h>
#endif
@ -11,10 +11,10 @@
namespace DB
{
namespace Cpu
namespace CPU
{
#if (defined(__x86_64__) || defined(__i386__))
#if (defined(__x86_64__))
/// Our version is independent of -mxsave option, because we do dynamic dispatch.
inline UInt64 our_xgetbv(UInt32 xcr) noexcept
{
@ -30,7 +30,7 @@ inline UInt64 our_xgetbv(UInt32 xcr) noexcept
inline bool cpuid(UInt32 op, UInt32 sub_op, UInt32 * res) noexcept /// NOLINT
{
#if defined(__x86_64__) || defined(__i386__)
#if defined(__x86_64__)
__cpuid_count(op, sub_op, res[0], res[1], res[2], res[3]);
return true;
#else
@ -45,7 +45,7 @@ inline bool cpuid(UInt32 op, UInt32 sub_op, UInt32 * res) noexcept /// NOLINT
inline bool cpuid(UInt32 op, UInt32 * res) noexcept /// NOLINT
{
#if defined(__x86_64__) || defined(__i386__)
#if defined(__x86_64__)
__cpuid(op, res[0], res[1], res[2], res[3]);
return true;
#else
@ -98,7 +98,7 @@ inline bool cpuid(UInt32 op, UInt32 * res) noexcept /// NOLINT
OP(AMXTILE) \
OP(AMXINT8)
union CpuInfo
union CPUInfo
{
UInt32 info[4];
@ -110,9 +110,9 @@ union CpuInfo
UInt32 edx;
} registers;
inline explicit CpuInfo(UInt32 op) noexcept { cpuid(op, info); }
inline explicit CPUInfo(UInt32 op) noexcept { cpuid(op, info); }
inline CpuInfo(UInt32 op, UInt32 sub_op) noexcept { cpuid(op, sub_op, info); }
inline CPUInfo(UInt32 op, UInt32 sub_op) noexcept { cpuid(op, sub_op, info); }
};
#define DEF_NAME(X) inline bool have##X() noexcept;
@ -121,77 +121,77 @@ union CpuInfo
bool haveRDTSCP() noexcept
{
return (CpuInfo(0x80000001).registers.edx >> 27) & 1u;
return (CPUInfo(0x80000001).registers.edx >> 27) & 1u;
}
bool haveSSE() noexcept
{
return (CpuInfo(0x1).registers.edx >> 25) & 1u;
return (CPUInfo(0x1).registers.edx >> 25) & 1u;
}
bool haveSSE2() noexcept
{
return (CpuInfo(0x1).registers.edx >> 26) & 1u;
return (CPUInfo(0x1).registers.edx >> 26) & 1u;
}
bool haveSSE3() noexcept
{
return CpuInfo(0x1).registers.ecx & 1u;
return CPUInfo(0x1).registers.ecx & 1u;
}
bool havePCLMUL() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 1) & 1u;
return (CPUInfo(0x1).registers.ecx >> 1) & 1u;
}
bool haveSSSE3() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 9) & 1u;
return (CPUInfo(0x1).registers.ecx >> 9) & 1u;
}
bool haveSSE41() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 19) & 1u;
return (CPUInfo(0x1).registers.ecx >> 19) & 1u;
}
bool haveSSE42() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 20) & 1u;
return (CPUInfo(0x1).registers.ecx >> 20) & 1u;
}
bool haveF16C() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 29) & 1u;
return (CPUInfo(0x1).registers.ecx >> 29) & 1u;
}
bool havePOPCNT() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 23) & 1u;
return (CPUInfo(0x1).registers.ecx >> 23) & 1u;
}
bool haveAES() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 25) & 1u;
return (CPUInfo(0x1).registers.ecx >> 25) & 1u;
}
bool haveXSAVE() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 26) & 1u;
return (CPUInfo(0x1).registers.ecx >> 26) & 1u;
}
bool haveOSXSAVE() noexcept
{
return (CpuInfo(0x1).registers.ecx >> 27) & 1u;
return (CPUInfo(0x1).registers.ecx >> 27) & 1u;
}
bool haveAVX() noexcept
{
#if defined(__x86_64__) || defined(__i386__)
#if defined(__x86_64__)
// http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
// https://bugs.chromium.org/p/chromium/issues/detail?id=375968
return haveOSXSAVE() // implies haveXSAVE()
&& (our_xgetbv(0) & 6u) == 6u // XMM state and YMM state are enabled by OS
&& ((CpuInfo(0x1).registers.ecx >> 28) & 1u); // AVX bit
&& ((CPUInfo(0x1).registers.ecx >> 28) & 1u); // AVX bit
#else
return false;
#endif
@ -199,33 +199,33 @@ bool haveAVX() noexcept
bool haveFMA() noexcept
{
return haveAVX() && ((CpuInfo(0x1).registers.ecx >> 12) & 1u);
return haveAVX() && ((CPUInfo(0x1).registers.ecx >> 12) & 1u);
}
bool haveAVX2() noexcept
{
return haveAVX() && ((CpuInfo(0x7, 0).registers.ebx >> 5) & 1u);
return haveAVX() && ((CPUInfo(0x7, 0).registers.ebx >> 5) & 1u);
}
bool haveBMI1() noexcept
{
return (CpuInfo(0x7, 0).registers.ebx >> 3) & 1u;
return (CPUInfo(0x7, 0).registers.ebx >> 3) & 1u;
}
bool haveBMI2() noexcept
{
return (CpuInfo(0x7, 0).registers.ebx >> 8) & 1u;
return (CPUInfo(0x7, 0).registers.ebx >> 8) & 1u;
}
bool haveAVX512F() noexcept
{
#if defined(__x86_64__) || defined(__i386__)
#if defined(__x86_64__)
// https://software.intel.com/en-us/articles/how-to-detect-knl-instruction-support
return haveOSXSAVE() // implies haveXSAVE()
&& (our_xgetbv(0) & 6u) == 6u // XMM state and YMM state are enabled by OS
&& ((our_xgetbv(0) >> 5) & 7u) == 7u // ZMM state is enabled by OS
&& CpuInfo(0x0).registers.eax >= 0x7 // leaf 7 is present
&& ((CpuInfo(0x7, 0).registers.ebx >> 16) & 1u); // AVX512F bit
&& CPUInfo(0x0).registers.eax >= 0x7 // leaf 7 is present
&& ((CPUInfo(0x7, 0).registers.ebx >> 16) & 1u); // AVX512F bit
#else
return false;
#endif
@ -233,92 +233,92 @@ bool haveAVX512F() noexcept
bool haveAVX512DQ() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ebx >> 17) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ebx >> 17) & 1u);
}
bool haveRDSEED() noexcept
{
return CpuInfo(0x0).registers.eax >= 0x7 && ((CpuInfo(0x7, 0).registers.ebx >> 18) & 1u);
return CPUInfo(0x0).registers.eax >= 0x7 && ((CPUInfo(0x7, 0).registers.ebx >> 18) & 1u);
}
bool haveADX() noexcept
{
return CpuInfo(0x0).registers.eax >= 0x7 && ((CpuInfo(0x7, 0).registers.ebx >> 19) & 1u);
return CPUInfo(0x0).registers.eax >= 0x7 && ((CPUInfo(0x7, 0).registers.ebx >> 19) & 1u);
}
bool haveAVX512IFMA() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ebx >> 21) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ebx >> 21) & 1u);
}
bool havePCOMMIT() noexcept
{
return CpuInfo(0x0).registers.eax >= 0x7 && ((CpuInfo(0x7, 0).registers.ebx >> 22) & 1u);
return CPUInfo(0x0).registers.eax >= 0x7 && ((CPUInfo(0x7, 0).registers.ebx >> 22) & 1u);
}
bool haveCLFLUSHOPT() noexcept
{
return CpuInfo(0x0).registers.eax >= 0x7 && ((CpuInfo(0x7, 0).registers.ebx >> 23) & 1u);
return CPUInfo(0x0).registers.eax >= 0x7 && ((CPUInfo(0x7, 0).registers.ebx >> 23) & 1u);
}
bool haveCLWB() noexcept
{
return CpuInfo(0x0).registers.eax >= 0x7 && ((CpuInfo(0x7, 0).registers.ebx >> 24) & 1u);
return CPUInfo(0x0).registers.eax >= 0x7 && ((CPUInfo(0x7, 0).registers.ebx >> 24) & 1u);
}
bool haveAVX512PF() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ebx >> 26) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ebx >> 26) & 1u);
}
bool haveAVX512ER() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ebx >> 27) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ebx >> 27) & 1u);
}
bool haveAVX512CD() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ebx >> 28) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ebx >> 28) & 1u);
}
bool haveSHA() noexcept
{
return CpuInfo(0x0).registers.eax >= 0x7 && ((CpuInfo(0x7, 0).registers.ebx >> 29) & 1u);
return CPUInfo(0x0).registers.eax >= 0x7 && ((CPUInfo(0x7, 0).registers.ebx >> 29) & 1u);
}
bool haveAVX512BW() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ebx >> 30) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ebx >> 30) & 1u);
}
bool haveAVX512VL() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ebx >> 31) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ebx >> 31) & 1u);
}
bool havePREFETCHWT1() noexcept
{
return CpuInfo(0x0).registers.eax >= 0x7 && ((CpuInfo(0x7, 0).registers.ecx >> 0) & 1u);
return CPUInfo(0x0).registers.eax >= 0x7 && ((CPUInfo(0x7, 0).registers.ecx >> 0) & 1u);
}
bool haveAVX512VBMI() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ecx >> 1) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ecx >> 1) & 1u);
}
bool haveAVX512VBMI2() noexcept
{
return haveAVX512F() && ((CpuInfo(0x7, 0).registers.ecx >> 6) & 1u);
return haveAVX512F() && ((CPUInfo(0x7, 0).registers.ecx >> 6) & 1u);
}
bool haveRDRAND() noexcept
{
return CpuInfo(0x0).registers.eax >= 0x7 && ((CpuInfo(0x1).registers.ecx >> 30) & 1u);
return CPUInfo(0x0).registers.eax >= 0x7 && ((CPUInfo(0x1).registers.ecx >> 30) & 1u);
}
inline bool haveAMX() noexcept
{
#if defined(__x86_64__) || defined(__i386__)
#if defined(__x86_64__)
// http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
return haveOSXSAVE() // implies haveXSAVE()
&& ((our_xgetbv(0) >> 17) & 0x3) == 0x3; // AMX state are enabled by OS
@ -330,22 +330,22 @@ inline bool haveAMX() noexcept
bool haveAMXBF16() noexcept
{
return haveAMX()
&& ((CpuInfo(0x7, 0).registers.edx >> 22) & 1u); // AMX-BF16 bit
&& ((CPUInfo(0x7, 0).registers.edx >> 22) & 1u); // AMX-BF16 bit
}
bool haveAMXTILE() noexcept
{
return haveAMX()
&& ((CpuInfo(0x7, 0).registers.edx >> 24) & 1u); // AMX-TILE bit
&& ((CPUInfo(0x7, 0).registers.edx >> 24) & 1u); // AMX-TILE bit
}
bool haveAMXINT8() noexcept
{
return haveAMX()
&& ((CpuInfo(0x7, 0).registers.edx >> 25) & 1u); // AMX-INT8 bit
&& ((CPUInfo(0x7, 0).registers.edx >> 25) & 1u); // AMX-INT8 bit
}
struct CpuFlagsCache
struct CPUFlagsCache
{
#define DEF_NAME(X) static inline bool have_##X = have##X();
CPU_ID_ENUMERATE(DEF_NAME)
@ -354,4 +354,3 @@ struct CpuFlagsCache
}
}

2
src/Common/FieldVisitorConvertToNumber.h
View File

@ -91,7 +91,7 @@ public:
if constexpr (std::is_floating_point_v<T>)
return x.getValue().template convertTo<T>() / x.getScaleMultiplier().template convertTo<T>();
else
return (x.getValue() / x.getScaleMultiplier()). template convertTo<T>();
return (x.getValue() / x.getScaleMultiplier()).template convertTo<T>();
}
T operator() (const AggregateFunctionStateData &) const

8
src/Common/ProfileEvents.cpp
View File

@ -312,7 +312,7 @@ The server successfully detected this situation and will download merged part fr
M(ParallelReplicasStealingLeftoversMicroseconds, "Time spent collecting orphaned segments") \
M(ParallelReplicasCollectingOwnedSegmentsMicroseconds, "Time spent collecting segments meant by hash") \
\
M(PerfCpuCycles, "Total cycles. Be wary of what happens during CPU frequency scaling.") \
M(PerfCPUCycles, "Total cycles. Be wary of what happens during CPU frequency scaling.") \
M(PerfInstructions, "Retired instructions. Be careful, these can be affected by various issues, most notably hardware interrupt counts.") \
M(PerfCacheReferences, "Cache accesses. Usually, this indicates Last Level Cache accesses, but this may vary depending on your CPU. This may include prefetches and coherency messages; again this depends on the design of your CPU.") \
M(PerfCacheMisses, "Cache misses. Usually this indicates Last Level Cache misses; this is intended to be used in conjunction with the PERFCOUNTHWCACHEREFERENCES event to calculate cache miss rates.") \
@ -321,12 +321,12 @@ The server successfully detected this situation and will download merged part fr
M(PerfBusCycles, "Bus cycles, which can be different from total cycles.") \
M(PerfStalledCyclesFrontend, "Stalled cycles during issue.") \
M(PerfStalledCyclesBackend, "Stalled cycles during retirement.") \
M(PerfRefCpuCycles, "Total cycles; not affected by CPU frequency scaling.") \
M(PerfRefCPUCycles, "Total cycles; not affected by CPU frequency scaling.") \
\
M(PerfCpuClock, "The CPU clock, a high-resolution per-CPU timer") \
M(PerfCPUClock, "The CPU clock, a high-resolution per-CPU timer") \
M(PerfTaskClock, "A clock count specific to the task that is running") \
M(PerfContextSwitches, "Number of context switches") \
M(PerfCpuMigrations, "Number of times the process has migrated to a new CPU") \
M(PerfCPUMigrations, "Number of times the process has migrated to a new CPU") \
M(PerfAlignmentFaults, "Number of alignment faults. These happen when unaligned memory accesses happen; the kernel can handle these but it reduces performance. This happens only on some architectures (never on x86).") \
M(PerfEmulationFaults, "Number of emulation faults. The kernel sometimes traps on unimplemented instructions and emulates them for user space. This can negatively impact performance.") \
M(PerfMinEnabledTime, "For all events, minimum time that an event was enabled. Used to track event multiplexing influence") \

22
src/Common/TargetSpecific.cpp
View File

@ -1,7 +1,7 @@
#include <base/defines.h>
#include <Common/TargetSpecific.h>
#include <Common/CpuId.h>
#include <Common/CPUID.h>
namespace DB
{
@ -9,25 +9,25 @@ namespace DB
UInt32 getSupportedArchs()
{
UInt32 result = 0;
if (Cpu::CpuFlagsCache::have_SSE42)
if (CPU::CPUFlagsCache::have_SSE42)
result |= static_cast<UInt32>(TargetArch::SSE42);
if (Cpu::CpuFlagsCache::have_AVX)
if (CPU::CPUFlagsCache::have_AVX)
result |= static_cast<UInt32>(TargetArch::AVX);
if (Cpu::CpuFlagsCache::have_AVX2)
if (CPU::CPUFlagsCache::have_AVX2)
result |= static_cast<UInt32>(TargetArch::AVX2);
if (Cpu::CpuFlagsCache::have_AVX512F)
if (CPU::CPUFlagsCache::have_AVX512F)
result |= static_cast<UInt32>(TargetArch::AVX512F);
if (Cpu::CpuFlagsCache::have_AVX512BW)
if (CPU::CPUFlagsCache::have_AVX512BW)
result |= static_cast<UInt32>(TargetArch::AVX512BW);
if (Cpu::CpuFlagsCache::have_AVX512VBMI)
if (CPU::CPUFlagsCache::have_AVX512VBMI)
result |= static_cast<UInt32>(TargetArch::AVX512VBMI);
if (Cpu::CpuFlagsCache::have_AVX512VBMI2)
if (CPU::CPUFlagsCache::have_AVX512VBMI2)
result |= static_cast<UInt32>(TargetArch::AVX512VBMI2);
if (Cpu::CpuFlagsCache::have_AMXBF16)
if (CPU::CPUFlagsCache::have_AMXBF16)
result |= static_cast<UInt32>(TargetArch::AMXBF16);
if (Cpu::CpuFlagsCache::have_AMXTILE)
if (CPU::CPUFlagsCache::have_AMXTILE)
result |= static_cast<UInt32>(TargetArch::AMXTILE);
if (Cpu::CpuFlagsCache::have_AMXINT8)
if (CPU::CPUFlagsCache::have_AMXINT8)
result |= static_cast<UInt32>(TargetArch::AMXINT8);
return result;
}

18
src/Common/ThreadProfileEvents.cpp
View File

@ -6,10 +6,8 @@
#include "ProcfsMetricsProvider.h"
#include "hasLinuxCapability.h"
#include <filesystem>
#include <fstream>
#include <optional>
#include <unordered_set>
#include <fcntl.h>
#include <unistd.h>
@ -36,7 +34,7 @@ namespace ProfileEvents
extern const Event OSReadBytes;
extern const Event OSWriteBytes;
extern const Event PerfCpuCycles;
extern const Event PerfCPUCycles;
extern const Event PerfInstructions;
extern const Event PerfCacheReferences;
extern const Event PerfCacheMisses;
@ -45,12 +43,12 @@ namespace ProfileEvents
extern const Event PerfBusCycles;
extern const Event PerfStalledCyclesFrontend;
extern const Event PerfStalledCyclesBackend;
extern const Event PerfRefCpuCycles;
extern const Event PerfRefCPUCycles;
extern const Event PerfCpuClock;
extern const Event PerfCPUClock;
extern const Event PerfTaskClock;
extern const Event PerfContextSwitches;
extern const Event PerfCpuMigrations;
extern const Event PerfCPUMigrations;
extern const Event PerfAlignmentFaults;
extern const Event PerfEmulationFaults;
extern const Event PerfMinEnabledTime;
@ -218,7 +216,7 @@ thread_local PerfEventsCounters current_thread_counters;
// descriptions' source: http://man7.org/linux/man-pages/man2/perf_event_open.2.html
static const PerfEventInfo raw_events_info[] = {
HARDWARE_EVENT(PERF_COUNT_HW_CPU_CYCLES, PerfCpuCycles),
HARDWARE_EVENT(PERF_COUNT_HW_CPU_CYCLES, PerfCPUCycles),
HARDWARE_EVENT(PERF_COUNT_HW_INSTRUCTIONS, PerfInstructions),
HARDWARE_EVENT(PERF_COUNT_HW_CACHE_REFERENCES, PerfCacheReferences),
HARDWARE_EVENT(PERF_COUNT_HW_CACHE_MISSES, PerfCacheMisses),
@ -227,13 +225,13 @@ static const PerfEventInfo raw_events_info[] = {
HARDWARE_EVENT(PERF_COUNT_HW_BUS_CYCLES, PerfBusCycles),
HARDWARE_EVENT(PERF_COUNT_HW_STALLED_CYCLES_FRONTEND, PerfStalledCyclesFrontend),
HARDWARE_EVENT(PERF_COUNT_HW_STALLED_CYCLES_BACKEND, PerfStalledCyclesBackend),
HARDWARE_EVENT(PERF_COUNT_HW_REF_CPU_CYCLES, PerfRefCpuCycles),
HARDWARE_EVENT(PERF_COUNT_HW_REF_CPU_CYCLES, PerfRefCPUCycles),
// `cpu-clock` is a bit broken according to this: https://stackoverflow.com/a/56967896
SOFTWARE_EVENT(PERF_COUNT_SW_CPU_CLOCK, PerfCpuClock),
SOFTWARE_EVENT(PERF_COUNT_SW_CPU_CLOCK, PerfCPUClock),
SOFTWARE_EVENT(PERF_COUNT_SW_TASK_CLOCK, PerfTaskClock),
SOFTWARE_EVENT(PERF_COUNT_SW_CONTEXT_SWITCHES, PerfContextSwitches),
SOFTWARE_EVENT(PERF_COUNT_SW_CPU_MIGRATIONS, PerfCpuMigrations),
SOFTWARE_EVENT(PERF_COUNT_SW_CPU_MIGRATIONS, PerfCPUMigrations),
SOFTWARE_EVENT(PERF_COUNT_SW_ALIGNMENT_FAULTS, PerfAlignmentFaults),
SOFTWARE_EVENT(PERF_COUNT_SW_EMULATION_FAULTS, PerfEmulationFaults),

8
src/Coordination/KeeperConstants.cpp
View File

@ -85,7 +85,7 @@
M(OSReadChars) \
M(OSWriteChars) \
\
M(PerfCpuCycles) \
M(PerfCPUCycles) \
M(PerfInstructions) \
M(PerfCacheReferences) \
M(PerfCacheMisses) \
@ -94,12 +94,12 @@
M(PerfBusCycles) \
M(PerfStalledCyclesFrontend) \
M(PerfStalledCyclesBackend) \
M(PerfRefCpuCycles) \
M(PerfRefCPUCycles) \
\
M(PerfCpuClock) \
M(PerfCPUClock) \
M(PerfTaskClock) \
M(PerfContextSwitches) \
M(PerfCpuMigrations) \
M(PerfCPUMigrations) \
M(PerfAlignmentFaults) \
M(PerfEmulationFaults) \
M(PerfMinEnabledTime) \

2
src/Core/AccurateComparison.h
View File

@ -152,7 +152,7 @@ bool notEqualsOp(A a, B b)
}
/// Converts numeric to an equal numeric of other type.
/// When `strict` is `true` check that result exactly same as input, otherwise just check overflow
/// When `strict` is `true` check that result exactly the same as input, otherwise just check overflow
template <typename From, typename To, bool strict = true>
inline bool NO_SANITIZE_UNDEFINED convertNumeric(From value, To & result)
{

51
src/Core/SettingsFields.cpp
View File

@ -1,8 +1,7 @@
#include <Core/SettingsFields.h>
#include <Core/Field.h>
#include <Core/AccurateComparison.h>
#include <Common/getNumberOfPhysicalCPUCores.h>
#include <Common/FieldVisitorConvertToNumber.h>
#include <Common/logger_useful.h>
#include <DataTypes/DataTypeMap.h>
#include <DataTypes/DataTypeString.h>
@ -13,6 +12,7 @@
#include <cmath>
namespace DB
{
namespace ErrorCodes
@ -20,6 +20,7 @@ namespace ErrorCodes
extern const int SIZE_OF_FIXED_STRING_DOESNT_MATCH;
extern const int CANNOT_PARSE_BOOL;
extern const int CANNOT_PARSE_NUMBER;
extern const int CANNOT_CONVERT_TYPE;
}
@ -48,9 +49,51 @@ namespace
T fieldToNumber(const Field & f)
{
if (f.getType() == Field::Types::String)
{
return stringToNumber<T>(f.get<const String &>());
}
else if (f.getType() == Field::Types::UInt64)
{
T result;
if (!accurate::convertNumeric(f.get<UInt64>(), result))
throw Exception(ErrorCodes::CANNOT_CONVERT_TYPE, "Field value {} is out of range of {} type", f, demangle(typeid(T).name()));
return result;
}
else if (f.getType() == Field::Types::Int64)
{
T result;
if (!accurate::convertNumeric(f.get<Int64>(), result))
throw Exception(ErrorCodes::CANNOT_CONVERT_TYPE, "Field value {} is out of range of {} type", f, demangle(typeid(T).name()));
return result;
}
else if (f.getType() == Field::Types::Bool)
{
return T(f.get<bool>());
}
else if (f.getType() == Field::Types::Float64)
{
Float64 x = f.get<Float64>();
if constexpr (std::is_floating_point_v<T>)
{
return T(x);
}
else
{
if (!isFinite(x))
{
/// Conversion of infinite values to integer is undefined.
throw Exception(ErrorCodes::CANNOT_CONVERT_TYPE, "Cannot convert infinite value to integer type");
}
else if (x > Float64(std::numeric_limits<T>::max()) || x < Float64(std::numeric_limits<T>::lowest()))
{
throw Exception(ErrorCodes::CANNOT_CONVERT_TYPE, "Cannot convert out of range floating point value to integer type");
}
else
return T(x);
}
}
else
return applyVisitor(FieldVisitorConvertToNumber<T>(), f);
throw Exception(ErrorCodes::CANNOT_CONVERT_TYPE, "Invalid value {} of the setting, which needs {}", f, demangle(typeid(T).name()));
}
Map stringToMap(const String & str)
@ -174,7 +217,7 @@ namespace
if (f.getType() == Field::Types::String)
return stringToMaxThreads(f.get<const String &>());
else
return applyVisitor(FieldVisitorConvertToNumber<UInt64>(), f);
return fieldToNumber<UInt64>(f);
}
}

2
src/Dictionaries/YAMLRegExpTreeDictionarySource.cpp
View File

@ -227,7 +227,7 @@ void parseMatchNode(UInt64 parent_id, UInt64 & id, const YAML::Node & node, Resu
if (!match.contains(key_name))
{
throw Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, "Yaml match rule must contain key {}", key_name);
throw Exception(ErrorCodes::INVALID_CONFIG_PARAMETER, "YAML match rule must contain key {}", key_name);
}
for (const auto & [key, node_] : match)
{

2
src/Functions/FunctionsJSON.h
View File

@ -5,7 +5,7 @@
#include <base/range.h>
#include <Common/CpuId.h>
#include <Common/CPUID.h>
#include <Common/typeid_cast.h>
#include <Common/assert_cast.h>

6
src/Functions/divide/divide.cpp
View File

@ -1,5 +1,5 @@
#include "divide.h"
#include <Common/CpuId.h>
#include <Common/CPUID.h>
#if defined(__x86_64__)
namespace SSE2
@ -26,9 +26,9 @@ template <typename A, typename B, typename ResultType>
void divideImpl(const A * __restrict a_pos, B b, ResultType * __restrict c_pos, size_t size)
{
#if defined(__x86_64__)
if (DB::Cpu::CpuFlagsCache::have_AVX2)
if (DB::CPU::CPUFlagsCache::have_AVX2)
AVX2::divideImpl(a_pos, b, c_pos, size);
else if (DB::Cpu::CpuFlagsCache::have_SSE2)
else if (DB::CPU::CPUFlagsCache::have_SSE2)
SSE2::divideImpl(a_pos, b, c_pos, size);
#else
Generic::divideImpl(a_pos, b, c_pos, size);

10
src/Functions/serverConstants.cpp
View File

@ -51,12 +51,12 @@ namespace
};
class FunctionTcpPort : public FunctionConstantBase<FunctionTcpPort, UInt16, DataTypeUInt16>
class FunctionTCPPort : public FunctionConstantBase<FunctionTCPPort, UInt16, DataTypeUInt16>
{
public:
static constexpr auto name = "tcpPort";
static FunctionPtr create(ContextPtr context) { return std::make_shared<FunctionTcpPort>(context); }
explicit FunctionTcpPort(ContextPtr context) : FunctionConstantBase(context->getTCPPort(), context->isDistributed()) {}
static FunctionPtr create(ContextPtr context) { return std::make_shared<FunctionTCPPort>(context); }
explicit FunctionTCPPort(ContextPtr context) : FunctionConstantBase(context->getTCPPort(), context->isDistributed()) {}
};
@ -153,9 +153,9 @@ REGISTER_FUNCTION(ServerUUID)
factory.registerFunction<FunctionServerUUID>();
}
REGISTER_FUNCTION(TcpPort)
REGISTER_FUNCTION(TCPPort)
{
factory.registerFunction<FunctionTcpPort>();
factory.registerFunction<FunctionTCPPort>();
}
REGISTER_FUNCTION(Timezone)

6
src/IO/ConnectionTimeouts.cpp
View File

@ -20,7 +20,7 @@ ConnectionTimeouts ConnectionTimeouts::getTCPTimeoutsWithoutFailover(const Setti
.withConnectionTimeout(settings.connect_timeout)
.withSendTimeout(settings.send_timeout)
.withReceiveTimeout(settings.receive_timeout)
.withTcpKeepAliveTimeout(settings.tcp_keep_alive_timeout)
.withTCPKeepAliveTimeout(settings.tcp_keep_alive_timeout)
.withHandshakeTimeout(settings.handshake_timeout_ms)
.withHedgedConnectionTimeout(settings.hedged_connection_timeout_ms)
.withReceiveDataTimeout(settings.receive_data_timeout_ms);
@ -40,8 +40,8 @@ ConnectionTimeouts ConnectionTimeouts::getHTTPTimeouts(const Settings & settings
.withConnectionTimeout(settings.http_connection_timeout)
.withSendTimeout(settings.http_send_timeout)
.withReceiveTimeout(settings.http_receive_timeout)
.withHttpKeepAliveTimeout(http_keep_alive_timeout)
.withTcpKeepAliveTimeout(settings.tcp_keep_alive_timeout)
.withHTTPKeepAliveTimeout(http_keep_alive_timeout)
.withTCPKeepAliveTimeout(settings.tcp_keep_alive_timeout)
.withHandshakeTimeout(settings.handshake_timeout_ms);
}

4
src/IO/ConnectionTimeouts.h
View File

@ -16,8 +16,8 @@ struct Settings;
M(secure_connection_timeout, withSecureConnectionTimeout) \
M(send_timeout, withSendTimeout) \
M(receive_timeout, withReceiveTimeout) \
M(tcp_keep_alive_timeout, withTcpKeepAliveTimeout) \
M(http_keep_alive_timeout, withHttpKeepAliveTimeout) \
M(tcp_keep_alive_timeout, withTCPKeepAliveTimeout) \
M(http_keep_alive_timeout, withHTTPKeepAliveTimeout) \
M(hedged_connection_timeout, withHedgedConnectionTimeout) \
M(receive_data_timeout, withReceiveDataTimeout) \
M(handshake_timeout, withHandshakeTimeout) \

5
src/IO/S3/Credentials.cpp
View File

@ -22,7 +22,6 @@ namespace ErrorCodes
# include <aws/core/utils/UUID.h>
# include <aws/core/http/HttpClientFactory.h>
# include <IO/S3/PocoHTTPClientFactory.h>
# include <aws/core/utils/HashingUtils.h>
# include <aws/core/platform/FileSystem.h>
@ -31,9 +30,7 @@ namespace ErrorCodes
# include <IO/S3/Client.h>
# include <fstream>
# include <base/EnumReflection.h>
# include <boost/algorithm/string.hpp>
# include <boost/algorithm/string/split.hpp>
# include <boost/algorithm/string/classification.hpp>
# include <Poco/Exception.h>
@ -755,7 +752,7 @@ S3CredentialsProviderChain::S3CredentialsProviderChain(
configuration.put_request_throttler,
Aws::Http::SchemeMapper::ToString(Aws::Http::Scheme::HTTP));
/// See MakeDefaultHttpResourceClientConfiguration().
/// See MakeDefaultHTTPResourceClientConfiguration().
/// This is part of EC2 metadata client, but unfortunately it can't be accessed from outside
/// of contrib/aws/aws-cpp-sdk-core/source/internal/AWSHttpResourceClient.cpp
aws_client_configuration.maxConnections = 2;

4
src/IO/S3/PocoHTTPClient.cpp
View File

@ -146,9 +146,9 @@ ConnectionTimeouts getTimeoutsFromConfiguration(const PocoHTTPClientConfiguratio
.withConnectionTimeout(Poco::Timespan(client_configuration.connectTimeoutMs * 1000))
.withSendTimeout(Poco::Timespan(client_configuration.requestTimeoutMs * 1000))
.withReceiveTimeout(Poco::Timespan(client_configuration.requestTimeoutMs * 1000))
.withTcpKeepAliveTimeout(Poco::Timespan(
.withTCPKeepAliveTimeout(Poco::Timespan(
client_configuration.enableTcpKeepAlive ? client_configuration.tcpKeepAliveIntervalMs * 1000 : 0))
.withHttpKeepAliveTimeout(Poco::Timespan(
.withHTTPKeepAliveTimeout(Poco::Timespan(
client_configuration.http_keep_alive_timeout_ms * 1000)); /// flag indicating whether keep-alive is enabled is set to each session upon creation
}

4
src/Interpreters/Context.cpp
View File

@ -1534,7 +1534,7 @@ void Context::addExternalTable(const String & table_name, TemporaryTableHolder &
std::lock_guard lock(mutex);
if (external_tables_mapping.end() != external_tables_mapping.find(table_name))
throw Exception(ErrorCodes::TABLE_ALREADY_EXISTS, "Temporary table {} already exists.", backQuoteIfNeed(table_name));
throw Exception(ErrorCodes::TABLE_ALREADY_EXISTS, "Temporary table {} already exists", backQuoteIfNeed(table_name));
external_tables_mapping.emplace(table_name, std::make_shared<TemporaryTableHolder>(std::move(temporary_table)));
}
@ -4513,7 +4513,7 @@ void Context::setClientConnectionId(uint32_t connection_id_)
client_info.connection_id = connection_id_;
}
void Context::setHttpClientInfo(ClientInfo::HTTPMethod http_method, const String & http_user_agent, const String & http_referer)
void Context::setHTTPClientInfo(ClientInfo::HTTPMethod http_method, const String & http_user_agent, const String & http_referer)
{
client_info.http_method = http_method;
client_info.http_user_agent = http_user_agent;

2
src/Interpreters/Context.h
View File

@ -630,7 +630,7 @@ public:
void setClientInterface(ClientInfo::Interface interface);
void setClientVersion(UInt64 client_version_major, UInt64 client_version_minor, UInt64 client_version_patch, unsigned client_tcp_protocol_version);
void setClientConnectionId(uint32_t connection_id);
void setHttpClientInfo(ClientInfo::HTTPMethod http_method, const String & http_user_agent, const String & http_referer);
void setHTTPClientInfo(ClientInfo::HTTPMethod http_method, const String & http_user_agent, const String & http_referer);
void setForwardedFor(const String & forwarded_for);
void setQueryKind(ClientInfo::QueryKind query_kind);
void setQueryKindInitial();

4
src/Interpreters/InterpreterShowFunctionsQuery.cpp
View File

@ -25,13 +25,13 @@ String InterpreterShowFunctionsQuery::getRewrittenQuery()
const auto & query = query_ptr->as<ASTShowFunctionsQuery &>();
DatabasePtr systemDb = DatabaseCatalog::instance().getSystemDatabase();
DatabasePtr system_db = DatabaseCatalog::instance().getSystemDatabase();
String rewritten_query = fmt::format(
R"(
SELECT *
FROM {}.{})",
systemDb->getDatabaseName(),
system_db->getDatabaseName(),
functions_table);
if (!query.like.empty())

4
src/Interpreters/Session.cpp
View File

@ -429,11 +429,11 @@ void Session::setClientConnectionId(uint32_t connection_id)
prepared_client_info->connection_id = connection_id;
}
void Session::setHttpClientInfo(ClientInfo::HTTPMethod http_method, const String & http_user_agent, const String & http_referer)
void Session::setHTTPClientInfo(ClientInfo::HTTPMethod http_method, const String & http_user_agent, const String & http_referer)
{
if (session_context)
{
session_context->setHttpClientInfo(http_method, http_user_agent, http_referer);
session_context->setHTTPClientInfo(http_method, http_user_agent, http_referer);
}
else
{

2
src/Interpreters/Session.h
View File

@ -65,7 +65,7 @@ public:
void setClientInterface(ClientInfo::Interface interface);
void setClientVersion(UInt64 client_version_major, UInt64 client_version_minor, UInt64 client_version_patch, unsigned client_tcp_protocol_version);
void setClientConnectionId(uint32_t connection_id);
void setHttpClientInfo(ClientInfo::HTTPMethod http_method, const String & http_user_agent, const String & http_referer);
void setHTTPClientInfo(ClientInfo::HTTPMethod http_method, const String & http_user_agent, const String & http_referer);
void setForwardedFor(const String & forwarded_for);
void setQuotaClientKey(const String & quota_key);
void setConnectionClientVersion(UInt64 client_version_major, UInt64 client_version_minor, UInt64 client_version_patch, unsigned client_tcp_protocol_version);

4
src/Parsers/Kusto/KustoFunctions/KQLFunctionFactory.cpp
View File

@ -359,7 +359,7 @@ std::unique_ptr<IParserKQLFunction> KQLFunctionFactory::get(String & kql_functio
return std::make_unique<ExtractAll>();
case KQLFunctionValue::extract_json:
return std::make_unique<ExtractJson>();
return std::make_unique<ExtractJSON>();
case KQLFunctionValue::has_any_index:
return std::make_unique<HasAnyIndex>();
@ -389,7 +389,7 @@ std::unique_ptr<IParserKQLFunction> KQLFunctionFactory::get(String & kql_functio
return std::make_unique<ParseCSV>();
case KQLFunctionValue::parse_json:
return std::make_unique<ParseJson>();
return std::make_unique<ParseJSON>();
case KQLFunctionValue::parse_url:
return std::make_unique<ParseURL>();

4
src/Parsers/Kusto/KustoFunctions/KQLStringFunctions.cpp
View File

@ -240,7 +240,7 @@ bool ExtractAll::convertImpl(String & out, IParser::Pos & pos)
return true;
}
bool ExtractJson::convertImpl(String & out, IParser::Pos & pos)
bool ExtractJSON::convertImpl(String & out, IParser::Pos & pos)
{
String datatype = "String";
ParserKeyword s_kql("typeof");
@ -431,7 +431,7 @@ bool ParseCSV::convertImpl(String & out, IParser::Pos & pos)
return true;
}
bool ParseJson::convertImpl(String & out, IParser::Pos & pos)
bool ParseJSON::convertImpl(String & out, IParser::Pos & pos)
{
const String fn_name = getKQLFunctionName(pos);
if (fn_name.empty())

4
src/Parsers/Kusto/KustoFunctions/KQLStringFunctions.h
View File

@ -62,7 +62,7 @@ protected:
bool convertImpl(String & out, IParser::Pos & pos) override;
};
class ExtractJson : public IParserKQLFunction
class ExtractJSON : public IParserKQLFunction
{
protected:
const char * getName() const override { return "extract_json(), extractjson()"; }
@ -125,7 +125,7 @@ protected:
bool convertImpl(String & out, IParser::Pos & pos) override;
};
class ParseJson : public IParserKQLFunction
class ParseJSON : public IParserKQLFunction
{
protected:
const char * getName() const override { return "parse_json()"; }

4
src/Processors/Formats/Impl/AvroRowInputFormat.cpp
View File

@ -212,7 +212,7 @@ static AvroDeserializer::DeserializeFn createDecimalDeserializeFn(const avro::No
};
}
static std::string nodeToJson(avro::NodePtr root_node)
static std::string nodeToJSON(avro::NodePtr root_node)
{
std::ostringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
ss.exceptions(std::ios::failbit);
@ -641,7 +641,7 @@ AvroDeserializer::DeserializeFn AvroDeserializer::createDeserializeFn(const avro
throw Exception(ErrorCodes::ILLEGAL_COLUMN,
"Type {} is not compatible with Avro {}:\n{}",
target_type->getName(), avro::toString(root_node->type()), nodeToJson(root_node));
target_type->getName(), avro::toString(root_node->type()), nodeToJSON(root_node));
}
AvroDeserializer::SkipFn AvroDeserializer::createSkipFn(const avro::NodePtr & root_node)

2
src/Processors/Formats/Impl/ConstantExpressionTemplate.cpp
View File

@ -16,7 +16,6 @@
#include <Interpreters/TreeRewriter.h>
#include <Interpreters/Context.h>
#include <Interpreters/convertFieldToType.h>
#include <Interpreters/ExpressionActions.h>
#include <Interpreters/castColumn.h>
#include <IO/ReadHelpers.h>
#include <Parsers/ASTExpressionList.h>
@ -28,7 +27,6 @@
#include <Processors/Formats/Impl/ConstantExpressionTemplate.h>
#include <Parsers/ExpressionElementParsers.h>
#include <boost/functional/hash.hpp>
#include <base/sort.h>
namespace DB

2
src/Processors/QueryPlan/PartsSplitter.cpp
View File

@ -54,7 +54,7 @@ public:
Values getValue(size_t part_idx, size_t mark) const
{
const auto & index = parts[part_idx].data_part->index;
const auto & index = parts[part_idx].data_part->getIndex();
Values values(index.size());
for (size_t i = 0; i < values.size(); ++i)
{

8
src/Server/HTTPHandler.cpp
View File

@ -125,7 +125,7 @@ namespace ErrorCodes
namespace
{
bool tryAddHttpOptionHeadersFromConfig(HTTPServerResponse & response, const Poco::Util::LayeredConfiguration & config)
bool tryAddHTTPOptionHeadersFromConfig(HTTPServerResponse & response, const Poco::Util::LayeredConfiguration & config)
{
if (config.has("http_options_response"))
{
@ -153,7 +153,7 @@ bool tryAddHttpOptionHeadersFromConfig(HTTPServerResponse & response, const Poco
void processOptionsRequest(HTTPServerResponse & response, const Poco::Util::LayeredConfiguration & config)
{
/// If can add some headers from config
if (tryAddHttpOptionHeadersFromConfig(response, config))
if (tryAddHTTPOptionHeadersFromConfig(response, config))
{
response.setKeepAlive(false);
response.setStatusAndReason(HTTPResponse::HTTP_NO_CONTENT);
@ -496,7 +496,7 @@ bool HTTPHandler::authenticateUser(
else if (request.getMethod() == HTTPServerRequest::HTTP_POST)
http_method = ClientInfo::HTTPMethod::POST;
session->setHttpClientInfo(http_method, request.get("User-Agent", ""), request.get("Referer", ""));
session->setHTTPClientInfo(http_method, request.get("User-Agent", ""), request.get("Referer", ""));
session->setForwardedFor(request.get("X-Forwarded-For", ""));
session->setQuotaClientKey(quota_key);
@ -1065,7 +1065,7 @@ void HTTPHandler::handleRequest(HTTPServerRequest & request, HTTPServerResponse
response.set("X-ClickHouse-Server-Display-Name", server_display_name);
if (!request.get("Origin", "").empty())
tryAddHttpOptionHeadersFromConfig(response, server.config());
tryAddHTTPOptionHeadersFromConfig(response, server.config());
/// For keep-alive to work.
if (request.getVersion() == HTTPServerRequest::HTTP_1_1)

16
src/Storages/MergeTree/IMergeTreeDataPart.cpp
View File

@ -313,13 +313,13 @@ IMergeTreeDataPart::IMergeTreeDataPart(
const IMergeTreeDataPart * parent_part_)
: DataPartStorageHolder(data_part_storage_)
, storage(storage_)
, mutable_name(name_)
, name(mutable_name)
, info(info_)
, index_granularity_info(storage_, part_type_)
, part_type(part_type_)
, parent_part(parent_part_)
, parent_part_name(parent_part ? parent_part->name : "")
, mutable_name(name_)
{
if (parent_part)
{
@ -342,6 +342,19 @@ IMergeTreeDataPart::~IMergeTreeDataPart()
decrementTypeMetric(part_type);
}
const IMergeTreeDataPart::Index & IMergeTreeDataPart::getIndex() const
{
return index;
}
void IMergeTreeDataPart::setIndex(Columns index_)
{
index = std::move(index_);
}
void IMergeTreeDataPart::setName(const String & new_name)
{
mutable_name = new_name;
@ -842,6 +855,7 @@ void IMergeTreeDataPart::loadIndex()
for (size_t i = 0; i < key_size; ++i)
{
loaded_index[i]->shrinkToFit();
loaded_index[i]->protect();
if (loaded_index[i]->size() != marks_count)
throw Exception(ErrorCodes::CANNOT_READ_ALL_DATA, "Cannot read all data from index file {}(expected size: "

23
src/Storages/MergeTree/IMergeTreeDataPart.h
View File

@ -75,6 +75,7 @@ public:
using ColumnSizeByName = std::unordered_map<std::string, ColumnSize>;
using NameToNumber = std::unordered_map<std::string, size_t>;
using Index = Columns;
using IndexSizeByName = std::unordered_map<std::string, ColumnSize>;
using Type = MergeTreeDataPartType;
@ -212,10 +213,6 @@ public:
const MergeTreeData & storage;
private:
String mutable_name;
mutable MergeTreeDataPartState state{MergeTreeDataPartState::Temporary};
public:
const String & name; // const ref to private mutable_name
MergeTreePartInfo info;
@ -309,12 +306,6 @@ public:
/// Throws an exception if state of the part is not in affordable_states
void assertState(const std::initializer_list<MergeTreeDataPartState> & affordable_states) const;
/// Primary key (correspond to primary.idx file).
/// Always loaded in RAM. Contains each index_granularity-th value of primary key tuple.
/// Note that marks (also correspond to primary key) is not always in RAM, but cached. See MarkCache.h.
using Index = Columns;
Index index;
MergeTreePartition partition;
/// Amount of rows between marks
@ -369,6 +360,9 @@ public:
/// Version of part metadata (columns, pk and so on). Managed properly only for replicated merge tree.
int32_t metadata_version;
const Index & getIndex() const;
void setIndex(Columns index_);
/// For data in RAM ('index')
UInt64 getIndexSizeInBytes() const;
UInt64 getIndexSizeInAllocatedBytes() const;
@ -567,6 +561,10 @@ public:
mutable std::atomic<time_t> last_removal_attempt_time = 0;
protected:
/// Primary key (correspond to primary.idx file).
/// Always loaded in RAM. Contains each index_granularity-th value of primary key tuple.
/// Note that marks (also correspond to primary key) are not always in RAM, but cached. See MarkCache.h.
Index index;
/// Total size of all columns, calculated once in calcuateColumnSizesOnDisk
ColumnSize total_columns_size;
@ -623,6 +621,9 @@ protected:
void initializeIndexGranularityInfo();
private:
String mutable_name;
mutable MergeTreeDataPartState state{MergeTreeDataPartState::Temporary};
/// In compact parts order of columns is necessary
NameToNumber column_name_to_position;
@ -660,7 +661,7 @@ private:
virtual void appendFilesOfIndexGranularity(Strings & files) const;
/// Loads index file.
/// Loads the index file.
void loadIndex();
void appendFilesOfIndex(Strings & files) const;

4
src/Storages/MergeTree/MergeTreeData.cpp
View File

@ -6824,7 +6824,7 @@ Block MergeTreeData::getMinMaxCountProjectionBlock(
{
for (const auto & part : real_parts)
{
const auto & primary_key_column = *part->index[0];
const auto & primary_key_column = *part->getIndex()[0];
auto & min_column = assert_cast<ColumnAggregateFunction &>(*partition_minmax_count_columns[pos]);
insert(min_column, primary_key_column[0]);
}
@ -6835,7 +6835,7 @@ Block MergeTreeData::getMinMaxCountProjectionBlock(
{
for (const auto & part : real_parts)
{
const auto & primary_key_column = *part->index[0];
const auto & primary_key_column = *part->getIndex()[0];
auto & max_column = assert_cast<ColumnAggregateFunction &>(*partition_minmax_count_columns[pos]);
insert(max_column, primary_key_column[primary_key_column.size() - 1]);
}

2
src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp
View File

@ -1087,7 +1087,7 @@ MarkRanges MergeTreeDataSelectExecutor::markRangesFromPKRange(
MarkRanges res;
size_t marks_count = part->index_granularity.getMarksCount();
const auto & index = part->index;
const auto & index = part->getIndex();
if (marks_count == 0)
return res;

2
src/Storages/MergeTree/MergedBlockOutputStream.cpp
View File

@ -181,7 +181,7 @@ MergedBlockOutputStream::Finalizer MergedBlockOutputStream::finalizePartAsync(
new_part->rows_count = rows_count;
new_part->modification_time = time(nullptr);
new_part->index = writer->releaseIndexColumns();
new_part->setIndex(writer->releaseIndexColumns());
new_part->checksums = checksums;
new_part->setBytesOnDisk(checksums.getTotalSizeOnDisk());
new_part->setBytesUncompressedOnDisk(checksums.getTotalSizeUncompressedOnDisk());

2
src/Storages/MergeTree/MutateTask.cpp
View File

@ -899,7 +899,7 @@ void finalizeMutatedPart(
new_data_part->rows_count = source_part->rows_count;
new_data_part->index_granularity = source_part->index_granularity;
new_data_part->index = source_part->index;
new_data_part->setIndex(source_part->getIndex());
new_data_part->minmax_idx = source_part->minmax_idx;
new_data_part->modification_time = time(nullptr);

2
src/Storages/StorageURL.cpp
View File

@ -1511,7 +1511,7 @@ void StorageURL::processNamedCollectionResult(Configuration & configuration, con
&& configuration.http_method != Poco::Net::HTTPRequest::HTTP_PUT)
throw Exception(
ErrorCodes::BAD_ARGUMENTS,
"Http method can be POST or PUT (current: {}). For insert default is POST, for select GET",
"HTTP method can be POST or PUT (current: {}). For insert default is POST, for select GET",
configuration.http_method);
configuration.format = collection.getOrDefault<String>("format", "auto");

2
tests/queries/0_stateless/01557_field_infinite_convert_to_number.sql
View File

@ -1 +1 @@
SET max_threads = nan; -- { serverError 70 }
SET max_threads = nan; -- { serverError CANNOT_CONVERT_TYPE }

0
tests/queries/0_stateless/02992_settings_overflow.reference Normal file
View File

1
tests/queries/0_stateless/02992_settings_overflow.sql Normal file
View File

@ -0,0 +1 @@
SET max_threads = -1; -- { serverError CANNOT_CONVERT_TYPE }

1
tests/queries/1_stateful/00098_primary_key_memory_allocated.reference Normal file
View File

@ -0,0 +1 @@
1 1 1

1
tests/queries/1_stateful/00098_primary_key_memory_allocated.sql Normal file
View File

@ -0,0 +1 @@
SELECT primary_key_bytes_in_memory < 16000, primary_key_bytes_in_memory_allocated < 16000, primary_key_bytes_in_memory_allocated / primary_key_bytes_in_memory < 1.1 FROM system.parts WHERE database = 'test' AND table = 'hits';

6
utils/check-style/check-style
View File

@ -442,3 +442,9 @@ ls -1d $ROOT_PATH/contrib/*-cmake | xargs -I@ find @ -name 'CMakeLists.txt' -or
# DOS/Windows newlines
find $ROOT_PATH/{base,src,programs,utils,docs} -name '*.md' -or -name '*.h' -or -name '*.cpp' -or -name '*.js' -or -name '*.py' -or -name '*.html' | xargs grep -l -P '\r$' && echo "^ Files contain DOS/Windows newlines (\r\n instead of \n)."
# Wrong spelling of abbreviations, e.g. SQL is right, Sql is wrong. XMLHttpRequest is very wrong.
find $ROOT_PATH/{src,base,programs,utils} -name '*.h' -or -name '*.cpp' |
grep -vP $EXCLUDE_DIRS |
xargs grep -P 'Sql|Html|Xml|Cpu|Tcp|Udp|Http|Db|Json|Yaml' | grep -v -P 'RabbitMQ|Azure|Aws|aws|Avro|IO/S3' &&
echo "Abbreviations such as SQL, XML, HTTP, should be in all caps. For example, SQL is right, Sql is wrong. XMLHttpRequest is very wrong."