Merge branch 'master' into debug_in_ci

docs/en/operations/allocation-profiling.md

@@ -0,0 +1,207 @@
+---
+slug: /en/operations/allocation-profiling
+sidebar_label: "Allocation profiling"
+title: "Allocation profiling"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Allocation profiling
+
+ClickHouse uses [jemalloc](https://github.com/jemalloc/jemalloc) as its global allocator that comes with some tools for allocation sampling and profiling.  
+To make allocation profiling more convenient, `SYSTEM` commands are provided along 4LW commands in Keeper.
+
+## Sampling allocations and flushing heap profiles
+
+If we want to sample and profile allocations in `jemalloc`, we need to start ClickHouse/Keeper with profiling enabled using environment variable `MALLOC_CONF`.
+
+```sh
+MALLOC_CONF=background_thread:true,prof:true
+```
+
+`jemalloc` will sample allocation and store the information internally.
+
+We can tell `jemalloc` to flush current profile by running:
+
+<Tabs groupId="binary">
+<TabItem value="clickhouse" label="ClickHouse">
+
+    SYSTEM JEMALLOC FLUSH PROFILE
+
+</TabItem>
+<TabItem value="keeper" label="Keeper">
+
+    echo jmfp | nc localhost 9181
+
+</TabItem>
+</Tabs>
+
+By default, heap profile file will be generated in `/tmp/jemalloc_clickhouse._pid_._seqnum_.heap` where `_pid_` is the PID of ClickHouse and `_seqnum_` is the global sequence number for the current heap profile.  
+For Keeper, the default file is `/tmp/jemalloc_keeper._pid_._seqnum_.heap` following the same rules.
+
+A different location can be defined by appending the `MALLOC_CONF` environment variable with `prof_prefix` option.  
+For example, if we want to generate profiles in `/data` folder where the prefix for filename will be `my_current_profile` we can run ClickHouse/Keeper with following environment variable:
+```sh
+MALLOC_CONF=background_thread:true,prof:true,prof_prefix:/data/my_current_profile
+```
+Generated file will append to prefix PID and sequence number.
+
+## Analyzing heap profiles
+
+After we generated heap profiles, we need to analyze them.  
+For that, we need to use `jemalloc`'s tool called [jeprof](https://github.com/jemalloc/jemalloc/blob/dev/bin/jeprof.in) which can be installed in multiple ways:
+- installing `jemalloc` using system's package manager
+- cloning [jemalloc repo](https://github.com/jemalloc/jemalloc) and running autogen.sh from the root folder that will provide you with `jeprof` script inside the `bin` folder
+
+:::note
+`jeprof` uses `addr2line` to generate stacktraces which can be really slow.  
+If that’s the case, we recommend installing an [alternative implementation](https://github.com/gimli-rs/addr2line) of the tool.
+
+```
+git clone https://github.com/gimli-rs/addr2line
+cd addr2line
+cargo b --examples -r
+cp ./target/release/examples/addr2line path/to/current/addr2line
+```
+:::
+
+There are many different formats to generate from the heap profile using `jeprof`.
+We recommend to run `jeprof --help` to check usage and many different options the tool provides. 
+
+In general, `jeprof` command will look like this:
+
+```sh
+jeprof path/to/binary path/to/heap/profile --output_format [ > output_file]
+```
+
+If we want to compare which allocations happened between 2 profiles we can set the base argument:
+
+```sh
+jeprof path/to/binary --base path/to/first/heap/profile path/to/second/heap/profile --output_format [ > output_file]
+```
+
+For example:
+
+- if we want to generate a text file with each procedure written per line:
+
+```sh
+jeprof path/to/binary path/to/heap/profile --text > result.txt
+```
+
+- if we want to generate a PDF file with call-graph:
+
+```sh
+jeprof path/to/binary path/to/heap/profile --pdf > result.pdf
+```
+
+### Generating flame graph
+
+`jeprof` allows us to generate collapsed stacks for building flame graphs.
+
+We need to use `--collapsed` argument:
+
+```sh
+jeprof path/to/binary path/to/heap/profile --collapsed > result.collapsed
+```
+
+After that, we can use many different tools to visualize collapsed stacks.
+
+Most popular would be [FlameGraph](https://github.com/brendangregg/FlameGraph) which contains a script called `flamegraph.pl`:
+
+```sh
+cat result.collapsed | /path/to/FlameGraph/flamegraph.pl --color=mem --title="Allocation Flame Graph" --width 2400 > result.svg
+```
+
+Another interesting tool is [speedscope](https://www.speedscope.app/) that allows you to analyze collected stacks in a more interactive way.
+
+## Controlling allocation profiler during runtime
+
+If ClickHouse/Keeper were started with enabled profiler, they support additional commands for disabling/enabling allocation profiling during runtime.
+Using those commands, it's easier to profile only specific intervals.
+
+Disable profiler:
+
+<Tabs groupId="binary">
+<TabItem value="clickhouse" label="ClickHouse">
+
+    SYSTEM JEMALLOC DISABLE PROFILE
+
+</TabItem>
+<TabItem value="keeper" label="Keeper">
+
+    echo jmdp | nc localhost 9181
+
+</TabItem>
+</Tabs>
+
+Enable profiler:
+
+<Tabs groupId="binary">
+<TabItem value="clickhouse" label="ClickHouse">
+
+    SYSTEM JEMALLOC ENABLE PROFILE
+
+</TabItem>
+<TabItem value="keeper" label="Keeper">
+
+    echo jmep | nc localhost 9181
+
+</TabItem>
+</Tabs>
+
+It's also possible to control the initial state of the profiler by setting `prof_active` option which is enabled by default.  
+For example, if we don't want to sample allocations during startup but only after we enable the profiler, we can start ClickHouse/Keeper with following environment variable:
+```sh
+MALLOC_CONF=background_thread:true,prof:true,prof_active:false
+```
+
+and enable profiler at a later point.
+
+## Additional options for profiler
+
+`jemalloc` has many different options available related to profiler which can be controlled by modifying `MALLOC_CONF` environment variable.
+For example, interval between allocation samples can be controlled with `lg_prof_sample`.  
+If you want to dump heap profile every N bytes you can enable it using `lg_prof_interval`.  
+
+We recommend to check `jemalloc`s [reference page](https://jemalloc.net/jemalloc.3.html) for such options.
+
+## Other resources
+
+ClickHouse/Keeper expose `jemalloc` related metrics in many different ways.
+
+:::warning Warning
+It's important to be aware that none of these metrics are synchronized with each other and values may drift.
+:::
+
+### System table `asynchronous_metrics`
+
+```sql
+SELECT *
+FROM system.asynchronous_metrics
+WHERE metric ILIKE '%jemalloc%'
+FORMAT Vertical
+```
+
+[Reference](/en/operations/system-tables/asynchronous_metrics)
+
+### System table `jemalloc_bins`
+
+Contains information about memory allocations done via jemalloc allocator in different size classes (bins) aggregated from all arenas.
+
+[Reference](/en/operations/system-tables/jemalloc_bins)
+
+### Prometheus
+
+All `jemalloc` related metrics from `asynchronous_metrics` are also exposed using Prometheus endpoint in both ClickHouse and Keeper.
+
+[Reference](/en/operations/server-configuration-parameters/settings#prometheus)
+
+### `jmst` 4LW command in Keeper
+
+Keeper supports `jmst` 4LW command which returns [basic allocator statistics](https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Basic-Allocator-Statistics).
+
+Example:
+```sh
+echo jmst | nc localhost 9181
+```

docs/en/operations/settings/settings-formats.md

@@ -1597,7 +1597,13 @@ Result:
 
 Use ANSI escape sequences to paint colors in Pretty formats.
 
-Enabled by default.
+possible values:
+
+-   `0` — Disabled. Pretty formats do not use ANSI escape sequences.
+-   `1` — Enabled. Pretty formats will use ANSI escape sequences except for `NoEscapes` formats.
+-   `auto` - Enabled if `stdout` is a terminal except for `NoEscapes` formats.
+
+Default value is `auto`. 
 
 ### output_format_pretty_grid_charset {#output_format_pretty_grid_charset}
 

docs/ru/operations/settings/settings.md

@@ -2796,6 +2796,17 @@ SELECT TOP 3 name, value FROM system.settings;
 3. │ max_block_size          │ 65505   │
    └─────────────────────────┴─────────┘
 ```
+### output_format_pretty_color {#output_format_pretty_color}
+
+Включает/выключает управляющие последовательности ANSI в форматах Pretty. 
+
+Возможные значения:
+
+-   `0` — выключена. Не исползует ANSI последовательности в форматах Pretty.
+-   `1` — включена. Исползует ANSI последовательности с исключением форматов `NoEscapes`.
+-   `auto` - включена если `stdout` является терминалом с исключением форматов `NoEscapes`.
+
+Значение по умолчанию: `auto`
 
 ## system_events_show_zero_values {#system_events_show_zero_values}
 

src/Common/AsyncLoader.cpp

@@ -36,7 +36,7 @@ static constexpr size_t PRINT_MESSAGE_EACH_N_SECONDS = 5;
 
 void logAboutProgress(Poco::Logger * log, size_t processed, size_t total, AtomicStopwatch & watch)
 {
-    if (processed % PRINT_MESSAGE_EACH_N_OBJECTS == 0 || watch.compareAndRestart(PRINT_MESSAGE_EACH_N_SECONDS))
+    if (total && (processed % PRINT_MESSAGE_EACH_N_OBJECTS == 0 || watch.compareAndRestart(PRINT_MESSAGE_EACH_N_SECONDS)))
     {
         LOG_INFO(log, "Processed: {}%", processed * 100.0 / total);
         watch.restart();

src/Common/StackTrace.cpp

@@ -32,6 +32,10 @@ std::atomic<bool> show_addresses = true;
 
 bool shouldShowAddress(const void * addr)
 {
+    /// Likely inline frame
+    if (!addr)
+        return false;
+
     /// If the address is less than 4096, most likely it is a nullptr dereference with offset,
     /// and showing this offset is secure nevertheless.
     /// NOTE: 4096 is the page size on x86 and it can be different on other systems,
@@ -203,20 +207,24 @@ static void * getCallerAddress(const ucontext_t & context)
 #endif
 }
 
-// FIXME: looks like this is used only for Sentry but duplicates the whole algo, maybe replace?
+void StackTrace::forEachFrame(
-void StackTrace::symbolize(
+    const StackTrace::FramePointers & frame_pointers,
-    const StackTrace::FramePointers & frame_pointers, [[maybe_unused]] size_t offset, size_t size, StackTrace::Frames & frames)
+    size_t offset,
+    size_t size,
+    std::function<void(const Frame &)> callback,
+    bool fatal)
 {
 #if defined(__ELF__) && !defined(OS_FREEBSD)
     const DB::SymbolIndex & symbol_index = DB::SymbolIndex::instance();
     std::unordered_map<std::string, DB::Dwarf> dwarfs;
 
-    for (size_t i = 0; i < offset; ++i)
+    using enum DB::Dwarf::LocationInfoMode;
-        frames[i].virtual_addr = frame_pointers[i];
+    const auto mode = fatal ? FULL_WITH_INLINE : FAST;
 
     for (size_t i = offset; i < size; ++i)
     {
-        StackTrace::Frame & current_frame = frames[i];
+        StackTrace::Frame current_frame;
+        std::vector<DB::Dwarf::SymbolizedFrame> inline_frames;
         current_frame.virtual_addr = frame_pointers[i];
         const auto * object = symbol_index.findObject(current_frame.virtual_addr);
         uintptr_t virtual_offset = object ? uintptr_t(object->address_begin) : 0;
@@ -230,26 +238,41 @@ void StackTrace::symbolize(
                 auto dwarf_it = dwarfs.try_emplace(object->name, object->elf).first;
 
                 DB::Dwarf::LocationInfo location;
-                std::vector<DB::Dwarf::SymbolizedFrame> inline_frames;
                 if (dwarf_it->second.findAddress(
-                        uintptr_t(current_frame.physical_addr), location, DB::Dwarf::LocationInfoMode::FAST, inline_frames))
+                        uintptr_t(current_frame.physical_addr), location, mode, inline_frames))
                 {
                     current_frame.file = location.file.toString();
                     current_frame.line = location.line;
                 }
             }
         }
-        else
-            current_frame.object = "?";
 
         if (const auto * symbol = symbol_index.findSymbol(current_frame.virtual_addr))
             current_frame.symbol = demangle(symbol->name);
-        else
+
-            current_frame.symbol = "?";
+        for (const auto & frame : inline_frames)
+        {
+            StackTrace::Frame current_inline_frame;
+            const String file_for_inline_frame = frame.location.file.toString();
+
+            current_inline_frame.file = "inlined from " + file_for_inline_frame;
+            current_inline_frame.line = frame.location.line;
+            current_inline_frame.symbol = frame.name;
+
+            callback(current_inline_frame);
+        }
+
+        callback(current_frame);
     }
 #else
-    for (size_t i = 0; i < size; ++i)
+    UNUSED(fatal);
-        frames[i].virtual_addr = frame_pointers[i];
+
+    for (size_t i = offset; i < size; ++i)
+    {
+        StackTrace::Frame current_frame;
+        current_frame.virtual_addr = frame_pointers[i];
+        callback(current_frame);
+    }
 #endif
 }
 
@@ -349,72 +372,52 @@ toStringEveryLineImpl([[maybe_unused]] bool fatal, const StackTraceRefTriple & s
     if (stack_trace.size == 0)
         return callback("<Empty trace>");
 
+    size_t frame_index = stack_trace.offset;
 #if defined(__ELF__) && !defined(OS_FREEBSD)
-
+    size_t inline_frame_index = 0;
-    using enum DB::Dwarf::LocationInfoMode;
+    auto callback_wrapper = [&](const StackTrace::Frame & frame)
-    const auto mode = fatal ? FULL_WITH_INLINE : FAST;
-
-    const DB::SymbolIndex & symbol_index = DB::SymbolIndex::instance();
-    std::unordered_map<String, DB::Dwarf> dwarfs;
-
-    for (size_t i = stack_trace.offset; i < stack_trace.size; ++i)
     {
-        std::vector<DB::Dwarf::SymbolizedFrame> inline_frames;
-        const void * virtual_addr = stack_trace.pointers[i];
-        const auto * object = symbol_index.findObject(virtual_addr);
-        uintptr_t virtual_offset = object ? uintptr_t(object->address_begin) : 0;
-        const void * physical_addr = reinterpret_cast<const void *>(uintptr_t(virtual_addr) - virtual_offset);
-
         DB::WriteBufferFromOwnString out;
-        out << i << ". ";
 
-        String file;
+        /// Inline frame
-        if (std::error_code ec; object && std::filesystem::exists(object->name, ec) && !ec)
+        if (!frame.virtual_addr)
         {
-            auto dwarf_it = dwarfs.try_emplace(object->name, object->elf).first;
+            out << frame_index << "." << inline_frame_index++ << ". ";
-
+        }
-            DB::Dwarf::LocationInfo location;
+        else
-
+        {
-            if (dwarf_it->second.findAddress(uintptr_t(physical_addr), location, mode, inline_frames))
+            out << frame_index++ << ". ";
-            {
+            inline_frame_index = 0;
-                file = location.file.toString();
-                out << file << ":" << location.line << ": ";
-            }
         }
 
-        if (const auto * const symbol = symbol_index.findSymbol(virtual_addr))
+        if (frame.file.has_value() && frame.line.has_value())
-            out << demangleAndCollapseNames(file, symbol->name);
+            out << *frame.file << ':' << *frame.line << ": ";
+
+        if (frame.symbol.has_value() && frame.file.has_value())
+            out << demangleAndCollapseNames(*frame.file, frame.symbol->data());
         else
             out << "?";
 
-        if (shouldShowAddress(physical_addr))
+        if (shouldShowAddress(frame.physical_addr))
         {
             out << " @ ";
-            DB::writePointerHex(physical_addr, out);
+            DB::writePointerHex(frame.physical_addr, out);
         }
 
-        out << " in " << (object ? object->name : "?");
+        if (frame.object.has_value())
-
+            out << " in " << *frame.object;
-        for (size_t j = 0; j < inline_frames.size(); ++j)
-        {
-            const auto & frame = inline_frames[j];
-            const String file_for_inline_frame = frame.location.file.toString();
-            callback(fmt::format(
-                "{}.{}. inlined from {}:{}: {}",
-                i,
-                j + 1,
-                file_for_inline_frame,
-                frame.location.line,
-                demangleAndCollapseNames(file_for_inline_frame, frame.name)));
-        }
 
         callback(out.str());
-    }
+    };
 #else
-    for (size_t i = stack_trace.offset; i < stack_trace.size; ++i)
+    auto callback_wrapper = [&](const StackTrace::Frame & frame)
-        if (const void * const addr = stack_trace.pointers[i]; shouldShowAddress(addr))
+    {
-            callback(fmt::format("{}. {}", i, addr));
+        if (frame.virtual_addr && shouldShowAddress(frame.virtual_addr))
+            callback(fmt::format("{}. {}", frame_index++, frame.virtual_addr));
+    };
 #endif
+
+    StackTrace::forEachFrame(stack_trace.pointers, stack_trace.offset, stack_trace.size, callback_wrapper, fatal);
 }
 
 void StackTrace::toStringEveryLine(std::function<void(std::string_view)> callback) const

src/Common/StackTrace.h

@@ -62,7 +62,14 @@ public:
 
     static std::string toString(void ** frame_pointers, size_t offset, size_t size);
     static void dropCache();
-    static void symbolize(const FramePointers & frame_pointers, size_t offset, size_t size, StackTrace::Frames & frames);
+
+    /// @param fatal - if true, will process inline frames (slower)
+    static void forEachFrame(
+        const FramePointers & frame_pointers,
+        size_t offset,
+        size_t size,
+        std::function<void(const Frame &)> callback,
+        bool fatal);
 
     void toStringEveryLine(std::function<void(std::string_view)> callback) const;
     static void toStringEveryLine(const FramePointers & frame_pointers, std::function<void(std::string_view)> callback);

src/Core/Settings.h

@@ -810,7 +810,7 @@ class IColumn;
     M(Bool, parallelize_output_from_storages, true, "Parallelize output for reading step from storage. It allows parallelizing query processing right after reading from storage if possible", 0) \
     M(String, insert_deduplication_token, "", "If not empty, used for duplicate detection instead of data digest", 0) \
     M(Bool, count_distinct_optimization, false, "Rewrite count distinct to subquery of group by", 0) \
-    M(Bool, throw_if_no_data_to_insert, true, "Enables or disables empty INSERTs, enabled by default", 0) \
+    M(Bool, throw_if_no_data_to_insert, true, "Allows or forbids empty INSERTs, enabled by default (throws an error on an empty insert)", 0) \
     M(Bool, compatibility_ignore_auto_increment_in_create_table, false, "Ignore AUTO_INCREMENT keyword in column declaration if true, otherwise return error. It simplifies migration from MySQL", 0) \
     M(Bool, multiple_joins_try_to_keep_original_names, false, "Do not add aliases to top level expression list on multiple joins rewrite", 0) \
     M(Bool, optimize_sorting_by_input_stream_properties, true, "Optimize sorting by sorting properties of input stream", 0) \
@@ -1048,7 +1048,7 @@ class IColumn;
     M(UInt64, output_format_pretty_max_rows, 10000, "Rows limit for Pretty formats.", 0) \
     M(UInt64, output_format_pretty_max_column_pad_width, 250, "Maximum width to pad all values in a column in Pretty formats.", 0) \
     M(UInt64, output_format_pretty_max_value_width, 10000, "Maximum width of value to display in Pretty formats. If greater - it will be cut.", 0) \
-    M(Bool, output_format_pretty_color, true, "Use ANSI escape sequences to paint colors in Pretty formats", 0) \
+    M(UInt64Auto, output_format_pretty_color, "auto", "Use ANSI escape sequences in Pretty formats. 0 - disabled, 1 - enabled, 'auto' - enabled if a terminal.", 0) \
     M(String, output_format_pretty_grid_charset, "UTF-8", "Charset for printing grid borders. Available charsets: ASCII, UTF-8 (default one).", 0) \
     M(UInt64, output_format_parquet_row_group_size, 1000000, "Target row group size in rows.", 0) \
     M(UInt64, output_format_parquet_row_group_size_bytes, 512 * 1024 * 1024, "Target row group size in bytes, before compression.", 0) \

src/Core/SettingsChangesHistory.h

@@ -87,7 +87,8 @@ static std::map<ClickHouseVersion, SettingsChangesHistory::SettingsChanges> sett
     {"23.12", {{"allow_suspicious_ttl_expressions", true, false, "It is a new setting, and in previous versions the behavior was equivalent to allowing."},
               {"input_format_parquet_allow_missing_columns", false, true, "Allow missing columns in Parquet files by default"},
               {"input_format_orc_allow_missing_columns", false, true, "Allow missing columns in ORC files by default"},
-              {"input_format_arrow_allow_missing_columns", false, true, "Allow missing columns in Arrow files by default"}}},
+              {"input_format_arrow_allow_missing_columns", false, true, "Allow missing columns in Arrow files by default"},
+              {"output_format_pretty_color", true, "auto", "Setting is changed to allow also for auto value, disabling ANSI escapes if output is not a tty"}}},
     {"23.9", {{"optimize_group_by_constant_keys", false, true, "Optimize group by constant keys by default"},
               {"input_format_json_try_infer_named_tuples_from_objects", false, true, "Try to infer named Tuples from JSON objects by default"},
               {"input_format_json_read_numbers_as_strings", false, true, "Allow to read numbers as strings in JSON formats by default"},

src/Daemon/SentryWriter.cpp

@@ -169,11 +169,9 @@ void SentryWriter::onFault(int sig, const std::string & error_message, const Sta
             };
 
             StackTrace::Frames frames;
-            StackTrace::symbolize(stack_trace.getFramePointers(), offset, stack_size, frames);
 
-            for (ssize_t i = stack_size - 1; i >= offset; --i)
+            auto sentry_add_stack_trace = [&](const StackTrace::Frame & current_frame)
             {
-                const StackTrace::Frame & current_frame = frames[i];
                 sentry_value_t sentry_frame = sentry_value_new_object();
                 UInt64 frame_ptr = reinterpret_cast<UInt64>(current_frame.virtual_addr);
 
@@ -190,7 +188,9 @@ void SentryWriter::onFault(int sig, const std::string & error_message, const Sta
                     sentry_value_set_by_key(sentry_frame, "lineno", sentry_value_new_int32(static_cast<int32_t>(current_frame.line.value())));
 
                 sentry_value_append(sentry_frames, sentry_frame);
-            }
+            };
+
+            StackTrace::forEachFrame(stack_trace.getFramePointers(), offset, stack_size, sentry_add_stack_trace, /* fatal= */ true);
         }
 
         /// Prepare data for https://develop.sentry.dev/sdk/event-payloads/threads/

src/Disks/IO/createReadBufferFromFileBase.cpp

@@ -36,8 +36,7 @@ std::unique_ptr<ReadBufferFromFileBase> createReadBufferFromFileBase(
     std::optional<size_t> file_size,
     int flags,
     char * existing_memory,
-    size_t alignment,
+    size_t alignment)
-    bool use_external_buffer)
 {
     if (file_size.has_value() && !*file_size)
         return std::make_unique<ReadBufferFromEmptyFile>();
@@ -149,8 +148,7 @@ std::unique_ptr<ReadBufferFromFileBase> createReadBufferFromFileBase(
                 existing_memory,
                 buffer_alignment,
                 file_size,
-                settings.local_throttler,
+                settings.local_throttler);
-                use_external_buffer);
         }
         else
             throw Exception(ErrorCodes::LOGICAL_ERROR, "Unknown read method");

src/Disks/IO/createReadBufferFromFileBase.h

@@ -21,6 +21,5 @@ std::unique_ptr<ReadBufferFromFileBase> createReadBufferFromFileBase(
     std::optional<size_t> file_size = {},
     int flags_ = -1,
     char * existing_memory = nullptr,
-    size_t alignment = 0,
+    size_t alignment = 0);
-    bool use_external_buffer = false);
 }

src/Formats/FormatFactory.cpp

@@ -5,6 +5,7 @@
 #include <Interpreters/Context.h>
 #include <Interpreters/ProcessList.h>
 #include <IO/SharedThreadPools.h>
+#include <IO/WriteHelpers.h>
 #include <Processors/Formats/IRowInputFormat.h>
 #include <Processors/Formats/IRowOutputFormat.h>
 #include <Processors/Formats/Impl/MySQLOutputFormat.h>
@@ -451,6 +452,7 @@ OutputFormatPtr FormatFactory::getOutputFormatParallelIfPossible(
         throw Exception(ErrorCodes::FORMAT_IS_NOT_SUITABLE_FOR_OUTPUT, "Format {} is not suitable for output", name);
 
     auto format_settings = _format_settings ? *_format_settings : getFormatSettings(context);
+    format_settings.is_writing_to_terminal = isWritingToTerminal(buf);
 
     const Settings & settings = context->getSettingsRef();
 
@@ -492,6 +494,7 @@ OutputFormatPtr FormatFactory::getOutputFormat(
 
     auto format_settings = _format_settings ? *_format_settings : getFormatSettings(context);
     format_settings.max_threads = context->getSettingsRef().max_threads;
+    format_settings.is_writing_to_terminal = format_settings.is_writing_to_terminal = isWritingToTerminal(buf);
 
     /** TODO: Materialization is needed, because formats can use the functions `IDataType`,
       *  which only work with full columns.

src/Formats/FormatSettings.h

@@ -4,7 +4,7 @@
 #include <Core/Defines.h>
 #include <base/types.h>
 #include <base/unit.h>
-
+#include <Core/SettingsFields.h>
 
 namespace DB
 {
@@ -34,6 +34,7 @@ struct FormatSettings
     bool null_as_default = true;
     bool decimal_trailing_zeros = false;
     bool defaults_for_omitted_fields = true;
+    bool is_writing_to_terminal = false;
 
     bool seekable_read = true;
     UInt64 max_rows_to_read_for_schema_inference = 25000;
@@ -272,7 +273,7 @@ struct FormatSettings
         UInt64 max_rows = 10000;
         UInt64 max_column_pad_width = 250;
         UInt64 max_value_width = 10000;
-        bool color = true;
+        SettingFieldUInt64Auto color{"auto"};
 
         bool output_format_pretty_row_numbers = false;
 

src/IO/AsynchronousReadBufferFromFile.h

@@ -67,10 +67,8 @@ public:
         char * existing_memory = nullptr,
         size_t alignment = 0,
         std::optional<size_t> file_size_ = std::nullopt,
-        ThrottlerPtr throttler_ = {},
+        ThrottlerPtr throttler_ = {})
-        bool use_external_buffer_ = false)
+        : AsynchronousReadBufferFromFileDescriptor(reader_, priority_, -1, buf_size, existing_memory, alignment, file_size_, throttler_)
-        : AsynchronousReadBufferFromFileDescriptor(
-            reader_, priority_, -1, buf_size, existing_memory, alignment, file_size_, throttler_, use_external_buffer_)
         , file_name(file_name_)
     {
         file = OpenedFileCache::instance().get(file_name, flags);

src/IO/AsynchronousReadBufferFromFileDescriptor.cpp

@@ -97,11 +97,7 @@ bool AsynchronousReadBufferFromFileDescriptor::nextImpl()
         /// No pending request. Do synchronous read.
 
         ProfileEventTimeIncrement<Microseconds> watch(ProfileEvents::SynchronousReadWaitMicroseconds);
-        if (!use_external_buffer)
+        result = asyncReadInto(memory.data(), memory.size(), DEFAULT_PREFETCH_PRIORITY).get();
-            result = asyncReadInto(memory.data(), memory.size(), DEFAULT_PREFETCH_PRIORITY).get();
-        else
-            /// External buffer will be substituted in place of internal_buffer (see CachedOnDiskReadBufferFromFile)
-            result = asyncReadInto(internal_buffer.begin(), internal_buffer.size(), DEFAULT_PREFETCH_PRIORITY).get();
     }
 
     chassert(result.size >= result.offset);
@@ -114,9 +110,8 @@ bool AsynchronousReadBufferFromFileDescriptor::nextImpl()
     if (bytes_read)
     {
         /// Adjust the working buffer so that it ignores `offset` bytes.
-        if (!use_external_buffer)
+        internal_buffer = Buffer(memory.data(), memory.data() + memory.size());
-            internal_buffer = Buffer(memory.data(), memory.data() + memory.size());
+        working_buffer = Buffer(memory.data() + result.offset, memory.data() + result.size);
-        working_buffer = Buffer(internal_buffer.begin() + result.offset, internal_buffer.begin() + result.size);
         pos = working_buffer.begin();
     }
 
@@ -142,15 +137,13 @@ AsynchronousReadBufferFromFileDescriptor::AsynchronousReadBufferFromFileDescript
     char * existing_memory,
     size_t alignment,
     std::optional<size_t> file_size_,
-    ThrottlerPtr throttler_,
+    ThrottlerPtr throttler_)
-    bool use_external_buffer_)
     : ReadBufferFromFileBase(buf_size, existing_memory, alignment, file_size_)
     , reader(reader_)
     , base_priority(priority_)
     , required_alignment(alignment)
     , fd(fd_)
     , throttler(throttler_)
-    , use_external_buffer(use_external_buffer_)
 {
     if (required_alignment > buf_size)
         throw Exception(
@@ -228,7 +221,7 @@ off_t AsynchronousReadBufferFromFileDescriptor::seek(off_t offset, int whence)
     file_offset_of_buffer_end = seek_pos;
     bytes_to_ignore = new_pos - seek_pos;
 
-    if (bytes_to_ignore >= internal_buffer.size() && !(use_external_buffer && internal_buffer.empty()))
+    if (bytes_to_ignore >= internal_buffer.size())
         throw Exception(ErrorCodes::LOGICAL_ERROR,
                         "Logical error in AsynchronousReadBufferFromFileDescriptor, bytes_to_ignore ({}"
                         ") >= internal_buffer.size() ({})", bytes_to_ignore, internal_buffer.size());

src/IO/AsynchronousReadBufferFromFileDescriptor.h

@@ -29,7 +29,6 @@ protected:
     size_t bytes_to_ignore = 0;           /// How many bytes should we ignore upon a new read request.
     int fd;
     ThrottlerPtr throttler;
-    bool use_external_buffer;
 
     bool nextImpl() override;
 
@@ -47,8 +46,7 @@ public:
         char * existing_memory = nullptr,
         size_t alignment = 0,
         std::optional<size_t> file_size_ = std::nullopt,
-        ThrottlerPtr throttler_ = {},
+        ThrottlerPtr throttler_ = {});
-        bool use_external_buffer_ = false);
 
     ~AsynchronousReadBufferFromFileDescriptor() override;
 

src/IO/WriteHelpers.h

@@ -29,6 +29,7 @@
 #include <Common/Exception.h>
 #include <Common/StringUtils/StringUtils.h>
 #include <Common/NaNUtils.h>
+#include <Common/typeid_cast.h>
 
 #include <IO/CompressionMethod.h>
 #include <IO/WriteBuffer.h>
@@ -36,6 +37,7 @@
 #include <IO/VarInt.h>
 #include <IO/DoubleConverter.h>
 #include <IO/WriteBufferFromString.h>
+#include <IO/WriteBufferFromFileDescriptor.h>
 
 #ifdef __clang__
 #pragma clang diagnostic push
@@ -1405,6 +1407,12 @@ void writePointerHex(const void * ptr, WriteBuffer & buf);
 
 String fourSpaceIndent(size_t indent);
 
+bool inline isWritingToTerminal(const WriteBuffer & buf)
+{
+    const auto * write_buffer_to_descriptor = typeid_cast<const WriteBufferFromFileDescriptor *>(&buf);
+    return write_buffer_to_descriptor && write_buffer_to_descriptor->getFD() == STDOUT_FILENO && isatty(STDOUT_FILENO);
+}
+
 }
 
 template<>

src/Interpreters/Aggregator.cpp

@@ -1797,8 +1797,8 @@ void Aggregator::writeToTemporaryFile(AggregatedDataVariants & data_variants, si
         rows,
         ReadableSize(uncompressed_size),
         ReadableSize(compressed_size),
-        static_cast<double>(uncompressed_size) / rows,
+        rows ? static_cast<double>(uncompressed_size) / rows : 0.0,
-        static_cast<double>(compressed_size) / rows,
+        rows ? static_cast<double>(compressed_size) / rows : 0.0,
         static_cast<double>(uncompressed_size) / compressed_size,
         static_cast<double>(rows) / elapsed_seconds,
         ReadableSize(static_cast<double>(uncompressed_size) / elapsed_seconds),

src/Interpreters/Cache/SLRUFileCachePriority.cpp

@@ -264,7 +264,6 @@ SLRUFileCachePriority::SLRUIterator::SLRUIterator(
 
 SLRUFileCachePriority::EntryPtr SLRUFileCachePriority::SLRUIterator::getEntry() const
 {
-    chassert(entry == lru_iterator.getEntry());
     return entry;
 }
 

src/Processors/Formats/Impl/PrettyBlockOutputFormat.cpp

@@ -12,8 +12,8 @@ namespace DB
 {
 
 PrettyBlockOutputFormat::PrettyBlockOutputFormat(
-    WriteBuffer & out_, const Block & header_, const FormatSettings & format_settings_, bool mono_block_)
+    WriteBuffer & out_, const Block & header_, const FormatSettings & format_settings_, bool mono_block_, bool color_)
-     : IOutputFormat(header_, out_), format_settings(format_settings_), serializations(header_.getSerializations()), mono_block(mono_block_)
+     : IOutputFormat(header_, out_), format_settings(format_settings_), serializations(header_.getSerializations()), color(color_), mono_block(mono_block_)
 {
 }
 
@@ -237,7 +237,7 @@ void PrettyBlockOutputFormat::writeChunk(const Chunk & chunk, PortKind port_kind
 
         const auto & col = header.getByPosition(i);
 
-        if (format_settings.pretty.color)
+        if (color)
             writeCString("\033[1m", out);
 
         if (col.type->shouldAlignRightInPrettyFormats())
@@ -255,7 +255,7 @@ void PrettyBlockOutputFormat::writeChunk(const Chunk & chunk, PortKind port_kind
                 writeChar(' ', out);
         }
 
-        if (format_settings.pretty.color)
+        if (color)
             writeCString("\033[0m", out);
     }
     writeCString(" ", out);
@@ -335,7 +335,7 @@ void PrettyBlockOutputFormat::writeValueWithPadding(
             reinterpret_cast<const UInt8 *>(serialized_value.data()), serialized_value.size(), 0, 1 + format_settings.pretty.max_value_width));
 
         const char * ellipsis = format_settings.pretty.charset == FormatSettings::Pretty::Charset::UTF8 ? "⋯" : "~";
-        if (format_settings.pretty.color)
+        if (color)
         {
             serialized_value += "\033[31;1m";
             serialized_value += ellipsis;

src/Processors/Formats/Impl/PrettyBlockOutputFormat.h

@@ -5,7 +5,6 @@
 #include <Formats/FormatSettings.h>
 #include <Formats/FormatFactory.h>
 
-
 namespace DB
 {
 
@@ -19,10 +18,8 @@ class PrettyBlockOutputFormat : public IOutputFormat
 {
 public:
     /// no_escapes - do not use ANSI escape sequences - to display in the browser, not in the console.
-    PrettyBlockOutputFormat(WriteBuffer & out_, const Block & header_, const FormatSettings & format_settings_, bool mono_block_);
+    PrettyBlockOutputFormat(WriteBuffer & out_, const Block & header_, const FormatSettings & format_settings_, bool mono_block_, bool color_);
-
     String getName() const override { return "PrettyBlockOutputFormat"; }
-
 protected:
     void consume(Chunk) override;
     void consumeTotals(Chunk) override;
@@ -57,6 +54,8 @@ protected:
         total_rows = 0;
     }
 
+    bool color;
+
 private:
     bool mono_block;
     /// For mono_block == true only
@@ -73,13 +72,8 @@ void registerPrettyFormatWithNoEscapesAndMonoBlock(FormatFactory & factory, cons
             const Block & sample,
             const FormatSettings & format_settings)
         {
-            if (no_escapes)
+            bool color = !no_escapes && format_settings.pretty.color.valueOr(format_settings.is_writing_to_terminal);
-            {
+            return std::make_shared<OutputFormat>(buf, sample, format_settings, mono_block, color);
-                FormatSettings changed_settings = format_settings;
-                changed_settings.pretty.color = false;
-                return std::make_shared<OutputFormat>(buf, sample, changed_settings, mono_block);
-            }
-            return std::make_shared<OutputFormat>(buf, sample, format_settings, mono_block);
         });
         if (!mono_block)
             factory.markOutputFormatSupportsParallelFormatting(name);

src/Processors/Formats/Impl/PrettyCompactBlockOutputFormat.cpp

@@ -48,8 +48,8 @@ GridSymbols ascii_grid_symbols {
 
 }
 
-PrettyCompactBlockOutputFormat::PrettyCompactBlockOutputFormat(WriteBuffer & out_, const Block & header, const FormatSettings & format_settings_, bool mono_block_)
+PrettyCompactBlockOutputFormat::PrettyCompactBlockOutputFormat(WriteBuffer & out_, const Block & header, const FormatSettings & format_settings_, bool mono_block_, bool color_)
-    : PrettyBlockOutputFormat(out_, header, format_settings_, mono_block_)
+    : PrettyBlockOutputFormat(out_, header, format_settings_, mono_block_, color_)
 {
 }
 
@@ -87,18 +87,18 @@ void PrettyCompactBlockOutputFormat::writeHeader(
             for (size_t k = 0; k < max_widths[i] - name_widths[i]; ++k)
                 writeCString(grid_symbols.dash, out);
 
-            if (format_settings.pretty.color)
+            if (color)
                 writeCString("\033[1m", out);
             writeString(col.name, out);
-            if (format_settings.pretty.color)
+            if (color)
                 writeCString("\033[0m", out);
         }
         else
         {
-            if (format_settings.pretty.color)
+            if (color)
                 writeCString("\033[1m", out);
             writeString(col.name, out);
-            if (format_settings.pretty.color)
+            if (color)
                 writeCString("\033[0m", out);
 
             for (size_t k = 0; k < max_widths[i] - name_widths[i]; ++k)

src/Processors/Formats/Impl/PrettyCompactBlockOutputFormat.h

@@ -13,7 +13,7 @@ namespace DB
 class PrettyCompactBlockOutputFormat : public PrettyBlockOutputFormat
 {
 public:
-    PrettyCompactBlockOutputFormat(WriteBuffer & out_, const Block & header, const FormatSettings & format_settings_, bool mono_block_);
+    PrettyCompactBlockOutputFormat(WriteBuffer & out_, const Block & header, const FormatSettings & format_settings_, bool mono_block_, bool color);
     String getName() const override { return "PrettyCompactBlockOutputFormat"; }
 
 private:

src/Processors/Formats/Impl/PrettySpaceBlockOutputFormat.cpp

@@ -48,18 +48,18 @@ void PrettySpaceBlockOutputFormat::writeChunk(const Chunk & chunk, PortKind port
             for (ssize_t k = 0; k < std::max(0z, static_cast<ssize_t>(max_widths[i] - name_widths[i])); ++k)
                 writeChar(' ', out);
 
-            if (format_settings.pretty.color)
+            if (color)
                 writeCString("\033[1m", out);
             writeString(col.name, out);
-            if (format_settings.pretty.color)
+            if (color)
                 writeCString("\033[0m", out);
         }
         else
         {
-            if (format_settings.pretty.color)
+            if (color)
                 writeCString("\033[1m", out);
             writeString(col.name, out);
-            if (format_settings.pretty.color)
+            if (color)
                 writeCString("\033[0m", out);
 
             for (ssize_t k = 0; k < std::max(0z, static_cast<ssize_t>(max_widths[i] - name_widths[i])); ++k)

src/Processors/Formats/Impl/PrettySpaceBlockOutputFormat.h

@@ -11,8 +11,8 @@ namespace DB
 class PrettySpaceBlockOutputFormat : public PrettyBlockOutputFormat
 {
 public:
-    PrettySpaceBlockOutputFormat(WriteBuffer & out_, const Block & header, const FormatSettings & format_settings_, bool mono_block_)
+    PrettySpaceBlockOutputFormat(WriteBuffer & out_, const Block & header, const FormatSettings & format_settings_, bool mono_block_, bool color_)
-        : PrettyBlockOutputFormat(out_, header, format_settings_, mono_block_) {}
+        : PrettyBlockOutputFormat(out_, header, format_settings_, mono_block_, color_) {}
 
     String getName() const override { return "PrettySpaceBlockOutputFormat"; }
 

src/Storages/MergeTree/RPNBuilder.cpp

@@ -14,7 +14,9 @@
 #include <Columns/ColumnConst.h>
 #include <Columns/ColumnSet.h>
 
+#include <Functions/indexHint.h>
 #include <Functions/IFunction.h>
+#include <Functions/IFunctionAdaptors.h>
 
 #include <Storages/KeyDescription.h>
 
@@ -390,6 +392,15 @@ size_t RPNBuilderFunctionTreeNode::getArgumentsSize() const
     }
     else
     {
+        // indexHint arguments are stored inside of `FunctionIndexHint` class,
+        // because they are used only for index analysis.
+        if (dag_node->function_base->getName() == "indexHint")
+        {
+            const auto * adaptor = typeid_cast<const FunctionToFunctionBaseAdaptor *>(dag_node->function_base.get());
+            const auto * index_hint = typeid_cast<const FunctionIndexHint *>(adaptor->getFunction().get());
+            return index_hint->getActions()->getOutputs().size();
+        }
+
         return dag_node->children.size();
     }
 }
@@ -409,6 +420,15 @@ RPNBuilderTreeNode RPNBuilderFunctionTreeNode::getArgumentAt(size_t index) const
     }
     else
     {
+        // indexHint arguments are stored inside of `FunctionIndexHint` class,
+        // because they are used only for index analysis.
+        if (dag_node->function_base->getName() == "indexHint")
+        {
+            const auto * adaptor = typeid_cast<const FunctionToFunctionBaseAdaptor *>(dag_node->function_base.get());
+            const auto * index_hint = typeid_cast<const FunctionIndexHint *>(adaptor->getFunction().get());
+            return RPNBuilderTreeNode(index_hint->getActions()->getOutputs()[index], tree_context);
+        }
+
         return RPNBuilderTreeNode(dag_node->children[index], tree_context);
     }
 }

tests/integration/test_storage_kafka/test.py

@@ -4450,7 +4450,7 @@ def test_block_based_formats_1(kafka_cluster):
                      kafka_group_name = '{topic}',
                      kafka_format = 'PrettySpace';
 
-        INSERT INTO test.kafka SELECT number * 10 as key, number * 100 as value FROM numbers(5) settings max_block_size=2, optimize_trivial_insert_select=0;
+        INSERT INTO test.kafka SELECT number * 10 as key, number * 100 as value FROM numbers(5) settings max_block_size=2, optimize_trivial_insert_select=0, output_format_pretty_color=1;
     """
     )
 

tests/integration/test_storage_rabbitmq/test.py

@@ -3156,7 +3156,7 @@ def test_block_based_formats_1(rabbitmq_cluster):
     )
 
     instance.query(
-        "INSERT INTO test.rabbitmq SELECT number * 10 as key, number * 100 as value FROM numbers(5) settings max_block_size=2, optimize_trivial_insert_select=0;"
+        "INSERT INTO test.rabbitmq SELECT number * 10 as key, number * 100 as value FROM numbers(5) settings max_block_size=2, optimize_trivial_insert_select=0, output_format_pretty_color=1;"
     )
     insert_messages = []
 

tests/queries/0_stateless/00085_visible_width_of_tuple_of_dates.sql

@@ -1 +1,2 @@
+SET output_format_pretty_color=1;
 SELECT (toDate('2000-01-01'), toDate('2000-01-01')) AS x FORMAT PrettyCompact;

tests/queries/0_stateless/00089_group_by_arrays_of_fixed.sql

@@ -1 +1 @@
-SELECT arr, count() AS c FROM (SELECT arrayMap(x -> x % 2, groupArray(number)) AS arr FROM (SELECT number FROM system.numbers LIMIT 10000) GROUP BY number % ((number * 0xABCDEF0123456789 % 1234) + 1)) GROUP BY arr ORDER BY c DESC, arr ASC;
+SELECT arr, count() AS c FROM (SELECT arrayMap(x -> x % 2, arraySort(groupArray(number))) AS arr FROM (SELECT number FROM system.numbers LIMIT 10000) GROUP BY number % ((number * 0xABCDEF0123456789 % 1234) + 1)) GROUP BY arr ORDER BY c DESC, arr ASC;

tests/queries/0_stateless/00098_k_union_all.sql

@@ -1,3 +1,4 @@
+SET output_format_pretty_color=1;
 SELECT 1 FORMAT PrettySpace;
 SELECT 1 UNION ALL SELECT 1 FORMAT PrettySpace;
 SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1 FORMAT PrettySpace;

tests/queries/0_stateless/00298_enum_width_and_cast.sql

@@ -1,5 +1,6 @@
 DROP TABLE IF EXISTS enum;
 
+SET output_format_pretty_color=1;
 CREATE TABLE enum (x Enum8('Hello' = -100, '\\' = 0, '\t\\t' = 111), y UInt8) ENGINE = TinyLog;
 INSERT INTO enum (y) VALUES (0);
 SELECT * FROM enum ORDER BY x, y FORMAT PrettyCompact;

tests/queries/0_stateless/00405_PrettyCompactMonoBlock.sh

@@ -5,10 +5,10 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
 . "$CURDIR"/../shell_config.sh
 
 echo 'one block'
-${CLICKHOUSE_LOCAL} --query="SELECT * FROM numbers(2)" --format PrettyCompactMonoBlock
+${CLICKHOUSE_LOCAL} --query="SELECT * FROM numbers(2) SETTINGS output_format_pretty_color=1" --format PrettyCompactMonoBlock
 echo 'two blocks'
-${CLICKHOUSE_LOCAL} --query="SELECT * FROM numbers(1) UNION ALL SELECT * FROM numbers(1)" --format PrettyCompactMonoBlock
+${CLICKHOUSE_LOCAL} --query="SELECT * FROM numbers(1) UNION ALL SELECT * FROM numbers(1) SETTINGS output_format_pretty_color=1" --format PrettyCompactMonoBlock
 echo 'extremes'
-${CLICKHOUSE_LOCAL} --query="SELECT * FROM numbers(3)" --format PrettyCompactMonoBlock --extremes=1
+${CLICKHOUSE_LOCAL} --query="SELECT * FROM numbers(3) SETTINGS output_format_pretty_color=1" --format PrettyCompactMonoBlock --extremes=1
 echo 'totals'
-${CLICKHOUSE_LOCAL} --query="SELECT sum(number) FROM numbers(3) GROUP BY number%2 WITH TOTALS ORDER BY number%2" --format PrettyCompactMonoBlock
+${CLICKHOUSE_LOCAL} --query="SELECT sum(number) FROM numbers(3) GROUP BY number%2 WITH TOTALS ORDER BY number%2 SETTINGS output_format_pretty_color=1" --format PrettyCompactMonoBlock

tests/queries/0_stateless/00405_output_format_pretty_color.reference

@@ -0,0 +1,363 @@
+0
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     1 │ 1     │ (1,'1') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     2 │ 2     │ (2,'2') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     5 │ 5     │ (5,'5') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     7 │ 7     │ (7,'7') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     8 │ 8     │ (8,'8') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+ hello   world   tuple     sometimes_nulls
+
+     0   0       (0,'0')              ᴺᵁᴸᴸ 
+     1   1       (1,'1')                 1 
+     2   2       (2,'2')                 2 
+     3   3       (3,'3')              ᴺᵁᴸᴸ 
+     4   4       (4,'4')                 1 
+ hello   world   tuple     sometimes_nulls
+
+     5   5       (5,'5')                 2 
+     6   6       (6,'6')              ᴺᵁᴸᴸ 
+     7   7       (7,'7')                 1 
+     8   8       (8,'8')                 2 
+     9   9       (9,'9')              ᴺᵁᴸᴸ 
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     1 │ 1     │ (1,'1') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     2 │ 2     │ (2,'2') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     5 │ 5     │ (5,'5') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     7 │ 7     │ (7,'7') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     8 │ 8     │ (8,'8') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+ hello   world   tuple     sometimes_nulls
+
+     0   0       (0,'0')              ᴺᵁᴸᴸ 
+     1   1       (1,'1')                 1 
+     2   2       (2,'2')                 2 
+     3   3       (3,'3')              ᴺᵁᴸᴸ 
+     4   4       (4,'4')                 1 
+ hello   world   tuple     sometimes_nulls
+
+     5   5       (5,'5')                 2 
+     6   6       (6,'6')              ᴺᵁᴸᴸ 
+     7   7       (7,'7')                 1 
+     8   8       (8,'8')                 2 
+     9   9       (9,'9')              ᴺᵁᴸᴸ 
+1
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     1 │ 1     │ (1,'1') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     2 │ 2     │ (2,'2') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     5 │ 5     │ (5,'5') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     7 │ 7     │ (7,'7') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     8 │ 8     │ (8,'8') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+ hello   world   tuple     sometimes_nulls
+
+     0   0       (0,'0')              ᴺᵁᴸᴸ 
+     1   1       (1,'1')                 1 
+     2   2       (2,'2')                 2 
+     3   3       (3,'3')              ᴺᵁᴸᴸ 
+     4   4       (4,'4')                 1 
+ hello   world   tuple     sometimes_nulls
+
+     5   5       (5,'5')                 2 
+     6   6       (6,'6')              ᴺᵁᴸᴸ 
+     7   7       (7,'7')                 1 
+     8   8       (8,'8')                 2 
+     9   9       (9,'9')              ᴺᵁᴸᴸ 
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     1 │ 1     │ (1,'1') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     2 │ 2     │ (2,'2') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     5 │ 5     │ (5,'5') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     7 │ 7     │ (7,'7') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     8 │ 8     │ (8,'8') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+ hello   world   tuple     sometimes_nulls
+
+     0   0       (0,'0')              ᴺᵁᴸᴸ 
+     1   1       (1,'1')                 1 
+     2   2       (2,'2')                 2 
+     3   3       (3,'3')              ᴺᵁᴸᴸ 
+     4   4       (4,'4')                 1 
+ hello   world   tuple     sometimes_nulls
+
+     5   5       (5,'5')                 2 
+     6   6       (6,'6')              ᴺᵁᴸᴸ 
+     7   7       (7,'7')                 1 
+     8   8       (8,'8')                 2 
+     9   9       (9,'9')              ᴺᵁᴸᴸ 
+auto
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     1 │ 1     │ (1,'1') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     2 │ 2     │ (2,'2') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     5 │ 5     │ (5,'5') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     7 │ 7     │ (7,'7') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     8 │ 8     │ (8,'8') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+ hello   world   tuple     sometimes_nulls
+
+     0   0       (0,'0')              ᴺᵁᴸᴸ 
+     1   1       (1,'1')                 1 
+     2   2       (2,'2')                 2 
+     3   3       (3,'3')              ᴺᵁᴸᴸ 
+     4   4       (4,'4')                 1 
+ hello   world   tuple     sometimes_nulls
+
+     5   5       (5,'5')                 2 
+     6   6       (6,'6')              ᴺᵁᴸᴸ 
+     7   7       (7,'7')                 1 
+     8   8       (8,'8')                 2 
+     9   9       (9,'9')              ᴺᵁᴸᴸ 
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     1 │ 1     │ (1,'1') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     2 │ 2     │ (2,'2') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┏━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ hello ┃ world ┃ tuple   ┃ sometimes_nulls ┃
+┡━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│     5 │ 5     │ (5,'5') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+├───────┼───────┼─────────┼─────────────────┤
+│     7 │ 7     │ (7,'7') │               1 │
+├───────┼───────┼─────────┼─────────────────┤
+│     8 │ 8     │ (8,'8') │               2 │
+├───────┼───────┼─────────┼─────────────────┤
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     0 │ 0     │ (0,'0') │            ᴺᵁᴸᴸ │
+│     1 │ 1     │ (1,'1') │               1 │
+│     2 │ 2     │ (2,'2') │               2 │
+│     3 │ 3     │ (3,'3') │            ᴺᵁᴸᴸ │
+│     4 │ 4     │ (4,'4') │               1 │
+└───────┴───────┴─────────┴─────────────────┘
+┌─hello─┬─world─┬─tuple───┬─sometimes_nulls─┐
+│     5 │ 5     │ (5,'5') │               2 │
+│     6 │ 6     │ (6,'6') │            ᴺᵁᴸᴸ │
+│     7 │ 7     │ (7,'7') │               1 │
+│     8 │ 8     │ (8,'8') │               2 │
+│     9 │ 9     │ (9,'9') │            ᴺᵁᴸᴸ │
+└───────┴───────┴─────────┴─────────────────┘
+ hello   world   tuple     sometimes_nulls
+
+     0   0       (0,'0')              ᴺᵁᴸᴸ 
+     1   1       (1,'1')                 1 
+     2   2       (2,'2')                 2 
+     3   3       (3,'3')              ᴺᵁᴸᴸ 
+     4   4       (4,'4')                 1 
+ hello   world   tuple     sometimes_nulls
+
+     5   5       (5,'5')                 2 
+     6   6       (6,'6')              ᴺᵁᴸᴸ 
+     7   7       (7,'7')                 1 
+     8   8       (8,'8')                 2 
+     9   9       (9,'9')              ᴺᵁᴸᴸ 

tests/queries/0_stateless/00405_output_format_pretty_color.sql

@@ -0,0 +1,32 @@
+SET output_format_pretty_color = 0;
+SHOW SETTING output_format_pretty_color;
+
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT Pretty;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompact;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettySpace;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompactMonoBlock;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyNoEscapes;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompactNoEscapes;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettySpaceNoEscapes;
+
+SET output_format_pretty_color = 1;
+SHOW SETTING output_format_pretty_color;
+
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT Pretty;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompact;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettySpace;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompactMonoBlock;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyNoEscapes;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompactNoEscapes;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettySpaceNoEscapes;
+
+SET output_format_pretty_color = 'auto';
+SHOW SETTING output_format_pretty_color;
+
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT Pretty;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompact;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettySpace;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompactMonoBlock;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyNoEscapes;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompactNoEscapes;
+SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettySpaceNoEscapes;

tests/queries/0_stateless/00405_pretty_formats.sql

@@ -1,3 +1,5 @@
+SET output_format_pretty_color = 1;
+
 SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT Pretty;
 SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettyCompact;
 SELECT number AS hello, toString(number) AS world, (hello, world) AS tuple, nullIf(hello % 3, 0) AS sometimes_nulls FROM system.numbers LIMIT 10 SETTINGS max_block_size = 5 FORMAT PrettySpace;

tests/queries/0_stateless/00476_pretty_formats_and_widths.sql

@@ -1,3 +1,4 @@
+SET output_format_pretty_color=1;
 SELECT toUInt64(round(exp10(number))) AS x, toString(x) AS s FROM system.numbers LIMIT 10 FORMAT Pretty;
 SELECT toUInt64(round(exp10(number))) AS x, toString(x) AS s FROM system.numbers LIMIT 10 FORMAT PrettyCompact;
 SELECT toUInt64(round(exp10(number))) AS x, toString(x) AS s FROM system.numbers LIMIT 10 FORMAT PrettySpace;

tests/queries/0_stateless/00818_inner_join_bug_3567.sql

@@ -1,3 +1,4 @@
+SET output_format_pretty_color = 1;
 SET allow_experimental_analyzer = 1;
 
 DROP TABLE IF EXISTS table1;

tests/queries/0_stateless/01018_ambiguous_column.sql

@@ -1,3 +1,4 @@
+SET output_format_pretty_color=1;
 SET allow_experimental_analyzer = 1;
 
 select * from system.one cross join system.one;

tests/queries/0_stateless/01074_partial_revokes.sql

@@ -43,7 +43,7 @@ REVOKE SELECT ON db.* FROM test_user_01074;
 GRANT SELECT ON db.table TO test_user_01074;
 REVOKE SELECT(col1) ON db.table FROM test_user_01074;
 SHOW GRANTS FOR test_user_01074;
-SELECT * FROM system.grants WHERE user_name = 'test_user_01074' format Pretty;
+SELECT * FROM system.grants WHERE user_name = 'test_user_01074' SETTINGS output_format_pretty_color=1 FORMAT Pretty;
 
 SELECT '--cleanup';
 REVOKE SELECT ON *.* FROM test_user_01074;
@@ -73,7 +73,7 @@ SELECT '--grant option 1';
 GRANT SELECT ON *.* TO test_user_01074 WITH GRANT OPTION;
 REVOKE GRANT OPTION FOR SELECT(col1) ON db.table FROM test_user_01074;
 SHOW GRANTS FOR test_user_01074;
-SELECT * FROM system.grants WHERE user_name = 'test_user_01074' format Pretty;
+SELECT * FROM system.grants WHERE user_name = 'test_user_01074' SETTINGS output_format_pretty_color=1 FORMAT Pretty;
 
 SELECT '--cleanup';
 REVOKE SELECT ON *.* FROM test_user_01074;

tests/queries/0_stateless/01293_pretty_max_value_width.sql

@@ -1,3 +1,4 @@
+SET output_format_pretty_color=1;
 SELECT 'привет' AS x, 'мир' AS y FORMAT Pretty;
 
 SET output_format_pretty_max_value_width = 5;

tests/queries/0_stateless/01472_many_rows_in_totals.sql

@@ -1,3 +1,5 @@
+set output_format_pretty_color=1;
+
 -- Disable external aggregation because it may produce several blocks instead of one.
 set max_bytes_before_external_group_by = 0;
 set output_format_write_statistics = 0;

tests/queries/0_stateless/01509_output_format_pretty_row_numbers.sql

@@ -1,3 +1,4 @@
+SET output_format_pretty_color=1;
 SELECT * FROM numbers(10) FORMAT Pretty;
 SELECT * FROM numbers(10) FORMAT PrettyCompact;
 SELECT * FROM numbers(10) FORMAT PrettyCompactMonoBlock;

tests/queries/0_stateless/01670_neighbor_lc_bug.sql

@@ -40,6 +40,7 @@ FROM
     ORDER BY val_string, rowNr
 )
 ORDER BY rowNr, val_string, str_m1, str_p1, val_low, low_m1, low_p1
+SETTINGS output_format_pretty_color=1
 format PrettyCompact;
 
 drop table if exists neighbor_test;

tests/queries/0_stateless/01671_merge_join_and_constants.sql

@@ -1,3 +1,4 @@
+SET output_format_pretty_color=1;
 SET allow_experimental_analyzer = 1;
 
 DROP TABLE IF EXISTS table1;

tests/queries/0_stateless/02152_bool_type.sql

@@ -1,3 +1,5 @@
+SET output_format_pretty_color=1;
+
 SELECT CAST('True', 'Bool');
 SELECT CAST('TrUe', 'Bool');
 SELECT CAST('true', 'Bool');

tests/queries/0_stateless/02375_pretty_formats.sql.j2

@@ -3,6 +3,6 @@
                   'PrettySpaceNoEscapesMonoBlock'] -%}
 
 select '{{ format }}';
-select number as x, number + 1 as y from numbers(4) settings max_block_size=2 format {{ format }};
+select number as x, number + 1 as y from numbers(4) settings max_block_size=2, output_format_pretty_color=1 format {{ format }};
 
 {% endfor -%}

tests/queries/0_stateless/02418_do_not_return_empty_blocks_from_ConvertingAggregatedToChunksTransform.sh

@@ -13,7 +13,7 @@ ${CLICKHOUSE_CURL} \
      from numbers_mt(1e6)
      where number = 42
      group by number
-     settings max_threads = 10, max_bytes_before_external_group_by = 1, group_by_two_level_threshold = 1
+     settings max_threads = 10, max_bytes_before_external_group_by = 1, group_by_two_level_threshold = 1, output_format_pretty_color=1
      format PrettyCompact"
 
 ${CLICKHOUSE_CURL} \
@@ -24,5 +24,5 @@ ${CLICKHOUSE_CURL} \
     from numbers_mt(1e6)
     where number = 42
     group by number
-    settings max_threads = 10, max_bytes_before_external_group_by = 0, group_by_two_level_threshold = 1
+    settings max_threads = 10, max_bytes_before_external_group_by = 0, group_by_two_level_threshold = 1, output_format_pretty_color=1
     format PrettyCompact"

tests/queries/0_stateless/02421_formats_with_totals_and_extremes.sql.j2

@@ -1,5 +1,6 @@
 -- Tags: no-fasttest
 
+set output_format_pretty_color=1;
 set output_format_write_statistics=0;
 
 {% for format in ['CSV', 'TSV', 'XML', 'Vertical', 'Pretty', 'JSON', 'JSONCompact'] -%}

tests/queries/0_stateless/02798_explain_settings_not_applied_bug.sql

@@ -1,3 +1,4 @@
+SET output_format_pretty_color=1;
 SET read_in_order_two_level_merge_threshold=1000000;
 
 DROP TABLE IF EXISTS t;

tests/queries/0_stateless/02896_union_distinct_http_format.sh

@@ -4,5 +4,5 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
 # shellcheck source=../shell_config.sh
 . "$CURDIR"/../shell_config.sh
 
-curl -d@- -sS "${CLICKHOUSE_URL}" <<< 'SELECT 1 UNION DISTINCT SELECT 1 FORMAT PrettyCompactMonoBlock'
+curl -d@- -sS "${CLICKHOUSE_URL}" <<< 'SELECT 1 UNION DISTINCT SELECT 1 SETTINGS output_format_pretty_color=1 FORMAT PrettyCompactMonoBlock'
-curl -d@- -sS "${CLICKHOUSE_URL}" <<< 'SELECT * FROM (SELECT 1 as a UNION DISTINCT SELECT 2 as a) ORDER BY a FORMAT PrettyCompactMonoBlock'
+curl -d@- -sS "${CLICKHOUSE_URL}" <<< 'SELECT * FROM (SELECT 1 as a UNION DISTINCT SELECT 2 as a) ORDER BY a SETTINGS output_format_pretty_color=1 FORMAT PrettyCompactMonoBlock'

tests/queries/0_stateless/02962_analyzer_resolve_group_by_on_shards.sql

@@ -0,0 +1,20 @@
+SELECT NULL AND (toDate(-2147483647, NULL) AND NULL)
+FROM remote('127.0.0.{1,2}', view(
+    SELECT
+        NULL AND NULL,
+        NULL,
+        toDate(toDate('0.0001048577', toDate(NULL, 10 AND (toDate(257, 9223372036854775807, NULL) AND NULL AND NULL) AND NULL, 7, NULL), NULL, NULL) AND NULL AND -2147483648, NULL, NULL) AND NULL
+    FROM system.one
+    WHERE toDate(toDate(NULL, NULL, NULL), NULL)
+    GROUP BY
+        GROUPING SETS ((NULL))
+));
+
+SELECT NULL AND (toDate(-2147483647, NULL) AND NULL)
+FROM remote('127.0.0.{1,2}', view(
+    SELECT NULL
+    FROM system.one
+    WHERE toDate(toDate(NULL, NULL, NULL), NULL)
+    GROUP BY
+        GROUPING SETS ((''))
+));

tests/queries/0_stateless/02962_indexHint_rpn_construction.sql

@@ -0,0 +1,20 @@
+CREATE TABLE tab
+(
+    `foo` Array(LowCardinality(String)),
+    INDEX idx foo TYPE bloom_filter GRANULARITY 1
+)
+ENGINE = MergeTree
+PRIMARY KEY tuple();
+
+INSERT INTO tab SELECT if(number % 2, ['value'], [])
+FROM system.numbers
+LIMIT 10000;
+
+SELECT *
+FROM tab
+PREWHERE indexHint(indexHint(-1, 0.))
+WHERE has(foo, 'b');
+
+SELECT *
+FROM tab
+PREWHERE indexHint(0);

utils/check-style/aspell-ignore/en/aspell-dict.txt

@@ -282,6 +282,7 @@ FilesystemMainPathTotalINodes
 FilesystemMainPathUsedBytes
 FilesystemMainPathUsedINodes
 FixedString
+FlameGraph
 Flink
 ForEach
 FreeBSD
@@ -1130,6 +1131,7 @@ authenticators
 autocompletion
 autodetect
 autodetected
+autogen
 autogenerate
 autogenerated
 autogeneration
@@ -1718,6 +1720,7 @@ javaHashUTF
 jbod
 jdbc
 jemalloc
+jeprof
 joinGet
 json
 jsonMergePatch