thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-10 01:25:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

renamed setting

Browse Source
This commit is contained in:
zvonand 2023-09-07 12:00:25 +03:00
parent 83405a8808
commit c36a3fbd39
10 changed files with 37 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/en/operations/settings/settings.md
View File

@ -4660,7 +4660,7 @@ The default value is `false`.
<validate_tcp_client_information>true</validate_tcp_client_information>
```
## ignore_eacces_multidirectory_globs {#ignore_eacces_multidirectory_globs}
## ignore_access_denied_multidirectory_globs {#ignore_access_denied_multidirectory_globs}
Allows to ignore 'permission denied' errors when using multi-directory `{}` globs for [File](../../sql-reference/table-functions/file.md#globs_in_path) and [HDFS](../../sql-reference/table-functions/hdfs.md) storages.
This setting is only applicable when multi directory `{}` glob is used.
@ -4689,12 +4689,12 @@ It happens because a multi-directory glob requires a recursive search in _all_ a
If this setting is on, all inaccessible directories will be silently skipped, even if they are explicitly specified inside `{}`.
```sql
SELECT _path, _file FROM file('my_directory/{data1/f1,data2/f2}.csv', CSV) SETTINGS ignore_eacces_multidirectory_globs = 0;
SELECT _path, _file FROM file('my_directory/{data1/f1,data2/f2}.csv', CSV) SETTINGS ignore_access_denied_multidirectory_globs = 0;
Code: 1001. DB::Exception: std::__1::__fs::filesystem::filesystem_error: filesystem error: in directory_iterator::directory_iterator(...): Permission denied
```
```sql
SELECT _path, _file FROM file('my_directory/{data1/f1,data2/f2}.csv', CSV) SETTINGS ignore_eacces_multidirectory_globs = 1;
SELECT _path, _file FROM file('my_directory/{data1/f1,data2/f2}.csv', CSV) SETTINGS ignore_access_denied_multidirectory_globs = 1;
┌─_path───────────────────┬─_file───────┐
│ <full path to file> │ <file name> │

4
docs/en/sql-reference/table-functions/file.md
View File

@ -141,7 +141,7 @@ Multiple path components can have globs. For being processed file must exist and
- `*` — Substitutes any number of any characters except `/` including empty string.
- `?` — Substitutes any single character.
- `{some_string,another_string,yet_another_one}` — Substitutes any of strings `'some_string', 'another_string', 'yet_another_one'`. In case at least one of strings contains `/`, `'permission denied'` errors may be ignored using [ignore_eacces_multidirectory_globs](/docs/en/operations/settings/settings.md#ignore_eacces_multidirectory_globs) setting for file & HDFS.
- `{some_string,another_string,yet_another_one}` — Substitutes any of strings `'some_string', 'another_string', 'yet_another_one'`. In case at least one of strings contains `/`, `'permission denied'` errors may be ignored using [ignore_access_denied_multidirectory_globs](/docs/en/operations/settings/settings.md#ignore_access_denied_multidirectory_globs) setting for file & HDFS.
- `{N..M}` — Substitutes any number in range from N to M including both borders.
- `**` - Fetches all files inside the folder recursively.
@ -210,7 +210,7 @@ SELECT count(*) FROM file('big_dir/**/file002', 'CSV', 'name String, value UInt3
- [engine_file_allow_create_multiple_files](/docs/en/operations/settings/settings.md#engine_file_allow_create_multiple_files) - allows to create a new file on each insert if format has suffix. Disabled by default.
- [engine_file_skip_empty_files](/docs/en/operations/settings/settings.md#engine_file_skip_empty_files) - allows to skip empty files while reading. Disabled by default.
- [storage_file_read_method](/docs/en/operations/settings/settings.md#engine-file-emptyif-not-exists) - method of reading data from storage file, one of: read, pread, mmap (only for clickhouse-local). Default value: `pread` for clickhouse-server, `mmap` for clickhouse-local.
- [ignore_eacces_multidirectory_globs](/docs/en/operations/settings/settings.md#ignore_eacces_multidirectory_globs) - allows to ignore permission denied errors for multi-directory globs.
- [ignore_access_denied_multidirectory_globs](/docs/en/operations/settings/settings.md#ignore_access_denied_multidirectory_globs) - allows to ignore permission denied errors for multi-directory globs.
**See Also**

4
docs/en/sql-reference/table-functions/hdfs.md
View File

@ -45,7 +45,7 @@ Multiple path components can have globs. For being processed file should exists
- `*` — Substitutes any number of any characters except `/` including empty string.
- `?` — Substitutes any single character.
- `{some_string,another_string,yet_another_one}` — Substitutes any of strings `'some_string', 'another_string', 'yet_another_one'`. In case at least one of strings contains `/`, `'permission denied'` errors may be ignored using [ignore_eacces_multidirectory_globs](/docs/en/operations/settings/settings.md#ignore_eacces_multidirectory_globs) setting.
- `{some_string,another_string,yet_another_one}` — Substitutes any of strings `'some_string', 'another_string', 'yet_another_one'`. In case at least one of strings contains `/`, `'permission denied'` errors may be ignored using [ignore_access_denied_multidirectory_globs](/docs/en/operations/settings/settings.md#ignore_access_denied_multidirectory_globs) setting.
- `{N..M}` — Substitutes any number in range from N to M including both borders.
Constructions with `{}` are similar to the [remote table function](../../sql-reference/table-functions/remote.md)).
@ -102,7 +102,7 @@ FROM hdfs('hdfs://hdfs1:9000/big_dir/file{0..9}{0..9}{0..9}', 'CSV', 'name Strin
- [hdfs_truncate_on_insert](/docs/en/operations/settings/settings.md#hdfs-truncate-on-insert) - allows to truncate file before insert into it. Disabled by default.
- [hdfs_create_multiple_files](/docs/en/operations/settings/settings.md#hdfs_allow_create_multiple_files) - allows to create a new file on each insert if format has suffix. Disabled by default.
- [hdfs_skip_empty_files](/docs/en/operations/settings/settings.md#hdfs_skip_empty_files) - allows to skip empty files while reading. Disabled by default.
- [ignore_eacces_multidirectory_globs](/docs/en/operations/settings/settings.md#ignore_eacces_multidirectory_globs) - allows to ignore permission denied errors for multi-directory globs.
- [ignore_access_denied_multidirectory_globs](/docs/en/operations/settings/settings.md#ignore_access_denied_multidirectory_globs) - allows to ignore permission denied errors for multi-directory globs.
**See Also**

6
docs/ru/operations/settings/settings.md
View File

@ -4207,7 +4207,7 @@ SELECT toFloat64('1.7091'), toFloat64('1.5008753E7') SETTINGS precise_float_pars
└─────────────────────┴──────────────────────────┘
```
## ignore_eacces_multidirectory_globs {#ignore_eacces_multidirectory_globs}
## ignore_access_denied_multidirectory_globs {#ignore_access_denied_multidirectory_globs}
Позволяет игнорировать ошибку 'permission denied', возникающую при использовании шаблона `{}`, содержащего `/` внутри себя.
Работает для [File](../../sql-reference/table-functions/file.md#globs_in_path) и [HDFS](../../sql-reference/table-functions/hdfs.md).
@ -4237,12 +4237,12 @@ my_directory/
Если данная настройка имеет значение 1, то недоступные директории будут тихо пропущены, даже если они явно указаны внутри `{}`.
```sql
SELECT _path, _file FROM file('my_directory/{data1/f1,data2/f2}.csv', CSV) SETTINGS ignore_eacces_multidirectory_globs = 0;
SELECT _path, _file FROM file('my_directory/{data1/f1,data2/f2}.csv', CSV) SETTINGS ignore_access_denied_multidirectory_globs = 0;
Code: 1001. DB::Exception: std::__1::__fs::filesystem::filesystem_error: filesystem error: in directory_iterator::directory_iterator(...): Permission denied
```
```sql
SELECT _path, _file FROM file('my_directory/{data1/f1,data2/f2}.csv', CSV) SETTINGS ignore_eacces_multidirectory_globs = 1;
SELECT _path, _file FROM file('my_directory/{data1/f1,data2/f2}.csv', CSV) SETTINGS ignore_access_denied_multidirectory_globs = 1;
┌─_path───────────────────┬─_file───────┐
│ <full path to file> │ <file name> │

2
docs/ru/sql-reference/table-functions/file.md
View File

@ -79,7 +79,7 @@ SELECT * FROM file('test.csv', 'CSV', 'column1 UInt32, column2 UInt32, column3 U
- `*` — заменяет любое количество любых символов кроме `/`, включая отсутствие символов.
- `?` — заменяет ровно один любой символ.
- `{some_string,another_string,yet_another_one}` — заменяет любую из строк `'some_string', 'another_string', 'yet_another_one'`. В случае, если в какой-либо из строк содержится `/`, то ошибки доступа (permission denied) к существующим, но недоступным директориям/файлам могут быть проигнорированы при помощи настройки [ignore_eacces_multidirectory_globs](/docs/ru/operations/settings/settings.md#ignore_eacces_multidirectory_globs).
- `{some_string,another_string,yet_another_one}` — заменяет любую из строк `'some_string', 'another_string', 'yet_another_one'`. В случае, если в какой-либо из строк содержится `/`, то ошибки доступа (permission denied) к существующим, но недоступным директориям/файлам могут быть проигнорированы при помощи настройки [ignore_access_denied_multidirectory_globs](/docs/ru/operations/settings/settings.md#ignore_access_denied_multidirectory_globs).
- `{N..M}` — заменяет любое число в интервале от `N` до `M` включительно (может содержать ведущие нули).
Конструкция с `{}` аналогична табличной функции [remote](remote.md).

4
docs/ru/sql-reference/table-functions/hdfs.md
View File

@ -43,7 +43,7 @@ LIMIT 2
- `*` — Заменяет любое количество любых символов кроме `/`, включая отсутствие символов.
- `?` — Заменяет ровно один любой символ.
- `{some_string,another_string,yet_another_one}` — Заменяет любую из строк `'some_string', 'another_string', 'yet_another_one'`. В случае, если в какой-либо из строк содержится `/`, то ошибки доступа (permission denied) к существующим, но недоступным директориям/файлам могут быть проигнорированы при помощи настройки [ignore_eacces_multidirectory_globs](/docs/ru/operations/settings/settings.md#ignore_eacces_multidirectory_globs).
- `{some_string,another_string,yet_another_one}` — Заменяет любую из строк `'some_string', 'another_string', 'yet_another_one'`. В случае, если в какой-либо из строк содержится `/`, то ошибки доступа (permission denied) к существующим, но недоступным директориям/файлам могут быть проигнорированы при помощи настройки [ignore_access_denied_multidirectory_globs](/docs/ru/operations/settings/settings.md#ignore_access_denied_multidirectory_globs).
- `{N..M}` — Заменяет любое число в интервале от `N` до `M` включительно (может содержать ведущие нули).
Конструкция с `{}` аналогична табличной функции [remote](remote.md).
@ -61,5 +61,5 @@ LIMIT 2
**Смотрите также**
- [Виртуальные столбцы](index.md#table_engines-virtual_columns)
- Параметр [ignore_eacces_multidirectory_globs](/docs/ru/operations/settings/settings.md#ignore_eacces_multidirectory_globs)
- Параметр [ignore_access_denied_multidirectory_globs](/docs/ru/operations/settings/settings.md#ignore_access_denied_multidirectory_globs)

2
src/Core/Settings.h
View File

@ -797,7 +797,7 @@ class IColumn;
M(Timezone, session_timezone, "", "This setting can be removed in the future due to potential caveats. It is experimental and is not suitable for production usage. The default timezone for current session or query. The server default timezone if empty.", 0) \
M(Bool, allow_create_index_without_type, false, "Allow CREATE INDEX query without TYPE. Query will be ignored. Made for SQL compatibility tests.", 0)\
M(Bool, create_index_ignore_unique, false, "Ignore UNIQUE keyword in CREATE UNIQUE INDEX. Made for SQL compatibility tests.", 0) \
M(Bool, ignore_eacces_multidirectory_globs, false, "Ignore access denied errors when processing multi-directory globs for file & HDFS.", 0)\
M(Bool, ignore_access_denied_multidirectory_globs, false, "Ignore access denied errors when processing multi-directory globs for file & HDFS.", 0)\
// End of COMMON_SETTINGS
// Please add settings related to formats into the FORMAT_FACTORY_SETTINGS and move obsolete settings to OBSOLETE_SETTINGS.

20
src/Storages/HDFS/StorageHDFS.cpp
View File

@ -74,7 +74,7 @@ namespace
std::vector<StorageHDFS::PathWithInfo> LSWithRegexpMatching(const String & path_for_ls,
const HDFSFSPtr & fs,
const String & for_match,
bool ignore_eacces_multidirectory_globs);
bool ignore_access_denied_multidirectory_globs);
/*
* When `{...}` has any `/`s, it must be processed in a different way:
@ -91,7 +91,7 @@ namespace
re2::RE2 & matcher,
const size_t max_depth,
const size_t next_slash_after_glob_pos,
bool ignore_eacces_multidirectory_globs)
bool ignore_access_denied_multidirectory_globs)
{
/// We don't need to go all the way in every directory if max_depth is reached
/// as it is upper limit of depth by simply counting `/`s in curly braces
@ -104,8 +104,8 @@ namespace
{
// ignore file not found (as in LSWithRegexpMatching)
// keep throw other exception, libhdfs3 doesn't have function to get exception type, so use errno.
// ignore permission denied if ignore_eacces_multidirectory_globs is true
if (!(ignore_eacces_multidirectory_globs && errno == EACCES))
// ignore permission denied if ignore_access_denied_multidirectory_globs is true
if (!(ignore_access_denied_multidirectory_globs && errno == EACCES))
throw Exception(
ErrorCodes::ACCESS_DENIED, "Cannot list directory {}: {}", path_for_ls, String(hdfsGetLastError()));
}
@ -134,7 +134,7 @@ namespace
{
std::vector<StorageHDFS::PathWithInfo> result_part = LSWithRegexpMatching(
fs::path(full_path) / "" , fs, suffix_with_globs.substr(next_slash_after_glob_pos),
ignore_eacces_multidirectory_globs);
ignore_access_denied_multidirectory_globs);
std::move(result_part.begin(), result_part.end(), std::back_inserter(result));
}
}
@ -142,7 +142,7 @@ namespace
{
std::vector<StorageHDFS::PathWithInfo> result_part = LSWithFoldedRegexpMatching(
fs::path(full_path), fs, processed_suffix + dir_or_file_name, suffix_with_globs,
matcher, max_depth - 1, next_slash_after_glob_pos, ignore_eacces_multidirectory_globs);
matcher, max_depth - 1, next_slash_after_glob_pos, ignore_access_denied_multidirectory_globs);
std::move(result_part.begin(), result_part.end(), std::back_inserter(result));
}
}
@ -156,7 +156,7 @@ namespace
const String & path_for_ls,
const HDFSFSPtr & fs,
const String & for_match,
bool ignore_eacces_multidirectory_globs)
bool ignore_access_denied_multidirectory_globs)
{
const size_t first_glob_pos = for_match.find_first_of("*?{");
const bool has_glob = first_glob_pos != std::string::npos;
@ -199,7 +199,7 @@ namespace
if (slashes_in_glob)
{
return LSWithFoldedRegexpMatching(fs::path(prefix_without_globs), fs, "", suffix_with_globs, matcher,
slashes_in_glob, next_slash_after_glob_pos, ignore_eacces_multidirectory_globs);
slashes_in_glob, next_slash_after_glob_pos, ignore_access_denied_multidirectory_globs);
}
HDFSFileInfo ls;
@ -233,7 +233,7 @@ namespace
if (re2::RE2::FullMatch(file_name, matcher))
{
std::vector<StorageHDFS::PathWithInfo> result_part = LSWithRegexpMatching(fs::path(full_path) / "", fs,
suffix_with_globs.substr(next_slash_after_glob_pos), ignore_eacces_multidirectory_globs);
suffix_with_globs.substr(next_slash_after_glob_pos), ignore_access_denied_multidirectory_globs);
/// Recursion depth is limited by pattern. '*' works only for depth = 1, for depth = 2 pattern path is '*/*'. So we do not need additional check.
std::move(result_part.begin(), result_part.end(), std::back_inserter(result));
}
@ -261,7 +261,7 @@ namespace
HDFSBuilderWrapper builder = createHDFSBuilder(uri_without_path + "/", context->getGlobalContext()->getConfigRef());
HDFSFSPtr fs = createHDFSFS(builder.get());
auto res = LSWithRegexpMatching("/", fs, path_from_uri, context->getSettingsRef().ignore_eacces_multidirectory_globs);
auto res = LSWithRegexpMatching("/", fs, path_from_uri, context->getSettingsRef().ignore_access_denied_multidirectory_globs);
return res;
}
}

24
src/Storages/StorageFile.cpp
View File

@ -104,7 +104,7 @@ void listFilesWithRegexpMatchingImpl(
const std::string & path_for_ls,
const std::string & for_match,
size_t & total_bytes_to_read,
bool ignore_eacces_multidirectory_globs,
bool ignore_access_denied_multidirectory_globs,
std::vector<std::string> & result,
bool recursive = false);
@ -123,14 +123,14 @@ void listFilesWithFoldedRegexpMatchingImpl(const std::string & path_for_ls,
size_t & total_bytes_to_read,
const size_t max_depth,
const size_t next_slash_after_glob_pos,
bool ignore_eacces_multidirectory_globs,
bool ignore_access_denied_multidirectory_globs,
std::vector<std::string> & result)
{
if (!max_depth)
return;
const fs::directory_iterator end;
fs::directory_iterator it = ignore_eacces_multidirectory_globs
fs::directory_iterator it = ignore_access_denied_multidirectory_globs
? fs::directory_iterator(path_for_ls, fs::directory_options::skip_permission_denied)
: fs::directory_iterator(path_for_ls);
for (; it != end; ++it)
@ -150,7 +150,7 @@ void listFilesWithFoldedRegexpMatchingImpl(const std::string & path_for_ls,
{
listFilesWithRegexpMatchingImpl(fs::path(full_path) / "" ,
suffix_with_globs.substr(next_slash_after_glob_pos),
total_bytes_to_read, ignore_eacces_multidirectory_globs, result);
total_bytes_to_read, ignore_access_denied_multidirectory_globs, result);
}
}
else if (it->is_directory())
@ -158,7 +158,7 @@ void listFilesWithFoldedRegexpMatchingImpl(const std::string & path_for_ls,
listFilesWithFoldedRegexpMatchingImpl(fs::path(full_path), processed_suffix + dir_or_file_name,
suffix_with_globs, matcher, total_bytes_to_read,
max_depth - 1, next_slash_after_glob_pos,
ignore_eacces_multidirectory_globs, result);
ignore_access_denied_multidirectory_globs, result);
}
}
@ -171,7 +171,7 @@ void listFilesWithRegexpMatchingImpl(
const std::string & path_for_ls,
const std::string & for_match,
size_t & total_bytes_to_read,
bool ignore_eacces_multidirectory_globs,
bool ignore_access_denied_multidirectory_globs,
std::vector<std::string> & result,
bool recursive)
{
@ -231,7 +231,7 @@ void listFilesWithRegexpMatchingImpl(
{
listFilesWithFoldedRegexpMatchingImpl(fs::path(prefix_without_globs), "", suffix_with_globs, matcher,
total_bytes_to_read, slashes_in_glob, next_slash_after_glob_pos,
ignore_eacces_multidirectory_globs, result);
ignore_access_denied_multidirectory_globs, result);
return;
}
@ -257,12 +257,12 @@ void listFilesWithRegexpMatchingImpl(
{
listFilesWithRegexpMatchingImpl(fs::path(full_path).append(it->path().string()) / "",
looking_for_directory ? suffix_with_globs.substr(next_slash_after_glob_pos) : current_glob,
total_bytes_to_read, ignore_eacces_multidirectory_globs, result, recursive);
total_bytes_to_read, ignore_access_denied_multidirectory_globs, result, recursive);
}
else if (looking_for_directory && re2::RE2::FullMatch(file_name, matcher))
/// Recursion depth is limited by pattern. '*' works only for depth = 1, for depth = 2 pattern path is '*/*'. So we do not need additional check.
listFilesWithRegexpMatchingImpl(fs::path(full_path) / "", suffix_with_globs.substr(next_slash_after_glob_pos),
total_bytes_to_read, ignore_eacces_multidirectory_globs, result);
total_bytes_to_read, ignore_access_denied_multidirectory_globs, result);
}
}
}
@ -271,10 +271,10 @@ std::vector<std::string> listFilesWithRegexpMatching(
const std::string & path_for_ls,
const std::string & for_match,
size_t & total_bytes_to_read,
bool ignore_eacces_multidirectory_globs)
bool ignore_access_denied_multidirectory_globs)
{
std::vector<std::string> result;
listFilesWithRegexpMatchingImpl(path_for_ls, for_match, total_bytes_to_read, ignore_eacces_multidirectory_globs, result);
listFilesWithRegexpMatchingImpl(path_for_ls, for_match, total_bytes_to_read, ignore_access_denied_multidirectory_globs, result);
return result;
}
@ -439,7 +439,7 @@ Strings StorageFile::getPathsList(const String & table_path, const String & user
else
{
/// We list only non-directory files.
paths = listFilesWithRegexpMatching("/", path, total_bytes_to_read, context->getSettingsRef().ignore_eacces_multidirectory_globs);
paths = listFilesWithRegexpMatching("/", path, total_bytes_to_read, context->getSettingsRef().ignore_access_denied_multidirectory_globs);
can_be_directory = false;
}

2
tests/queries/0_stateless/02771_multidirectory_globs_storage_file.sh
View File

@ -33,6 +33,6 @@ ${CLICKHOUSE_CLIENT} --query "SELECT *, _file FROM file('${user_files_path}/${CL
# Add a directory to test against permission_denied
rm -rf ${user_files_path:?}/${CLICKHOUSE_TEST_UNIQUE_NAME:?}/dir_inaccessible/ && mkdir -p ${user_files_path}/${CLICKHOUSE_TEST_UNIQUE_NAME}/dir_inaccessible/ && chmod 000 ${user_files_path}/${CLICKHOUSE_TEST_UNIQUE_NAME}/dir_inaccessible/
${CLICKHOUSE_CLIENT} --query "SELECT *, _file FROM file('${user_files_path}/${CLICKHOUSE_TEST_UNIQUE_NAME}/dir{?/subdir?1/da,2/subdir2?/da}ta1.csv', CSV) SETTINGS ignore_eacces_multidirectory_globs = 1;"
${CLICKHOUSE_CLIENT} --query "SELECT *, _file FROM file('${user_files_path}/${CLICKHOUSE_TEST_UNIQUE_NAME}/dir{?/subdir?1/da,2/subdir2?/da}ta1.csv', CSV) SETTINGS ignore_access_denied_multidirectory_globs = 1;"
rm -rf ${user_files_path:?}/${CLICKHOUSE_TEST_UNIQUE_NAME:?}