thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-19 16:20:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Follow-up to "Implement support of encrypted elements in configuration file"

Browse Source 
Cf. PR #50986

- rename XML attribute "encryption_codec" to "encrypted_by"
This commit is contained in:
Robert Schulze 2023-07-26 11:19:20 +00:00
parent d4737ca033
commit aa25ce9e3d
No known key found for this signature in database
GPG Key ID: 26703B55FB13728A
13 changed files with 110 additions and 76 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

67
docs/en/operations/configuration-files.md
View File

@ -67,7 +67,7 @@ Substitutions can also be performed from ZooKeeper. To do this, specify the attr
## Encrypting Configuration {#encryption}
You can use symmetric encryption to encrypt a configuration element, for example, a password field. To do so, first configure the [encryption codec](../sql-reference/statements/create/table.md#encryption-codecs), then add attribute `encryption_codec` with the name of the encryption codec as value to the element to encrypt.
You can use symmetric encryption to encrypt a configuration element, for example, a password field. To do so, first configure the [encryption codec](../sql-reference/statements/create/table.md#encryption-codecs), then add attribute `encrypted_by` with the name of the encryption codec as value to the element to encrypt.
Unlike attributes `from_zk`, `from_env` and `incl` (or element `include`), no substitution, i.e. decryption of the encrypted value, is performed in the preprocessed file. Decryption happens only at runtime in the server process.
@ -75,19 +75,22 @@ Example:
```xml
<clickhouse>
<encryption_codecs>
<aes_128_gcm_siv>
<key_hex>00112233445566778899aabbccddeeff</key_hex>
</aes_128_gcm_siv>
</encryption_codecs>
<interserver_http_credentials>
<user>admin</user>
<password encryption_codec="AES_128_GCM_SIV">961F000000040000000000EEDDEF4F453CFE6457C4234BD7C09258BD651D85</password>
<password encrypted_by="AES_128_GCM_SIV">961F000000040000000000EEDDEF4F453CFE6457C4234BD7C09258BD651D85</password>
</interserver_http_credentials>
</clickhouse>
```
To get the encrypted value `encrypt_decrypt` example application may be used.
To encrypt a value, you can use the (example) program `encrypt_decrypt`:
Example:
@ -138,12 +141,17 @@ Here you can see default config written in YAML: [config.yaml.example](https://g
There are some differences between YAML and XML formats in terms of ClickHouse configurations. Here are some tips for writing a configuration in YAML format.
You should use a Scalar node to write a key-value pair:
An XML tag with a text value is represented by a YAML key-value pair
``` yaml
key: value
```
To create a node, containing other nodes you should use a Map:
Corresponding XML:
``` xml
<key>value</value>
```
A nested XML node is represented by a YAML map:
``` yaml
map_key:
key1: val1
@ -151,7 +159,16 @@ map_key:
key3: val3
```
To create a list of values or nodes assigned to one tag you should use a Sequence:
Corresponding XML:
``` xml
<map_key>
<key1>val1</key1>
<key2>val2</key2>
<key3>val3</key3>
</map_key>
```
To create the same XML tag multiple times, use a YAML sequence:
``` yaml
seq_key:
- val1
@ -162,8 +179,22 @@ seq_key:
key3: val5
```
If you want to write an attribute for a Sequence or Map node, you should use a @ prefix before the attribute key. Note, that @ is reserved by YAML standard, so you should also to wrap it into double quotes:
Corresponding XML:
```xml
<seq_key>val1</seq_key>
<seq_key>val2</seq_key>
<seq_key>
<key1>val3</key1>
</seq_key>
<seq_key>
<map>
<key2>val4</key2>
<key3>val5</key3>
</map>
</seq_key>
```
To provide an XML attribute, you can use an attribute key with a `@` prefix. Note that `@` is reserved by YAML standard, so must be wrapped in double quotes:
``` yaml
map:
"@attr1": value1
@ -171,16 +202,14 @@ map:
key: 123
```
From that Map we will get these XML nodes:
Corresponding XML:
``` xml
<map attr1="value1" attr2="value2">
<key>123</key>
</map>
```
You can also set attributes for Sequence:
It is also possible to use attributes in YAML sequence:
``` yaml
seq:
- "@attr1": value1
@ -189,13 +218,25 @@ seq:
- abc
```
So, we can get YAML config equal to this XML one:
Corresponding XML:
``` xml
<seq attr1="value1" attr2="value2">123</seq>
<seq attr1="value1" attr2="value2">abc</seq>
```
The aforementioned syntax does not allow to express XML text nodes with XML attributes as YAML. This special case can be achieved using an
`#text` attribute key:
```yaml
map_key:
"@attr1": value1
"#text": value2
```
Corresponding XML:
```xml
<map_key attr1="value1">value2</map>
```
## Implementation Details {#implementation-details}
For each config file, the server also generates `file-preprocessed.xml` files when starting. These files contain all the completed substitutions and overrides, and they are intended for informational use. If ZooKeeper substitutions were used in the config files but ZooKeeper is not available on the server start, the server loads the configuration from the preprocessed file.

7
docs/ru/operations/configuration-files.md
View File

@ -87,7 +87,7 @@ $ cat /etc/clickhouse-server/users.d/alice.xml
## Шифрование {#encryption}
Вы можете использовать симметричное шифрование для зашифровки элемента конфигурации, например, поля password. Чтобы это сделать, сначала настройте [кодек шифрования](../sql-reference/statements/create/table.md#encryption-codecs), затем добавьте аттибут`encryption_codec` с именем кодека шифрования как значение к элементу, который надо зашифровать.
Вы можете использовать симметричное шифрование для зашифровки элемента конфигурации, например, поля password. Чтобы это сделать, сначала настройте [кодек шифрования](../sql-reference/statements/create/table.md#encryption-codecs), затем добавьте аттибут`encrypted_by` с именем кодека шифрования как значение к элементу, который надо зашифровать.
В отличии от аттрибутов `from_zk`, `from_env` и `incl` (или элемента `include`), подстановка, т.е. расшифровка зашифрованного значения, не выподняется в файле предобработки. Расшифровка происходит только во время исполнения в серверном процессе.
@ -95,15 +95,18 @@ $ cat /etc/clickhouse-server/users.d/alice.xml
```xml
<clickhouse>
<encryption_codecs>
<aes_128_gcm_siv>
<key_hex>00112233445566778899aabbccddeeff</key_hex>
</aes_128_gcm_siv>
</encryption_codecs>
<interserver_http_credentials>
<user>admin</user>
<password encryption_codec="AES_128_GCM_SIV">961F000000040000000000EEDDEF4F453CFE6457C4234BD7C09258BD651D85</password>
<password encrypted_by="AES_128_GCM_SIV">961F000000040000000000EEDDEF4F453CFE6457C4234BD7C09258BD651D85</password>
</interserver_http_credentials>
</clickhouse>
```

20
src/Common/Config/ConfigProcessor.cpp
View File

@ -192,13 +192,13 @@ static void mergeAttributes(Element & config_element, Element & with_element)
std::string ConfigProcessor::encryptValue(const std::string & codec_name, const std::string & value)
{
EncryptionMethod method = getEncryptionMethod(codec_name);
CompressionCodecEncrypted codec(method);
EncryptionMethod encryption_method = toEncryptionMethod(codec_name);
CompressionCodecEncrypted codec(encryption_method);
Memory<> memory;
memory.resize(codec.getCompressedReserveSize(static_cast<UInt32>(value.size())));
auto bytes_written = codec.compress(value.data(), static_cast<UInt32>(value.size()), memory.data());
auto encrypted_value = std::string(memory.data(), bytes_written);
std::string encrypted_value(memory.data(), bytes_written);
std::string hex_value;
boost::algorithm::hex(encrypted_value.begin(), encrypted_value.end(), std::back_inserter(hex_value));
return hex_value;
@ -206,8 +206,8 @@ std::string ConfigProcessor::encryptValue(const std::string & codec_name, const
std::string ConfigProcessor::decryptValue(const std::string & codec_name, const std::string & value)
{
EncryptionMethod method = getEncryptionMethod(codec_name);
CompressionCodecEncrypted codec(method);
EncryptionMethod encryption_method = toEncryptionMethod(codec_name);
CompressionCodecEncrypted codec(encryption_method);
Memory<> memory;
std::string encrypted_value;
@ -223,7 +223,7 @@ std::string ConfigProcessor::decryptValue(const std::string & codec_name, const
memory.resize(codec.readDecompressedBlockSize(encrypted_value.data()));
codec.decompress(encrypted_value.data(), static_cast<UInt32>(encrypted_value.size()), memory.data());
std::string decrypted_value = std::string(memory.data(), memory.size());
std::string decrypted_value(memory.data(), memory.size());
return decrypted_value;
}
@ -234,7 +234,7 @@ void ConfigProcessor::decryptRecursive(Poco::XML::Node * config_root)
if (node->nodeType() == Node::ELEMENT_NODE)
{
Element & element = dynamic_cast<Element &>(*node);
if (element.hasAttribute("encryption_codec"))
if (element.hasAttribute("encrypted_by"))
{
const NodeListPtr children = element.childNodes();
if (children->length() != 1)
@ -244,8 +244,8 @@ void ConfigProcessor::decryptRecursive(Poco::XML::Node * config_root)
if (text_node->nodeType() != Node::TEXT_NODE)
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Encrypted node {} should have text node", node->nodeName());
auto encryption_codec = element.getAttribute("encryption_codec");
text_node->setNodeValue(decryptValue(encryption_codec, text_node->getNodeValue()));
auto encrypted_by = element.getAttribute("encrypted_by");
text_node->setNodeValue(decryptValue(encrypted_by, text_node->getNodeValue()));
}
decryptRecursive(node);
}
@ -775,7 +775,7 @@ ConfigProcessor::LoadedConfig ConfigProcessor::loadConfigWithZooKeeperIncludes(
void ConfigProcessor::decryptEncryptedElements(LoadedConfig & loaded_config)
{
CompressionCodecEncrypted::Configuration::instance().tryLoad(*loaded_config.configuration, "encryption_codecs");
CompressionCodecEncrypted::Configuration::instance().load(*loaded_config.configuration, "encryption_codecs");
Node * config_root = getRootNode(loaded_config.preprocessed_xml.get());
decryptRecursive(config_root);
loaded_config.configuration = new Poco::Util::XMLConfiguration(loaded_config.preprocessed_xml);

4
src/Common/examples/encrypt_decrypt.cpp
View File

@ -3,7 +3,7 @@
#include <Compression/CompressionCodecEncrypted.h>
#include <iostream>
/** This test program encrypts or decrypts text values using a symmetric encryption codec like AES_128_GCM_SIV or AES_256_GCM_SIV.
/** This program encrypts or decrypts text values using a symmetric encryption codec like AES_128_GCM_SIV or AES_256_GCM_SIV.
* Keys for codecs are loaded from <encryption_codecs> section of configuration file.
*
* How to use:
@ -32,7 +32,7 @@ int main(int argc, char ** argv)
DB::ConfigProcessor processor(argv[1], false, true);
auto loaded_config = processor.loadConfig();
DB::CompressionCodecEncrypted::Configuration::instance().tryLoad(*loaded_config.configuration, "encryption_codecs");
DB::CompressionCodecEncrypted::Configuration::instance().load(*loaded_config.configuration, "encryption_codecs");
if (action == "-e")
std::cout << processor.encryptValue(codec_name, value) << std::endl;

42
src/Compression/CompressionCodecEncrypted.cpp
View File

@ -31,14 +31,14 @@ namespace ErrorCodes
extern const int BAD_ARGUMENTS;
}
EncryptionMethod getEncryptionMethod(const std::string & name)
EncryptionMethod toEncryptionMethod(const std::string & name)
{
if (name == "AES_128_GCM_SIV")
return AES_128_GCM_SIV;
else if (name == "AES_256_GCM_SIV")
return AES_256_GCM_SIV;
else
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Wrong encryption method. Got {}", name);
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown encryption method. Got {}", name);
}
namespace
@ -48,34 +48,22 @@ namespace
String getMethodName(EncryptionMethod Method)
{
if (Method == AES_128_GCM_SIV)
{
return "AES_128_GCM_SIV";
}
else if (Method == AES_256_GCM_SIV)
{
return "AES_256_GCM_SIV";
}
else
{
return "";
}
}
/// Get method code (used for codec, to understand which one we are using)
uint8_t getMethodCode(EncryptionMethod Method)
{
if (Method == AES_128_GCM_SIV)
{
return static_cast<uint8_t>(CompressionMethodByte::AES_128_GCM_SIV);
}
else if (Method == AES_256_GCM_SIV)
{
return static_cast<uint8_t>(CompressionMethodByte::AES_256_GCM_SIV);
}
else
{
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Wrong encryption method. Got {}", getMethodName(Method));
}
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown encryption method. Got {}", getMethodName(Method));
}
} // end of namespace
@ -105,17 +93,11 @@ const String empty_nonce = {"\0\0\0\0\0\0\0\0\0\0\0\0", actual_nonce_size};
UInt64 methodKeySize(EncryptionMethod Method)
{
if (Method == AES_128_GCM_SIV)
{
return 16;
}
else if (Method == AES_256_GCM_SIV)
{
return 32;
}
else
{
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Wrong encryption method. Got {}", getMethodName(Method));
}
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown encryption method. Got {}", getMethodName(Method));
}
std::string lastErrorString()
@ -130,17 +112,11 @@ std::string lastErrorString()
auto getMethod(EncryptionMethod Method)
{
if (Method == AES_128_GCM_SIV)
{
return EVP_aead_aes_128_gcm_siv;
}
else if (Method == AES_256_GCM_SIV)
{
return EVP_aead_aes_256_gcm_siv;
}
else
{
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Wrong encryption method. Got {}", getMethodName(Method));
}
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown encryption method. Got {}", getMethodName(Method));
}
/// Encrypt plaintext with particular algorithm and put result into ciphertext_and_tag.
@ -206,17 +182,11 @@ size_t decrypt(std::string_view ciphertext, char * plaintext, EncryptionMethod m
auto getMethod(EncryptionMethod Method)
{
if (Method == AES_128_GCM_SIV)
{
return EVP_aes_128_gcm;
}
else if (Method == AES_256_GCM_SIV)
{
return EVP_aes_256_gcm;
}
else
{
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Wrong encryption method. Got {}", getMethodName(Method));
}
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown encryption method. Got {}", getMethodName(Method));
}
/// Encrypt plaintext with particular algorithm and put result into ciphertext_and_tag.

4
src/Compression/CompressionCodecEncrypted.h
View File

@ -18,8 +18,8 @@ enum EncryptionMethod
MAX_ENCRYPTION_METHOD
};
/// Get method for string name. Throw exception for wrong name.
EncryptionMethod getEncryptionMethod(const std::string & name);
/// Get encryption method for string name. Throw exception for wrong name.
EncryptionMethod toEncryptionMethod(const std::string & name);
/** This codec encrypts and decrypts blocks with AES-128 in
* GCM-SIV mode (RFC-8452), which is the only cipher currently

7
tests/integration/test_config_decryption/configs/config.xml
View File

@ -1,4 +1,5 @@
<clickhouse>
<encryption_codecs>
<aes_128_gcm_siv>
<key_hex>00112233445566778899aabbccddeeff</key_hex>
@ -7,6 +8,8 @@
<key_hex>00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff</key_hex>
</aes_256_gcm_siv>
</encryption_codecs>
<max_table_size_to_drop encryption_codec="AES_128_GCM_SIV">96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</max_table_size_to_drop>
<max_partition_size_to_drop encryption_codec="AES_256_GCM_SIV">97260000000B0000000000BFFF70C4DA718754C1DA0E2F25FF9246D4783F7FFEC4089EC1CC14</max_partition_size_to_drop>
<max_table_size_to_drop encrypted_by="AES_128_GCM_SIV">96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</max_table_size_to_drop>
<max_partition_size_to_drop encrypted_by="AES_256_GCM_SIV">97260000000B0000000000BFFF70C4DA718754C1DA0E2F25FF9246D4783F7FFEC4089EC1CC14</max_partition_size_to_drop>
</clickhouse>

6
tests/integration/test_config_decryption/configs/config.yaml
View File

@ -3,9 +3,11 @@ encryption_codecs:
key_hex: 00112233445566778899aabbccddeeff
aes_256_gcm_siv:
key_hex: 00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff
max_table_size_to_drop:
'#text': 96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C
'@encryption_codec': AES_128_GCM_SIV
'@encrypted_by': AES_128_GCM_SIV
max_partition_size_to_drop:
'@encryption_codec': AES_256_GCM_SIV
'@encrypted_by': AES_256_GCM_SIV
'#text': 97260000000B0000000000BFFF70C4DA718754C1DA0E2F25FF9246D4783F7FFEC4089EC1CC14

8
tests/integration/test_config_decryption/configs/config_invalid_chars.xml
View File

@ -1,4 +1,5 @@
<clickhouse>
<encryption_codecs>
<aes_128_gcm_siv>
<key_hex>00112233445566778899aabbccddeeff</key_hex>
@ -7,6 +8,9 @@
<key_hex>00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff</key_hex>
</aes_256_gcm_siv>
</encryption_codecs>
<max_table_size_to_drop encryption_codec="AES_128_GCM_SIV">--96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</max_table_size_to_drop>
<max_partition_size_to_drop encryption_codec="AES_256_GCM_SIV">97260000000B0000000000BFFF70C4DA718754C1DA0E2F25FF9246D4783F7FFEC4089EC1CC14</max_partition_size_to_drop>
<!-- Dash prefix leads to invalid hex-encoding -->
<max_table_size_to_drop encrypted_by="AES_128_GCM_SIV">--96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</max_table_size_to_drop>
<max_partition_size_to_drop encrypted_by="AES_256_GCM_SIV">97260000000B0000000000BFFF70C4DA718754C1DA0E2F25FF9246D4783F7FFEC4089EC1CC14</max_partition_size_to_drop>
</clickhouse>

6
tests/integration/test_config_decryption/configs/config_no_encryption_key.xml
View File

@ -1,3 +1,7 @@
<clickhouse>
<max_table_size_to_drop encryption_codec="AES_128_GCM_SIV">96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</max_table_size_to_drop>
<!-- section "encryption_codec" is not specified -->
<max_table_size_to_drop encrypted_by="AES_128_GCM_SIV">96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</max_table_size_to_drop>
</clickhouse>

6
tests/integration/test_config_decryption/configs/config_subnodes.xml
View File

@ -1,10 +1,14 @@
<clickhouse>
<encryption_codecs>
<aes_128_gcm_siv>
<key_hex>00112233445566778899aabbccddeeff</key_hex>
</aes_128_gcm_siv>
</encryption_codecs>
<interserver_http_credentials encryption_codec="AES_128_GCM_SIV">
<!-- tags with "encrypted_by" must not have nested tags -->
<interserver_http_credentials encrypted_by="AES_128_GCM_SIV">
<password>96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</password>
</interserver_http_credentials>
</clickhouse>

7
tests/integration/test_config_decryption/configs/config_wrong_method.xml
View File

@ -1,4 +1,5 @@
<clickhouse>
<encryption_codecs>
<aes_128_gcm_siv>
<key_hex>00112233445566778899aabbccddeeff</key_hex>
@ -7,6 +8,8 @@
<key_hex>00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff</key_hex>
</aes_256_gcm_siv>
</encryption_codecs>
<max_table_size_to_drop encryption_codec="AES_128_GCM_SIV">96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</max_table_size_to_drop>
<max_partition_size_to_drop encryption_codec="WRONG">97260000000B0000000000BFFF70C4DA718754C1DA0E2F25FF9246D4783F7FFEC4089EC1CC14</max_partition_size_to_drop>
<max_table_size_to_drop encrypted_by="AES_128_GCM_SIV">96260000000B0000000000E8FE3C087CED2205A5071078B29FD5C3B97F824911DED3217E980C</max_table_size_to_drop>
<max_partition_size_to_drop encrypted_by="WRONG">97260000000B0000000000BFFF70C4DA718754C1DA0E2F25FF9246D4783F7FFEC4089EC1CC14</max_partition_size_to_drop>
</clickhouse>

2
tests/integration/test_config_decryption/test_wrong_settings.py
View File

@ -15,7 +15,7 @@ def start_clickhouse(config, err_msg):
def test_wrong_method():
start_clickhouse(
"configs/config_wrong_method.xml", "Wrong encryption method. Got WRONG"
"configs/config_wrong_method.xml", "Unknown encryption method. Got WRONG"
)