Commit Graph

479 Commits

Author SHA1 Message Date
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs
Poco::File to std::filesystem
2021-05-31 23:17:02 +03:00
Denis Glazachev
0d2c239938 Allow recursive execution paths 2021-05-24 00:07:52 +04:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
Vitaliy Zakaznikov
03ebd18d01 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into ldap-user-dn-detection 2021-05-18 14:08:28 -04:00
Arslan G
dde9e30043 Merge branch 'master' into add-sqlite-support 2021-05-17 14:38:24 +03:00
Arslan G
ea51840f85 Init SQLite integration 2021-05-17 14:02:35 +03:00
Vitaly Baranov
1dc415596c Fix race condition in ContextAccess. 2021-05-17 10:03:26 +03:00
kssenii
0527f0ea33 Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-12 16:54:18 +03:00
kssenii
393b026f2e Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-11 16:49:23 +03:00
Amos Bird
264cff6415
Projections
TODO (suggested by Nikolai)

1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies

3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)

Also need to figure out how prewhere works for projections, and
row_filter_policies.

wip
2021-05-11 18:12:23 +08:00
Alexey Milovidov
037d06bb2a Fix warnings by PVS-Studio 2021-05-08 23:50:12 +03:00
Alexey Milovidov
604daa9581 Fix a bunch of warnings from PVS-Studio 2021-05-08 19:09:17 +03:00
Alexey Milovidov
2fbecf36ca More fixes for PVS-Studio 2021-05-08 18:20:40 +03:00
kssenii
02288359c5 Less manual concatenation of paths 2021-05-08 13:59:55 +03:00
Alexey Milovidov
1fda866881 Merge branch 'master' into normalize-bigint 2021-05-02 21:24:45 +03:00
Vladimir
3b7c68196b
Merge pull request #23429 from Jokser/disk-s3-restart-possibility 2021-05-01 15:28:14 +03:00
Maksim Kita
318c4bb80d Add examples folder filter to ya.make.in 2021-04-30 11:25:52 +03:00
Alexey Milovidov
8f01af62d9 Merge branch 'master' into normalize-bigint 2021-04-25 06:57:44 +03:00
Pavel Kovalenko
10cf881a08 Add SYSTEM RESTART DISK command. 2021-04-20 22:23:54 +03:00
Denis Glazachev
902d519b40 Merge branch 'master' into ldap-user-dn-detection
* master: (694 commits)
  Fix integration test test_storage_kafka failed error
  Fix test 00163_column_oriented_formats failed error
  Read ORC file by stripe to reduce memory cost
  Function toDateTime decimal overflow ubsan fix
  Revert "[RFC] Fix memory tracking with min_bytes_to_use_mmap_io"
  Zlib use attribute constructor for functable initialization
  Translate to Russian (clickhouse-client documentation)
  Simple key dictionary primary key wrong order fix
  Disable hedged requests
  Added integration test
  Revert "Function `arrayFold` for folding over array with accumulator"
  Fix documentation for the GRANT command.
  Added system query reload model
  Make function `unhex` case insensitive for compatibility
  Improve documentation for CREATE ROW POLICY command #2.
  Add exception message
  Fix tidy
  Fix waiting for all connections closed on shutdown.
  Disable postgresql_port in perf tests
  Mark 01605_adaptive_granularity_block_borders as long
  ...
2021-04-19 18:36:07 +04:00
Maksim Kita
abf2a87ce7 Added system query reload model 2021-04-17 20:59:08 +03:00
songenjie
f731739fab fix suggests 2021-04-14 10:05:41 +08:00
songenjie
564136ec46 [clickhouse][server][dll][alter]support fetch part 2021-04-13 12:40:33 +08:00
Denis Glazachev
9a5032585f Merge branch 'master' into ldap-user-dn-detection
* master: (860 commits)
  Update version_date.tsv after release 21.2.8.31
  Update version_date.tsv after release 21.3.5.42
  Fixed typos
  Add metric to track how much time is spend during waiting for Buffer layer lock
  Safer SCOPE_EXIT
  Add SCOPE_EXIT_SAFE/SCOPE_EXIT_MEMORY_SAFE helpers
  Lock MEMORY_LIMIT_EXCEEDED in ThreadStatus::detachQuery()
  Update CHANGELOG.md
  Reset timeouts to default
  Add Third party service info
  Disable table function view in expression (#21465)
  fix test 01702_system_query_log
  Remove strange fsync on coordination logs rotation
  add test
  MemoryStorage sync comments and code
  Fix typos
  Support alter setting
  Handle not plain where tree in StorageMerge modifySelect
  Updated test
  Change Aggregatingmergetree to AggregatingMergeTree in docs
  ...
2021-04-07 16:27:52 +04:00
Denis Glazachev
a9e5532da6 Fix builds: soothe the linters 2021-03-29 12:27:16 +04:00
Denis Glazachev
0762e1a890 Implement config parsing and actual support for user_dn_detection section and user_dn placeholder substitution
Refactor some config parsing code
Rename some arguments to better reflect their meaning
Add documentation for user_dn_detection section and user_dn placeholder in config.xml and in docs
2021-03-29 02:23:20 +04:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
Pavel Kovalenko
39f3b17041 Merge remote-tracking branch 'origin/master' into unfreeze-partitions 2021-03-16 10:56:24 +03:00
Vitaly Baranov
37ce6e26d3 Add a new clause GRANTEES to commands CREATE USER and ALTER USER. 2021-03-14 19:12:42 +03:00
Vitaly Baranov
d6e0342c30 Improvements in implementations of the classes AccessRights and GrantedRoles. 2021-03-14 19:12:35 +03:00
Denis Glazachev
290a6d273e
Add Kerberos support for authenticating existing users when accessing over HTTP 2021-03-11 23:41:10 +03:00
Pavel Kovalenko
078dfce038 Add ALTER TABLE UNFREEZE command. 2021-02-24 17:28:30 +03:00
tavplubix
103d569885
Merge branch 'master' into fix_force_drop_materialized_view 2021-02-20 20:41:03 +03:00
Alexander Tokmakov
5cfd687dfb fix 2021-02-19 19:22:47 +03:00
Nikita Mikhaylov
5f8a6ab9c1 remove probably useless code 2021-02-16 23:54:05 +03:00
Vitaly Baranov
d8d2bd885c Fix calculation of interval's end in quota consumption. 2021-02-16 23:54:05 +03:00
Azat Khuzhin
06e8065ee6 Add missing sync of underlying files 2021-02-13 13:19:51 +03:00
JackyWoo
77c0f0a0e7 add quota type QUERY_SELECTS and QUERY_INSERTS 2021-02-07 10:57:18 +08:00
Alexey Milovidov
093108bf81 Normalize BigInt implementation 2021-01-27 03:54:57 +03:00
Denis Glazachev
30ab2830e0 Merge branch 'master' into ldap-role-mapping
* master: (605 commits)
  DOCSUP-4710: Added support numeric parameters in number and string data types (#18696)
  DOCSUP-5604: Edit and translate to Russian (#18929)
  Update version_date.tsv after release 21.1.2.15
  Usability improvement of clickhouse-test
  Update jit_large_requests.xml
  Update README.md
  Update images.json
  Make symbolizers available in fuzzer Docker image
  Update Dragonbox
  Speed up aggregate function sum
  Fix MSan report in Kerberos library
  Fix MSan error in rocksdb #19213
  Add more Fuzzer tasks
  Fixes
  Update comment for curl dependency for aws
  Disable curl for mariadb-connector-c (it is not required)
  Fix TSan
  Skip test for ANTLR
  DistributedBlockOutputStream: add more comments
  DistributedBlockOutputStream: Remove superfluous brackets for string construction
  ...
2021-01-18 22:55:05 +04:00
alexey-milovidov
72b142a00a
Merge branch 'master' into pg2ch 2021-01-06 23:18:59 +03:00
Denis Glazachev
8893fbcf8e Rename {username} to {user_name}
Add caching/checking of search_params
Adjust comments/doc
Use special authentication logic from ExternalAuthenticators::checkLDAPCredentials
2021-01-06 07:40:47 +04:00
Denis Glazachev
c8cf51b81e Merge branch 'master' into ldap-role-mapping
* master: (620 commits)
  Add test for some possible ambiguities in syntax
  Update PushingToViewsBlockOutputStream.h
  [For #18707] MySQL compatibility: support DIV and MOD operators
  Mark another flaky test
  Remove some headers
  Mark some TestFlows as flaky
  Fix error
  Fix errors
  One more test
  Arcadia does not support distributed queries
  Add a test for #14974
  Added a test from #15641
  More robust stateful test
  Update tests
  Remove bad code in HashJoin
  Update test
  Don't allow conversion between UUID and numeric types
  Remove pink screen with confusing questions about Kerberos
  Do not throw from Parser
  Fix the unexpected behaviour of show tables when antlr parser enabled (#18431)
  ...

# Conflicts:
#	programs/server/config.xml
#	src/Access/Authentication.cpp
#	src/Access/Authentication.h
2021-01-06 03:42:02 +04:00
alexey-milovidov
12ad7c31df
Merge pull request #18486 from ClickHouse/better_18167
Better version of #18167
2020-12-31 15:20:03 +03:00
Denis Glazachev
0c7b151800 Revisit mapped role management 2020-12-28 00:54:24 +04:00
kssenii
2f6cb7f2f5 Add storage PostgreSQL with read support 2020-12-27 12:15:57 +00:00
Denis Glazachev
07aa3fe30d Refine the caching 2020-12-25 03:46:08 +04:00
Denis Glazachev
b1e46ccef8 Remove unneeded include 2020-12-25 01:57:13 +04:00
Denis Glazachev
3ce0731630 Move caching and LDAP credential verification code to ExternalAuthenticators 2020-12-25 01:49:19 +04:00
Alexander Tokmakov
21c3fc0e16 check settings constraints in setProfile(...) 2020-12-24 20:48:54 +03:00
Vitaliy Zakaznikov
c12695ceed Merge branch 'master' of https://github.com/traceon/ClickHouse into ldap-role-mapping 2020-12-23 09:38:08 -05:00
Vitaliy Zakaznikov
26ca04c92d Merge branch 'master' of https://github.com/traceon/ClickHouse into ldap-cache-login 2020-12-23 08:17:12 -05:00
Alexey Milovidov
9c1516bd74 Slightly better for gcc-9 2020-12-21 03:41:22 +03:00
Alexey Milovidov
31b955e14a Slightly better for gcc-9 2020-12-21 03:38:53 +03:00
Denis Glazachev
c4b85f2dcd Simplify the code 2020-12-17 18:48:12 +04:00
Denis Glazachev
53db7e564c Do transformations based on prefix only 2020-12-17 18:29:05 +04:00
Vitaly Baranov
d7a3cc8c90 Don't show the "_temporary_and_external_tables" database in system tables
system.databases, system.tables, system.columns.
2020-12-17 11:42:25 +03:00
Alexander Tokmakov
76e73e4e8b fix 'Unknown setting profile' error 2020-12-16 23:49:56 +03:00
Vitaly Baranov
a98e2311f8
Merge pull request #17908 from vitlibar/fix-checking-introspection-grants
Fix checking introspection grants
2020-12-14 12:10:17 +03:00
Vitaly Baranov
1b3893bcab User with allow_ddl=0 cannot do DDL but can grant DDL. 2020-12-11 16:38:49 +03:00
Vitaly Baranov
648be453a4 User with allow_introspection_functions=0 cannot call introspection functions but can grant INTROSPECTION. 2020-12-11 16:38:49 +03:00
Vitaly Baranov
710ba6f617 Simplify class ContextAccess. 2020-12-11 16:38:45 +03:00
alexey-milovidov
d9b52f94f7
Merge pull request #17637 from amosbird/buildid
reload symbols and fix build-id
2020-12-08 14:55:35 +03:00
Alexey Milovidov
2c7b03ab6e Fix error 2020-12-02 22:20:47 +03:00
Alexey Milovidov
c9aa412151 Allow quotas to be keyed by proxy-forwarded IP address 2020-12-02 00:09:16 +03:00
Amos Bird
310918b06a
reload symbols and fix build-id 2020-11-30 22:30:55 +08:00
Denis Glazachev
d12f59388a Compilation fix 2020-11-24 22:48:15 +04:00
Denis Glazachev
47be319dea Refactor the cached ldap info locations
Add cached value access synchronization
2020-11-24 20:17:58 +04:00
Denis Glazachev
03b3a93a15 Compilation fix 2020-11-22 11:17:01 +04:00
Denis Glazachev
1a587b0c21 Merge branch 'master' into ldap-role-mapping
* master: (159 commits)
  Review fix.
  Update version_date.tsv after release 20.8.7.15
  wrong translation
  Update version_date.tsv after release 20.9.6.14
  Update version_date.tsv after release 20.10.5.10
  Update version_date.tsv after release 20.11.4.13
  Improvements in coverage images
  Fixed a problem with the translation of the document
  final_parallel
  final_parallel
  DOCSUP-4162: Document the system.replicated_fetches system table (#16900)
  Update settings.md
  Update settings.md
  Less verbose logging when fetch is impossible
  Don't add tons of client coverage files in stateful tests with coverage
  More compatible watches in TestKeeper
  Trying to make read_in_order_many_parts more stable
  trigger CI
  Update version_date.tsv after release 20.6.10.2
  Update visibleWidth.cpp
  ...
2020-11-22 00:58:55 +04:00
Denis Glazachev
e9a3a97cb9 Improve regex instance creation code 2020-11-22 00:44:54 +04:00
Denis Glazachev
fb481649ec Change naming
Add serialization of new params in getStorageParamsJSON()
2020-11-21 19:08:02 +04:00
Denis Glazachev
78acf226db Revert user directory name change 2020-11-21 18:08:40 +04:00
Tai White
43fce34eec Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into ldap-cache-login 2020-11-20 20:56:36 +01:00
Denis Glazachev
a0a50c1eb6 Change some config parameters to handle placeholders 2020-11-20 23:31:21 +04:00
Denis Glazachev
facdd225aa Add regex syntax checks 2020-11-20 20:59:56 +04:00
Denis Glazachev
be184272d8 Compilation fixes 2020-11-20 02:26:52 +04:00
Denis Glazachev
279853b16a WIP: Implement group extraction and role mapping 2020-11-20 02:02:18 +04:00
Alexey Milovidov
24f4fa6edf Follow Arcadia ya.make rules 2020-11-17 00:16:50 +03:00
Alexey Milovidov
3df04ce0c2 Follow Arcadia ya.make rules 2020-11-16 21:24:58 +03:00
Denis Glazachev
a35088d681 Add ldap_ prefix to var names 2020-11-10 00:20:34 +04:00
Denis Glazachev
0e704ec278
Merge branch 'master' into ldap-cache-login 2020-11-10 00:10:22 +04:00
Alexander Tokmakov
5cdfcfb307 remove other stringstreams 2020-11-09 22:12:44 +03:00
Alexander Tokmakov
62ff00ee8b use WriteBuffer in formatAST(...) 2020-11-09 19:05:40 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Alexey Milovidov
b56486510f Improve Arcadia 2020-11-05 12:55:01 +03:00
Vitaliy Zakaznikov
e6d04b4780 Merge branch 'master' of https://github.com/traceon/ClickHouse into ldap-cache-login 2020-11-03 16:53:23 -05:00
Maxim Akhmedov
3627fabfb9 Remove -g0 form Arcadia build settings. 2020-10-29 17:37:23 +03:00
Alexander Kuzmenkov
ba34145817
Merge pull request #16425 from filimonov/minumum
Fix typos reported by codespell
2020-10-28 08:07:18 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell 2020-10-27 12:04:03 +01:00
Alexey Milovidov
32ed8c9681 Fix trailing whitespace 2020-10-26 22:12:40 +03:00
Denis Glazachev
1eb7c31011 Add "verification_cooldown enabled" check 2020-10-26 16:44:36 +04:00
Denis Glazachev
f2a6696362 Implement verification_cooldown LDAP server connection param 2020-10-26 16:44:36 +04:00
alexey-milovidov
6cd7d46f93
Update AccessControlManager.cpp 2020-10-23 09:13:26 +03:00
Denis Glazachev
a6439aba44 More specific log messages for each access storage type 2020-10-22 20:47:52 +04:00
alexey-milovidov
f351b52851
Update AccessControlManager.cpp 2020-10-22 07:03:17 +03:00
Denis Glazachev
691b28e98b Add a log message after a storage is added 2020-10-22 02:43:02 +04:00
Vitaly Baranov
235a493a25
Merge pull request #12736 from traceon/ldap-any-user-authentication
Add LDAP user directory support for locally non-existent users
2020-10-19 23:58:35 +03:00
Denis Glazachev
bf14cb58e2 Use count() 2020-10-14 20:54:09 +04:00
Denis Glazachev
f26b7573a2 Fix pre-C++20 compiler builds 2020-10-14 16:58:54 +04:00
Denis Glazachev
e3ef8bbc48 GCC 9/10 compilation fix 2020-10-14 05:29:25 +04:00
Vitaly Baranov
eddd26cf3a Fix deadlock in InterpreterGrantQuery. 2020-10-13 01:03:47 +03:00
Vitaly Baranov
b8019f2bc7 Fix deadlocks in RoleCache. 2020-10-12 21:40:05 +03:00
Denis Glazachev
e348ec17b2 Refactor role handling 2020-10-09 00:57:23 +04:00
Denis Glazachev
555f056a4c Revert "user not found" detection in loginImpl() and getIDOfLoggedUserImpl() 2020-10-06 20:32:06 +04:00
Denis Glazachev
54446eeec6 Use ErrorCodes::UNKNOWN_USER 2020-10-06 20:00:29 +04:00
Denis Glazachev
fed6080273 Implement custom getIDOfLoggedUserImpl() 2020-10-06 19:54:22 +04:00
Denis Glazachev
e2f444ae85 Simplify loginImpl() and getIDOfLoggedUserImpl() 2020-10-06 19:37:35 +04:00
Denis Glazachev
950a07835f Stylistic changes 2020-10-06 19:23:08 +04:00
Denis Glazachev
2fc6a4ea9c Add log_and_mask_exceptions flag to login() 2020-10-05 00:24:09 +04:00
Denis Glazachev
1eb8ecf050 Fix compilation 2020-10-04 23:56:25 +04:00
Denis Glazachev
7f47719768 Refactor exception handling in login() et al.
Simplify LDAPClient and LDAPAccessStorage
2020-10-04 23:55:58 +04:00
Denis Glazachev
00a354cd37 Manually remove storages in reverse order in MultipleAccessStorage d-tor 2020-10-04 22:00:56 +04:00
Denis Glazachev
82475088f9 Fix "user has been dropped" issue 2020-10-03 17:31:02 +04:00
Denis Glazachev
68ccd59a74 Synch with internal memory_storage
Fix exception message
2020-10-03 00:32:13 +04:00
Denis Glazachev
ab2c37cead Serialize all calls to ldap lib 2020-10-03 00:31:14 +04:00
Denis Glazachev
8f2c23e6de Merge branch 'master' into ldap-any-user-authentication
* master: (224 commits)
  Update in.md
  Add a test for embedded configs
  Allow to run without /proc/self/maps
  Update adopters.md
  More convenient
  Enable embedded configs for AArch64
  More results
  More handy
  More handy
  Added a comment
  Adjustments
  Adjustments
  Adjustments
  Added new results
  Step 1: make adding hardware benchmark results more convenient
  Revert "Avoid deadlocks in Log/TinyLog"
  Fix MSan report in QueryLog
  add some disabled tests from arcadia to skip_list.json
  Update skip_list.json
  Bump CI. [2]
  ...

# Conflicts:
#	docker/test/testflows/runner/Dockerfile
2020-09-25 17:33:50 +04:00
Vitaly Baranov
0605808ed1
Merge pull request #14492 from vitlibar/use-previous-users-xml-if-failed-to-reload
Use previous users.xml if failed to reload
2020-09-24 16:25:19 +03:00
Denis Glazachev
2c6b6673f2 Remove extra declaration 2020-09-21 02:57:34 +04:00
Denis Glazachev
03481f7a3a Synchronize the code with respect to IAccessStorage::login() functionality 2020-09-21 02:51:38 +04:00
Denis Glazachev
442b1407cf Merge branch 'master' into ldap-any-user-authentication
* master: (375 commits)
  Update type-conversion-functions.md
  Update maxmap.md
  Update maxmap.md
  Update maxmap.md
  Update single_fixed_string_groupby.xml
  Alter remove column properties and TTLs (#14742)
  better fixed string group by support
  Fix incorrect key condition of fixed strings.
  constant output order
  more tests for  #14646
  Maybe fix MSan report in base64
  Proper exception message for wrong number of arguments of CAST
  Added a test
  Fix buffer overflow in "bar" function
  Update convertMySQLDataType.cpp
  Fix clang-tidy
  Remove obsolete code from performance test
  Slightly better code
  Even more
  Even more
  ...

# Conflicts:
#	src/Interpreters/Context.cpp
2020-09-21 00:31:04 +04:00
Vitaly Baranov
46ab2f76cd
Merge pull request #14919 from vitlibar/add-login-function-to-access-storage
Add login() function to IAccessStorage
2020-09-20 16:58:22 +03:00
Vitaly Baranov
33efb36714 Use previous users.xml if failed to reload. 2020-09-19 16:21:12 +03:00
Vitaly Baranov
dbc837c148 IAccessStorage now can override the login() function. 2020-09-19 01:04:20 +03:00
Vitaly Baranov
74d5b43fe6
Merge pull request #14828 from vitlibar/change-columns-of-system-user-directories
Add 'params' column to system.user_directories table.
2020-09-15 18:25:50 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build (#14809) 2020-09-15 12:55:57 +03:00
Vitaly Baranov
0f53b449fd Add 'params' column to system.user_directories table. 2020-09-15 01:51:53 +03:00
Denis Glazachev
c202364f01
Merge branch 'master' into ldap-any-user-authentication 2020-09-14 18:53:46 +04:00
alexey-milovidov
c2f762e20a
Merge pull request #14748 from vitlibar/skip-access-storages-with-same-path
Skip access storages with same path while reading the main config.
2020-09-12 14:20:02 +03:00
Vitaly Baranov
5e3260678c
Merge pull request #14561 from vitlibar/no-detach-threads
Use join() instead of detach() for threads
2020-09-12 02:04:08 +03:00
Vitaly Baranov
e2c2a679ef Skip access storages with same path while reading the main config. 2020-09-12 01:34:46 +03:00
Vitaly Baranov
bee629c971 Use join() instead of detach() for the lists_writing_thread in DiskAccessStorage. 2020-09-09 03:19:13 +03:00
Alexey Milovidov
e3924b8057 Fix "Arcadia" 2020-09-08 01:14:13 +03:00
Denis Glazachev
59e0c10aaf Merge branch 'master' into ldap-any-user-authentication 2020-09-03 19:36:19 +04:00
Vitaly Baranov
dda884d6d0
Update AccessControlManager.h
Add "override" for the destructor of AccessControlManager
2020-09-03 17:20:14 +03:00
alexey-milovidov
bfa5a9ef03
Merge pull request #13987 from vitlibar/fix-grant-all-on-table
Fix GRANT ALL statement when executed on a non-global level
2020-09-02 05:06:32 +03:00
alexey-milovidov
04c88ca9e4
Update AccessFlags.h 2020-09-02 05:06:21 +03:00
alexey-milovidov
5a529f171c
Merge pull request #13988 from vitlibar/user-directories-mixed-style
Add storages from <user_directories> after ones from <users_config> and <access_control_path>.
2020-09-02 05:03:34 +03:00
Denis Glazachev
9b4cd06051 Merge branch 'master' into ldap-any-user-authentication
* master: (414 commits)
  Add .reference
  Update 01460_mark_inclusion_search_crash.sql
  Fix bug in mark inclusion search.
  cosmetic fixes for performance report
  cosmetic fixes in performance report
  Add more docs about functional tests
  Add ability to specify Default codec for columns (#14049)
  better
  done
  Disable force TTL on optimise
  Update CMakeLists.txt
  Update custom parts of storage AST only if it has extended definition
  DOCSUP-712: Documented the ttl_only_drop_parts setting (#13823)
  Small fixes
  gix comment and useDefault*(), add tests for nullables
  Fix a build for old some OS with old find
  Update hdfs.md
  Fix 01085_max_distributed_connections flackiness
  Fix 00974_distributed_join_on flackiness (by allow retries and hide logs)
  Update adopters.md
  ...
2020-08-30 11:56:38 +04:00
Denis Glazachev
7ffb618f6e Add missing proper findOrGenerateImpl() implementation to MultipleAccessStorage class 2020-08-28 16:05:08 +04:00
Denis Glazachev
f91d57adac Adjust naming 2020-08-28 12:06:06 +04:00
Amos Bird
078b14610d
ALTER MODIFY SAMPLE BY 2020-08-27 22:31:30 +08:00
Denis Glazachev
c72765187b GCC 9 compilation fix 2020-08-27 12:36:31 +04:00
Denis Glazachev
3d6e56cd61 Maintain the list and update role changes in cached users 2020-08-27 00:34:33 +04:00
Denis Glazachev
1768db2c2f Add findOrGenerate() to IAccessStorage interface (falls back to find() by default)
Use findOrGenerate() when setting user in Context (authentication path)
Implement findOrGenerateImpl() and findImpl() in LDAPAccessStorage
2020-08-26 22:09:26 +04:00
Vitaly Baranov
7ac4bd7d1e Add storages from <user_directories> after ones from <users_config> and <access_control_path>. 2020-08-26 13:45:35 +03:00
Denis Glazachev
1cc9b81cdd Disallow multiple ldap sections in user_directories section 2020-08-24 16:02:47 +04:00
Vitaly Baranov
2a96151516 Fix GRANT ALL statement when executed on a non-global level. 2020-08-22 01:59:52 +03:00
Denis Glazachev
ec52a165af Style fixes 2020-08-20 12:46:42 +04:00
Denis Glazachev
cbc9285bd6 Merge branch 'master' into ldap-any-user-authentication
* master: (43 commits)
  Publish list of tests that failed the concurrent fast test
  fix test
  Use gnu++2a instead of c++2a for unbundled build to fix numeric_limits<__int128>
  fix test
  Update 01453_fixsed_string_sort.sql
  Added test.
  Fix ColumnString::updatePermutationWithCollation.
  Add support for extended precision integers and decimals (#13097)
  remove retries
  Check that ya.make files are auto-generated
  Fix "Arcadia"
  ISSUES-4006 trigger CI again
  fixed typo arrayCompact
  Stratify nans comparison in arrayCompact function
  Remove even more useless code
  Remove useless code around zkutil
  Fix 00956_sensitive_data_masking flackiness
  Update docker/test/stress/run.sh
  add exclusive DDLGuard for database
  Fix handling embedded config.
  ...
2020-08-20 11:41:43 +04:00
Denis Glazachev
bdfea652c1 Change user_template to roles
Change top_enclosing_storage to access_control_manager
Simplify the lookup in peer storages
2020-08-20 11:39:27 +04:00
Vitaly Baranov
c800941ab7 Fix handling embedded config. 2020-08-18 18:08:50 +03:00
Denis Glazachev
58f73ff041 Merge branch 'master' into ldap-any-user-authentication
* master: (30 commits)
  Documentation improve:  Translate [select] section into Chinese to improve readability (#13814)
  Update adopters.md
  partially disable test with MaterializeMySQL
  one more unroll for arerage.cpp
  Update adopters.md
  Update adopters.md
  Update adopters.md
  Update adopters.md
  fix #13819
  DOCSUP-1888: Documented the input_format_avro_allow_missing_fields setting (#13671)
  Fix "Arcadia" and "Unbundled" builds
  DOCSUP-928: Documented the groupArraySample function (#13791)
  Documentation about ReplacingMergeTree extended with type DateTime64 for column (#13498)
  doc: update quotas.md (#13400)
  Fix 01356_initialize_aggregation in unbundled build (change topKWeighted order)
  Fix style
  Add normalizedQueryHash function with tests
  Fix topK/topKWeighted merge (wtih non-default parameters)
  Add test
  Add function "normalizeQuery"
  ...

# Conflicts:
#	programs/server/Server.cpp
#	src/Access/AccessControlManager.cpp
#	src/Access/AccessControlManager.h
2020-08-18 14:54:02 +04:00
Vitaly Baranov
29a6558d33 Add system table system.user_directories 2020-08-16 19:15:39 +03:00
Vitaly Baranov
0759dff12b Support <user_directories> section in the main config. 2020-08-16 19:15:38 +03:00
Vitaly Baranov
2909ed1bc0 Better initialization of access storages. Make list of access storages dynamic. 2020-08-16 19:15:34 +03:00
Vitaly Baranov
ad03ff3887 Rename storages users.xml=>users_xml, disk=>local_directory. 2020-08-16 16:42:57 +03:00
Vitaly Baranov
a77b262444 Fix typo. 2020-08-16 16:42:57 +03:00
Vitaly Baranov
25d463f257 Fix renaming in MemoryAccessStorage. 2020-08-16 16:42:57 +03:00
Vitaly Baranov
d1e193f02e Check name of inserted entities in precedent storages. 2020-08-16 16:42:57 +03:00
Vitaly Baranov
35158f8bfe Prefer users from users.xml in case of duplication. 2020-08-16 16:42:57 +03:00
Vitaly Baranov
0caf592941 Remove exception about duplicates when multiple access storages keeps entities with the same name. 2020-08-16 16:42:57 +03:00
Denis Glazachev
5b0524f9dc Allow empty 'user_template' (defaults to user name 'default')
Do not show 'missing template user' error message to client, log at server side
2020-08-15 16:17:07 +04:00
Denis Glazachev
7375dc5d66 Merge branch 'master' into ldap-any-user-authentication
* master: (956 commits)
  Remove ZooKeeper from unit tests
  Revert check location of Docker compose files @qoega.
  Update developer-instruction.md
  trigger the CI
  Update settings.md
  Update settings.md
  Fix
  fix sync 2
  Fix tests.
  Add docker for style check
  fix sync 1
  passwd and group location error
  Hotfix for pushdown with StorageMerge (#13679)
  Fix error with batch aggregation and -Array combinator
  Fix cassandra build on macos
  Update adopters.md
  Update adopters.md
  Fix build after merge
  Fix shared build
  log error message
  ...

# Conflicts:
#	tests/testflows/ldap/docker-compose/clickhouse-service.yml
2020-08-15 14:18:40 +04:00
Alexey Milovidov
edd89a8610 Fix half of typos 2020-08-08 03:47:03 +03:00
Vitaly Baranov
8d6e04835c
Merge pull request #13199 from vitlibar/fix-reading-row-policies-from-users-xml
Fix parsing row policies from users.xml
2020-08-07 23:32:28 +03:00
Vitaly Baranov
3778b7db90 Fix reading row policies from users.xml when names of databases or tables contain dots. 2020-08-04 20:48:57 +03:00
Vitaly Baranov
dadebadcac Print correct error message in log for unknown settings in users.xml 2020-08-04 00:20:33 +03:00
Alexey Milovidov
6f690b7c0d Normalize ya.make files, fix "Arcadia" build 2020-08-02 16:57:38 +03:00
Vitaly Baranov
7c4ae5ee65 Add the parameter custom_settings_prefixes to the server config. 2020-07-31 20:57:49 +03:00
Vitaly Baranov
442f3de5a8 Implement custom settings. 2020-07-31 20:57:43 +03:00
Vitaly Baranov
56665a15f7 Rework and rename the template class SettingsCollection => BaseSettings. 2020-07-31 20:54:18 +03:00
Vitaly Baranov
e40a8ac176 Use names of the settings instead of their indices in SettingsConstraints and settings profiles. 2020-07-31 19:11:27 +03:00
Vitaly Baranov
300727afa3 Rework the StringField* classes and make conversion String => StringField* explicit. 2020-07-31 19:11:27 +03:00
Vitaly Baranov
18e3f1f60d Split SettingsCollection.h into 3 files: SettingsFields.h, SettingsEnums.h, SettingsCollection.h 2020-07-31 19:11:27 +03:00
Vitaly Baranov
90602b869a Make SettingsChanges a class. 2020-07-31 19:11:27 +03:00
Denis Glazachev
fc557eebe4 Merge branch 'master' into ldap-any-user-authentication
* master: (219 commits)
  Fix link in the documentation (#12769)
  add mapAdd and mapSubtract functions (#11735)
  DOCSUP-1916 split reference.md into separate files (#12963)
  DOCSUP-790: docs for mutations_sync setting (#12639)
  Adding extra xfails for some ldap tests.
  Fix JSON
  Additions
  Added instruction
  Add benchmark instructions
  Add benchmark results for OmniSci
  [docs] maybe fix build
  Fix terrible unbearable data rot. CC @blinkov
  Block structure mismatch is a program error
  Merging #12548 - Correction to `merge_with_ttl_timeout` logic by @excitoon (#12982)
  Update entrypoint.sh
  Added test.
  Fix crash in ColumnTuple::updatePermutation
  DOCS-590: parallel_distributed_insert_select (#12980)
  Added test.
  Fix header for totals and extremes in QueryPipeline::unitePipelines.
  ...
2020-07-29 20:23:19 +04:00
Vitaly Baranov
f440953b87
Merge pull request #12646 from vitlibar/fix-create-user-if-not-exists
CREATE USER IF NOT EXISTS now doesn't throw exception if the user exists.
2020-07-28 17:43:01 +03:00
Denis Glazachev
6c1643d3c5 Remove unneeded logging 2020-07-24 14:11:00 +04:00
Denis Glazachev
79332c561e Fix local variable naming 2020-07-24 13:52:03 +04:00
Denis Glazachev
3b3404c326 Style fix
Remove unused declarations
2020-07-23 22:10:57 +04:00
Denis Glazachev
5d6b5101fe Implement LDAPAccessStorage and integrate it into AccessControlManager
Rename ExternalAuthenticators::setConfig to setConfiguration
Revisit LDAP servers config section comments
Add user_directories config section with comments (only for ldap)
Fix bug in MemoryAccessStorage::insertImpl
2020-07-23 21:55:24 +04:00
Vitaly Baranov
0093425201 CREATE USER IF NOT EXISTS now doesn't throw exception if the user exists. 2020-07-22 16:41:42 +03:00
alesapin
a3262cceee Better exception during directories creation 2020-07-21 14:21:13 +03:00
Denis Glazachev
b68d7b6c24
Merge branch 'master' into ldap-per-user-authentication 2020-07-11 21:37:52 +04:00
Denis Glazachev
2a3a0d47b3 Style fix 2020-07-11 21:31:00 +04:00
Denis Glazachev
3e68368b59 Refactor ExternalAuthenticators configuration process 2020-07-11 21:06:01 +04:00
Denis Glazachev
af98e74afd Gracefully handle the case when ExternalAuthenticators instance is not created (yet) 2020-07-11 02:42:48 +04:00
Vitaly Baranov
30e3d61b01 Fix calculating implicit access rights. 2020-07-10 17:16:43 +03:00
Denis Glazachev
9effacfbc1 Merge branch 'master' into ldap-per-user-authentication
* master: (1102 commits)
  Update README.md
  Update README.md
  Update README.md
  Update index.md
  [docs] add intrdocution for statements page (#12189)
  Revert "Run perf tests with memory sampling (for allocations >1M)"
  Sanitize LINK_LIBRARIES property for the directories (#12160)
  [docs] refactor Domains overview (#12186)
  DOCS-647: toStartOfSecond (#12190)
  [docs] add intrdocution for commercial page (#12187)
  DOCSUP-1348 Russian translation for new functions (#133) (#12194)
  changelog fixes
  Update index.md (#12191)
  Update zh kafka.md title (#12192)
  Added test for #3767
  style fix for #12152
  Tests for fixed issues #10846 and #7347
  changelog fixes
  [docs] introduction for special table engines (#12170)
  [docs] introduction for third-party interfaces (#12175)
  ...

# Conflicts:
#	src/Access/ya.make
#	src/Common/ErrorCodes.cpp
2020-07-08 00:42:09 +04:00
Vitaly Baranov
03b36c262e Improve REVOKE command: now it requires only grant/admin option for only
access which will be revoked.
REVOKE ALL FROM user1 now revokes all granted roles.
2020-07-02 12:54:24 +03:00
Vitaly Baranov
c39eb8f71b Fix partial revokes (complex cases). 2020-06-30 18:47:02 +03:00
Vitaly Baranov
eb27814fbe Fix access rights: cannot grant INTROSPECTION when allow_introspection_functions=0. 2020-06-29 16:43:31 +03:00
Vitaly Baranov
f3f005d5b9
Merge pull request #12015 from vitlibar/fix-access-rights-allow-ddl-0
Fix calculation of access rights when allow_ddl = 0
2020-06-29 15:14:22 +03:00
Vitaly Baranov
bd72bd6e10 Fix access rights: cannot grant DDL when allow_ddl=0 2020-06-28 21:38:14 +03:00
alesapin
11f88340a5 Merge branch 'mongo' of https://github.com/ageraab/ClickHouse into storage_mongodb 2020-06-26 16:03:06 +03:00
sundy-li
0a4af8f0a7 add SYSTEM DROP REPLICA 2020-06-23 12:12:30 +08:00
Denis Glazachev
5db60202b6 Merge branch 'master' into ldap-per-user-authentication 2020-06-19 00:11:08 +04:00
Denis Glazachev
7317acb609 Silently reject empty passwords. Empty user names are rejected verbosely. 2020-06-18 23:33:59 +04:00
Vitaly Baranov
9fe47df2e8 Support multiple users/roles in SHOW CREATE USER(ROLE, etc.) and SHOW GRANTS FOR commands.
Support syntax "SHOW CREATE USER ALL" and "SHOW GRANTS FOR ALL".
2020-06-15 22:07:47 +03:00
Vitaly Baranov
4bd00b02e2 Improve syntax of CREATE QUOTA. Now resource types and key types could be written with underscores.
Also rename columns key_type=>keys and source=>storage in table system.quotas.
2020-06-15 20:10:34 +03:00
Vitaly Baranov
7d1951a79b Improve messages for errors in access storages. 2020-06-15 20:10:34 +03:00
Vitaly Baranov
92b9f4a88d Rename ExtendedRoleSet => RolesOrUsersSet. 2020-06-15 20:10:34 +03:00
Vitaly Baranov
9f31184d76 Support for multiple names in one CREATE/ALTER command. 2020-06-15 20:10:28 +03:00
Vitaly Baranov
3ffcb8e790 Fix casting values of settings while reading profiles from users.xml. 2020-06-15 01:44:25 +03:00
Vitaly Baranov
ca2fb59321 Fix calculating full names of row policies. 2020-06-15 01:44:25 +03:00
Denis Glazachev
276fcd8903 Add/rename parameters that control TLS 2020-06-12 21:59:47 +04:00
Denis Glazachev
04f222f85b Tell OpenLDAP too create a new SSL/TLS context for each connection 2020-06-12 16:48:00 +04:00
Denis Glazachev
9e3a28a6b8 Merge branch 'master' into ldap-per-user-authentication
* master: (414 commits)
  Update file.md
  Update merge.md
  Update dictionary.md
  Update external-data.md
  Update distributed.md
  Update null.md
  Update set.md
  Update join.md
  Update url.md
  Update view.md
  Update materializedview.md
  Update memory.md
  Update buffer.md
  Update generate.md
  removed a sentence about global lock during rename (#11577)
  greatCircleAngle en translation (#11584)
  Update configuration-files.md
  try fix flacky test
  Update why.html
  Update rich.html
  ...

# Conflicts:
#	src/Common/ErrorCodes.cpp
#	utils/ci/jobs/quick-build/run.sh
2020-06-11 03:06:17 +04:00
Denis Glazachev
848330b37a Expect <ldap_servers> in main config.xml 2020-06-11 02:48:15 +04:00
tavplubix
686c0539db
Merge pull request #11487 from ClickHouse/improve_dns_cache
Use DNSResolver to check if user is allowed to connect from an address
2020-06-07 19:55:03 +03:00
Alexander Tokmakov
0cf8015f4a use DNSResolver to check if user is allowed to connect 2020-06-07 00:02:29 +03:00
Denis Glazachev
0197627f3f Added checks for empty server name 2020-06-03 22:52:12 +04:00
Alexander Kuzmenkov
1c33918f07 style 2020-06-03 17:17:41 +03:00
Alexander Kuzmenkov
07e4bb7050 Remove assorted synonyms of LOGICAL_ERROR.
We don't need any special handling for them on the client, and, on the
contrary, have to handle them as logical errors in tests.
2020-06-03 16:18:42 +03:00
Denis Glazachev
b28def8f4c Stylistic changes 2020-06-03 15:20:53 +04:00
Denis Glazachev
48f3d4094a Remove irrelevant stuff 2020-06-03 01:06:44 +04:00
Denis Glazachev
c427524bc8 Simplefy ExternalAuthenticators exposure to isCorrectPassword() 2020-06-03 01:02:31 +04:00
Denis Glazachev
b7caa154e2 Add ARCADIA_BUILD check 2020-06-03 00:37:14 +04:00
Denis Glazachev
6ff0550e4e Move parseExternalAuthenticators functionality into the c-tor of ExternalAuthenticators 2020-06-02 13:37:02 +04:00
Denis Glazachev
c61cbe8e98 Merge branch 'master' into ldap-per-user-authentication
* master: (114 commits)
  Update PushingToViewsBlockOutputStream.cpp
  Update PushingToViewsBlockOutputStream.cpp
  make clang-10 happy
  Fix sync_async test (remove timeout)
  CLICKHOUSEDOCS-631: temporary_files_codec, join_on_disk_max_files_to_merge settings. (#11242)
  Suppress output of cancelled queries in clickhouse-client #9473
  Better log messages in ConfigReloader
  fix select from StorageJoin
  Fix unit tests under MSan
  Added test.
  Fix build.
  Fix arguments for AggregateFunctionQuantile/
  Update style.md
  Add a guide on error messages.
  Report dictionary name on dictionary load errors.
  more types in ASOF JOIN (#11301)
  Fix part_log test
  Update test.
  Add perftest.
  Parallel processing for PushingToViewsBlockOutputStream::writeSuffix
  ...
2020-06-02 12:42:29 +04:00
Denis Glazachev
e8144976fe Merge branch 'master' into ldap-per-user-authentication 2020-05-31 15:09:16 +04:00
Denis Glazachev
165dc4e109 Disable system static OpenLDAP linking support (due to fPIC mismatch) 2020-05-31 15:04:56 +04:00
Alexey Milovidov
25f941020b Remove namespace pollution 2020-05-31 00:57:37 +03:00
Denis Glazachev
246900c1ac Compilation fix 2020-05-29 18:33:50 +04:00
alexey-milovidov
304c6a1ee3
Merge pull request #11278 from vitlibar/fix-crash-set-default-role-with-wrong-args
Fix crash when SET DEFAULT ROLE is called with wrong arguments.
2020-05-29 16:24:42 +03:00
Denis Glazachev
9fb0a95c75 Compilation fix: add missing "/include"
Style fix
2020-05-29 16:14:42 +04:00
Denis Glazachev
d9ca9cd9b2 Compilation fix
Typo fix
2020-05-29 14:00:12 +04:00
Denis Glazachev
f1cfc7b472 Rename LDAP_PASSWORD to LDAP_SERVER and use "ldap_server" as a string key
Some refactoring
2020-05-29 11:47:01 +04:00
Vitaly Baranov
2d12b4d3ac Fix crash when SET DEFAULT ROLE is called with wrong arguments. 2020-05-29 09:54:27 +03:00
Denis Glazachev
f9d4136792 Merge branch 'master' into ldap-per-user-authentication 2020-05-29 10:52:44 +04:00
Vitaly Baranov
d7cc703233
Merge pull request #11080 from vitlibar/add-authentication-type-to-system-users
Show authentication type in system.users table
2020-05-28 20:20:41 +03:00
Vitaly Baranov
98172deffc Show authentication type in table system.users and while executing SHOW CREATE USER query. 2020-05-28 08:34:10 +03:00
Denis Glazachev
d4fd018715 Fix linking errors in parser-related test executables 2020-05-28 02:54:14 +04:00
Denis Glazachev
2863de750e Merge branch 'master' into ldap-per-user-authentication 2020-05-28 01:30:52 +04:00
Denis Glazachev
d74f1357d4 Add LDAP authentication support 2020-05-28 01:06:33 +04:00
Maxim Akhmedov
e09bcb4290 Make possible adding nested storages to MultipleAccessStorage in run-time. 2020-05-26 17:05:06 +03:00
Vitaly Baranov
2c8a355f19
Merge pull request #11081 from vitlibar/fix-no-password-mode
Fix settings NO_PASSWORD authentication mode in users.xml.
2020-05-26 14:20:34 +03:00
Vitaly Baranov
eeb4cbc433 Fix settings NO_PASSWORD authentication mode in users.xml. 2020-05-25 11:35:26 +03:00
Alexey Milovidov
7e1813825b Return old names of macros 2020-05-24 01:24:01 +03:00
Alexey Milovidov
9d24908e53 Progress on task 2020-05-23 20:52:11 +03:00
Alexey Milovidov
241f8c5431 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" \+ [^+]+ \+ "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" \+ ([^+]+) \+ "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/' 2020-05-23 20:10:21 +03:00
Alexey Milovidov
f69cbdcbfc find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" \+ [^+]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" \+ ([^+]+)\);/\1_FORMATTED(\2, "\3{}", \4);/' 2020-05-23 20:09:37 +03:00
Alexey Milovidov
f68d1ceb4f find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}\5{}\7{}", \4, \6, \8);/' 2020-05-23 20:02:09 +03:00
Alexey Milovidov
8042e5febe find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}\5{}", \4, \6);/' 2020-05-23 19:58:15 +03:00
Alexey Milovidov
8d2e80a5e2 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+"\)' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+, "[^"]+")\)/\1_FORMATTED(\2)/' 2020-05-23 19:42:39 +03:00
ageraab
cb24d47472
Merge branch 'master' into mongo 2020-05-16 02:56:22 +03:00
bobrovskij artemij
95677432e5 MongoDB engine (read-only) 2020-05-14 23:59:03 +03:00
Vitaly Baranov
bf2f38881d Fix compilation. 2020-05-14 14:12:20 +03:00
Vitaly Baranov
c30587196a Add system tables for users, roles and grants. 2020-05-14 14:12:20 +03:00