Vitaly Baranov
41d8621d97
Fix replacing dictionaries: StorageID is now updated immediately for every dictionary during CREATE OR REPLACE
...
even if the dictionary is not loaded yet or loading right now.
2024-03-17 12:10:52 +01:00
Maksim Kita
2a327107b6
Updated implementation
2024-01-25 14:31:49 +03:00
Alexey Milovidov
1910434174
Remove CurrentStatusInfo
2023-08-30 20:48:26 +02:00
Azat Khuzhin
0586a27432
Charge only server memory for dictionaries
...
Right now the memory will be counted for query/user for dictionary, but
only if it load by user (via SYSTEM RELOAD QUERY or via dictGet()), but
it could be also loaded in backgrounad (due to lifetime, or
update_field, so it is like Buffer, only server memory should be
charged.
v2: mark test as long
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
2023-05-21 22:53:52 +02:00
Sema Checherinda
4dd86a406a
Merge pull request #48543 from azat/mv-uniq-thread-group
...
Use one ThreadGroup while pushing to materialized views (and some refactoring for ThreadGroup)
2023-04-11 11:47:46 +02:00
Azat Khuzhin
79b83c4fd2
Remove superfluous includes of logger_userful.h from headers
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-10 17:59:30 +02:00
Azat Khuzhin
5b2b20a0b0
Rename ThreadGroupStatus to ThreadGroup
...
There are methods like getThreadGroup() and ThreadGroupSwitcher class,
so seems that this is logical.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-07 15:31:48 +02:00
Sema Checherinda
3c6deddd1d
work with comments on PR
2023-03-16 19:55:58 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
filimonov
0762783fbd
Fix incorrect trace log line on dict reload
2022-10-24 11:58:37 +02:00
Azat Khuzhin
2d7de7f683
Remove dictionaries from prometheus metrics on DETACH/DROP
...
Fixes : #23436 (cc @kitaisreal)
Introduced-in: #9622 (cc @YiuRULE)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-06 12:58:07 +03:00
Amos Bird
4a5e4274f0
base should not depend on Common
2022-04-29 10:26:35 +08:00
Maksim Kita
b1a956c5f1
clang-tidy check performance-move-const-arg fix
2022-03-02 18:15:27 +00:00
Azat Khuzhin
162f96f8e1
Get back detachQueryIfNotDetached() into ExternalLoader
...
ExternalLoader from ThreadPool (async loading) is done from the server
context, not from the query context, and the context will already go
away, so we should detachQueryIfNotDetached() to avoid trigger assertion
in ThreadStatus.
CI: https://s3.amazonaws.com/clickhouse-test-reports/34001/8cace291d17fa9553a98b2a1e8bf15b30fe5a1bd/stateless_tests__debug__actions__[3/3].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 16:25:33 +03:00
Azat Khuzhin
b0c862c297
Fix memory accounting for queries that uses < max_untracker_memory
...
MemoryTracker starts accounting memory directly only after per-thread
allocation exceeded max_untracker_memory (or memory_profiler_step).
But even memory under this limit should be accounted too, and there is
code to do this in ThreadStatus dtor, however due to
PullingAsyncPipelineExecutor detached the query from thread group that
memory was not accounted.
So remove CurrentThread::detachQueryIfNotDetached() from threads that
uses ThreadFromGlobalPool since it has ThreadStatus, and the query will
be detached using CurrentThread::defaultThreadDeleter.
Note, that before this patch memory accounting works for HTTP queries
due to it had been accounted from ParallelFormattingOutputFormat, but
not for TCP.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 16:25:33 +03:00
Vitaly Baranov
a140f4127e
Fix ExternalLoader without additional mutex.
2021-10-13 10:02:42 +03:00
Azat Khuzhin
2bab572caf
Fix lock-order-inversion between periodic dictionary reload and config reload
...
Integration tests found [1], TSan report:
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=11)
Cycle in lock order graph: M3152 (0x7b9000000058) => M3153 (0x7b9000000438) => M3152
Mutex M3153 acquired here while holding mutex M3152 in main thread:
3 std::__1::lock_guard<std::__1::recursive_mutex>::lock_guard(std::__1::recursive_mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x15716b81)
4 DB::ExternalLoader::addConfigRepository() const obj-x86_64-linux-gnu/../src/Interpreters/ExternalLoader.cpp:1283:21 (clickhouse+0x15716b81)
5 DB::Context::loadOrReloadDictionaries() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1453:62 (clickhouse+0x155d7bad)
6 DB::Server::main()::$_1::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:852:29 (clickhouse+0x9b1944c)
13 DB::ConfigReloader::reloadIfNewer() obj-x86_64-linux-gnu/../src/Common/Config/ConfigReloader.cpp:137:13 (clickhouse+0x17045e2e)
14 DB::ConfigReloader::ConfigReloader() obj-x86_64-linux-gnu/../src/Common/Config/ConfigReloader.cpp:33:9 (clickhouse+0x17044e51)
16 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:803:33 (clickhouse+0x9b0c41d)
17 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse+0x19ffc08b)
18 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:405:25 (clickhouse+0x9b03ebe)
19 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse+0x1a01c246)
20 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:183:20 (clickhouse+0x9b02576)
21 main obj-x86_64-linux-gnu/../programs/main.cpp:372:12 (clickhouse+0x9b00a3a)
Mutex M3152 acquired here while holding mutex M3153 in thread T2:
3 std::__1::lock_guard<std::__1::mutex>::lock_guard() obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x155d63b8)
4 DB::Context::getExternalDictionariesLoader() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1337:21 (clickhouse+0x155d63b8)
5 DB::Context::getExternalDictionariesLoader() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1332:41 (clickhouse+0x155d6359)
6 DB::DatabaseDictionary::tryGetTable() const obj-x86_64-inux-gnu/../src/Databases/DatabaseDictionary.cpp:76:38 (clickhouse+0x157819ad)
7 DB::DatabaseCatalog::getTableImpl() const obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:285:28 (clickhouse+0x1564a1fa)
8 DB::DatabaseCatalog::getTable() const obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:656:16 (clickhouse+0x1564fa2a)
9 DB::JoinedTables::getLeftTableStorage() obj-x86_64-linux-gnu/../src/Interpreters/JoinedTables.cpp:219:40 (clickhouse+0x15eeef45)
10 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:321:33 (clickhouse+0x15b792be)
19 DB::ClickHouseDictionarySource::doInvalidateQuery() const obj-x86_64-linux-gnu/../src/Dictionaries/ClickHouseDictionarySource.cpp:207:36 (clickhouse+0x12872d2d)
20 DB::ClickHouseDictionarySource::isModified() const obj-x86_64-linux-gnu/../src/Dictionaries/ClickHouseDictionarySource.cpp:144:25 (clickhouse+0x12872534)
21 DB::IDictionary::isModified() const <null> (clickhouse+0x128ce39b)
22 DB::ExternalLoader::LoadingDispatcher::reloadOutdated() obj-x86_64-linux-gnu/../src/Interpreters/ExternalLoader.cpp:660:50 (clickhouse+0x157305f7)
23 DB::ExternalLoader::PeriodicUpdater::doPeriodicUpdates() obj-x86_64-linux-gnu/../src/Interpreters/ExternalLoader.cpp:1248:36 (clickhouse+0x1572fff7)
[1]: https://clickhouse-test-reports.s3.yandex.net/29856/42ca2b4bb241827edf69bbd6938d6b19c31935f1/integration_tests_(thread).html#fail1
2021-10-10 04:04:05 +03:00
Alexey Milovidov
fe6b7c77c7
Rename "common" to "base"
2021-10-02 10:13:14 +03:00
Alexander Tokmakov
d82a794e8c
fix logical race on dictionaries loading
2021-09-10 12:54:22 +03:00
Maksim Kita
67e9b85951
Merge ext into common
2021-06-16 23:28:41 +03:00
Azat Khuzhin
4c47d91c94
Attach background thread for dictionary reload to the reload query
2021-05-09 14:30:31 +03:00
Vitaly Baranov
46f4c60839
Small simplification in ExternalLoader.
2021-03-23 15:15:44 +03:00
Vitaly Baranov
090e558da4
Merge pull request #21844 from Avogar/catboost-deadlock
...
Fix deadlock in first catboost model execution
2021-03-23 15:01:04 +03:00
Maksim Kita
a65bfaeab3
Fixed tests
2021-03-18 17:03:22 +03:00
Pavel Kruglov
bf888b7e5f
Fix deadlock in first catboost model execution
2021-03-17 21:57:40 +03:00
Nikolai Kochetov
c7236a9359
Try fix tests.
2020-10-27 19:12:53 +03:00
Nikolai Kochetov
71298ba496
Move fix to ExternalLoader::checkLoaded.
2020-10-27 17:21:51 +03:00
Vitaly Baranov
cce970e40c
Use join() instead of detach() for loading threads in ExternalLoader.
2020-09-09 03:19:13 +03:00
Alexander Tokmakov
f6949b2f47
use dictionary ids
2020-07-15 22:25:31 +03:00
Alexey Milovidov
25f941020b
Remove namespace pollution
2020-05-31 00:57:37 +03:00
Alexey Milovidov
7e1813825b
Return old names of macros
2020-05-24 01:24:01 +03:00
Alexey Milovidov
af2a59e49b
Progress on task
2020-05-24 00:50:34 +03:00
Alexey Milovidov
d853bc39c5
Apply all transformations again
2020-05-24 00:41:48 +03:00
Alexey Milovidov
c87fb91532
Progress on task
2020-05-24 00:41:35 +03:00
Alexey Milovidov
93626701b8
Apply all transformations again
2020-05-24 00:27:27 +03:00
Alexey Milovidov
ce0619dabf
Progress on task
2020-05-24 00:26:45 +03:00
Alexey Milovidov
cd6d3c7509
Progress on task
2020-05-23 22:55:06 +03:00
Alexey Milovidov
d5b7883451
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_[^\_(]+\([^,]+, "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_[^\_(]+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5{}\7{}\9", \4, \6, \8);/'
2020-05-23 22:41:08 +03:00
Alexey Milovidov
29762240de
Remove duplicate whitespaces (preparation)
2020-05-23 22:31:54 +03:00
Alexey Milovidov
7fed65cbe2
Remove duplicate whitespaces (preparation)
2020-05-23 22:14:58 +03:00
Alexey Milovidov
9d2a0d2dd7
Apply all transformations again
2020-05-23 21:59:49 +03:00
Alexey Milovidov
a2ad11897f
Remove duplicate whitespaces (preparation)
2020-05-23 21:53:58 +03:00
Alexey Milovidov
1f13515a65
Make all LOG in single line (preparation)
2020-05-23 21:31:37 +03:00
Alexey Milovidov
241f8c5431
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" \+ [^+]+ \+ "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" \+ ([^+]+) \+ "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/'
2020-05-23 20:10:21 +03:00
Alexey Milovidov
e391b77d81
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/'
2020-05-23 19:56:05 +03:00
Vitaly Baranov
dd34cd73e2
Move initialization of DatabaseWithDictionaries to constructor.
2020-04-30 20:22:20 +03:00
Alexey Milovidov
0a7edce036
Checkpoint
2020-04-22 09:01:33 +03:00
Vitaly Baranov
d62345bbaa
Show information about not loaded dictionaries in system.columns, system.tables
...
and in "SHOW TABLES" executed in any database with the "Dictionary" engine.
2020-04-18 12:50:54 +03:00
Vitaly Baranov
d8b76232fe
Remove useless logging from ExternalLoader.
2020-04-07 21:39:34 +03:00
Ivan Lezhankin
06446b4f08
dbms/ → src/
2020-04-03 18:14:31 +03:00