thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-04 05:22:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
146,365 Commits 9,182 Branches 1,782 Tags 2.3 GiB
8a202d91ad
Commit Graph

1360 Commits

Author SHA1 Message Date
Antonio Andelic
ad2196155c Add test for system node modification 2022-09-02 08:14:06 +00:00
Antonio Andelic
b7eebfc626 Correctly check if the node is using system path 2022-09-02 07:47:12 +00:00
Antonio Andelic
7164e59b54 Merge branch 'master' into keeper-linearizable-reads 2022-09-01 08:52:31 +00:00
Antonio Andelic
fa4c7259f9 Fix typos 2022-09-01 08:46:28 +00:00
Antonio Andelic
c7d1dd942c Address PR comments 2022-09-01 07:56:06 +00:00
Antonio Andelic
21cdb9a500 Don't remove from cache if it wasn't added during rollback 2022-08-30 08:14:53 +00:00
Antonio Andelic
2416ddfae7 Add extra message about startup preprocessing 2022-08-30 08:14:31 +00:00
Antonio Andelic
57471d11db Fix uncommitted auth 2022-08-29 12:55:30 +00:00
Antonio Andelic
50af0b1938 Add bunch of comments for request processing 2022-08-29 10:50:18 +00:00
Antonio Andelic
b4af21b6fb Small fixes 2022-08-29 07:06:51 +00:00
Antonio Andelic
bff2b232ef Cache deltas for path 2022-08-26 14:03:38 +00:00
Antonio Andelic
26d68dce23 Fix style 2022-08-26 10:55:07 +00:00
Antonio Andelic
dae65178b7 Improve preprocessing performance 2022-08-26 10:46:14 +00:00
Antonio Andelic
5d45f2faa9 Add comments 2022-08-23 08:22:03 +00:00
Antonio Andelic
46882791cd Fix build 2022-08-23 07:29:47 +00:00
Antonio Andelic
444ac9af6c Merge branch 'master' into keeper-linearizable-reads 2022-08-23 07:03:35 +00:00
Antonio Andelic
99d23cb67e Merge branch 'master' into keeper-linearizable-reads 2022-08-23 07:03:20 +00:00
Antonio Andelic
a87a762b0c
Merge pull request #39973 from ClickHouse/keeper-listen-host 
Support `interserver_listen_host` in Keeper
 2022-08-23 09:03:08 +02:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
Antonio Andelic
9396834762 Addressed comments 2022-08-17 09:29:08 +00:00
Antonio Andelic
24ee3e022d
Merge branch 'master' into keeper-listen-host 2022-08-09 09:39:27 +02:00
Alexey Milovidov
48bed932ec
Merge pull request #39931 from HarryLeeIBM/hlee-s390x-coordination 
Fix Endian issue in KeeperSnapshotManager for s390x
 2022-08-09 04:24:32 +03:00
HarryLeeIBM
42323cfb92 Use constexpr char array as magic number 2022-08-08 10:59:09 -07:00
Antonio Andelic
6b935a252f Support multiple listen hosts 2022-08-08 09:08:19 +00:00
Alexey Milovidov
adc880f362
Merge pull request #39867 from ClickHouse/keeper-block-memory-tracker-commit 
Block memory tracker in Keeper during commit
 2022-08-06 02:02:29 +03:00
HarryLeeIBM
cbd01a3a73 Fix Endian issue in KeeperSnapshotManager 2022-08-05 11:07:42 -07:00
Antonio Andelic
07a85513e8 Merge branch 'master' into keeper-create-snapshot-on-exit 2022-08-04 13:12:26 +00:00
Antonio Andelic
10d7259c2b Add log for snapshot on exit 2022-08-04 13:12:24 +00:00
Antonio Andelic
0a7d2e7b8a Use LockMemoryExceptionThread 2022-08-04 13:03:05 +00:00
Antonio Andelic
278921be3b Fix style 2022-08-04 07:25:28 +00:00
Antonio Andelic
fa98338ce1 Add comment for memory tracker blocking 2022-08-04 07:13:42 +00:00
Antonio Andelic
56ca93ca3f Block memory tracker in Keeper during commit 2022-08-03 14:14:57 +00:00
Antonio Andelic
05467e315f
Merge pull request #39698 from ClickHouse/update-digest-version 
Update Keeper version for digest
 2022-08-03 13:13:00 +02:00
Antonio Andelic
b3b3c371f0
Update KeeperStorage.h 2022-08-02 09:20:02 +02:00
Antonio Andelic
7cdad88327 Create snapshot on shutdown 2022-07-30 11:34:17 +00:00
Antonio Andelic
21becb3030 Update version for digest 2022-07-29 07:36:19 +00:00
Antonio Andelic
e3cb65a99a Rollback on failed preprocess 2022-07-29 06:51:53 +00:00
Antonio Andelic
c0a5d45258 Rollback on failed PreAppend 2022-07-28 14:06:45 +00:00
Antonio Andelic
1090d6bca7 Rollback request if storing log fails 2022-07-28 12:55:29 +00:00
Antonio Andelic
e62526720f Address PR comments 2022-07-27 07:51:30 +00:00
Antonio Andelic
04b03b6a90 Don't allow to overwrite on startup 2022-07-25 12:38:48 +00:00
Antonio Andelic
afb6cb6824 Add KeeperContext 2022-07-23 14:45:31 +00:00
Antonio Andelic
6aff87d4b5 Ignore system paths from snapshots and logstore 2022-07-23 13:46:59 +00:00
Antonio Andelic
46a6fbd7aa Small polish 2022-07-22 11:03:07 +00:00
Antonio Andelic
3040ff0959 Merge branch 'master' into keeper-version-check 2022-07-22 08:07:55 +00:00
Antonio Andelic
75476d5110 Store only root system node 2022-07-22 08:07:38 +00:00
Antonio Andelic
fdd79e62da
Merge pull request #39393 from ClickHouse/keeper-disable-digest-default 
Disable real-time digest in Keeper by default
 2022-07-21 15:15:22 +02:00
Antonio Andelic
7d30ab80c4 Fix compatibility 2022-07-21 09:31:06 +00:00
Alexey Milovidov
19c4c0bd6e
Update CoordinationSettings.cpp 2022-07-21 06:41:42 +03:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Antonio Andelic
a4450225c7 Disable digest by default 2022-07-19 17:14:44 +00:00
Antonio Andelic
a3e00faaf9 Fix unit tests 2022-07-19 13:14:36 +00:00
Antonio Andelic
ce570b6ee3 Add logs and 4LW for api version 2022-07-19 09:02:57 +00:00
Antonio Andelic
e6ded88ea3 Small refactoring 2022-07-19 08:51:12 +00:00
Antonio Andelic
cfc741030f Revert to the version with path 2022-07-18 14:30:15 +00:00
Antonio Andelic
48db20d5a1 Add 4LW for api version 2022-07-18 12:19:16 +00:00
Antonio Andelic
b8ffb151dc Some small polishing 2022-07-18 08:52:52 +00:00
Antonio Andelic
7d7c4ce4cd Add ApiVersion request 2022-07-18 08:44:38 +00:00
Antonio Andelic
73e0c35ab0 Use 4LW for api version 2022-07-15 10:13:39 +00:00
Antonio Andelic
d81758898a Remove extra newline 2022-07-13 15:26:25 +00:00
Antonio Andelic
f05f22aa13 Merge branch 'master' into keeper-version-check 2022-07-13 15:25:48 +00:00
Antonio Andelic
3b5bdd1e2a Add test for current API 2022-07-13 12:41:16 +00:00
Antonio Andelic
c296e84cab fix unit tests in debug mode 2022-07-13 12:33:18 +00:00
Antonio Andelic
838a11a85a Rename RaftAppendResult 2022-07-13 11:35:06 +00:00
Antonio Andelic
2634f80956 Remove trailing whitespaces 2022-07-13 11:06:58 +00:00
Antonio Andelic
b49ac48bfd Merge branch 'master' into keeper-linearizable-reads 2022-07-13 09:30:21 +00:00
Nikita Mikhaylov
1f0280bcde
Fix path retrieval for Keeper's state (#39148) 2022-07-13 11:25:34 +02:00
Antonio Andelic
ee4828b084 Fix list watches 2022-07-13 09:23:05 +00:00
Antonio Andelic
8bd86c39f6 Add valid values for read_mode setting 2022-07-12 08:00:31 +00:00
Antonio Andelic
fd1e5d43a0 fix unit tests in debug mode 2022-07-12 07:52:50 +00:00
Antonio Andelic
5f661e3e2b Merge branch 'master' into keeper-linearizable-reads 2022-07-12 07:33:34 +00:00
Antonio Andelic
0555477ec1 Remove debug logs 2022-07-12 07:05:29 +00:00
Antonio Andelic
6dcdf0fcb1 Throw CORRUPTED_DATA in debug mode for invalid checksum 2022-07-12 06:41:58 +00:00
Antonio Andelic
1f5b3b9922 Add restrictions for modification of internal paths 2022-07-11 13:06:24 +00:00
Antonio Andelic
568ba5d3cc Address PR comments 2022-07-11 12:56:09 +00:00
Antonio Andelic
efc7f5c534 Merge branch 'master' into keeper-persist-state 2022-07-11 12:43:06 +00:00
Antonio Andelic
41dd4e13f4 Merge branch 'master' into keeper-version-check 2022-07-11 09:38:20 +00:00
Antonio Andelic
e4f7635df7 Format 2022-07-10 19:10:27 +00:00
Antonio Andelic
183f5cc940 Add checksum to the state file 2022-07-10 15:01:38 +00:00
Antonio Andelic
2b26865f90 Unit test fixes 2022-07-10 14:40:08 +00:00
Antonio Andelic
4622084d2a Add test for persisted state 2022-07-10 09:47:04 +00:00
Antonio Andelic
f60074e9ec Make state durable 2022-07-10 09:33:49 +00:00
Antonio Andelic
37f799550b
Merge pull request #38072 from lingpeng0314/master 
Add Keeper related monitoring data
 2022-07-10 09:39:57 +02:00
Alexander Tokmakov
7630fc76ae avoid weird exception 2022-07-07 17:19:05 +02:00
Antonio Andelic
e96af48e28 Polish 2022-07-07 11:35:02 +00:00
alesapin
bd8a3ee841
Merge pull request #38338 from ClickHouse/zookeeper-add-extra-list-argument 
Extend ZooKeeper list request with support for filtering persistent or ephemeral nodes only
 2022-07-05 12:53:51 +02:00
Antonio Andelic
db94955319
Merge pull request #38556 from ClickHouse/keeper-dont-rollback-session-id 
Don't rollback SessionID request in Keeper
 2022-07-05 10:36:56 +02:00
alesapin
dde76824ca Fix ub 2022-07-02 16:02:42 +02:00
Antonio Andelic
8c77632881 Fix style 2022-07-01 14:58:19 +00:00
Antonio Andelic
97e32e6ad9 Use commit_ext in unit tests 2022-07-01 14:55:26 +00:00
alesapin
919baf331e Review fixes 2022-07-01 15:57:24 +02:00
Antonio Andelic
5a97c0fc8b Finalize requests in separate thread 2022-07-01 13:19:28 +00:00
Antonio Andelic
ae0fe1ac85 Add setting for read mode 2022-07-01 09:30:59 +00:00
alesapin
66705eacb0 Add test for keeper mntr command 2022-07-01 00:07:22 +02:00
Antonio Andelic
7333e36639 Merge branch 'master' into keeper-linearizable-reads 2022-06-29 13:59:31 +00:00
Antonio Andelic
2de659715b Don't rollback SessionID request 2022-06-29 10:21:37 +00:00
Antonio Andelic
7a8fbbf132 Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-29 08:31:03 +00:00
Antonio Andelic
1d26446f84 Define new list request type 2022-06-29 08:30:39 +00:00
Antonio Andelic
77144322d7 Add extra checks for read result 2022-06-29 07:39:39 +00:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to 
Replace where possible by std::lock_guard which is more light-weight.
 2022-06-28 19:19:06 +00:00
Antonio Andelic
73000a042d Don't reconnect manually 2022-06-28 13:44:30 +00:00
Antonio Andelic
6d7050cb42 Additional refactoring for request processing 2022-06-28 09:54:16 +00:00
Antonio Andelic
23c705222f Process read requests in bg thread 2022-06-28 08:48:19 +00:00
Antonio Andelic
a6c628cab4 Small fixes for queues 2022-06-27 14:34:27 +00:00
Antonio Andelic
dc5cc33769 Process outside of lock 2022-06-27 14:22:53 +00:00
Antonio Andelic
53e906fdd5 Improve batching by processing single request from each session 2022-06-27 12:05:27 +00:00
Antonio Andelic
8a3531c2db Refactor batching requests logic 2022-06-27 07:30:59 +00:00
Antonio Andelic
0d906e311d PR review fixes 2022-06-27 06:53:09 +00:00
Antonio Andelic
76c4fd9c8a Some fixes for batch reads 2022-06-24 14:42:24 +00:00
Antonio Andelic
ecd545fe4f Batch read requests 2022-06-24 11:36:12 +00:00
Antonio Andelic
91637bf79a Merge branch 'master' into keeper-linearizable-reads 2022-06-23 14:07:00 +00:00
Antonio Andelic
2e71ae97a1 Add unit test for different list request types 2022-06-23 10:28:12 +00:00
Antonio Andelic
ae3d9fd962 Expose list request type in internal client 2022-06-23 10:28:12 +00:00
Antonio Andelic
568c957eb2 Fix TestKeeper 2022-06-23 10:28:12 +00:00
Antonio Andelic
3a71b63b5d Add list request type 2022-06-23 10:28:12 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
lingpeng0314
f06b19bdbf Add keeper monitoring data 2022-06-20 11:32:18 +08:00
Antonio Andelic
d56894e04e Don't rollback session_id 2022-06-18 19:16:17 +00:00
Antonio Andelic
d2f3d581eb Use fast lin reads in jepsen 2022-06-18 18:43:02 +00:00
Antonio Andelic
9fe06019c0 Some small fixes 2022-06-17 18:42:46 +00:00
Antonio Andelic
125e1bd182 Initial version for linearizable reads 2022-06-17 18:19:09 +00:00
Antonio Andelic
ac0b7ab20b Fix backwards compatibility with older snapshots 2022-06-15 13:46:27 +00:00
Antonio Andelic
7e1f64002d Address review comments 2022-06-15 12:48:30 +00:00
Antonio Andelic
6e55593398 Fix remove preprocess 2022-06-14 13:23:46 +00:00
Antonio Andelic
b7bd5a8eb1 Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-14 12:51:35 +00:00
Maksim Kita
da8b1b1eba
Merge pull request #38025 from kitaisreal/use-base-sort-instead-of-standard 
Use pdqsort instead of standard sort
 2022-06-14 12:13:35 +02:00
Antonio Andelic
5d4b289550 Update stats in update delta 2022-06-14 09:14:53 +00:00
Antonio Andelic
bf27fe284c Use std::list for deltas 2022-06-14 07:37:02 +00:00
Antonio Andelic
7e99e9fe43 Fix child num assert 2022-06-14 07:17:11 +00:00
Antonio Andelic
e5504f1b33 Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-13 15:43:10 +00:00
Maksim Kita
98a89b50ff Use pdqsort instead of standard sort 2022-06-13 15:31:08 +02:00
mergify[bot]
2bf9818af0
Merge branch 'master' into keeper-broken-to-detached 2022-06-13 10:04:54 +00:00
alesapin
aff6b7a8d6 Fix keeper storage 2022-06-12 00:01:41 +02:00
alesapin
aa1fb9b2e7 Remove trash 2022-06-10 15:27:27 +02:00
alesapin
4c574f30af Add comment 2022-06-10 15:12:14 +02:00
alesapin
682eb6bcc9 Add checks for numChildren 2022-06-10 15:10:45 +02:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros 
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.

__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
 2022-06-10 10:22:31 +02:00
Antonio Andelic
2c37fe3d7b Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-06 12:39:55 +00:00
Antonio Andelic
3de1d10d93 move comment 2022-06-06 12:37:03 +00:00
Antonio Andelic
6bb2d897b4 Merge branch 'master' into keeper-broken-to-detached 2022-06-06 12:19:34 +00:00
Antonio Andelic
fbdc499533 Extract more logic into a function 2022-06-06 12:19:13 +00:00
Michael Lex
041b8f6a22
Fix bug in clickhouse-keeper: dead_session_check_period was passed as micros instead of millis. (#37824) 2022-06-03 15:35:45 +02:00
mergify[bot]
d115ddb40c
Merge branch 'master' into keeper-broken-to-detached 2022-06-02 07:39:13 +00:00
Alexey Milovidov
b5f48a7d3f Merge branch 'master' of github.com:ClickHouse/ClickHouse into llvm-14 2022-06-01 22:09:58 +02:00
mergify[bot]
4aefcf29e3
Merge branch 'revert-37534-revert-37036-keeper-preprocess-operations' into keeper-real-time-digest 2022-06-01 10:17:49 +00:00
mergify[bot]
cbb26d1677
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-01 10:16:41 +00:00
Alexey Milovidov
4bb04f913f Fix clang-tidy-14 2022-05-31 17:20:07 +02:00
Azat Khuzhin
d86181d3cd keeper: store only unique session IDs for watches 
This should speed up keeper, especially in case of incorrect usage (like
the case that had been fixed in #37640), especially in case on non
release build.

And also this should fix SIGKILL in stress tests.

You will find some details for one of such SIGKILL in `<details>` tag [1]:

<details>

    $ pigz -cd clickhouse-server.stress.log.gz | tail
    2022.05.27 16:17:24.882971 [ 637 ] {} <Trace> BackgroundSchedulePool/BgSchPool: Waiting for threads to finish.
    2022.05.27 16:17:24.896749 [ 637 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 4.09 MiB.
    2022.05.27 16:17:24.907163 [ 637 ] {} <Debug> Application: Shut down storages.
    2022.05.27 16:17:24.907233 [ 637 ] {} <Debug> Application: Waiting for current connections to servers for tables to finish.
    2022.05.27 16:17:24.934335 [ 637 ] {} <Information> Application: Closed all listening sockets. Waiting for 1 outstanding connections.
    2022.05.27 16:17:29.843491 [ 637 ] {} <Information> Application: Closed connections to servers for tables. But 1 remain. Probably some tables of other users cannot finish their connections after context shutdown.
    2022.05.27 16:17:29.843632 [ 637 ] {} <Debug> KeeperDispatcher: Shutting down storage dispatcher
    2022.05.27 16:17:34.612616 [ 688 ] {} <Test> virtual Coordination::ZooKeeperRequest::~ZooKeeperRequest(): Processing of request xid=2147483647 took 10000 ms
    2022.05.27 16:17:54.612109 [ 3176 ] {} <Debug> KeeperTCPHandler: Session #12 expired
    2022.05.27 16:19:59.823038 [ 635 ] {} <Fatal> Application: Child process was terminated by signal 9 (KILL). If it is not done by 'forcestop' command or manually, the possible cause is OOM Killer (see 'dmesg' and look at the '/var/log/kern.log' for the details).

    Thread 26 (Thread 0x7f1c7703f700 (LWP 708)):
    0  0x000000000b074b2a in __tsan::MemoryAccessImpl(__tsan::ThreadState*, unsigned long, int, bool, bool, unsigned long long*, __tsan::Shadow) ()
    1  0x000000000b08630c in __tsan::MemoryAccessRange(__tsan::ThreadState*, unsigned long, unsigned long, unsigned long, bool) ()
    2  0x000000000b01ff03 in memmove ()
    3  0x000000001bbc8996 in std::__1::__move<long, long> (__first=0xb8600000d83304, __last=<optimized out>, __result=0x7f1c021cd000) at ../contrib/libcxx/include/__algorithm/move.h:57
    4  std::__1::move<long*, long*> (__first=0xb8600000d83304, __last=<optimized out>, __result=0x7f1c021cd000) at ../contrib/libcxx/include/__algorithm/move.h:70
    5  std::__1::vector<long, std::__1::allocator<long> >::erase (this=0x7b1400584c48, __position=...) at ../contrib/libcxx/include/vector:1608
    6  DB::KeeperStorage::clearDeadWatches (this=0x7b5800001ad8, this@entry=0x7b5800001800, session_id=session_id@entry=12) at ../src/Coordination/KeeperStorage.cpp:1228
    7  0x000000001bbc5c55 in DB::KeeperStorage::processRequest (this=0x7b5800001800, zk_request=..., session_id=12, time=1, new_last_zxid=..., check_acl=true) at ../src/Coordination/KeeperStorage.cpp:1122
    8  0x000000001bba06a3 in DB::KeeperStateMachine::commit (this=<optimized out>, log_idx=3549, data=...) at ../src/Coordination/KeeperStateMachine.cpp:143
    9  0x000000001bba6193 in nuraft::state_machine::commit_ext (this=0x7b4c00001f98, params=...) at ../contrib/NuRaft/include/libnuraft/state_machine.hxx:75
    10 0x00000000202c5a55 in nuraft::raft_server::commit_app_log (this=this@entry=0x7b6c00002a18, idx_to_commit=idx_to_commit@entry=3549, le=..., need_to_handle_commit_elem=true, initial_commit_exec=false) at ../contrib/NuRaft/src/handle_commit.cxx:311
    11 0x00000000202c4f98 in nuraft::raft_server::commit_in_bg_exec (this=<optimized out>, this@entry=0x7b6c00002a18, timeout_ms=timeout_ms@entry=0, initial_commit_exec=false) at ../contrib/NuRaft/src/handle_commit.cxx:241
    12 0x00000000202c4613 in nuraft::raft_server::commit_in_bg (this=this@entry=0x7b6c00002a18) at ../contrib/NuRaft/src/handle_commit.cxx:149
    ...
    Thread 28 (Thread 0x7f1c7603d700 (LWP 710)):
    0  0x00007f1d22a6d110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
    1  0x00007f1d22a650a3 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
    2  0x000000000b0337b0 in pthread_mutex_lock ()
    3  0x00000000221884da in std::__1::__libcpp_mutex_lock (__m=0x7b4c00002088) at ../contrib/libcxx/include/__threading_support:303
    4  std::__1::mutex::lock (this=0x7b4c00002088) at ../contrib/libcxx/src/mutex.cpp:33
    5  0x000000001bba4188 in std::__1::lock_guard<std::__1::mutex>::lock_guard (__m=..., this=<optimized out>) at ../contrib/libcxx/include/__mutex_base:91
    6  DB::KeeperStateMachine::getDeadSessions (this=0x7b4c00001f98) at ../src/Coordination/KeeperStateMachine.cpp:360
    7  0x000000001bb79b4b in DB::KeeperServer::getDeadSessions (this=0x7b4400012700) at ../src/Coordination/KeeperServer.cpp:572
    8  0x000000001bb64d1a in DB::KeeperDispatcher::sessionCleanerTask (this=<optimized out>, this@entry=0x7b640001c218) at ../src/Coordination/KeeperDispatcher.cpp:399
    ...
    Thread 1 (Thread 0x7f1d227148c0 (LWP 637)):
    0  0x00007f1d22a69376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
    1  0x000000000b0895e0 in __tsan::call_pthread_cancel_with_cleanup(int (*)(void*), void (*)(void*), void*) ()
    2  0x000000000b017091 in pthread_cond_wait ()
    3  0x0000000020569d98 in Poco::EventImpl::waitImpl (this=0x7b2000008798) at ../contrib/poco/Foundation/src/Event_POSIX.cpp:106
    4  0x000000001bb636cf in Poco::Event::wait (this=0x7b2000008798) at ../contrib/poco/Foundation/include/Poco/Event.h:97
    5  ThreadFromGlobalPool::join (this=<optimized out>) at ../src/Common/ThreadPool.h:217
    6  DB::KeeperDispatcher::shutdown (this=0x7b640001c218) at ../src/Coordination/KeeperDispatcher.cpp:322
    7  0x0000000019ca8bfc in DB::Context::shutdownKeeperDispatcher (this=<optimized out>) at ../src/Interpreters/Context.cpp:2111
    8  0x000000000b0a979b in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_9::operator()() const (this=0x7ffcde44f0a0) at ../programs/server/Server.cpp:1407

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37593/2613149f6bf4f242bbbf2c3c8539b5176fd77286/stress_test__thread__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-30 11:50:48 +03:00
Antonio Andelic
6a984cfa6f
Update Changelog.cpp 2022-05-27 09:45:09 +02:00
Antonio Andelic
ea1f04fed4 Test broken logs folder 2022-05-27 07:42:09 +00:00
Antonio Andelic
1ad5fe5ced Move broken logs to detached folder 2022-05-26 13:37:20 +00:00
Antonio Andelic
d9dd11bb66
Revert "Revert "Add support for preprocessing ZooKeeper operations in clickhouse-keeper"" 2022-05-26 08:14:55 +02:00
Antonio Andelic
6a962549d5
Revert "Add support for preprocessing ZooKeeper operations in clickhouse-keeper" 2022-05-25 16:45:32 +02:00
Antonio Andelic
9e5a635da2 Fix rollback 2022-05-25 14:21:26 +00:00
Antonio Andelic
fe72cd7478 Merge branch 'master' into keeper-real-time-digest 2022-05-25 07:41:31 +00:00
Antonio Andelic
61e38b9e82 Allow comitting of logs without digest 2022-05-25 07:41:00 +00:00
Antonio Andelic
c55f4e470a Small fix 2022-05-24 13:11:11 +00:00
Antonio Andelic
85e7118300 Fix integrations test 2022-05-24 09:10:45 +00:00
Antonio Andelic
e91e7fdba7 Fix style 2022-05-23 15:33:29 +00:00
Antonio Andelic
c268296fc6 Fix single node force recovery and add tests 2022-05-23 09:53:46 +00:00
Antonio Andelic
2b8f71b4a8 Fix style 2022-05-23 07:55:23 +00:00
Antonio Andelic
ea1cbff0d8 Fix unit tests 2022-05-23 07:51:26 +00:00
Antonio Andelic
f379f225fa Add ephemerals on preprocess 2022-05-23 06:48:39 +00:00
Antonio Andelic
d8c247cfdc Add error 2022-05-20 10:14:51 +00:00
Antonio Andelic
25ab817c2b Revert formatting 2022-05-20 10:12:09 +00:00
Antonio Andelic
c133f815bd Merge branch 'keeper-preprocess-operations' into keeper-real-time-digest 2022-05-20 07:57:47 +00:00
Antonio Andelic
f5759e2d7e Cache nodes latest state 2022-05-19 12:30:57 +00:00
Antonio Andelic
6fba1c96ed Define small test for digest check 2022-05-17 13:53:12 +00:00
Antonio Andelic
c5e4598447 Calculate digest on preprocess 2022-05-17 10:23:51 +00:00
Antonio Andelic
639ceb84b1 Add comment and fix typo 2022-05-17 10:15:12 +00:00
alesapin
f958203b6c
Update src/Coordination/KeeperStorage.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-05-17 11:55:54 +02:00
alesapin
e75aa445a6
Update src/Coordination/KeeperStorage.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-05-17 11:36:12 +02:00
Antonio Andelic
fc5a79f186 Polishing changes 2022-05-17 08:16:55 +00:00
Antonio Andelic
b88f3fcfb1 Merge branch 'keeper-preprocess-operations' into keeper-real-time-digest 2022-05-17 06:52:30 +00:00
Antonio Andelic
573e42d0dd Address PR comments 2022-05-17 06:45:51 +00:00
Antonio Andelic
2205c01697 Add to snapshot and config for digest 2022-05-16 13:36:31 +00:00
Antonio Andelic
99b7e23812 Add version for digest 2022-05-16 13:36:31 +00:00
Antonio Andelic
2ec61a8d7f Initial implementation of digest 2022-05-16 13:36:31 +00:00
Antonio Andelic
6bd0b0043d Add zxid to snapshot 2022-05-16 13:35:54 +00:00
Antonio Andelic
0dccafda2e Fetch zxid independently of log idx 2022-05-16 13:35:54 +00:00
Antonio Andelic
adb8ac4fda Change log level 2022-05-16 13:15:31 +00:00
Antonio Andelic
f3646cea41 Rename preprocess test 2022-05-11 12:17:29 +00:00
Antonio Andelic
65af47a90b Merge branch 'master' into keeper-preprocess-operations 2022-05-11 12:11:17 +00:00
Antonio Andelic
e6d187001c Fix unit tests and modify messages 2022-05-11 12:10:17 +00:00
Antonio Andelic
b2aa1802cd Add unit test for basic CRUD with preprocessing 2022-05-11 09:08:55 +00:00
Antonio Andelic
0fb11ab3ff Rename uncommitted state 2022-05-11 09:08:39 +00:00
Antonio Andelic
15672a8374 Fix integration tests 2022-05-11 07:53:32 +00:00
Antonio Andelic
79080ff0cd Fix style 2022-05-10 13:43:45 +00:00
Antonio Andelic
8610d74177 init cversion 2022-05-10 13:31:39 +00:00
Antonio Andelic
88007809c2 Add rollback support 2022-05-10 13:04:35 +00:00
Antonio Andelic
02319f92a3 Define auth for sessions with preprocess 2022-05-10 12:53:18 +00:00
Antonio Andelic
f8d0aa4bc1 Fix update delta 2022-05-10 09:53:15 +00:00
Antonio Andelic
7eeb463fdb Fix local 2022-05-10 07:00:38 +00:00
Antonio Andelic
a988cfa27b Small fixes 2022-05-09 13:23:31 +00:00
Antonio Andelic
a89b57aeb6 Fix commit 2022-05-09 10:19:20 +00:00
Antonio Andelic
f5bdef4435 Remove indentation 2022-05-09 09:57:06 +00:00
Antonio Andelic
7c7bac180f Rename variables with short names 2022-05-09 09:52:43 +00:00
Antonio Andelic
246a5043b4 Rewrite commit with single lambda 2022-05-09 09:52:39 +00:00
Antonio Andelic
285bb21b91 Define close session 2022-05-09 09:35:16 +00:00
Antonio Andelic
9796527890 Support Auth check 2022-05-09 09:16:05 +00:00
Antonio Andelic
ad7226e151 Add processLocal for read requests 2022-05-09 08:32:25 +00:00
Antonio Andelic
7f6fa9fe83 Fix splitting impl of basic operations 2022-05-09 07:02:42 +00:00
Antonio Andelic
cff68aa31f Precommit implementation 2022-05-09 07:02:42 +00:00
Antonio Andelic
94ba3d129c Define preprocess with updated nodes 2022-05-09 07:02:42 +00:00
Antonio Andelic
c4cf1c863b Add precommit initial 2022-05-09 07:02:42 +00:00
alesapin
f4cdd86ae8 Force keeper snapshot equality on different replicas 2022-05-05 19:23:01 +02:00
alesapin
b76406fc8f
Merge pull request #36910 from ClickHouse/add_test 
Fix bug in keeper which could lead to corrupted compressed logs
 2022-05-04 23:44:45 +02:00
alesapin
bc50ddc37a Fix typos in text 2022-05-04 18:43:50 +02:00
alesapin
c6556da5fc Fix bug which can lead to corrupted logs in compressed format 2022-05-04 18:08:00 +02:00
alesapin
0dffd7c44a Fix coordination 2022-05-03 20:20:07 +02:00
Dmitry Novik
2ed5a4013a
Revert "Revert "Memory overcommit: continue query execution if memory is available"" 2022-05-03 00:45:13 +02:00
alesapin
f0b7af0aa2
Revert "Memory overcommit: continue query execution if memory is available" 2022-05-03 00:36:50 +02:00
Dmitry Novik
71b6f89166
Merge pull request #35637 from ClickHouse/memory-overcommit-free 
Memory overcommit: continue query execution if memory is available
 2022-05-02 19:00:18 +02:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase 
Reorganize source files so that base won't depend on Common
 2022-04-30 09:40:58 +03:00
mergify[bot]
e9fde5d067
Merge branch 'master' into memory-overcommit-free 2022-04-29 19:31:59 +00:00
alesapin
61dfbd4a83
Merge pull request #36758 from ClickHouse/warn_user_if_setting_is_strange 
Add small script for keeper check
 2022-04-29 12:10:36 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
alesapin
6e742be0e9
Update KeeperServer.cpp 2022-04-28 19:09:46 +02:00
alesapin
db9cb4cf09
Merge pull request #36736 from ClickHouse/keeper-id-guards 
Add an extra check for RAFT config change
 2022-04-28 18:22:39 +02:00
alesapin
3107b2bcd6 Add small script for stupid keeper check 2022-04-28 18:13:54 +02:00
Antonio Andelic
44ff5f1697 Add some checks for endpoint change 2022-04-28 08:54:05 +00:00
Antonio Andelic
9646487c09 Address PR review comments 2022-04-26 07:32:02 +00:00
Antonio Andelic
cf022542bd Merge branch 'master' into keeper-recovery-mode 2022-04-26 06:50:02 +00:00
alesapin
1fffa56f63 Merge branch 'master' into memory-overcommit-free 2022-04-25 20:21:08 +02:00
mergify[bot]
eefc302779
Merge branch 'master' into fix-keeper-hostname-check 2022-04-22 14:07:53 +00:00
Antonio Andelic
4e389d8df1 Rename to hostname, add tests 2022-04-22 08:16:14 +00:00
Maksim Kita
57444fc7d3
Merge pull request #36444 from rschu1ze/clang-tidy-fixes 
Clang tidy fixes
 2022-04-21 16:11:27 +02:00
Antonio Andelic
6ac3784af7 Add config example 2022-04-21 13:49:36 +00:00
Antonio Andelic
1778ddda97 Simplify code 2022-04-21 13:32:12 +00:00
Antonio Andelic
792b06cc66 Add config for host check 2022-04-21 13:32:12 +00:00
Antonio Andelic
b2770fbe1c Add setting for disabling checks 2022-04-21 13:32:12 +00:00
Antonio Andelic
03779fd463 Merge branch 'master' into keeper-recovery-mode 2022-04-21 12:34:18 +00:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings 
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
 2022-04-20 10:29:05 +02:00
Antonio Andelic
bb0d941add Run tests on a single cluster 2022-04-20 08:06:17 +00:00
Antonio Andelic
bbb0be6a44
Merge pull request #36402 from rschu1ze/clang-tidy-contains 
Activate clang-tidy warning "readability-container-contains"
 2022-04-20 08:15:46 +02:00
Antonio Andelic
e129f8ce00 Small fix 2022-04-19 12:39:34 +00:00
Antonio Andelic
70f3d3a863 Merge branch 'master' into keeper-recovery-mode 2022-04-19 12:32:48 +00:00
Antonio Andelic
3e77340a81
Allow clickhouse-keeper to serve 4-letter commands before quorum (#35992) 
* Use async start for clichkouse-keeper

* Return non active server for 4lw commands if quorum not achieved
 2022-04-19 14:03:00 +02:00
Antonio Andelic
73db184ad6 Ignore requests on leader while in recovery mode 2022-04-19 10:23:54 +00:00
Antonio Andelic
608c0996d0 Fix tests 2022-04-19 08:59:44 +00:00
Antonio Andelic
6c3bf0a5d3 Format files 2022-04-19 08:08:36 +00:00
Antonio Andelic
272965fc44 Refactoring 2022-04-19 08:08:35 +00:00
Antonio Andelic
4ecb66c1ad Support recovery with four letter commands 2022-04-19 08:08:35 +00:00
Antonio Andelic
ff2ebe113c WIP 2022-04-19 08:08:35 +00:00
Antonio Andelic
dbd88a5acb Use 1 leader quorum for recovery 2022-04-19 08:08:35 +00:00
Antonio Andelic
0e1ba927bd Add argument for force recovery 2022-04-19 08:08:35 +00:00
Antonio Andelic
bb4bc17af1 Force recover from configuration 2022-04-19 08:08:35 +00:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
Alexey Milovidov
f86808914c Fix strange trash in Keeper 2022-04-18 04:44:30 +02:00
alesapin
c0317c818c Fix bug in keeper 2022-04-14 17:33:46 +02:00
zhanglistar
0ae820550d fix ut error sometimes 2022-04-12 14:30:53 +08:00
Antonio Andelic
bf1f34ddb3 Fix unit tests 2022-04-06 07:43:48 +00:00
Antonio Andelic
d296eeee2d Small changes for Keeper 2022-04-05 13:56:28 +00:00
alesapin
d0bafe93d6 Fix name 2022-03-19 21:02:11 +01:00
alesapin
bf5b3a856d Rename some variables in keeper 2022-03-17 11:55:15 +01:00
zhangyuli1
3cba1177ee remove wchc from four_letter_word_white_list 2022-03-16 17:40:49 +08:00
Maksim Kita
65c52298b6 Fix clang-tidy warnings in Compression, Coordination, Core folders 2022-03-14 18:17:35 +00:00
alesapin
71ecd6be74
Merge pull request #35004 from ClickHouse/more_keeper_sanity_checks 
More keeper sanity checks
 2022-03-07 19:53:59 +01:00
alesapin
0c93c9e21d
Merge pull request #35010 from bigo-sg/keeperstatopt 
Keeper atomic stat
 2022-03-07 12:34:28 +01:00
alesapin
e81bbfb5b5 Review fixes 2022-03-07 12:13:37 +01:00
alesapin
4d86bf6fdf Merge branch 'master' into more_keeper_sanity_checks 2022-03-07 11:41:04 +01:00
zhanglistar
9efc8a1d38 Fix min/max stat 2022-03-07 11:11:14 +08:00
alesapin
5db9018ae0 Fix unitialized variable 2022-03-04 13:26:56 +01:00
alesapin
2ab920a404 Review fixes 2022-03-04 13:14:38 +01:00
alesapin
c33808d3db Fix read old records from logs 2022-03-03 15:18:03 +01:00
alesapin
f24fa16184 Fix address check 2022-03-03 11:29:43 +01:00
zhanglistar
9ee0d2c8a0 keeper atomic stat 2022-03-03 14:30:22 +08:00
alesapin
04d4c52e01 Add sanity checks for keeper 2022-03-02 20:37:59 +01:00
alesapin
74ff3ee4bc Add sanity check for localhost 2022-03-02 20:02:02 +01:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
alesapin
576ff0b40d
Merge pull request #34715 from azat/fix-opened-file-cache-race 
Workaround for a bug in NuRaft library
 2022-03-02 13:58:43 +01:00
Azat Khuzhin
54ceadd4b3 Avoid busy polling in keeper while searching for changelog files to delete 
Fixes: #34534
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-27 18:27:49 +03:00
alesapin
fd95d7a498
Merge pull request #34486 from bigo-sg/keeperVersion 
Keeper: add version to SnapshotableHashTable
 2022-02-23 13:35:49 +03:00
alesapin
9d21911ca7
Merge pull request #34534 from bigo-sg/changelogAsyncDel 
Keeper changelog async delete useless logs
 2022-02-23 10:30:23 +03:00
zhanglistar
0ca819197d delete useless header in changelog 2022-02-22 19:13:55 +08:00
zhanglistar
1291483094 changelog modify log 2022-02-22 19:09:45 +08:00
zhanglistar
96fe231c35 Changelog delete useless sleep in dctor 2022-02-22 19:07:33 +08:00
zhanglistar
6db770f841 Changelog add some logs and adjust thread exiting logic 2022-02-22 18:55:31 +08:00
zhangxiao871
a06e4b991e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into yandex-master 2022-02-22 11:10:47 +08:00
zhangxiao871
ad124a4d10 compatible. 2022-02-22 11:10:14 +08:00
zhanglistar
0299fd296d Use ConcurrentBoundedQueue instead of boost spsc queue. 2022-02-22 09:34:33 +08:00
Azat Khuzhin
0025110a16 Replace exit() with abort() in case of NuRaft errors 
CI founds [1]:

    WARNING: ThreadSanitizer: data race (pid=99)
      Write of size 8 at 0x7b14002192b0 by thread T27:
        12 std::__1::map<>
        13 DB::OpenedFileCache::~OpenedFileCache() obj-x86_64-linux-gnu/../src/IO/OpenedFileCache.h:27:7 (clickhouse+0xac66de6)
        14 cxa_at_exit_wrapper(void*) crtstuff.c (clickhouse+0xaa3646f)
        15 decltype(*(std::__1::forward<nuraft::raft_server*&>(fp0)).*fp()) std::__1::__invoke<void ()(), nuraft::raft_server*&, void>()

      Previous read of size 8 at 0x7b14002192b0 by thread T37 (mutexes: write M732116415018761216):
        4 DB::OpenedFileCache::get() obj-x86_64-linux-gnu/../src/IO/OpenedFileCache.h:47:37 (clickhouse+0xac66784)

    Thread T27 'nuraft_commit' (tid=193, running) created by main thread at:
    ...
    Thread T37 'MergeMutate' (tid=204, running) created by main thread at:
    ...

But it also reports that the mutex was already destroyed:

    Mutex M732116415018761216 is already destroyed.

The problem is that [nuraft can call `exit()`](1707a7572a/src/handle_commit.cxx (L157-L158)) which will call atexit handlers:

    2022.02.17 22:54:03.495450 [ 193 ] {} <Error> RaftInstance: background committing thread encounter err Memory limit (total) exceeded: would use 56.56 GiB (attempt to allocate chunk of 8192 bytes), maximum: 55.82 GiB, exiting to protect the system

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/33057/5a8cf3ac98808dadf125068a33ed9c622998a484/fuzzer_astfuzzertsan,actions//report.html

Let's replace exit() with abort() to avoid this.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-19 13:05:08 +03:00
alesapin
efb9a6d0fa
Merge pull request #34584 from bigo-sg/keerpSnapMemOpt 
Keeper memory optimization by not holding snapshot in memory
 2022-02-18 10:34:31 +03:00
zhanglistar
677b34d41d
Merge branch 'ClickHouse:master' into keeperVersion 2022-02-18 14:07:30 +08:00
zhanglistar
03c648e317
Update src/Coordination/KeeperStateMachine.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-02-17 18:15:25 +08:00
zhanglistar
cd0f7d5b16 Fix buffer not alloced bug 2022-02-17 16:33:35 +08:00
zhanglistar
7868153098 no std::cerr in KeeperSnapshotManager.h 2022-02-17 10:49:33 +08:00
zhanglistar
f2b2723a3d add buffer alloc and errno 2022-02-17 10:34:22 +08:00
zhanglistar
213a3481b5 move mmap file2buffer to function 2022-02-15 20:25:19 +08:00
alesapin
c2bdcbbe77
Merge pull request #34587 from bigo-sg/changelogNoclone 
Keeper change log no need to clone log entry
 2022-02-15 14:06:28 +03:00
zhanglistar
64d40fad2b
Merge branch 'master' into keeperVersion 2022-02-15 09:22:54 +08:00
Maksim Kita
db245cfbc9
Merge pull request #34523 from bigo-sg/nolockappendentries 
Keeper no lock on append_entries
 2022-02-14 21:28:10 +01:00
zhanglistar
b790903a61 keeper Changelog no need clone log entry 2022-02-14 22:43:37 +08:00
zhanglistar
ceeed2e928 keeper no memory snapshot 2022-02-14 21:34:55 +08:00
zhanglistar
edcea7dc31 keeper snapshot memory opt. 2022-02-14 19:43:08 +08:00
alexey-milovidov
ea71dc9d11
Merge pull request #34510 from kitaisreal/table-functions-insert-partition-by-refactoring 
Improve performance of insert into table functions URL, S3, File, HDFS
 2022-02-12 10:14:00 +03:00
zhanglistar
1dad40e25f Delete useless header 2022-02-12 00:04:08 +08:00
zhanglistar
5906b36095 Keeper changelog async delete useless logs 2022-02-11 23:58:15 +08:00
zhanglistar
e07c5751fa No lock on append_entries 2022-02-11 17:18:55 +08:00
Maksim Kita
13cbf79ecb Improve performance of insert into table functions URL, S3, File, HDFS 2022-02-10 20:06:23 +00:00
zhanglistar
4d73fb7fb7 Fix ut in SnapshotableHashTable 2022-02-10 17:56:41 +08:00
zhanglistar
baeea7c635 Fix typo in SnapshotableHashTable.h 2022-02-10 17:37:49 +08:00
zhanglistar
87350f3552 Keeper SnapshotableHashTable fix clean bug. 2022-02-10 17:33:12 +08:00
zhanglistar
220e4dd25a Keeper: add version to SnapshotableHashTable 2022-02-10 17:13:46 +08:00
zhanglistar
72ffcbbb05 keeper SnapshotableHashTable clean opt. 2022-02-10 15:43:39 +08:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
zhangxiao871
74796c6e01 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into yandex-master 2022-01-26 17:51:46 +08:00
alesapin
ab2aca2699 Fix bug in keeper which can lead to inconsistent snapshots 2022-01-24 13:17:40 +03:00
alesapin
3fed7c0f55 Fix race condition 2022-01-22 23:28:43 +03:00
alesapin
2ee3f70330 Fix erase 2022-01-22 22:36:23 +03:00
alesapin
7771249730 Fix tests 2022-01-22 19:30:45 +03:00
alesapin
460c01f2ea Fix path 2022-01-22 18:29:36 +03:00
alesapin
706c055656 Buildable 2022-01-21 21:09:46 +03:00
alesapin
a7f9377835 Fix style 2022-01-21 18:01:45 +03:00
alesapin
dd1a361960 Fix typo 2022-01-21 17:57:05 +03:00
alesapin
ab2146d2a0 Better hash set and less locks 2022-01-21 17:26:50 +03:00
alesapin
9248a1ae03 Not so ugly interface 2022-01-21 16:35:28 +03:00
zhangxiao871
be3d3886b1 Fix tests 2022-01-20 10:49:16 +08:00
alesapin
35b6b11a5d Fix hash 2022-01-19 16:47:12 +03:00
alesapin
044cc3a00e Control snapshot size better 2022-01-19 16:38:11 +03:00
alesapin
9ea6b8c2d3 Some interm state 2022-01-19 14:46:29 +03:00
alesapin
3d325aacf6 Merge branch 'master' into better_hashmap 2022-01-19 12:31:23 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl 
RFC: Split headers, move SystemLog into module, more forward declarations
 2022-01-18 17:22:49 +03:00
alesapin
07f3d08a1a Fix stupid bug 2022-01-17 17:54:09 +03:00
alesapin
4fb8761a22 Merge branch 'master' into better_hashmap 2022-01-17 15:51:50 +03:00
alesapin
b2271cc2d9
Merge pull request #33288 from JackyWoo/add_lower_bound_session_timeout_to_keeper 
Add lower bound session timeout to keeper
 2022-01-17 14:06:15 +03:00
mergify[bot]
085492614b
Merge branch 'master' into add_lower_bound_session_timeout_to_keeper 2022-01-13 12:21:46 +00:00
zhangxiao871
deebfd1e29 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into yandex-master 2022-01-13 17:50:10 +08:00
zhangxiao871
264cb193bd fix use-of-uninitialized-value 2022-01-13 17:49:39 +08:00
小路
2d1d640a48
Update ZooKeeperDataReader.cpp 2022-01-13 17:31:54 +08:00
zhangxiao871
b9aa46c5a6 fix session_id_counter for keeper-converter 2022-01-12 17:56:51 +08:00
zhangxiao871
f9342ee2be try fix build and test 2022-01-12 17:51:18 +08:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog 
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
 2022-01-10 22:01:41 +03:00
alesapin
0860acb4ef
Merge pull request #33246 from nicelulu/fix_acl_map 
Fix acl map num
 2022-01-08 17:48:00 +03:00
JackyWoo
679d18c5e1 reset last_latency when reset keeper stats 2022-01-07 22:17:03 +08:00
zhangxiao871
30eb0921b0 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into yandex-master 2022-01-07 17:30:32 +08:00
枢木
a6c1bd0935 Add config to enable ipv4 or ipv6. 2022-01-07 17:22:20 +08:00
zhangxiao871
06a6e7f583 fix build 2022-01-06 22:44:01 +08:00
zhangxiao871
377cc208d0 add time 2022-01-06 21:14:45 +08:00
alesapin
f8114126ed Use HashMap and arena with free lists for keeper 2021-12-30 19:21:49 +03:00
JackyWoo
569ce62e8d use session_timeout as session timeout uper bound 2021-12-30 17:18:51 +08:00
JackyWoo
d35e5f8319 add lower bound session timeout to keeper 2021-12-29 20:59:01 +08:00
alesapin
7e1767542c
Merge pull request #33249 from nicelulu/fix_acl 
clickhouse-keeper acl consistent with zookeeper, accept generated digest
 2021-12-29 13:21:10 +03:00
zhangxiao871
daa4c9e33b clickhouse-keeper acl consistent with zookeeper, accept generated digest 2021-12-28 17:17:01 +08:00
zhangxiao871
9303b5f0ce better process empty acls 2021-12-28 15:54:00 +08:00
zhangxiao871
2e9979d672 Fix ACLMap num 2021-12-28 15:44:07 +08:00
alesapin
8bb6205293 Fix keeper log messages 2021-12-27 15:23:44 +03:00
Alexey Milovidov
f03cb4c762 Cleanup trash from Kafka and HDFS 2021-12-25 06:10:59 +03:00
zhanglistar
e4006b2cfd
Merge branch 'ClickHouse:master' into keepersizeopt 2021-12-17 14:35:19 +08:00
zhanglistar
bc4a2ca073
Update KeeperStorage.cpp 
modify potential overflow sub.
 2021-12-16 09:25:38 +08:00
bharatnc
a3dfb89927 remove unused headers in test 2021-12-11 08:30:22 -08:00
zhanglistar
2af221df18 1. update node size in set processor. 2021-12-11 22:24:59 +08:00
zhanglistar
de3aad085d 1. fix ut fail. 2021-12-10 10:26:28 +08:00
zhanglistar
2509e879db 1. snapshot add size of Node. 2021-12-09 23:09:56 +08:00
zhanglistar
91f26edeca
Update KeeperStorage.h 
delete trailing whitespaces.
 2021-12-09 18:15:53 +08:00
zhanglistar
359257483a
Update KeeperStorage.h 2021-12-09 18:05:11 +08:00
zhanglistar
21f50f95b2
Update KeeperStorage.h 
fix code style
 2021-12-09 18:04:29 +08:00
zhanglistar
368fa9144a 1. opt keeper sizeInBytes caculation. 2021-12-08 14:52:21 +08:00
alesapin
437591f2fa Add check for duplicate hostnames and IDs in KeeperConfig 2021-12-02 14:46:33 +03:00
alesapin
646cf38213
Merge pull request #28981 from JackyWoo/add_4_letter_words_commands 
Add four letter commands to keeper
 2021-11-24 13:32:36 +03:00
Kruglov Pavel
b63b47f0f0
Merge pull request #31265 from Avogar/fix-write-buffers 
Fix and refactor WriteBiffer-s a little
 2021-11-23 16:46:09 +03:00
alesapin
35de9e42d9
Merge branch 'master' into add_4_letter_words_commands 2021-11-22 18:02:02 +03:00
Alexander Tokmakov
856502fa81 log long operations in Keeper 2021-11-20 20:39:31 +03:00
alesapin
1afef1733b Ignore overhead 2021-11-19 18:22:46 +03:00
alesapin
587a1a622d Fix tests 2021-11-19 16:03:01 +03:00
alesapin
d1eba74c04 Trim string 2021-11-19 13:52:03 +03:00
alesapin
7820af7cbd More compatible output format 2021-11-19 12:48:38 +03:00
alesapin
f33e93e305 Merge branch 'add_4_letter_words_commands' of github.com:JackyWoo/ClickHouse into JackyWoo_add_4_letter_words_commands 2021-11-19 12:31:03 +03:00
alesapin
3f8f08e816 Slightly better 2021-11-19 12:30:58 +03:00
JackyWoo
33396b054e fix build error 2021-11-19 15:52:35 +08:00
alesapin
04dba4c360 Merge branch 'add_4_letter_words_commands' of github.com:JackyWoo/ClickHouse into JackyWoo_add_4_letter_words_commands 2021-11-18 23:18:12 +03:00
alesapin
dcec086573 Small refactoring 2021-11-18 23:17:22 +03:00
JackyWoo
0d19f2a485 ignore clang-tidy for IntNode in gtest_coordination 2021-11-18 21:27:43 +08:00
JackyWoo
afd8a321b4 ignore explicit-constructor in gtest_coordination.cpp 2021-11-18 18:34:44 +08:00
JackyWoo
27d50edc59 adjust log output 2021-11-18 15:49:43 +08:00
JackyWoo
007366b506 remove sizeOf method from SnapshotableHashTable 2021-11-18 12:35:32 +08:00
alesapin
52cd5a3c36 Merge branch 'add_4_letter_words_commands' of github.com:JackyWoo/ClickHouse into JackyWoo_add_4_letter_words_commands 2021-11-17 19:50:33 +03:00
JackyWoo
4438e4c778 fix stress test error 2021-11-17 21:21:12 +08:00
alesapin
5f3eca4cd1 Merge branch 'master' into JackyWoo_add_4_letter_words_commands 2021-11-17 12:59:20 +03:00
JackyWoo
0991eddcd4 fix build error 2021-11-15 10:16:11 +08:00
JackyWoo
a2f3337ca1 code style change 2021-11-12 20:48:42 +08:00
avogar
c521a9131a Small refactoring of WriteBiffer-s 2021-11-11 02:11:18 +03:00
JackyWoo
b480b40ff7 add docs 2021-11-09 19:40:35 +08:00
JackyWoo
094f79c47b fix code style error 2021-11-09 17:49:42 +08:00
JackyWoo
c71fb3337f add more 4lwd commands 2021-11-09 17:39:28 +08:00
alesapin
13d39fdbd4 Fix tautology 2021-11-08 17:59:38 +03:00
alesapin
e64efbb2b1 Fix Bug in Keeper with inability to start 2021-11-08 15:53:45 +03:00
JackyWoo
ec1ad60e9d fix seg fault and add test 2021-11-02 17:13:35 +08:00
JackyWoo
89f7c7eef6 fix check style error 2021-11-01 13:26:25 +08:00
JackyWoo
f24d430773 add sizeOf util function to SnapshotableHashTable 2021-11-01 13:20:42 +08:00
JackyWoo
715f982096 remove not used error code in FourLetterCommand.cpp 2021-10-29 18:00:38 +08:00
JackyWoo
694306ee62 add 4lw white list to keeper 2021-10-29 17:54:25 +08:00
JackyWoo
92a0b949c6 ignore superdigest whem dump keeper configuration 2021-10-28 19:05:19 +08:00
JackyWoo
239a13ef9a fix clang13 build error 2021-10-28 18:27:36 +08:00
JackyWoo
6f672df326 fix PVS check error 2021-10-28 16:01:30 +08:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
JackyWoo
4908f714f4 dump new coordination settings 2021-10-27 23:57:25 +08:00
JackyWoo
e5b0eedd31 adjust code style for keeper 4lw cmd 2021-10-27 22:26:53 +08:00
JackyWoo
a60663e33d add 4lw commands to keeper 2021-10-27 20:26:42 +08:00
alesapin
e530682529 Build fixes 2021-10-19 22:47:04 +03:00
alesapin
28b96ff20f
Update gtest_coordination.cpp 2021-10-19 18:38:20 +03:00
alesapin
18cceedc00 Fix build and update for clickhouse-keeper 2021-10-19 17:29:49 +03:00
alesapin
9f939364e3 Add some comments 2021-10-19 17:10:09 +03:00
alesapin
da14e5e784 Fix typos 2021-10-19 16:49:36 +03:00
alesapin
042eebd981 Add test for leader remove 2021-10-19 16:37:28 +03:00
alesapin
647856658f Add synchronization and active wait 2021-10-19 16:11:29 +03:00
alesapin
2d4b601d38 Better 2021-10-19 15:00:26 +03:00
alesapin
f2d266acce Merge branch 'master' into update_keeper_config 2021-10-19 10:16:57 +03:00
alesapin
a992895b09 Debug 2021-10-19 10:14:53 +03:00
alesapin
0a838d5926 Remove fuzzers 2021-10-18 18:28:17 +03:00
alesapin
bfe2a937eb At least able to start 2021-10-18 18:27:51 +03:00
alesapin
f1fe96e194 Merge branch 'master' into debug_keeper 2021-10-18 10:16:07 +03:00
alesapin
90ff7f05fd Start keeper asynchronously if has connection to other nodes 2021-10-14 13:21:41 +03:00
Maksim Kita
c6eece5a03 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
46ba649821 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
04047f76c7 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
d55561422f Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
aae409b321 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
afdc2fe50d Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
c9b6c2661b Refactor ConcurrentBoundedQueue 2021-10-14 00:33:18 +03:00
tavplubix
83a9a8d4dc
Update KeeperStateMachine.cpp 2021-10-11 23:23:56 +03:00
Azat Khuzhin
a5a2c5ef8a Move KEEPER_DEFAULT_PATH into separate header 2021-10-03 14:34:03 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
alesapin
3cd3850edf
Update gtest_coordination.cpp 2021-09-28 11:51:02 +03:00
alesapin
eb5402622e Better test 2021-09-28 10:24:01 +03:00
alesapin
608644436c Fix tidy 2021-09-27 17:21:10 +03:00
alesapin
a73064679d Compress keeper snapshots with default ZSTD codec 2021-09-27 14:54:04 +03:00
Alexey Milovidov
cd7f9d981c Remove ya.make 2021-09-25 04:22:54 +03:00
mergify[bot]
d69fc35dd9
Merge branch 'master' into compressed_logs 2021-09-22 19:43:16 +00:00
alesapin
f980a414ee Better tests and ability to work without compression 2021-09-22 13:38:06 +03:00
alesapin
3a11cc59b0 Rename back 2021-09-22 12:16:56 +03:00
alesapin
334a7ab8db Change default extension to zst 2021-09-21 23:43:44 +03:00
alesapin
eec44929f1 Fix clang tidy 2021-09-21 23:40:14 +03:00
alesapin
5a8c801635 Remove unused variable 2021-09-21 18:50:21 +03:00
alesapin
ac7579f187 Compressed logs for keeper 2021-09-21 17:29:05 +03:00
alesapin
f983381e45
Merge pull request #29030 from nicelulu/fix-keeper-converter 
keeper-converter add the last logfile that is less than the zxid.
 2021-09-20 13:49:07 +03:00
zhangxiao871
1faa98a48a Fix deserializeCheckVersionTxn version. 2021-09-16 18:34:13 +08:00
zhangxiao871
c86832b1d7 fix deserialize log order. 2021-09-15 01:19:19 +08:00
zhangxiao871
ba33fbbf1d add the last logfile that is less than the zxid. 2021-09-15 01:14:14 +08:00
alesapin
36a11af351 Merge branch 'master' into add_test_logs_level 2021-09-04 15:48:47 +03:00
alesapin
e1f2fe8c5d
Merge branch 'master' into better_session_expiration 2021-09-03 17:38:38 +03:00
alesapin
497c225203 Test log level for CI 2021-09-03 13:07:40 +03:00
alesapin
9281c4786b Fix queue test 2021-09-03 10:10:19 +03:00
alesapin
a949329830 Review fixes 2021-09-02 23:37:34 +03:00
alesapin
3a3c3acd18
Update src/Coordination/SessionExpiryQueue.h 
Co-authored-by: tavplubix <avtokmakov@yandex-team.ru>
 2021-09-02 23:19:44 +03:00
alesapin
9bad77f806 Fix clang tidy 2021-09-02 19:16:29 +03:00
alesapin
010985ce87 Add comment 2021-09-02 14:54:32 +03:00
alesapin
c5470864e8 Fixup 2021-09-02 14:43:34 +03:00
alesapin
4c613f30b3 Simplier sessions expiration in Keeper 2021-09-02 14:40:54 +03:00
alesapin
9b1b3ec916 Fix style check 2021-09-02 14:02:37 +03:00
alesapin
d681ffabf8 Comment 2021-09-02 12:22:51 +03:00
alesapin
a8003e444b Review fixes 2021-09-02 12:20:46 +03:00
alesapin
319f03d66f Fix PVS 2021-09-01 10:51:08 +03:00
alesapin
a1ac4fc7e0 Comments 2021-08-31 14:02:26 +03:00
alesapin
49d247ca40 Fix compaction after snapshot 2021-08-31 13:59:39 +03:00
alesapin
e8ab58bc87 Remove strange comment 2021-08-31 13:07:05 +03:00
alesapin
167372e48a Add comment 2021-08-31 12:57:22 +03:00
alesapin
decbd99338 Remove debug logs 2021-08-31 12:50:11 +03:00
alesapin
ff0ed1b290 Fix unit test 2021-08-31 12:42:15 +03:00
alesapin
a5400928ab Interm 2021-08-31 12:12:11 +03:00
alesapin
2160a2f8f7 Followup 2021-08-30 16:23:31 +03:00
alesapin
00935c3bb3 Trying to debug keeper logs disappearance 2021-08-30 16:07:58 +03:00
alesapin
fde3cc1315
Merge pull request #28197 from ClickHouse/fix_requests_push 
Fix rare case when watch response received before request response
 2021-08-27 20:20:30 +03:00
alesapin
44390a88ec
Merge pull request #28152 from ClickHouse/fix_rotate_log_interval_change 2021-08-27 12:29:15 +03:00
mergify[bot]
6c34bac4ca
Merge branch 'master' into fix_requests_push 2021-08-27 07:14:48 +00:00
alesapin
23325c3fa6 Fix rare case when watch response received before request response 2021-08-26 19:00:41 +03:00
alesapin
ebfac8cfbb More correct list watches semantics in ClickHouse Keeper 2021-08-26 14:50:08 +03:00
alesapin
64bc3285e9 Fix merge with master 2021-08-26 13:14:06 +03:00
alesapin
137a3681a0 Merge branch 'master' into fix_rotate_log_interval_change 2021-08-26 10:27:56 +03:00
alesapin
96b78f83a9 Fix typo 2021-08-26 00:04:11 +03:00
alesapin
e44f9cd42d ClickHouse Keeper: Fix endless logs when rotate_interval changed 2021-08-25 21:11:52 +03:00
alesapin
dc576e952a
Update KeeperStorage.cpp 2021-08-25 12:31:02 +03:00
alesapin
916e6cc9f2 Fix style 2021-08-24 17:06:10 +03:00
alesapin
95cf0634dc Clickhouse-keeper: renames and comments 2021-08-24 15:30:31 +03:00
zhangxiao871
f34787838a Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into fix_create_znode 2021-08-18 10:37:49 +08:00
Mikhail Filimonov
af6249c8ec
Less include <Common/Stopwatch.h> 2021-08-11 16:20:59 +02:00
zhangxiao871
3d3b1658c5 Fix clickhouse-keeper create znode exists and empty condition. 2021-08-03 17:59:08 +08:00
Denny Crane
1eb71713de
copypaste error 2021-07-21 16:26:45 -03:00
alesapin
ed34844d02 Fix build 2021-07-10 11:42:25 +03:00
alesapin
60b22aaac8 Better message 2021-07-09 16:03:23 +03:00
alesapin
b2fb551bd4 Fix several bugs in ZooKeeper snapshots deserialization 2021-07-09 16:00:50 +03:00
alesapin
a92bd49229 Supress PVS 2021-06-22 23:23:26 +03:00
alesapin
bf0a4864ac Add support for set/get ACL commands 2021-06-22 13:49:35 +03:00
alesapin
71e9689ba6 Fix PVS warning 2021-06-21 22:59:19 +03:00
alesapin
731edc9a6d Fixes in ACLs 2021-06-21 18:45:45 +03:00
alesapin
76cee4e3cf Debugging 2021-06-21 16:58:39 +03:00
alesapin
b5dae909dd Add some tests 2021-06-18 21:36:19 +03:00
alesapin
2d8f45a098 Add some initialization 2021-06-18 11:55:59 +03:00
alesapin
1747c254dc Remove unused flag 2021-06-17 21:36:50 +03:00
alesapin
dbe4ba8c2c Merge branch 'update_buffer_size_in_nuraft' into zookeeper_snapshots 2021-06-17 19:51:11 +03:00
alesapin
dc893f9644 Update NuRaft buffer to 64 bit size 2021-06-17 19:37:08 +03:00
alesapin
1a6abb4db4 Better 2021-06-17 19:32:50 +03:00
alesapin
d513d14b2c Add some functions for data conversion 2021-06-17 16:29:11 +03:00
alesapin
39e843d9c7 Some code for snapshot deserialization 2021-06-15 09:45:19 +03:00
alesapin
312bab0f32 Fix empty check 2021-05-28 18:37:23 +03:00
alesapin
4fac425fd2 Fix loop 2021-05-28 16:07:16 +03:00
alesapin
b4b9d16fb0 Remove debug 2021-05-28 14:55:01 +03:00
alesapin
a1efea20dc Better ACLs storage on disk (update snapshot version) 2021-05-28 14:52:19 +03:00
alesapin
4d59590b7f Slightly better 2021-05-27 11:15:46 +03:00
mergify[bot]
d1338230f4
Merge branch 'master' into keeper_auth 2021-05-26 13:55:00 +00:00
alesapin
359668a20b Fix benign race (detected by clang-12) in Keeper snapshots 2021-05-25 18:05:51 +03:00
alesapin
d72d28a0a5 Fix clang tidy 2021-05-24 18:53:11 +03:00
alesapin
77d4002498 Fix unit tests 2021-05-24 15:26:23 +03:00
alesapin
11ac483e12 Add tests 2021-05-24 15:18:04 +03:00
alesapin
cd815efaee Merge branch 'master' into keeper_auth 2021-05-23 20:57:18 +03:00
alesapin
472a41b287 Superdigest support 2021-05-23 20:54:42 +03:00
alesapin
bdb52bb643 Fixup ACL 2021-05-22 19:21:52 +03:00
alesapin
b9e9c9cf23 More correct implementation 2021-05-22 19:07:47 +03:00
alesapin
8c06f81130 Followup fix 2021-05-22 10:50:23 +03:00
alesapin
8c63f0f8e2 Fix race condition on keeper shutdown 2021-05-22 10:46:12 +03:00
alesapin
6e593cda7b Working test 2021-05-22 10:38:50 +03:00
alesapin
ec8394ed1d Initial implementation 2021-05-22 00:19:22 +03:00
alesapin
31181095e7 Review fixes 2021-05-18 17:08:56 +03:00
alesapin
bae419be36 Fix typo 2021-05-15 18:29:07 +03:00
alesapin
852608c937 Don't abort on Auth request 2021-05-15 18:01:00 +03:00
alesapin
bfa23d7e02 Merge branch 'master' into standalone_keeper 2021-05-13 11:03:07 +03:00
alesapin
f2a8b5b2c6 Fix concurrent snapshot read/write 2021-05-12 23:28:01 +03:00
alesapin
43ee9f0a3a Check for directory owner 2021-05-12 17:05:44 +03:00
alesapin
591abbaded Better error codes in Keeper while no leader alive 2021-05-11 16:54:36 +03:00
Alexey Milovidov
367f7fe6c9 Fix warnings by PVS-Studio 2021-05-08 23:57:08 +03:00
Alexey Milovidov
e905883c75 More fixes for PVS-Studio 2021-05-08 19:12:31 +03:00
Alexey Milovidov
8b9c058141 Fix bad code in Keeper (found by PVS-Studio) 2021-05-08 19:11:31 +03:00
Alexey Milovidov
241c7a94da Fix bad code in Keeper (found by PVS-Studio) 2021-05-08 19:11:08 +03:00
alesapin
ce848ff2f0 Fix tidy 2021-04-17 17:06:49 +03:00
alesapin
1b4e3ea1d4 Fix unit tests with new API 2021-04-16 23:07:40 +03:00
alesapin
5c7fa239e2 One more typo 2021-04-16 22:08:52 +03:00
alesapin
8ff272cf7c Fix typo 2021-04-16 21:35:03 +03:00
alesapin
690045c67f Fix nasty bug 2021-04-16 21:31:23 +03:00
alesapin
7ed32dddf0 More checks 2021-04-16 17:00:12 +03:00
alesapin
7f69910e97 Followup fix 2021-04-16 16:56:57 +03:00
alesapin
539f6cda8b Followup fix 2021-04-16 16:56:05 +03:00
alesapin
aecab0a1f6 Followup fix 2021-04-16 16:55:39 +03:00
alesapin
e724952c03 Fix obvious bug 2021-04-16 16:53:48 +03:00
alesapin
f168bfae6d More optimal 2021-04-16 16:50:09 +03:00
alesapin
50103ff0bc Merge branch 'master' into keeper_bench_mark 2021-04-15 11:30:33 +03:00
alesapin
b115b8af3d Something working 2021-04-13 14:55:08 +03:00
alesapin
bc8ebb1028 Simplify build 2021-04-12 16:11:43 +03:00
alesapin
338ff1615c Simplier config 2021-04-12 15:40:01 +03:00
alesapin
ef34c95c7e Add SSL to keeper (both client and internal) 2021-04-12 15:25:52 +03:00
alesapin
07442b1170 Fix stupid sed 2021-04-08 17:24:05 +03:00
alesapin
c28a3b860c Fix Coordination darwin build 2021-04-08 17:17:57 +03:00
alesapin
a1164a7e4c More consistent 2021-04-07 13:21:53 +03:00
alesapin
36c0e601a9 Better non-dirty fix 2021-04-07 13:18:07 +03:00
alesapin
05eeec16c1 Fix potential segfault on Keeper startup 2021-04-07 11:49:10 +03:00
alesapin
a40209e84d Remove strange fsync on coordination logs rotation 2021-04-06 15:25:15 +03:00
alesapin
8fc01195ce Some improvements 2021-04-01 15:19:39 +03:00
alesapin
4ce81a91e5 Remove unused file 2021-03-29 12:16:58 +03:00
alesapin
be132a32a2 More renames 2021-03-29 11:24:56 +03:00
alesapin
e936bb1dae Rename files 2021-03-29 10:58:42 +03:00
alesapin
9bdeb436c2 Fix typo 2021-03-26 15:06:36 +03:00
alesapin
2db57f0f16 Followup fix 2021-03-26 14:18:31 +03:00
alesapin
331c5b6636 Fix startup one more time 2021-03-26 13:55:39 +03:00
alesapin
ba5c151037 Fix race condition on snapshots 2021-03-26 13:20:07 +03:00
alesapin
9d8b21a04d Fix ephemeral node removal 2021-03-24 11:12:37 +03:00
alesapin
0c525b4ec4 Add an ability to run from .deb and .tgz package 2021-03-23 15:07:21 +03:00
alesapin
043b3cc7b5 Fix startup when leadership changed 2021-03-22 13:45:22 +03:00
alesapin
81c408cb7f Return meta and storage from snapshot 2021-03-19 11:08:43 +03:00
alesapin
2654147113 Fix on fix 2021-03-19 00:14:43 +03:00
alesapin
0137a6baac Add test founding bug 2021-03-18 23:55:11 +03:00
alesapin
077a2019b6 Found first real bug with jepsen 2021-03-16 15:36:54 +03:00
alesapin
ba9e1e5a8d Some initial code 
Add some java magic

Allow to connect with old session id

More angry nemesis and fixes

Angry

Fix style

Split to files

Better wrappers

Better structure

Add set test and split to separate files (I think something broken now)

Better

Missed files
 2021-03-12 22:07:57 +03:00
alesapin
94248dd205 Fix one more nasty bug 2021-03-08 11:53:52 +03:00
alesapin
fb264c12d3 Blind fix 2021-03-08 00:40:32 +03:00
alesapin
a8b7469857 Fix session timeout update 2021-03-06 17:14:38 +03:00
alesapin
1bfc27ab6f Fix shutdown 2021-03-05 16:06:47 +03:00
alesapin
ce047b4fe1 Remove unused method 2021-03-05 14:06:37 +03:00
alesapin
15ea9a9c0c Async version 2021-03-05 13:40:24 +03:00
alesapin
aa754a3ca8 Test restore from snapshot 2021-03-04 16:40:43 +03:00
alesapin
77dbe3fee8 Get rid of network order 2021-03-04 16:02:30 +03:00
alesapin
27011f086e More style 2021-03-04 15:31:05 +03:00
alesapin
42c1a65b0f Style 2021-03-04 15:30:26 +03:00
alesapin
c8423249db Less strict checks 2021-03-04 15:01:56 +03:00
alesapin
5565c37b4f tests and fixes for off by one error 2021-03-04 14:22:59 +03:00
alesapin
3a0b9102d8 Merge branch 'fix_nukeeper_default_path' into persistent_nukeeper_snapshot_storage 2021-03-04 11:30:54 +03:00
alesapin
d28463d70b Use path as default prefix for coordination logs 2021-03-04 11:29:24 +03:00
alesapin
7556a3d723 Enable snapshots 2021-03-04 11:00:26 +03:00
alesapin
4e175bd988 Useful comments 2021-03-03 19:27:08 +03:00
alesapin
e6083c4dc4 Fix style 2021-03-03 19:25:47 +03:00
alesapin
7e28bfeb9e Fix log store start 2021-03-03 18:37:31 +03:00
alesapin
186b39f0d5 Followup fix 2021-03-03 15:31:21 +03:00
alesapin
0136d7d51a Get rid of unused field 2021-03-03 15:29:00 +03:00
alesapin
37fc5faa6f Fix nasty serialization bug 2021-03-03 15:21:21 +03:00
alesapin
1707e7f1c3 Rename file 2021-03-03 14:22:38 +03:00
alesapin
e615299ae6 Fix some bugs and add snapshots validation 2021-03-03 14:10:24 +03:00
alesapin
07e39ef47a Merge branch 'master' into persistent_nukeeper_snapshot_storage 2021-03-03 11:12:01 +03:00
alesapin
ee185bc536 Start from committed plus 1 2021-03-02 18:58:02 +03:00
alesapin
346d2b1d27 Deepbugging 2021-03-02 18:19:05 +03:00
alesapin
8e6252b25f Use snapshot object in serialization 2021-03-02 17:30:56 +03:00
alesapin
10e16e39ea Written synchronous snapshots 2021-03-02 16:18:04 +03:00
alesapin
48bf55aef9 Even better tests 2021-03-02 15:37:00 +03:00
alesapin
08ccae32f3 Fix nasty bug 2021-03-02 15:34:18 +03:00
turbo jason
3a6307a990 [ClickHouse][LOG]correct shutdown timeout log 2021-03-02 15:13:54 +08:00
alesapin
40c7455d33 Test for snapshot removal 2021-03-01 19:02:15 +03:00
alesapin
321dd3e76b Fix bug 2021-03-01 18:32:27 +03:00
alesapin
b7b6fd7cb3 Max snapshots on disk 2021-03-01 17:54:08 +03:00
alesapin
73d3c20554 Test simple serde 2021-03-01 17:40:32 +03:00
alesapin
03960b1eed Some compileable code 2021-03-01 16:33:34 +03:00
alesapin
91bc4478d7 Merge branch 'master' into persistent_nukeeper_snapshot_storage 2021-03-01 13:18:35 +03:00
alesapin
11f2a271a2 Remove useless unit test 2021-03-01 10:40:00 +03:00
alesapin
83e2f1ca64 More fixes 2021-02-26 17:54:59 +03:00
alesapin
12f74d4e4a Fix build 2021-02-26 16:56:57 +03:00
alesapin
8130726794 Move storage to new map 2021-02-26 16:53:34 +03:00
alesapin
56d9d99d1c Fix style and bayan 2021-02-26 09:59:40 +03:00
alesapin
84102a22e7 Trying to make testkeeper better in single server mode 2021-02-25 23:30:55 +03:00
alesapin
15face72c5 Stl containers 2021-02-25 22:52:22 +03:00
alesapin
356e548267 Experimenting 2021-02-25 17:23:12 +03:00
alesapin
cf42824756 Experimenting 2021-02-25 17:22:51 +03:00
alesapin
f8b81eeeb0 Get rid of own zxid 2021-02-25 11:34:05 +03:00
alesapin
a1901383ae Improve performance and fix condition variable wait 2021-02-24 14:54:14 +03:00
alesapin
b0065334e2 Summing raft test more stable 2021-02-24 00:27:10 +03:00
alesapin
d7c3dfe5fa Don't allow to start with broken changelogs 2021-02-22 23:51:58 +03:00
alesapin
1aa0c0bff0 More stable test 2021-02-21 23:36:05 +03:00
alesapin
f8a581daba Remove possibly redundant allocations 2021-02-21 20:42:19 +03:00
alesapin
331de3d653 Trying to add tests more stable 2021-02-21 00:08:19 +03:00
alesapin
48d5d75282 Make unit test more stable 2021-02-20 23:47:18 +03:00
alesapin
9f520f42c1 Fix style 2021-02-20 18:56:55 +03:00
alesapin
0c2cf3cf30 Calculate checksum with siphash 2021-02-20 18:36:56 +03:00
alesapin
e7f792c94d Fix typos 2021-02-20 14:28:39 +03:00
alesapin
89dd15a91d Checksum for header and logging, better names 2021-02-20 14:04:38 +03:00
alesapin
ad374ec095 Rename file 2021-02-19 19:05:26 +03:00
alesapin
12d05c2792 Better startup 2021-02-19 17:47:53 +03:00
alesapin
068c9cfbf7 Fix logs level 2021-02-19 17:13:29 +03:00
alesapin
86a74ca6b5 Fix size deserialization 2021-02-19 12:24:50 +03:00
alesapin
b72b13bab0 Better list performance 2021-02-19 10:25:55 +03:00
alesapin
88a6d4e206 Revert "Revert "Better list requests"" 
This reverts commit 839d6f7072.
 2021-02-19 10:05:52 +03:00
alesapin
839d6f7072 Revert "Better list requests" 
This reverts commit ee98b2a472.
 2021-02-19 00:09:41 +03:00
alesapin
ee98b2a472 Better list requests 2021-02-19 00:03:16 +03:00
alesapin
0336764426 Fix tidy one more time 2021-02-18 18:51:16 +03:00
alesapin
904b4754cc Fix tidy 2021-02-18 14:47:37 +03:00
alesapin
7f815325ba More tests for broken changelog read 2021-02-18 14:42:09 +03:00
alesapin
5b597fdf44 Force sync setting and ability to start with broken log 2021-02-18 13:23:48 +03:00
alesapin
ff663dc511 Fsync at server shutdown 2021-02-17 23:36:25 +03:00
alesapin
6e244e7bb1 Trying without fsync 2021-02-17 22:32:38 +03:00
alesapin
9396bae2e2 More reliable test keeper tests 2021-02-17 14:53:47 +03:00
alesapin
acf843a01a Slightly more optimal 2021-02-17 12:00:12 +03:00
alesapin
af95db2fcf Test log storage instead of changelog 2021-02-17 11:00:17 +03:00
alesapin
dfaa79b88e Add missed file 2021-02-17 10:10:46 +03:00
alesapin
d3e87701d4 Persistent storage 2021-02-16 22:02:18 +03:00
alesapin
b76b8013ba Fix tests 2021-02-16 20:47:12 +03:00
alesapin
b029f3e5cf Border test 2021-02-16 20:32:35 +03:00
alesapin
16bcd9d247 Add changelog tests 2021-02-16 20:28:54 +03:00
alesapin
daee018ea8 Merge branch 'master' into persistent_nukeeper_log_storage 2021-02-16 11:42:54 +03:00
alesapin
ed9f2b5eb9 Linkable code 2021-02-15 21:01:01 +03:00
alesapin
5401116988 Compileable code 2021-02-15 20:59:40 +03:00
alesapin
3ce3360379 Some initial code 2021-02-15 18:36:25 +03:00
alesapin
c1550814ca Disable snapshots for tests 2021-02-15 10:30:08 +03:00
alesapin
447fcfa1c9 Fix build 2021-02-11 15:12:01 +03:00
alesapin
99a471e047 Add ability to start as follower 2021-02-11 13:25:10 +03:00
alesapin
0acd018361 Fix typo 2021-02-11 12:58:02 +03:00
alesapin
74630acff5 More debug in test 2021-02-11 12:49:49 +03:00
alesapin
b49b7f859d Simplify startup with fixed config 2021-02-11 12:17:57 +03:00
alesapin
b4d53886a3 Add sync request/response 2021-02-10 16:01:05 +03:00
alesapin
e07bdad5c0 Fix test build 2021-02-10 14:44:21 +03:00
alesapin
57d8d81d59 Fix style 2021-02-10 13:02:09 +03:00
alesapin
c95140d906 Better startup and non-verbose logging by default 2021-02-10 12:28:53 +03:00
alesapin
51c221f993 Fix outdated session kill 2021-02-09 21:29:06 +03:00
alesapin
c78f3ba204 Missed file 2021-02-09 18:39:15 +03:00
alesapin
b130fbfd78 Add coordination settings 2021-02-09 17:47:18 +03:00
alesapin
b8baf3a443 Fix some warnings 2021-02-08 22:40:44 +03:00
alesapin
f2feeb9b19 Missing fix 2021-02-08 18:20:12 +03:00
alesapin
d57613aa18 Fix 0_o build in arcadia 2021-02-08 16:50:03 +03:00
alesapin
add89c17f2 Less serde in responses, watches on followers 2021-02-08 16:06:55 +03:00
alesapin
786e687b2f Trying to avoid unlimited wait 2021-02-08 14:01:50 +03:00
alesapin
ab98040003 More logs 2021-02-05 10:15:28 +03:00
alesapin
c5312bf362 Trying to disable suspicious parameter 2021-02-04 22:29:46 +03:00
alesapin
d85e9b496c Fix gcc-10 build 2021-02-04 16:22:30 +03:00
alesapin
da51ea1794 Simplify shutdown and requests processing 2021-02-04 15:07:41 +03:00
alesapin
3ddca91a5b Fix build one more time 2021-02-04 12:39:07 +03:00
alesapin
e7a83868dd Fix build 2021-02-04 11:28:11 +03:00
alesapin
1ff87ac6f9 Add background session lifetime control 2021-02-03 23:32:15 +03:00
alesapin
0c3ef018bb Fix ya.make 2021-02-03 10:15:57 +03:00
alesapin
c334bdca1f Fix NuKeeper server parameters 2021-02-03 10:11:19 +03:00
alesapin
d265e3b419 Less timeouts 2021-02-02 11:02:25 +03:00
alesapin
365bf65f5a Fix install script 2021-02-01 17:14:59 +03:00
alesapin
57c9b6c864 Fix build without nuraft 2021-02-01 16:18:17 +03:00
alesapin
0fb7bc6cbe clang-tidy is very smart 2021-02-01 15:10:55 +03:00
alesapin
eb5c77f558 Fix some races and better reaction to leader change 2021-02-01 14:27:26 +03:00
alesapin
67412bd529 Don't accept connections if we don't see leader + race fix from upstream 2021-02-01 10:51:10 +03:00
alesapin
bac8cc55d2 Now we answer from follower nodes 2021-01-29 15:39:04 +03:00
alesapin
8d0d2ca8e0 Add some partition tests 2021-01-28 15:07:26 +03:00
alesapin
a33963e211 Better raft server startup 2021-01-27 20:54:25 +03:00
alesapin
7af28e758a Process read requests without raft 2021-01-27 16:37:58 +03:00
alesapin
817eb100a1 Better shutdown 2021-01-26 17:08:31 +03:00
alesapin
e8a320cfd0 Fix more warnings 2021-01-26 14:10:44 +03:00
alesapin
71dca6dc00 Tidy fixes 2021-01-26 11:17:19 +03:00
alesapin
ddeb008bbb Replace ulong with size_t 2021-01-26 10:52:34 +03:00
alesapin
10cec45e53 Fix obvious deadlock 2021-01-26 10:47:04 +03:00
alesapin
43a2aae368 Add non working cmake 2021-01-25 17:59:10 +03:00
alesapin
d7e805ad99 Comment 2021-01-25 17:47:03 +03:00
alesapin
1576800289 Remove races 2021-01-25 17:36:06 +03:00
alesapin
97b9dba460 Multinode config 2021-01-25 17:10:18 +03:00
alesapin
dea4b5009b Some server initialization 2021-01-25 15:29:12 +03:00
alesapin
7ff04d7532 Some fixes 2021-01-25 13:19:02 +03:00
alesapin
8b03329f4d Some logging 2021-01-22 23:04:47 +03:00
alesapin
c1e36cfe70 Something working 2021-01-22 19:04:57 +03:00
alesapin
c2e6d6cfe8 Starting nukeeper server 2021-01-21 23:01:25 +03:00
alesapin
4aa11b3494 Remove zkutil namespace from TestKeeperStorage 2021-01-21 18:09:48 +03:00
alesapin
61fe49194b First working snapshots 2021-01-21 17:34:34 +03:00
alesapin
d6b8dd7525 Dumb snapshoting 2021-01-21 16:53:10 +03:00