Commit Graph

936 Commits

Author SHA1 Message Date
Antonio Andelic
5662e0050c Define client for multi read 2022-09-16 09:55:02 +00:00
Antonio Andelic
296fe01fb8 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-16 08:00:15 +00:00
Antonio Andelic
39307f9dba Extract into function 2022-09-15 16:14:53 +00:00
Antonio Andelic
0dbd0c1f72 Merge branch 'master' into keeper-linearizable-reads 2022-09-15 16:03:18 +00:00
Antonio Andelic
ad7864cb5f enable S3 if client is built 2022-09-15 14:06:22 +00:00
Antonio Andelic
d3d1676352 Fix config update detection 2022-09-15 13:48:39 +00:00
Antonio Andelic
d4e4ac3801 Lock file during S3 snapshot upload 2022-09-15 09:11:55 +00:00
Antonio Andelic
9fdd2f2e61 Basic implementation for S3 snapshot upload 2022-09-15 09:11:55 +00:00
Antonio Andelic
3a9996f04c Add bg thread for snapshot upload to S3 2022-09-15 09:11:55 +00:00
Antonio Andelic
36f46891dd
Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown
Close sessions on Keeper shutdown
2022-09-13 19:09:34 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage
KeeperMap storage engine
2022-09-13 14:15:09 +02:00
alesapin
69d08b1007
Merge pull request #41075 from ClickHouse/keeper-fix-possible-segfault
Fix possible segfault during Keeper shutdown
2022-09-13 12:13:24 +02:00
Antonio Andelic
a6b5ffec5d Polishing 2022-09-13 09:51:46 +00:00
Antonio Andelic
b2cc6a8cc6 Use promise/future 2022-09-12 18:19:41 +00:00
Antonio Andelic
c2dfabe51c Use shared_ptr for Event 2022-09-12 12:25:39 +00:00
Antonio Andelic
e985ee3354 Close sessions on Keeper shutdown 2022-09-12 12:22:48 +00:00
Antonio Andelic
32483aeec8 Merge branch 'master' into keeper-storage 2022-09-09 12:24:01 +00:00
Antonio Andelic
951ee44229 Process read requests with snapshot install 2022-09-07 11:51:56 +00:00
Antonio Andelic
a377a5f1e9 Change assert to LOGICAL_ERROR 2022-09-07 09:24:50 +00:00
Antonio Andelic
4cdf18cc81 Shutdown storage after RAFT 2022-09-07 09:20:47 +00:00
Antonio Andelic
c6b17bd7f9 Don't preprocess if storage finalized 2022-09-07 09:14:03 +00:00
Antonio Andelic
f0ba3ee927
Fix typo 2022-09-05 09:51:02 +02:00
Antonio Andelic
110ad53300 Merge branch 'master' into keeper-linearizable-reads 2022-09-05 07:12:45 +00:00
Antonio Andelic
b816d46520
Merge branch 'master' into fix-keeper-system-path-check 2022-09-02 13:42:16 +02:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage 2022-09-02 12:13:07 +02:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Antonio Andelic
ad2196155c Add test for system node modification 2022-09-02 08:14:06 +00:00
Antonio Andelic
b7eebfc626 Correctly check if the node is using system path 2022-09-02 07:47:12 +00:00
Antonio Andelic
7164e59b54 Merge branch 'master' into keeper-linearizable-reads 2022-09-01 08:52:31 +00:00
Antonio Andelic
fa4c7259f9 Fix typos 2022-09-01 08:46:28 +00:00
Antonio Andelic
c7d1dd942c Address PR comments 2022-09-01 07:56:06 +00:00
Antonio Andelic
21cdb9a500 Don't remove from cache if it wasn't added during rollback 2022-08-30 08:14:53 +00:00
Antonio Andelic
2416ddfae7 Add extra message about startup preprocessing 2022-08-30 08:14:31 +00:00
Antonio Andelic
57471d11db Fix uncommitted auth 2022-08-29 12:55:30 +00:00
Antonio Andelic
50af0b1938 Add bunch of comments for request processing 2022-08-29 10:50:18 +00:00
Antonio Andelic
b4af21b6fb Small fixes 2022-08-29 07:06:51 +00:00
Antonio Andelic
bff2b232ef Cache deltas for path 2022-08-26 14:03:38 +00:00
Antonio Andelic
26d68dce23 Fix style 2022-08-26 10:55:07 +00:00
Antonio Andelic
dae65178b7 Improve preprocessing performance 2022-08-26 10:46:14 +00:00
Antonio Andelic
5d45f2faa9 Add comments 2022-08-23 08:22:03 +00:00
Antonio Andelic
46882791cd Fix build 2022-08-23 07:29:47 +00:00
Antonio Andelic
444ac9af6c Merge branch 'master' into keeper-linearizable-reads 2022-08-23 07:03:35 +00:00
Antonio Andelic
99d23cb67e Merge branch 'master' into keeper-linearizable-reads 2022-08-23 07:03:20 +00:00
Antonio Andelic
a87a762b0c
Merge pull request #39973 from ClickHouse/keeper-listen-host
Support `interserver_listen_host` in Keeper
2022-08-23 09:03:08 +02:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
Antonio Andelic
9396834762 Addressed comments 2022-08-17 09:29:08 +00:00
Antonio Andelic
24ee3e022d
Merge branch 'master' into keeper-listen-host 2022-08-09 09:39:27 +02:00
Alexey Milovidov
48bed932ec
Merge pull request #39931 from HarryLeeIBM/hlee-s390x-coordination
Fix Endian issue in KeeperSnapshotManager for s390x
2022-08-09 04:24:32 +03:00
HarryLeeIBM
42323cfb92 Use constexpr char array as magic number 2022-08-08 10:59:09 -07:00
Antonio Andelic
6b935a252f Support multiple listen hosts 2022-08-08 09:08:19 +00:00
Alexey Milovidov
adc880f362
Merge pull request #39867 from ClickHouse/keeper-block-memory-tracker-commit
Block memory tracker in Keeper during commit
2022-08-06 02:02:29 +03:00
HarryLeeIBM
cbd01a3a73 Fix Endian issue in KeeperSnapshotManager 2022-08-05 11:07:42 -07:00
Antonio Andelic
07a85513e8 Merge branch 'master' into keeper-create-snapshot-on-exit 2022-08-04 13:12:26 +00:00
Antonio Andelic
10d7259c2b Add log for snapshot on exit 2022-08-04 13:12:24 +00:00
Antonio Andelic
0a7d2e7b8a Use LockMemoryExceptionThread 2022-08-04 13:03:05 +00:00
Antonio Andelic
278921be3b Fix style 2022-08-04 07:25:28 +00:00
Antonio Andelic
fa98338ce1 Add comment for memory tracker blocking 2022-08-04 07:13:42 +00:00
Antonio Andelic
56ca93ca3f Block memory tracker in Keeper during commit 2022-08-03 14:14:57 +00:00
Antonio Andelic
05467e315f
Merge pull request #39698 from ClickHouse/update-digest-version
Update Keeper version for digest
2022-08-03 13:13:00 +02:00
Antonio Andelic
b3b3c371f0
Update KeeperStorage.h 2022-08-02 09:20:02 +02:00
Antonio Andelic
7cdad88327 Create snapshot on shutdown 2022-07-30 11:34:17 +00:00
Antonio Andelic
21becb3030 Update version for digest 2022-07-29 07:36:19 +00:00
Antonio Andelic
e3cb65a99a Rollback on failed preprocess 2022-07-29 06:51:53 +00:00
Antonio Andelic
c0a5d45258 Rollback on failed PreAppend 2022-07-28 14:06:45 +00:00
Antonio Andelic
1090d6bca7 Rollback request if storing log fails 2022-07-28 12:55:29 +00:00
Antonio Andelic
e62526720f Address PR comments 2022-07-27 07:51:30 +00:00
Antonio Andelic
04b03b6a90 Don't allow to overwrite on startup 2022-07-25 12:38:48 +00:00
Antonio Andelic
afb6cb6824 Add KeeperContext 2022-07-23 14:45:31 +00:00
Antonio Andelic
6aff87d4b5 Ignore system paths from snapshots and logstore 2022-07-23 13:46:59 +00:00
Antonio Andelic
46a6fbd7aa Small polish 2022-07-22 11:03:07 +00:00
Antonio Andelic
3040ff0959 Merge branch 'master' into keeper-version-check 2022-07-22 08:07:55 +00:00
Antonio Andelic
75476d5110 Store only root system node 2022-07-22 08:07:38 +00:00
Antonio Andelic
fdd79e62da
Merge pull request #39393 from ClickHouse/keeper-disable-digest-default
Disable real-time digest in Keeper by default
2022-07-21 15:15:22 +02:00
Antonio Andelic
7d30ab80c4 Fix compatibility 2022-07-21 09:31:06 +00:00
Alexey Milovidov
19c4c0bd6e
Update CoordinationSettings.cpp 2022-07-21 06:41:42 +03:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Antonio Andelic
a4450225c7 Disable digest by default 2022-07-19 17:14:44 +00:00
Antonio Andelic
a3e00faaf9 Fix unit tests 2022-07-19 13:14:36 +00:00
Antonio Andelic
ce570b6ee3 Add logs and 4LW for api version 2022-07-19 09:02:57 +00:00
Antonio Andelic
e6ded88ea3 Small refactoring 2022-07-19 08:51:12 +00:00
Antonio Andelic
cfc741030f Revert to the version with path 2022-07-18 14:30:15 +00:00
Antonio Andelic
48db20d5a1 Add 4LW for api version 2022-07-18 12:19:16 +00:00
Antonio Andelic
b8ffb151dc Some small polishing 2022-07-18 08:52:52 +00:00
Antonio Andelic
7d7c4ce4cd Add ApiVersion request 2022-07-18 08:44:38 +00:00
Antonio Andelic
73e0c35ab0 Use 4LW for api version 2022-07-15 10:13:39 +00:00
Antonio Andelic
d81758898a Remove extra newline 2022-07-13 15:26:25 +00:00
Antonio Andelic
f05f22aa13 Merge branch 'master' into keeper-version-check 2022-07-13 15:25:48 +00:00
Antonio Andelic
3b5bdd1e2a Add test for current API 2022-07-13 12:41:16 +00:00
Antonio Andelic
c296e84cab fix unit tests in debug mode 2022-07-13 12:33:18 +00:00
Antonio Andelic
838a11a85a Rename RaftAppendResult 2022-07-13 11:35:06 +00:00
Antonio Andelic
2634f80956 Remove trailing whitespaces 2022-07-13 11:06:58 +00:00
Antonio Andelic
b49ac48bfd Merge branch 'master' into keeper-linearizable-reads 2022-07-13 09:30:21 +00:00
Nikita Mikhaylov
1f0280bcde
Fix path retrieval for Keeper's state (#39148) 2022-07-13 11:25:34 +02:00
Antonio Andelic
ee4828b084 Fix list watches 2022-07-13 09:23:05 +00:00
Antonio Andelic
8bd86c39f6 Add valid values for read_mode setting 2022-07-12 08:00:31 +00:00
Antonio Andelic
fd1e5d43a0 fix unit tests in debug mode 2022-07-12 07:52:50 +00:00
Antonio Andelic
5f661e3e2b Merge branch 'master' into keeper-linearizable-reads 2022-07-12 07:33:34 +00:00
Antonio Andelic
0555477ec1 Remove debug logs 2022-07-12 07:05:29 +00:00
Antonio Andelic
6dcdf0fcb1 Throw CORRUPTED_DATA in debug mode for invalid checksum 2022-07-12 06:41:58 +00:00
Antonio Andelic
1f5b3b9922 Add restrictions for modification of internal paths 2022-07-11 13:06:24 +00:00
Antonio Andelic
568ba5d3cc Address PR comments 2022-07-11 12:56:09 +00:00
Antonio Andelic
efc7f5c534 Merge branch 'master' into keeper-persist-state 2022-07-11 12:43:06 +00:00
Antonio Andelic
41dd4e13f4 Merge branch 'master' into keeper-version-check 2022-07-11 09:38:20 +00:00
Antonio Andelic
e4f7635df7 Format 2022-07-10 19:10:27 +00:00
Antonio Andelic
183f5cc940 Add checksum to the state file 2022-07-10 15:01:38 +00:00
Antonio Andelic
2b26865f90 Unit test fixes 2022-07-10 14:40:08 +00:00
Antonio Andelic
4622084d2a Add test for persisted state 2022-07-10 09:47:04 +00:00
Antonio Andelic
f60074e9ec Make state durable 2022-07-10 09:33:49 +00:00
Antonio Andelic
37f799550b
Merge pull request #38072 from lingpeng0314/master
Add Keeper related monitoring data
2022-07-10 09:39:57 +02:00
Alexander Tokmakov
7630fc76ae avoid weird exception 2022-07-07 17:19:05 +02:00
Antonio Andelic
e96af48e28 Polish 2022-07-07 11:35:02 +00:00
alesapin
bd8a3ee841
Merge pull request #38338 from ClickHouse/zookeeper-add-extra-list-argument
Extend ZooKeeper list request with support for filtering persistent or ephemeral nodes only
2022-07-05 12:53:51 +02:00
Antonio Andelic
db94955319
Merge pull request #38556 from ClickHouse/keeper-dont-rollback-session-id
Don't rollback SessionID request in Keeper
2022-07-05 10:36:56 +02:00
alesapin
dde76824ca Fix ub 2022-07-02 16:02:42 +02:00
Antonio Andelic
8c77632881 Fix style 2022-07-01 14:58:19 +00:00
Antonio Andelic
97e32e6ad9 Use commit_ext in unit tests 2022-07-01 14:55:26 +00:00
alesapin
919baf331e Review fixes 2022-07-01 15:57:24 +02:00
Antonio Andelic
5a97c0fc8b Finalize requests in separate thread 2022-07-01 13:19:28 +00:00
Antonio Andelic
ae0fe1ac85 Add setting for read mode 2022-07-01 09:30:59 +00:00
alesapin
66705eacb0 Add test for keeper mntr command 2022-07-01 00:07:22 +02:00
Antonio Andelic
7333e36639 Merge branch 'master' into keeper-linearizable-reads 2022-06-29 13:59:31 +00:00
Antonio Andelic
2de659715b Don't rollback SessionID request 2022-06-29 10:21:37 +00:00
Antonio Andelic
7a8fbbf132 Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-29 08:31:03 +00:00
Antonio Andelic
1d26446f84 Define new list request type 2022-06-29 08:30:39 +00:00
Antonio Andelic
77144322d7 Add extra checks for read result 2022-06-29 07:39:39 +00:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to
Replace where possible by std::lock_guard which is more light-weight.
2022-06-28 19:19:06 +00:00
Antonio Andelic
73000a042d Don't reconnect manually 2022-06-28 13:44:30 +00:00
Antonio Andelic
6d7050cb42 Additional refactoring for request processing 2022-06-28 09:54:16 +00:00
Antonio Andelic
23c705222f Process read requests in bg thread 2022-06-28 08:48:19 +00:00
Antonio Andelic
a6c628cab4 Small fixes for queues 2022-06-27 14:34:27 +00:00
Antonio Andelic
dc5cc33769 Process outside of lock 2022-06-27 14:22:53 +00:00
Antonio Andelic
53e906fdd5 Improve batching by processing single request from each session 2022-06-27 12:05:27 +00:00
Antonio Andelic
8a3531c2db Refactor batching requests logic 2022-06-27 07:30:59 +00:00
Antonio Andelic
0d906e311d PR review fixes 2022-06-27 06:53:09 +00:00
Antonio Andelic
76c4fd9c8a Some fixes for batch reads 2022-06-24 14:42:24 +00:00
Antonio Andelic
ecd545fe4f Batch read requests 2022-06-24 11:36:12 +00:00
Antonio Andelic
91637bf79a Merge branch 'master' into keeper-linearizable-reads 2022-06-23 14:07:00 +00:00
Antonio Andelic
2e71ae97a1 Add unit test for different list request types 2022-06-23 10:28:12 +00:00
Antonio Andelic
ae3d9fd962 Expose list request type in internal client 2022-06-23 10:28:12 +00:00
Antonio Andelic
568c957eb2 Fix TestKeeper 2022-06-23 10:28:12 +00:00
Antonio Andelic
3a71b63b5d Add list request type 2022-06-23 10:28:12 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA)
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
2022-06-20 16:13:25 +02:00
lingpeng0314
f06b19bdbf Add keeper monitoring data 2022-06-20 11:32:18 +08:00
Antonio Andelic
d56894e04e Don't rollback session_id 2022-06-18 19:16:17 +00:00
Antonio Andelic
d2f3d581eb Use fast lin reads in jepsen 2022-06-18 18:43:02 +00:00
Antonio Andelic
9fe06019c0 Some small fixes 2022-06-17 18:42:46 +00:00
Antonio Andelic
125e1bd182 Initial version for linearizable reads 2022-06-17 18:19:09 +00:00
Antonio Andelic
ac0b7ab20b Fix backwards compatibility with older snapshots 2022-06-15 13:46:27 +00:00
Antonio Andelic
7e1f64002d Address review comments 2022-06-15 12:48:30 +00:00
Antonio Andelic
6e55593398 Fix remove preprocess 2022-06-14 13:23:46 +00:00
Antonio Andelic
b7bd5a8eb1 Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-14 12:51:35 +00:00
Maksim Kita
da8b1b1eba
Merge pull request #38025 from kitaisreal/use-base-sort-instead-of-standard
Use pdqsort instead of standard sort
2022-06-14 12:13:35 +02:00
Antonio Andelic
5d4b289550 Update stats in update delta 2022-06-14 09:14:53 +00:00
Antonio Andelic
bf27fe284c Use std::list for deltas 2022-06-14 07:37:02 +00:00
Antonio Andelic
7e99e9fe43 Fix child num assert 2022-06-14 07:17:11 +00:00
Antonio Andelic
e5504f1b33 Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-13 15:43:10 +00:00
Maksim Kita
98a89b50ff Use pdqsort instead of standard sort 2022-06-13 15:31:08 +02:00
mergify[bot]
2bf9818af0
Merge branch 'master' into keeper-broken-to-detached 2022-06-13 10:04:54 +00:00
alesapin
aff6b7a8d6 Fix keeper storage 2022-06-12 00:01:41 +02:00
alesapin
aa1fb9b2e7 Remove trash 2022-06-10 15:27:27 +02:00
alesapin
4c574f30af Add comment 2022-06-10 15:12:14 +02:00
alesapin
682eb6bcc9 Add checks for numChildren 2022-06-10 15:10:45 +02:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.

__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
2022-06-10 10:22:31 +02:00
Antonio Andelic
2c37fe3d7b Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-06 12:39:55 +00:00
Antonio Andelic
3de1d10d93 move comment 2022-06-06 12:37:03 +00:00
Antonio Andelic
6bb2d897b4 Merge branch 'master' into keeper-broken-to-detached 2022-06-06 12:19:34 +00:00
Antonio Andelic
fbdc499533 Extract more logic into a function 2022-06-06 12:19:13 +00:00
Michael Lex
041b8f6a22
Fix bug in clickhouse-keeper: dead_session_check_period was passed as micros instead of millis. (#37824) 2022-06-03 15:35:45 +02:00
mergify[bot]
d115ddb40c
Merge branch 'master' into keeper-broken-to-detached 2022-06-02 07:39:13 +00:00
Alexey Milovidov
b5f48a7d3f Merge branch 'master' of github.com:ClickHouse/ClickHouse into llvm-14 2022-06-01 22:09:58 +02:00
mergify[bot]
4aefcf29e3
Merge branch 'revert-37534-revert-37036-keeper-preprocess-operations' into keeper-real-time-digest 2022-06-01 10:17:49 +00:00
mergify[bot]
cbb26d1677
Merge branch 'master' into revert-37534-revert-37036-keeper-preprocess-operations 2022-06-01 10:16:41 +00:00
Alexey Milovidov
4bb04f913f Fix clang-tidy-14 2022-05-31 17:20:07 +02:00
Azat Khuzhin
d86181d3cd keeper: store only unique session IDs for watches
This should speed up keeper, especially in case of incorrect usage (like
the case that had been fixed in #37640), especially in case on non
release build.

And also this should fix SIGKILL in stress tests.

You will find some details for one of such SIGKILL in `<details>` tag [1]:

<details>

    $ pigz -cd clickhouse-server.stress.log.gz | tail
    2022.05.27 16:17:24.882971 [ 637 ] {} <Trace> BackgroundSchedulePool/BgSchPool: Waiting for threads to finish.
    2022.05.27 16:17:24.896749 [ 637 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 4.09 MiB.
    2022.05.27 16:17:24.907163 [ 637 ] {} <Debug> Application: Shut down storages.
    2022.05.27 16:17:24.907233 [ 637 ] {} <Debug> Application: Waiting for current connections to servers for tables to finish.
    2022.05.27 16:17:24.934335 [ 637 ] {} <Information> Application: Closed all listening sockets. Waiting for 1 outstanding connections.
    2022.05.27 16:17:29.843491 [ 637 ] {} <Information> Application: Closed connections to servers for tables. But 1 remain. Probably some tables of other users cannot finish their connections after context shutdown.
    2022.05.27 16:17:29.843632 [ 637 ] {} <Debug> KeeperDispatcher: Shutting down storage dispatcher
    2022.05.27 16:17:34.612616 [ 688 ] {} <Test> virtual Coordination::ZooKeeperRequest::~ZooKeeperRequest(): Processing of request xid=2147483647 took 10000 ms
    2022.05.27 16:17:54.612109 [ 3176 ] {} <Debug> KeeperTCPHandler: Session #12 expired
    2022.05.27 16:19:59.823038 [ 635 ] {} <Fatal> Application: Child process was terminated by signal 9 (KILL). If it is not done by 'forcestop' command or manually, the possible cause is OOM Killer (see 'dmesg' and look at the '/var/log/kern.log' for the details).

    Thread 26 (Thread 0x7f1c7703f700 (LWP 708)):
    0  0x000000000b074b2a in __tsan::MemoryAccessImpl(__tsan::ThreadState*, unsigned long, int, bool, bool, unsigned long long*, __tsan::Shadow) ()
    1  0x000000000b08630c in __tsan::MemoryAccessRange(__tsan::ThreadState*, unsigned long, unsigned long, unsigned long, bool) ()
    2  0x000000000b01ff03 in memmove ()
    3  0x000000001bbc8996 in std::__1::__move<long, long> (__first=0xb8600000d83304, __last=<optimized out>, __result=0x7f1c021cd000) at ../contrib/libcxx/include/__algorithm/move.h:57
    4  std::__1::move<long*, long*> (__first=0xb8600000d83304, __last=<optimized out>, __result=0x7f1c021cd000) at ../contrib/libcxx/include/__algorithm/move.h:70
    5  std::__1::vector<long, std::__1::allocator<long> >::erase (this=0x7b1400584c48, __position=...) at ../contrib/libcxx/include/vector:1608
    6  DB::KeeperStorage::clearDeadWatches (this=0x7b5800001ad8, this@entry=0x7b5800001800, session_id=session_id@entry=12) at ../src/Coordination/KeeperStorage.cpp:1228
    7  0x000000001bbc5c55 in DB::KeeperStorage::processRequest (this=0x7b5800001800, zk_request=..., session_id=12, time=1, new_last_zxid=..., check_acl=true) at ../src/Coordination/KeeperStorage.cpp:1122
    8  0x000000001bba06a3 in DB::KeeperStateMachine::commit (this=<optimized out>, log_idx=3549, data=...) at ../src/Coordination/KeeperStateMachine.cpp:143
    9  0x000000001bba6193 in nuraft::state_machine::commit_ext (this=0x7b4c00001f98, params=...) at ../contrib/NuRaft/include/libnuraft/state_machine.hxx:75
    10 0x00000000202c5a55 in nuraft::raft_server::commit_app_log (this=this@entry=0x7b6c00002a18, idx_to_commit=idx_to_commit@entry=3549, le=..., need_to_handle_commit_elem=true, initial_commit_exec=false) at ../contrib/NuRaft/src/handle_commit.cxx:311
    11 0x00000000202c4f98 in nuraft::raft_server::commit_in_bg_exec (this=<optimized out>, this@entry=0x7b6c00002a18, timeout_ms=timeout_ms@entry=0, initial_commit_exec=false) at ../contrib/NuRaft/src/handle_commit.cxx:241
    12 0x00000000202c4613 in nuraft::raft_server::commit_in_bg (this=this@entry=0x7b6c00002a18) at ../contrib/NuRaft/src/handle_commit.cxx:149
    ...
    Thread 28 (Thread 0x7f1c7603d700 (LWP 710)):
    0  0x00007f1d22a6d110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
    1  0x00007f1d22a650a3 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
    2  0x000000000b0337b0 in pthread_mutex_lock ()
    3  0x00000000221884da in std::__1::__libcpp_mutex_lock (__m=0x7b4c00002088) at ../contrib/libcxx/include/__threading_support:303
    4  std::__1::mutex::lock (this=0x7b4c00002088) at ../contrib/libcxx/src/mutex.cpp:33
    5  0x000000001bba4188 in std::__1::lock_guard<std::__1::mutex>::lock_guard (__m=..., this=<optimized out>) at ../contrib/libcxx/include/__mutex_base:91
    6  DB::KeeperStateMachine::getDeadSessions (this=0x7b4c00001f98) at ../src/Coordination/KeeperStateMachine.cpp:360
    7  0x000000001bb79b4b in DB::KeeperServer::getDeadSessions (this=0x7b4400012700) at ../src/Coordination/KeeperServer.cpp:572
    8  0x000000001bb64d1a in DB::KeeperDispatcher::sessionCleanerTask (this=<optimized out>, this@entry=0x7b640001c218) at ../src/Coordination/KeeperDispatcher.cpp:399
    ...
    Thread 1 (Thread 0x7f1d227148c0 (LWP 637)):
    0  0x00007f1d22a69376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
    1  0x000000000b0895e0 in __tsan::call_pthread_cancel_with_cleanup(int (*)(void*), void (*)(void*), void*) ()
    2  0x000000000b017091 in pthread_cond_wait ()
    3  0x0000000020569d98 in Poco::EventImpl::waitImpl (this=0x7b2000008798) at ../contrib/poco/Foundation/src/Event_POSIX.cpp:106
    4  0x000000001bb636cf in Poco::Event::wait (this=0x7b2000008798) at ../contrib/poco/Foundation/include/Poco/Event.h:97
    5  ThreadFromGlobalPool::join (this=<optimized out>) at ../src/Common/ThreadPool.h:217
    6  DB::KeeperDispatcher::shutdown (this=0x7b640001c218) at ../src/Coordination/KeeperDispatcher.cpp:322
    7  0x0000000019ca8bfc in DB::Context::shutdownKeeperDispatcher (this=<optimized out>) at ../src/Interpreters/Context.cpp:2111
    8  0x000000000b0a979b in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_9::operator()() const (this=0x7ffcde44f0a0) at ../programs/server/Server.cpp:1407

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37593/2613149f6bf4f242bbbf2c3c8539b5176fd77286/stress_test__thread__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-05-30 11:50:48 +03:00
Antonio Andelic
6a984cfa6f
Update Changelog.cpp 2022-05-27 09:45:09 +02:00
Antonio Andelic
ea1f04fed4 Test broken logs folder 2022-05-27 07:42:09 +00:00
Antonio Andelic
1ad5fe5ced Move broken logs to detached folder 2022-05-26 13:37:20 +00:00
Antonio Andelic
d9dd11bb66
Revert "Revert "Add support for preprocessing ZooKeeper operations in clickhouse-keeper"" 2022-05-26 08:14:55 +02:00
Antonio Andelic
6a962549d5
Revert "Add support for preprocessing ZooKeeper operations in clickhouse-keeper" 2022-05-25 16:45:32 +02:00
Antonio Andelic
9e5a635da2 Fix rollback 2022-05-25 14:21:26 +00:00
Antonio Andelic
fe72cd7478 Merge branch 'master' into keeper-real-time-digest 2022-05-25 07:41:31 +00:00
Antonio Andelic
61e38b9e82 Allow comitting of logs without digest 2022-05-25 07:41:00 +00:00
Antonio Andelic
c55f4e470a Small fix 2022-05-24 13:11:11 +00:00
Antonio Andelic
85e7118300 Fix integrations test 2022-05-24 09:10:45 +00:00
Antonio Andelic
e91e7fdba7 Fix style 2022-05-23 15:33:29 +00:00
Antonio Andelic
c268296fc6 Fix single node force recovery and add tests 2022-05-23 09:53:46 +00:00
Antonio Andelic
2b8f71b4a8 Fix style 2022-05-23 07:55:23 +00:00
Antonio Andelic
ea1cbff0d8 Fix unit tests 2022-05-23 07:51:26 +00:00
Antonio Andelic
f379f225fa Add ephemerals on preprocess 2022-05-23 06:48:39 +00:00
Antonio Andelic
d8c247cfdc Add error 2022-05-20 10:14:51 +00:00
Antonio Andelic
25ab817c2b Revert formatting 2022-05-20 10:12:09 +00:00
Antonio Andelic
c133f815bd Merge branch 'keeper-preprocess-operations' into keeper-real-time-digest 2022-05-20 07:57:47 +00:00
Antonio Andelic
f5759e2d7e Cache nodes latest state 2022-05-19 12:30:57 +00:00
Antonio Andelic
6fba1c96ed Define small test for digest check 2022-05-17 13:53:12 +00:00
Antonio Andelic
c5e4598447 Calculate digest on preprocess 2022-05-17 10:23:51 +00:00
Antonio Andelic
639ceb84b1 Add comment and fix typo 2022-05-17 10:15:12 +00:00
alesapin
f958203b6c
Update src/Coordination/KeeperStorage.cpp
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-05-17 11:55:54 +02:00
alesapin
e75aa445a6
Update src/Coordination/KeeperStorage.h
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-05-17 11:36:12 +02:00
Antonio Andelic
fc5a79f186 Polishing changes 2022-05-17 08:16:55 +00:00
Antonio Andelic
b88f3fcfb1 Merge branch 'keeper-preprocess-operations' into keeper-real-time-digest 2022-05-17 06:52:30 +00:00
Antonio Andelic
573e42d0dd Address PR comments 2022-05-17 06:45:51 +00:00
Antonio Andelic
2205c01697 Add to snapshot and config for digest 2022-05-16 13:36:31 +00:00
Antonio Andelic
99b7e23812 Add version for digest 2022-05-16 13:36:31 +00:00
Antonio Andelic
2ec61a8d7f Initial implementation of digest 2022-05-16 13:36:31 +00:00
Antonio Andelic
6bd0b0043d Add zxid to snapshot 2022-05-16 13:35:54 +00:00
Antonio Andelic
0dccafda2e Fetch zxid independently of log idx 2022-05-16 13:35:54 +00:00
Antonio Andelic
adb8ac4fda Change log level 2022-05-16 13:15:31 +00:00
Antonio Andelic
f3646cea41 Rename preprocess test 2022-05-11 12:17:29 +00:00
Antonio Andelic
65af47a90b Merge branch 'master' into keeper-preprocess-operations 2022-05-11 12:11:17 +00:00
Antonio Andelic
e6d187001c Fix unit tests and modify messages 2022-05-11 12:10:17 +00:00
Antonio Andelic
b2aa1802cd Add unit test for basic CRUD with preprocessing 2022-05-11 09:08:55 +00:00
Antonio Andelic
0fb11ab3ff Rename uncommitted state 2022-05-11 09:08:39 +00:00
Antonio Andelic
15672a8374 Fix integration tests 2022-05-11 07:53:32 +00:00
Antonio Andelic
79080ff0cd Fix style 2022-05-10 13:43:45 +00:00
Antonio Andelic
8610d74177 init cversion 2022-05-10 13:31:39 +00:00
Antonio Andelic
88007809c2 Add rollback support 2022-05-10 13:04:35 +00:00
Antonio Andelic
02319f92a3 Define auth for sessions with preprocess 2022-05-10 12:53:18 +00:00
Antonio Andelic
f8d0aa4bc1 Fix update delta 2022-05-10 09:53:15 +00:00
Antonio Andelic
7eeb463fdb Fix local 2022-05-10 07:00:38 +00:00
Antonio Andelic
a988cfa27b Small fixes 2022-05-09 13:23:31 +00:00
Antonio Andelic
a89b57aeb6 Fix commit 2022-05-09 10:19:20 +00:00
Antonio Andelic
f5bdef4435 Remove indentation 2022-05-09 09:57:06 +00:00
Antonio Andelic
7c7bac180f Rename variables with short names 2022-05-09 09:52:43 +00:00
Antonio Andelic
246a5043b4 Rewrite commit with single lambda 2022-05-09 09:52:39 +00:00
Antonio Andelic
285bb21b91 Define close session 2022-05-09 09:35:16 +00:00
Antonio Andelic
9796527890 Support Auth check 2022-05-09 09:16:05 +00:00
Antonio Andelic
ad7226e151 Add processLocal for read requests 2022-05-09 08:32:25 +00:00
Antonio Andelic
7f6fa9fe83 Fix splitting impl of basic operations 2022-05-09 07:02:42 +00:00
Antonio Andelic
cff68aa31f Precommit implementation 2022-05-09 07:02:42 +00:00
Antonio Andelic
94ba3d129c Define preprocess with updated nodes 2022-05-09 07:02:42 +00:00
Antonio Andelic
c4cf1c863b Add precommit initial 2022-05-09 07:02:42 +00:00
alesapin
f4cdd86ae8 Force keeper snapshot equality on different replicas 2022-05-05 19:23:01 +02:00
alesapin
b76406fc8f
Merge pull request #36910 from ClickHouse/add_test
Fix bug in keeper which could lead to corrupted compressed logs
2022-05-04 23:44:45 +02:00
alesapin
bc50ddc37a Fix typos in text 2022-05-04 18:43:50 +02:00
alesapin
c6556da5fc Fix bug which can lead to corrupted logs in compressed format 2022-05-04 18:08:00 +02:00
alesapin
0dffd7c44a Fix coordination 2022-05-03 20:20:07 +02:00
Dmitry Novik
2ed5a4013a
Revert "Revert "Memory overcommit: continue query execution if memory is available"" 2022-05-03 00:45:13 +02:00
alesapin
f0b7af0aa2
Revert "Memory overcommit: continue query execution if memory is available" 2022-05-03 00:36:50 +02:00
Dmitry Novik
71b6f89166
Merge pull request #35637 from ClickHouse/memory-overcommit-free
Memory overcommit: continue query execution if memory is available
2022-05-02 19:00:18 +02:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase
Reorganize source files so that base won't depend on Common
2022-04-30 09:40:58 +03:00
mergify[bot]
e9fde5d067
Merge branch 'master' into memory-overcommit-free 2022-04-29 19:31:59 +00:00
alesapin
61dfbd4a83
Merge pull request #36758 from ClickHouse/warn_user_if_setting_is_strange
Add small script for keeper check
2022-04-29 12:10:36 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
alesapin
6e742be0e9
Update KeeperServer.cpp 2022-04-28 19:09:46 +02:00
alesapin
db9cb4cf09
Merge pull request #36736 from ClickHouse/keeper-id-guards
Add an extra check for RAFT config change
2022-04-28 18:22:39 +02:00
alesapin
3107b2bcd6 Add small script for stupid keeper check 2022-04-28 18:13:54 +02:00
Antonio Andelic
44ff5f1697 Add some checks for endpoint change 2022-04-28 08:54:05 +00:00
Antonio Andelic
9646487c09 Address PR review comments 2022-04-26 07:32:02 +00:00
Antonio Andelic
cf022542bd Merge branch 'master' into keeper-recovery-mode 2022-04-26 06:50:02 +00:00