Skip to content

Instantly share code, notes, and snippets.

@rayjanoka

rayjanoka/bmhv3 journal (osd4)

Last active March 21, 2023 16:06
Show Gist options
  • Save rayjanoka/82a094110e1de07d75b5a961f1ee0d02 to your computer and use it in GitHub Desktop.
Save rayjanoka/82a094110e1de07d75b5a961f1ee0d02 to your computer and use it in GitHub Desktop.
Download ZIP
vitastor #1 journals
Raw
bmhv3 journal (osd4)
-- Logs begin at Tue 2023-03-21 12:40:24 UTC. --
Mar 21 15:42:41 bmhv3 node[467775]: Waiting to become master
Mar 21 15:42:46 bmhv3 node[467775]: Waiting to become master
Mar 21 15:42:51 bmhv3 node[467775]: Waiting to become master
Mar 21 15:42:56 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:01 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:06 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:11 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:12 bmhv3 etcd[465468]: Starting auto-compaction at revision 1022 (retention: 10 revisions)
Mar 21 15:43:12 bmhv3 etcd[465468]: Finished auto-compaction at revision 1022
Mar 21 15:43:12 bmhv3 etcd[465468]: store.index: compact 1022
Mar 21 15:43:12 bmhv3 etcd[465468]: finished scheduled compaction at 1022 (took 398.302834ms)
Mar 21 15:43:16 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:21 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:26 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:31 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:36 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:41 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:42 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:42 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:44 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:44 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:45 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:45 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:46 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:46 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:46 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:47 bmhv3 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:47 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:51 bmhv3 node[467775]: Waiting to become master
Mar 21 15:43:56 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:01 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:06 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:11 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:16 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:21 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:26 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:31 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:36 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:41 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:43 bmhv3 systemd-logind[990]: New session 25 of user ubuntu.
Mar 21 15:44:43 bmhv3 systemd[1]: Started Session 25 of user ubuntu.
Mar 21 15:44:46 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:51 bmhv3 node[467775]: Waiting to become master
Mar 21 15:44:56 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:01 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:06 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:11 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:16 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:21 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:26 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:31 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:36 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:41 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:46 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:51 bmhv3 node[467775]: Waiting to become master
Mar 21 15:45:56 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:01 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:06 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:11 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:16 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:21 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:26 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:31 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:36 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:41 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:46 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:52 bmhv3 node[467775]: Waiting to become master
Mar 21 15:46:57 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:02 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:07 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:12 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:17 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:22 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:27 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:32 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:37 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:38 bmhv3 etcd[465468]: closed an existing TCP streaming connection with peer cfefa00537b3d384 (stream Message writer)
Mar 21 15:47:38 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message writer)
Mar 21 15:47:42 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:44 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:44 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:44 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:44 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:44 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:47:44 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream Message (read tcp 10.10.0.167:43564->10.10.0.164:2380: i/o timeout)
Mar 21 15:47:44 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:47:44 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active
Mar 21 15:47:44 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:47:45 bmhv3 etcd[465468]: closed an existing TCP streaming connection with peer cfefa00537b3d384 (stream Message writer)
Mar 21 15:47:45 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message writer)
Mar 21 15:47:45 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:45 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:45 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:45 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:46 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:46 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:46 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:46 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:47 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:47 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:47 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:47 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:47 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:48 bmhv3 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:48 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:48 bmhv3 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:48 bmhv3 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:51 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:47:51 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream Message (read tcp 10.10.0.167:50380->10.10.0.164:2380: i/o timeout)
Mar 21 15:47:51 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:47:51 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active
Mar 21 15:47:51 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:47:52 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:56 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader)
Mar 21 15:47:56 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.167:43552->10.10.0.164:2380: i/o timeout)
Mar 21 15:47:56 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:47:56 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active
Mar 21 15:47:56 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader)
Mar 21 15:47:57 bmhv3 node[467775]: Waiting to become master
Mar 21 15:47:57 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:47:57 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream Message (read tcp 10.10.0.167:45822->10.10.0.164:2380: i/o timeout)
Mar 21 15:47:57 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:47:59 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active
Mar 21 15:47:59 bmhv3 etcd[465468]: closed an existing TCP streaming connection with peer cfefa00537b3d384 (stream Message writer)
Mar 21 15:47:59 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message writer)
Mar 21 15:47:59 bmhv3 etcd[465468]: failed to dial cfefa00537b3d384 on stream Message (dial tcp 10.10.0.164:2380: i/o timeout)
Mar 21 15:47:59 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:48:01 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active
Mar 21 15:48:01 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:48:02 bmhv3 node[467775]: Waiting to become master
Mar 21 15:48:07 bmhv3 node[467775]: Waiting to become master
Mar 21 15:48:07 bmhv3 etcd[465468]: closed an existing TCP streaming connection with peer cfefa00537b3d384 (stream Message writer)
Mar 21 15:48:07 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message writer)
Mar 21 15:48:07 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader)
Mar 21 15:48:07 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.167:45846->10.10.0.164:2380: i/o timeout)
Mar 21 15:48:07 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:48:07 bmhv3 etcd[465468]: peer cfefa00537b3d384 became active
Mar 21 15:48:07 bmhv3 etcd[465468]: established a TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader)
Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 [logterm: 21, index: 2714, vote: cb7af46e9829ddc8] ignored MsgVote from cfefa00537b3d384 [logterm: 21, index: 2714] at term 21
: lease is not expired (remaining ticks: 1)
Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 [term: 21] received a MsgAppResp message with higher term from cfefa00537b3d384 [term: 22]
Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 became follower at term 22
Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: raft.node: cb7af46e9829ddc8 lost leader cb7af46e9829ddc8 at term 22
Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 [term: 22] ignored a MsgAppResp message with lower term from af5ac16031361c6a [term: 21]
Mar 21 15:48:09 bmhv3 etcd[465468]: raft2023/03/21 15:48:09 INFO: cb7af46e9829ddc8 [term: 22] ignored a MsgAppResp message with lower term from 607474ecd5112191 [term: 21]
Mar 21 15:48:10 bmhv3 etcd[465468]: raft2023/03/21 15:48:10 INFO: cb7af46e9829ddc8 [logterm: 21, index: 2715, vote: 0] cast MsgVote for af5ac16031361c6a [logterm: 21, index: 2715] at term 22
Mar 21 15:48:10 bmhv3 etcd[465468]: raft2023/03/21 15:48:10 INFO: raft.node: cb7af46e9829ddc8 elected leader af5ac16031361c6a at term 22
Mar 21 15:48:10 bmhv3 node[467775]: failed to query etcd: timeout
Mar 21 15:48:10 bmhv3 node[467775]: events.js:174
Mar 21 15:48:10 bmhv3 node[467775]: throw er; // Unhandled 'error' event
Mar 21 15:48:10 bmhv3 node[467775]: ^
Mar 21 15:48:10 bmhv3 node[467775]: Error: socket hang up
Mar 21 15:48:10 bmhv3 node[467775]: at createHangUpError (_http_client.js:332:15)
Mar 21 15:48:10 bmhv3 node[467775]: at Socket.socketCloseListener (_http_client.js:373:25)
Mar 21 15:48:10 bmhv3 node[467775]: at Socket.emit (events.js:203:15)
Mar 21 15:48:10 bmhv3 node[467775]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:10 bmhv3 node[467775]: Emitted 'error' event at:
Mar 21 15:48:10 bmhv3 node[467775]: at Socket.socketCloseListener (_http_client.js:373:11)
Mar 21 15:48:10 bmhv3 node[467775]: at Socket.emit (events.js:203:15)
Mar 21 15:48:10 bmhv3 node[467775]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:10 bmhv3 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 15:48:10 bmhv3 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'.
Mar 21 15:48:12 bmhv3 etcd[465468]: health check for peer cfefa00537b3d384 could not connect: dial tcp 10.10.0.164:2380: i/o timeout
Mar 21 15:48:14 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader)
Mar 21 15:48:14 bmhv3 etcd[465468]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.167:58980->10.10.0.164:2380: i/o timeout)
Mar 21 15:48:14 bmhv3 etcd[465468]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:48:14 bmhv3 etcd[465468]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:48:17 bmhv3 etcd[465468]: health check for peer cfefa00537b3d384 could not connect: dial tcp 10.10.0.164:2380: i/o timeout
Mar 21 15:48:20 bmhv3 systemd[1]: vitastor-mon.service: Scheduled restart job, restart counter is at 1.
Mar 21 15:48:20 bmhv3 systemd[1]: Stopped Vitastor monitor.
Mar 21 15:48:20 bmhv3 systemd[1]: Started Vitastor monitor.
Mar 21 15:48:22 bmhv3 etcd[465468]: health check for peer cfefa00537b3d384 could not connect: dial tcp 10.10.0.164:2380: i/o timeout
Mar 21 15:48:22 bmhv3 node[475469]: failed to query etcd: timeout
Mar 21 15:48:22 bmhv3 node[475469]: events.js:174
Mar 21 15:48:22 bmhv3 node[475469]: throw er; // Unhandled 'error' event
Mar 21 15:48:22 bmhv3 node[475469]: ^
Mar 21 15:48:22 bmhv3 node[475469]: Error: socket hang up
Mar 21 15:48:22 bmhv3 node[475469]: at createHangUpError (_http_client.js:332:15)
Mar 21 15:48:22 bmhv3 node[475469]: at Socket.socketCloseListener (_http_client.js:373:25)
Mar 21 15:48:22 bmhv3 node[475469]: at Socket.emit (events.js:203:15)
Mar 21 15:48:22 bmhv3 node[475469]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:22 bmhv3 node[475469]: Emitted 'error' event at:
Mar 21 15:48:22 bmhv3 node[475469]: at Socket.socketCloseListener (_http_client.js:373:11)
Mar 21 15:48:22 bmhv3 node[475469]: at Socket.emit (events.js:203:15)
Mar 21 15:48:22 bmhv3 node[475469]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:22 bmhv3 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 15:48:22 bmhv3 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'.
Mar 21 15:48:24 bmhv3 etcd[465468]: raft2023/03/21 15:48:24 INFO: cb7af46e9829ddc8 [term: 22] received a MsgVote message with higher term from 607474ecd5112191 [term: 23]
Mar 21 15:48:24 bmhv3 etcd[465468]: raft2023/03/21 15:48:24 INFO: cb7af46e9829ddc8 became follower at term 23
Mar 21 15:48:24 bmhv3 etcd[465468]: raft2023/03/21 15:48:24 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732, vote: 0] cast MsgVote for 607474ecd5112191 [logterm: 22, index: 2732] at term 23
Mar 21 15:48:24 bmhv3 etcd[465468]: raft2023/03/21 15:48:24 INFO: raft.node: cb7af46e9829ddc8 lost leader af5ac16031361c6a at term 23
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732, vote: 607474ecd5112191] rejected MsgVote from af5ac16031361c6a [logterm: 22, index: 2732] at term 2
3
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 is starting a new election at term 23
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 became candidate at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 received MsgVoteResp from cb7af46e9829ddc8 at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732] sent MsgVote request to 46d9e5656b97850f at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732] sent MsgVote request to 607474ecd5112191 at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732] sent MsgVote request to af5ac16031361c6a at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 [logterm: 22, index: 2732] sent MsgVote request to cfefa00537b3d384 at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 received MsgVoteResp from af5ac16031361c6a at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 has received 2 MsgVoteResp votes and 0 vote rejections
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 received MsgVoteResp from 607474ecd5112191 at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 has received 3 MsgVoteResp votes and 0 vote rejections
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: cb7af46e9829ddc8 became leader at term 24
Mar 21 15:48:25 bmhv3 etcd[465468]: raft2023/03/21 15:48:25 INFO: raft.node: cb7af46e9829ddc8 elected leader cb7af46e9829ddc8 at term 24
Mar 21 15:48:27 bmhv3 etcd[465468]: health check for peer cfefa00537b3d384 could not connect: dial tcp 10.10.0.164:2380: i/o timeout
Mar 21 15:48:28 bmhv3 etcd[465468]: raft2023/03/21 15:48:28 WARN: cb7af46e9829ddc8 stepped down to follower since quorum is not active
Mar 21 15:48:28 bmhv3 etcd[465468]: raft2023/03/21 15:48:28 INFO: cb7af46e9829ddc8 became follower at term 24
Mar 21 15:48:28 bmhv3 etcd[465468]: raft2023/03/21 15:48:28 INFO: raft.node: cb7af46e9829ddc8 lost leader cb7af46e9829ddc8 at term 24
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 is starting a new election at term 24
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 became candidate at term 25
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 received MsgVoteResp from cb7af46e9829ddc8 at term 25
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to 46d9e5656b97850f at term 25
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to 607474ecd5112191 at term 25
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to af5ac16031361c6a at term 25
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to cfefa00537b3d384 at term 25
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 received MsgVoteResp from af5ac16031361c6a at term 25
Mar 21 15:48:29 bmhv3 etcd[465468]: raft2023/03/21 15:48:29 INFO: cb7af46e9829ddc8 has received 2 MsgVoteResp votes and 0 vote rejections
Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 is starting a new election at term 25
Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 became candidate at term 26
Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 received MsgVoteResp from cb7af46e9829ddc8 at term 26
Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to 46d9e5656b97850f at term 26
Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to 607474ecd5112191 at term 26
Mar 21 15:48:30 bmhv3 etcd[465468]: raft2023/03/21 15:48:30 INFO: cb7af46e9829ddc8 [logterm: 24, index: 2736] sent MsgVote request to af5ac16031361c6a at term 26
Raw
bmhv4 journal (osd3)
-- Logs begin at Fri 2023-03-17 18:39:52 UTC. -- [3740/5199]
Mar 21 15:42:18 bmhv4 node[149964]: Waiting to become master
Mar 21 15:42:23 bmhv4 node[149964]: Waiting to become master
Mar 21 15:42:28 bmhv4 node[149964]: Waiting to become master
Mar 21 15:42:33 bmhv4 node[149964]: Waiting to become master
Mar 21 15:42:38 bmhv4 node[149964]: Waiting to become master
Mar 21 15:42:43 bmhv4 node[149964]: Waiting to become master
Mar 21 15:42:48 bmhv4 node[149964]: Waiting to become master
Mar 21 15:42:53 bmhv4 node[149964]: Waiting to become master
Mar 21 15:42:58 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:03 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:08 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:12 bmhv4 etcd[144820]: store.index: compact 1022
Mar 21 15:43:12 bmhv4 etcd[144820]: finished scheduled compaction at 1022 (took 392.280926ms)
Mar 21 15:43:13 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:18 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:23 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:28 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:33 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:38 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:42 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:42 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:43 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:44 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:44 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:45 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:45 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:46 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:46 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:47 bmhv4 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:47 bmhv4 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:48 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:50 bmhv4 systemd[1]: Started libcontainer container c6129cf43061692a6c756adcc67c96fbc0bc5f6f411038632e7ccd01c7a6b1e0.
Mar 21 15:43:53 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:58 bmhv4 node[149964]: Waiting to become master
Mar 21 15:43:59 bmhv4 systemd[1]: Started libcontainer container 9e6dac849a8e3d0de5b67ec79ea45e5186531ae89751c82e57f86bed208ff832.
Mar 21 15:44:00 bmhv4 systemd[1]: Started libcontainer container eaa50b1961174b84be5b1df6deb956cd660fa03826d4e41e23b6da40562999b6.
Mar 21 15:44:00 bmhv4 systemd[1]: Started libcontainer container a8da5eb9efaec931320042dac1b4d5975ff2e2bb6fd73f9dd8cfcd1c45bca2dd.
Mar 21 15:44:00 bmhv4 systemd[1]: Started libcontainer container 5454035cd3a7110bec2507f833602a3a3227a8e59019df9ec42a37fec8be1809.
Mar 21 15:44:03 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:08 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:13 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:18 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:23 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:24 bmhv4 systemd-logind[992]: New session 132 of user ubuntu.
Mar 21 15:44:24 bmhv4 systemd[1]: Started Session 132 of user ubuntu.
Mar 21 15:44:28 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:33 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:38 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:43 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:48 bmhv4 node[149964]: Waiting to become master Mar 21 15:44:49 bmhv4 systemd[1]: Started libcontainer container cb852b8a5151ec80f7b3f6f69f869178c71d5fd478298e5d4d2bb65dc530691f.
Mar 21 15:44:53 bmhv4 node[149964]: Waiting to become master
Mar 21 15:44:58 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:03 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:08 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:13 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:18 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:23 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:28 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:33 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:38 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:43 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:46 bmhv4 systemd[1]: Started libcontainer container 804797332d24c3126e41699fe032bf376f6dd5f36f8a73be3a2dc49218251522.
Mar 21 15:45:47 bmhv4 systemd[1]: Started libcontainer container 4b58780c8e2bb55c3b06c0795db01f599497cb88328f83c2221a87b12b7f487d.
Mar 21 15:45:47 bmhv4 systemd[1]: Started libcontainer container e2736254d0254ac66c6fb221eadc879a23cf54e862c99b52b19b6ace833ea269.
Mar 21 15:45:48 bmhv4 systemd[1]: Started libcontainer container 5e10e33b52a9f93a89dda44cd4a683a3d622f0f89985bd95dca51464d1beddcf.
Mar 21 15:45:48 bmhv4 systemd[1]: Started libcontainer container ff18c58bd28cde69d352af0823616f78c0c40952012d724595b156caa52ac1cc.
Mar 21 15:45:48 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:53 bmhv4 node[149964]: Waiting to become master
Mar 21 15:45:58 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:03 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:08 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:13 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:18 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:23 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:28 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:33 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:38 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:43 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:48 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:53 bmhv4 node[149964]: Waiting to become master
Mar 21 15:46:58 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:03 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:05 bmhv4 systemd-logind[992]: New session 133 of user ubuntu.
Mar 21 15:47:05 bmhv4 systemd[1]: Started Session 133 of user ubuntu.
Mar 21 15:47:09 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:12 bmhv4 etcd[144820]: time="2023-03-21T15:47:12Z" level=warning msg="error reading websocket message: websocket: close 1006 (abnormal closure): unexpected EOF"
Mar 21 15:47:14 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:16 bmhv4 systemd[1]: Started libcontainer container bfb6c4f6c4e70041687810231dceb78939b2efb8c80394fcf5988c81de013caf.
Mar 21 15:47:19 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:24 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:29 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:34 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:39 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:44 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:49 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:54 bmhv4 node[149964]: Waiting to become master
Mar 21 15:47:59 bmhv4 node[149964]: Waiting to become master
Mar 21 15:48:02 bmhv4 etcd[144820]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:48:02 bmhv4 etcd[144820]: failed to read cfefa00537b3d384 on stream Message (read tcp 10.10.0.166:60352->10.10.0.164:2380: i/o timeout)
Mar 21 15:48:02 bmhv4 etcd[144820]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:48:02 bmhv4 etcd[144820]: peer cfefa00537b3d384 became active
Mar 21 15:48:02 bmhv4 etcd[144820]: established a TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:48:04 bmhv4 node[149964]: Waiting to become master
Mar 21 15:48:09 bmhv4 etcd[144820]: raft2023/03/21 15:48:09 INFO: af5ac16031361c6a [logterm: 21, index: 2714, vote: cb7af46e9829ddc8] ignored MsgVote from cfefa00537b3d384 [logterm: 21, index: 2714] at term 21
: lease is not expired (remaining ticks: 10)
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a is starting a new election at term 21
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a became candidate at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a received MsgVoteResp from af5ac16031361c6a at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a [logterm: 21, index: 2715] sent MsgVote request to 46d9e5656b97850f at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a [logterm: 21, index: 2715] sent MsgVote request to 607474ecd5112191 at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a [logterm: 21, index: 2715] sent MsgVote request to cb7af46e9829ddc8 at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a [logterm: 21, index: 2715] sent MsgVote request to cfefa00537b3d384 at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: raft.node: af5ac16031361c6a lost leader cb7af46e9829ddc8 at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a received MsgVoteResp rejection from cfefa00537b3d384 at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a has received 1 MsgVoteResp votes and 1 vote rejections
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a received MsgVoteResp from 607474ecd5112191 at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a has received 2 MsgVoteResp votes and 1 vote rejections
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a received MsgVoteResp from cb7af46e9829ddc8 at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a has received 3 MsgVoteResp votes and 1 vote rejections
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: af5ac16031361c6a became leader at term 22
Mar 21 15:48:10 bmhv4 etcd[144820]: raft2023/03/21 15:48:10 INFO: raft.node: af5ac16031361c6a elected leader af5ac16031361c6a at term 22
Mar 21 15:48:10 bmhv4 node[149964]: Waiting to become master
Mar 21 15:48:10 bmhv4 node[149964]: failed to query etcd: timeout
Mar 21 15:48:11 bmhv4 node[149964]: events.js:174
Mar 21 15:48:11 bmhv4 node[149964]: throw er; // Unhandled 'error' event
Mar 21 15:48:11 bmhv4 node[149964]: ^
Mar 21 15:48:11 bmhv4 node[149964]: Error: socket hang up
Mar 21 15:48:11 bmhv4 node[149964]: at createHangUpError (_http_client.js:332:15)
Mar 21 15:48:11 bmhv4 node[149964]: at Socket.socketCloseListener (_http_client.js:373:25)
Mar 21 15:48:11 bmhv4 node[149964]: at Socket.emit (events.js:203:15)
Mar 21 15:48:11 bmhv4 node[149964]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:11 bmhv4 node[149964]: Emitted 'error' event at:
Mar 21 15:48:11 bmhv4 node[149964]: at Socket.socketCloseListener (_http_client.js:373:11)
Mar 21 15:48:11 bmhv4 node[149964]: at Socket.emit (events.js:203:15)
Mar 21 15:48:11 bmhv4 node[149964]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:11 bmhv4 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 15:48:11 bmhv4 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'.
Mar 21 15:48:12 bmhv4 etcd[144820]: failed to reach the peerURL(http://10.10.0.164:2380) of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout)
Mar 21 15:48:12 bmhv4 etcd[144820]: cannot get the version of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout)
Mar 21 15:48:12 bmhv4 etcd[144820]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader)
Mar 21 15:48:12 bmhv4 etcd[144820]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.166:60366->10.10.0.164:2380: i/o timeout)
Mar 21 15:48:12 bmhv4 etcd[144820]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:48:15 bmhv4 etcd[144820]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:48:18 bmhv4 etcd[144820]: failed to reach the peerURL(http://10.10.0.164:2380) of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout)
Mar 21 15:48:18 bmhv4 etcd[144820]: cannot get the version of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout)
Mar 21 15:48:21 bmhv4 systemd[1]: vitastor-mon.service: Scheduled restart job, restart counter is at 1.
Mar 21 15:48:21 bmhv4 systemd[1]: Stopped Vitastor monitor.
Mar 21 15:48:21 bmhv4 systemd[1]: Started Vitastor monitor.
Mar 21 15:48:23 bmhv4 etcd[144820]: raft2023/03/21 15:48:23 WARN: af5ac16031361c6a stepped down to follower since quorum is not active
Mar 21 15:48:23 bmhv4 etcd[144820]: raft2023/03/21 15:48:23 INFO: af5ac16031361c6a became follower at term 22
Mar 21 15:48:23 bmhv4 etcd[144820]: raft2023/03/21 15:48:23 INFO: raft.node: af5ac16031361c6a lost leader af5ac16031361c6a at term 22
Mar 21 15:48:25 bmhv4 etcd[144820]: failed to reach the peerURL(http://10.10.0.164:2380) of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout)
Mar 21 15:48:25 bmhv4 etcd[144820]: cannot get the version of member cfefa00537b3d384 (Get http://10.10.0.164:2380/version: dial tcp 10.10.0.164:2380: i/o timeout)
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a is starting a new election at term 22
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a became candidate at term 23
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a received MsgVoteResp from af5ac16031361c6a at term 23
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732] sent MsgVote request to 46d9e5656b97850f at term 23
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732] sent MsgVote request to 607474ecd5112191 at term 23
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732] sent MsgVote request to cb7af46e9829ddc8 at term 23
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732] sent MsgVote request to cfefa00537b3d384 at term 23
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a received MsgVoteResp rejection from cb7af46e9829ddc8 at term 23
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a has received 1 MsgVoteResp votes and 1 vote rejections
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [term: 23] received a MsgVote message with higher term from cb7af46e9829ddc8 [term: 24]
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a became follower at term 24
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: af5ac16031361c6a [logterm: 22, index: 2732, vote: 0] cast MsgVote for cb7af46e9829ddc8 [logterm: 22, index: 2732] at term 24
Mar 21 15:48:25 bmhv4 etcd[144820]: raft2023/03/21 15:48:25 INFO: raft.node: af5ac16031361c6a elected leader cb7af46e9829ddc8 at term 24
Mar 21 15:48:25 bmhv4 node[168679]: failed to query etcd: {"error":"etcdserver: leader changed","message":"etcdserver: leader changed","code":14}
Mar 21 15:48:26 bmhv4 etcd[144820]: lost the TCP streaming connection with peer 607474ecd5112191 (stream Message reader)
Mar 21 15:48:26 bmhv4 etcd[144820]: failed to read 607474ecd5112191 on stream Message (read tcp 10.10.0.166:37116->10.10.0.171:2380: i/o timeout)
Mar 21 15:48:26 bmhv4 etcd[144820]: peer 607474ecd5112191 became inactive (message send to peer failed)
Mar 21 15:48:26 bmhv4 etcd[144820]: peer 607474ecd5112191 became active
Mar 21 15:48:26 bmhv4 etcd[144820]: established a TCP streaming connection with peer 607474ecd5112191 (stream Message reader)
Mar 21 15:48:29 bmhv4 etcd[144820]: raft2023/03/21 15:48:29 INFO: af5ac16031361c6a [term: 24] received a MsgVote message with higher term from cb7af46e9829ddc8 [term: 25]
Mar 21 15:48:29 bmhv4 etcd[144820]: raft2023/03/21 15:48:29 INFO: af5ac16031361c6a became follower at term 25
Mar 21 15:48:29 bmhv4 etcd[144820]: raft2023/03/21 15:48:29 INFO: af5ac16031361c6a [logterm: 24, index: 2736, vote: 0] cast MsgVote for cb7af46e9829ddc8 [logterm: 24, index: 2736] at term 25
Mar 21 15:48:29 bmhv4 etcd[144820]: raft2023/03/21 15:48:29 INFO: raft.node: af5ac16031361c6a lost leader cb7af46e9829ddc8 at term 25
Mar 21 15:48:30 bmhv4 node[168679]: failed to query etcd: timeout
Mar 21 15:48:30 bmhv4 node[168679]: events.js:174
Mar 21 15:48:30 bmhv4 node[168679]: throw er; // Unhandled 'error' event
Mar 21 15:48:30 bmhv4 node[168679]: ^
Mar 21 15:48:30 bmhv4 node[168679]: Error: socket hang up
Mar 21 15:48:30 bmhv4 node[168679]: at createHangUpError (_http_client.js:332:15)
Mar 21 15:48:30 bmhv4 node[168679]: at Socket.socketCloseListener (_http_client.js:373:25)
Mar 21 15:48:30 bmhv4 node[168679]: at Socket.emit (events.js:203:15)
Mar 21 15:48:30 bmhv4 node[168679]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:30 bmhv4 node[168679]: Emitted 'error' event at:
Mar 21 15:48:30 bmhv4 node[168679]: at Socket.socketCloseListener (_http_client.js:373:11)
Mar 21 15:48:30 bmhv4 node[168679]: at Socket.emit (events.js:203:15)
Mar 21 15:48:30 bmhv4 node[168679]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:30 bmhv4 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 15:48:30 bmhv4 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'.
Raw
bmhv7 journal (osd1)
-- Logs begin at Tue 2023-03-21 13:08:47 UTC. --
Mar 21 15:43:12 bmhv7 etcd[43768]: store.index: compact 1022
Mar 21 15:43:12 bmhv7 etcd[43768]: finished scheduled compaction at 1022 (took 392.040541ms)
Mar 21 15:43:36 bmhv7 systemd[1]: Created slice libcontainer container kubepods-burstable-pod9ac5c3e2_566f_48fd_9332_9c845c7e25d1.slice.
Mar 21 15:43:37 bmhv7 systemd-udevd[52478]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 21 15:43:37 bmhv7 systemd-udevd[52477]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 21 15:43:37 bmhv7 networkd-dispatcher[987]: WARNING:Unknown index 33 seen, reloading interface list
Mar 21 15:43:37 bmhv7 systemd-udevd[52478]: Using default interface naming scheme 'v245'.
Mar 21 15:43:37 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Link UP
Mar 21 15:43:37 bmhv7 systemd-udevd[52477]: Using default interface naming scheme 'v245'. Mar 21 15:43:37 bmhv7 kernel: eth0: renamed from tmp2e607 Mar 21 15:43:37 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Gained carrier Mar 21 15:43:37 bmhv7 networkd-dispatcher[987]: ERROR:Unknown interface index 33 seen even after reload Mar 21 15:43:37 bmhv7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Mar 21 15:43:37 bmhv7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): lxc2de1f96c479d: link becomes ready Mar 21 15:43:37 bmhv7 networkd-dispatcher[987]: WARNING:Unknown index 33 seen, reloading interface list
Mar 21 15:43:37 bmhv7 networkd-dispatcher[987]: ERROR:Unknown interface index 33 seen even after reload
Mar 21 15:43:37 bmhv7 systemd[1]: Started libcontainer container 2e60701dea55ace882d0aaffffdb788ba1adcff3a209ed36572157773e090036.
Mar 21 15:43:37 bmhv7 systemd[1]: Started libcontainer container 6469156091c79ef354fd42b729ad7935f1f5d7613b53d5f510eb6dd696ba6518.
Mar 21 15:43:38 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Gained IPv6LL
Mar 21 15:43:39 bmhv7 systemd[1]: Started libcontainer container a43c58055c9ff58e07cb3ba37b000582e90113e3d30f4a1e830ef1a3001f2e66. Mar 21 15:43:40 bmhv7 systemd[1]: Started libcontainer container e69e69fd116cc1680bd8e64baf347f11d5228740e5efb59ab3b3e7d2125d28b3.
Mar 21 15:43:41 bmhv7 systemd[1]: Started libcontainer container 92ebe5561c73a539e8f4d86e2c35b1f2db17670ef92a703e113634082c92cf1d.
Mar 21 15:43:41 bmhv7 systemd[1]: Started libcontainer container 4906f25538e1c29cb297716d18db0a36eac678107c033f4e2f74619a44fe2467.
Mar 21 15:43:41 bmhv7 systemd[1]: Started libcontainer container fb9df170f0612b08de30cf1e4e9644a08f535b592a1071bc297740b7d1ecbc48.
Mar 21 15:43:42 bmhv7 systemd[1]: Started libcontainer container be0851e05a65eefd3ae24938f6d67863334b234c1eb4e3510bd3545e6989fad0.
Mar 21 15:43:42 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:42 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:44 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:44 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:45 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:45 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:46 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:46 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:47 bmhv7 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:47 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:44:42 bmhv7 systemd[1]: Started libcontainer container 93a962bc9dfcacba9b1e2c84a967c9ee6de4d7328974a881633cfa5b3d41ced8.
Mar 21 15:44:59 bmhv7 systemd-logind[997]: New session 5 of user ubuntu.
Mar 21 15:44:59 bmhv7 systemd[1]: Started Session 5 of user ubuntu.
Mar 21 15:45:41 bmhv7 systemd[1]: Created slice libcontainer container kubepods-besteffort-podf7c4c46b_e818_4e46_ab66_ca8021decabc.slice.
Mar 21 15:45:54 bmhv7 kernel: nbd0: detected capacity change from 0 to 20971520
Mar 21 15:45:54 bmhv7 kernel: EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
Mar 21 15:45:55 bmhv7 systemd-udevd[53894]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 21 15:45:55 bmhv7 systemd-udevd[53894]: Using default interface naming scheme 'v245'.
Mar 21 15:45:55 bmhv7 networkd-dispatcher[987]: WARNING:Unknown index 35 seen, reloading interface list
Mar 21 15:45:55 bmhv7 systemd-udevd[53895]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 21 15:45:55 bmhv7 kernel: eth0: renamed from tmp7362b
Mar 21 15:45:55 bmhv7 systemd-networkd[946]: lxcc5c9eeefa2ef: Link UP
Mar 21 15:45:55 bmhv7 systemd-networkd[946]: tmp7362b: Failed to wait for the interface to be initialized: No such device
Mar 21 15:45:55 bmhv7 systemd-udevd[53895]: Using default interface naming scheme 'v245'.
Mar 21 15:45:55 bmhv7 systemd-networkd[946]: lxcc5c9eeefa2ef: Gained carrier
Mar 21 15:45:55 bmhv7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Mar 21 15:45:55 bmhv7 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): lxcc5c9eeefa2ef: link becomes ready
Mar 21 15:45:55 bmhv7 networkd-dispatcher[987]: ERROR:Unknown interface index 35 seen even after reload
Mar 21 15:45:55 bmhv7 networkd-dispatcher[987]: WARNING:Unknown index 35 seen, reloading interface list
Mar 21 15:45:55 bmhv7 networkd-dispatcher[987]: ERROR:Unknown interface index 35 seen even after reload
Mar 21 15:45:55 bmhv7 systemd[1]: Started libcontainer container 7362b5c17fd6a71070a9eba241c980f3f6a170b591d1e993d245cd2562af8a0e.
Mar 21 15:45:56 bmhv7 systemd[1]: Started libcontainer container 660f211d046a9cc1866bb33c409c3208ae4fc29067bd7c2893ea2a72cdc5f673.
Mar 21 15:45:56 bmhv7 systemd-networkd[946]: lxcc5c9eeefa2ef: Gained IPv6LL
Mar 21 15:46:01 bmhv7 systemd[1]: Started libcontainer container bc548033ca24f73d7f3178c5d054de083f6f64838c1878016b2a68021b9f6e84.
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 6 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 9 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 10 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 22 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 26 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 31 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 32 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 39 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 40 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 41 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 44 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 45 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 48 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 49 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 57 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 62 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 65 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 73 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 75 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 78 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 81 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 84 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 85 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 95 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 96 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 97 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 101 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 102 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 110 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 112 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 118 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 123 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 127 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 128 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 129 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 131 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 132 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 140 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 145 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 149 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 150 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 151 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 153 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 154 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 155 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 156 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 158 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 159 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 162 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 163 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 1 (testpool) PG 167 primary OSD from 2 to 1
...
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 192 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 202 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 210 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 212 primary OSD from 2 to 4
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 217 primary OSD from 2 to 4
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 220 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 221 primary OSD from 2 to 4
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 228 primary OSD from 2 to 4
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 230 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 232 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 235 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 253 primary OSD from 2 to 4
Mar 21 15:46:46 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 256 primary OSD from 2 to 1
Mar 21 15:46:46 bmhv7 node[43861]: PG configuration successfully changed
Mar 21 15:47:02 bmhv7 kernel: vitastor-osd[45920]: segfault at 559dad0fc6a8 ip 00007fd29a6808ad sp 00007ffd06ae11c0 error 4 in libvitastor_blk.so.0.8.6[7fd29a664000+2d000]
Mar 21 15:47:02 bmhv7 kernel: Code: 0f 1f 44 00 00 31 d2 48 f7 b7 c8 02 00 00 48 8d 42 01 49 39 c2 74 66 48 8b 8f c0 02 00 00 48 89 c2 48 c1 e2 05 48 8d 54 11 e0 <48> 83 7a 08 00 75 4c 4c 8b 87 80 02 00 00 48
8b 0a 48 8b 97 90 02
Mar 21 15:47:02 bmhv7 etcd[43768]: time="2023-03-21T15:47:02Z" level=warning msg="error reading websocket message: websocket: close 1006 (abnormal closure): unexpected EOF"
Mar 21 15:47:02 bmhv7 systemd[1]: [email protected]: Main process exited, code=killed, status=11/SEGV
Mar 21 15:47:02 bmhv7 systemd[1]: [email protected]: Failed with result 'signal'.
Mar 21 15:47:12 bmhv7 systemd[1]: [email protected]: Scheduled restart job, restart counter is at 1.
Mar 21 15:47:12 bmhv7 systemd[1]: Stopped Vitastor object storage daemon osd.1.
Mar 21 15:47:12 bmhv7 systemd[1]: Starting Vitastor object storage daemon osd.1...
Mar 21 15:47:12 bmhv7 systemd[1]: Started Vitastor object storage daemon osd.1.
Mar 21 15:47:12 bmhv7 systemd[1]: [email protected]: Succeeded.
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 1 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 2 primary OSD from 3 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 4 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 5 primary OSD from 4 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 6 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 7 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 8 primary OSD from 3 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 9 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 10 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 11 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 12 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 13 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 14 primary OSD from 3 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 15 primary OSD from 4 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 16 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 17 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 18 primary OSD from 1 to 0
...
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 243 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 244 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 245 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 246 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 247 primary OSD from 3 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 248 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 249 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 250 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 251 primary OSD from 3 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 252 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 253 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 254 primary OSD from 4 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 1 (testpool) PG 255 primary OSD from 1 to 0
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 1 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 2 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 3 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 7 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 10 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 11 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 14 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 16 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 17 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 19 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 24 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 25 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 26 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 31 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 35 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 40 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 41 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 42 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 44 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 46 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 47 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 48 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 49 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 55 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 57 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 59 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 60 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 62 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 64 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 73 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 79 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 80 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 83 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 85 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 87 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 88 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 89 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 90 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 94 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 96 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 99 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 110 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 113 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 116 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 120 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 121 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 123 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 125 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 127 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 130 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 136 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 140 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 141 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 144 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 145 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 146 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 150 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 155 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 159 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 160 primary OSD from 1 to 4
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 162 primary OSD from 1 to 3
Mar 21 15:47:18 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 163 primary OSD from 1 to 3
...
Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 243 primary OSD from 4 to 1
Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 245 primary OSD from 4 to 1
Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 246 primary OSD from 3 to 1
Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 249 primary OSD from 4 to 1
Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 250 primary OSD from 4 to 1
Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 255 primary OSD from 4 to 1
Mar 21 15:47:23 bmhv7 node[43861]: Moving pool 2 (ecpool) PG 256 primary OSD from 3 to 1
Mar 21 15:47:23 bmhv7 node[43861]: PG configuration successfully changed
Mar 21 15:47:28 bmhv7 systemd[1]: Started libcontainer container b40aab9cc637dc0409f532b7557546278bfef400958bb7416e85aea28ca069f3.
Mar 21 15:47:43 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Link DOWN
Mar 21 15:47:43 bmhv7 systemd-networkd[946]: lxc2de1f96c479d: Lost carrier
Mar 21 15:47:44 bmhv7 networkd-dispatcher[803668]: Interface "lxc2de1f96c479d" not found.
Mar 21 15:47:44 bmhv7 systemd[1]: networkd-dispatcher.service: Got notification message from PID 803668, but reception only permitted for main PID 987
Mar 21 15:47:44 bmhv7 networkd-dispatcher[987]: ERROR:Failed to get interface "lxc2de1f96c479d" status: Command '['/usr/bin/networkctl', 'status', '--no-pager', '--no-legend', '--', 'lxc2de1f96c479d']' returne
d non-zero exit status 1.
Mar 21 15:47:44 bmhv7 systemd[1]: run-netns-cni\x2d0c18c1b2\x2de692\x2d3636\x2df712\x2dbfcd32ab2781.mount: Succeeded.
Mar 21 15:47:44 bmhv7 systemd[18106]: run-netns-cni\x2d0c18c1b2\x2de692\x2d3636\x2df712\x2dbfcd32ab2781.mount: Succeeded.
Mar 21 15:47:44 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:44 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:44 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:44 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:44 bmhv7 systemd[1]: Removed slice libcontainer container kubepods-burstable-pod9ac5c3e2_566f_48fd_9332_9c845c7e25d1.slice.
Mar 21 15:47:45 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:45 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:45 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:45 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:46 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:46 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:46 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:46 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:47 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:47 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:47 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:47 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:48 bmhv7 kernel: IPv4: martian source 74.50.109.35 from 74.50.109.35, on dev enp33s0f0
Mar 21 15:47:48 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:47:48 bmhv7 kernel: IPv4: martian source 74.50.109.32 from 74.50.109.32, on dev enp33s0f0
Mar 21 15:47:48 bmhv7 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 2b e4 c8 08 06
Mar 21 15:48:09 bmhv7 etcd[43768]: raft2023/03/21 15:48:09 INFO: 607474ecd5112191 [logterm: 21, index: 2714, vote: 0] ignored MsgVote from cfefa00537b3d384 [logterm: 21, index: 2714] at term 21: lease is not e
xpired (remaining ticks: 10)
Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: 607474ecd5112191 [term: 21] received a MsgVote message with higher term from af5ac16031361c6a [term: 22]
Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: 607474ecd5112191 became follower at term 22
Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: 607474ecd5112191 [logterm: 21, index: 2715, vote: 0] cast MsgVote for af5ac16031361c6a [logterm: 21, index: 2715] at term 22
Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: raft.node: 607474ecd5112191 lost leader cb7af46e9829ddc8 at term 22
Mar 21 15:48:10 bmhv7 etcd[43768]: raft2023/03/21 15:48:10 INFO: raft.node: 607474ecd5112191 elected leader af5ac16031361c6a at term 22
Mar 21 15:48:10 bmhv7 node[43861]: failed to query etcd: timeout
Mar 21 15:48:10 bmhv7 node[43861]: events.js:174
Mar 21 15:48:10 bmhv7 node[43861]: throw er; // Unhandled 'error' event
Mar 21 15:48:10 bmhv7 node[43861]: ^
Mar 21 15:48:10 bmhv7 node[43861]: Error: socket hang up
Message from syslogd@bmhv7 at Mar 21 15:48:29 ...
kernel:[ 3446.142595] NMI watchdog: Watchdog detected hard LOCKUP on cpu 14
Message from syslogd@bmhv7 at Mar 21 15:48:29 ...
kernel:[ 3454.797679] NMI watchdog: Watchdog detected hard LOCKUP on cpu 15
Message from syslogd@bmhv7 at Mar 21 15:48:29 ...
kernel:[ 3457.801254] NMI watchdog: Watchdog detected hard LOCKUP on cpu 31
Message from syslogd@bmhv7 at Mar 21 15:48:29 ...
kernel:[ 3460.575290] watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [kworker/17:2:49099]
Mar 21 15:48:10 bmhv7 node[43861]: at createHangUpError (_http_client.js:332:15)
Mar 21 15:48:10 bmhv7 node[43861]: at Socket.socketCloseListener (_http_client.js:373:25)
Mar 21 15:48:10 bmhv7 node[43861]: at Socket.emit (events.js:203:15)
Mar 21 15:48:10 bmhv7 node[43861]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:10 bmhv7 node[43861]: Emitted 'error' event at:
Mar 21 15:48:10 bmhv7 node[43861]: at Socket.socketCloseListener (_http_client.js:373:11)
Mar 21 15:48:10 bmhv7 node[43861]: at Socket.emit (events.js:203:15)
Mar 21 15:48:10 bmhv7 node[43861]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:10 bmhv7 etcd[43768]: time="2023-03-21T15:48:10Z" level=warning msg="error reading websocket message: websocket: close 1006 (abnormal closure): unexpected EOF"
Mar 21 15:48:10 bmhv7 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 15:48:10 bmhv7 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'.
Mar 21 15:48:14 bmhv7 etcd[43768]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream MsgApp v2 reader)
Mar 21 15:48:14 bmhv7 etcd[43768]: failed to read cfefa00537b3d384 on stream MsgApp v2 (read tcp 10.10.0.171:46512->10.10.0.164:2380: i/o timeout)
Mar 21 15:48:14 bmhv7 etcd[43768]: peer cfefa00537b3d384 became inactive (message send to peer failed)
Mar 21 15:48:14 bmhv7 etcd[43768]: lost the TCP streaming connection with peer cfefa00537b3d384 (stream Message reader)
Mar 21 15:48:20 bmhv7 systemd[1]: vitastor-mon.service: Scheduled restart job, restart counter is at 1.
Mar 21 15:48:20 bmhv7 systemd[1]: Stopped Vitastor monitor.
Mar 21 15:48:20 bmhv7 systemd[1]: Started Vitastor monitor.
Mar 21 15:48:21 bmhv7 node[1125603]: Waiting to become master
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 is starting a new election at term 22
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 became candidate at term 23
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 received MsgVoteResp from 607474ecd5112191 at term 23
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 [logterm: 22, index: 2732] sent MsgVote request to 46d9e5656b97850f at term 23
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 [logterm: 22, index: 2732] sent MsgVote request to af5ac16031361c6a at term 23
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 [logterm: 22, index: 2732] sent MsgVote request to cb7af46e9829ddc8 at term 23
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 [logterm: 22, index: 2732] sent MsgVote request to cfefa00537b3d384 at term 23
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: raft.node: 607474ecd5112191 lost leader af5ac16031361c6a at term 23
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 received MsgVoteResp from cb7af46e9829ddc8 at term 23
Mar 21 15:48:24 bmhv7 etcd[43768]: raft2023/03/21 15:48:24 INFO: 607474ecd5112191 has received 2 MsgVoteResp votes and 0 vote rejections
Mar 21 15:48:25 bmhv7 node[1125603]: failed to query etcd: timeout
Mar 21 15:48:25 bmhv7 node[1125603]: events.js:174
Mar 21 15:48:25 bmhv7 node[1125603]: throw er; // Unhandled 'error' event
Mar 21 15:48:25 bmhv7 node[1125603]: ^
Mar 21 15:48:25 bmhv7 node[1125603]: Error: socket hang up
Mar 21 15:48:25 bmhv7 node[1125603]: at createHangUpError (_http_client.js:332:15)
Mar 21 15:48:25 bmhv7 node[1125603]: at Socket.socketCloseListener (_http_client.js:373:25)
Mar 21 15:48:25 bmhv7 node[1125603]: at Socket.emit (events.js:203:15)
Mar 21 15:48:25 bmhv7 node[1125603]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:25 bmhv7 node[1125603]: Emitted 'error' event at:
Mar 21 15:48:25 bmhv7 node[1125603]: at Socket.socketCloseListener (_http_client.js:373:11)
Mar 21 15:48:25 bmhv7 node[1125603]: at Socket.emit (events.js:203:15)
Mar 21 15:48:25 bmhv7 node[1125603]: at TCP._handle.close (net.js:607:12)
Mar 21 15:48:25 bmhv7 systemd[1]: vitastor-mon.service: Main process exited, code=exited, status=1/FAILURE
Mar 21 15:48:25 bmhv7 systemd[1]: vitastor-mon.service: Failed with result 'exit-code'.
Mar 21 15:48:25 bmhv7 etcd[43768]: raft2023/03/21 15:48:25 INFO: 607474ecd5112191 [logterm: 22, index: 2732, vote: 607474ecd5112191] rejected MsgVote from af5ac16031361c6a [logterm: 22, index: 2732] at term 23
Mar 21 15:48:25 bmhv7 etcd[43768]: raft2023/03/21 15:48:25 INFO: 607474ecd5112191 [term: 23] received a MsgVote message with higher term from cb7af46e9829ddc8 [term: 24]
Mar 21 15:48:25 bmhv7 etcd[43768]: raft2023/03/21 15:48:25 INFO: 607474ecd5112191 became follower at term 24
Raw
bmhv9 journal (osd2)
Mar 21 15:42:52 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:42:57 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:02 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:07 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:12 bmhv9 etcd[1775114]: store.index: compact 1022
Mar 21 15:43:12 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:12 bmhv9 etcd[1775114]: finished scheduled compaction at 1022 (took 394.570871ms)
Mar 21 15:43:17 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:22 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:27 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:32 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:37 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:42 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:42 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:42 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:44 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:44 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:45 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:45 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:46 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:46 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:47 bmhv9 kernel: IPv4: martian source 74.50.109.39 from 74.50.109.39, on dev enp33s0f0
Mar 21 15:43:47 bmhv9 kernel: ll header: 00000000: ff ff ff ff ff ff a0 36 9f 8c 37 dc 08 06
Mar 21 15:43:47 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:52 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:43:57 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:02 bmhv9 systemd-logind[1052]: New session 25 of user ubuntu.
Mar 21 15:44:02 bmhv9 systemd[1]: Started Session 25 of user ubuntu.
Mar 21 15:44:02 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:07 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:12 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:17 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:22 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:27 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:29 bmhv9 systemd[1]: Started libcontainer container 7a789cf3d4a7555ff33c58ae0a8ef52fe660d8e5f77b32950e9be086d8f17ab7.
Mar 21 15:44:30 bmhv9 systemd[1]: Started libcontainer container 65817557137d7633369240e89e947d42aab40395d74c5c688de4f4534a4a8e79.
Mar 21 15:44:30 bmhv9 systemd[1]: Started libcontainer container a1a121ec70a2485f960acf4c8d34ebf1c2665ca6c86452b2e25fe9fba82b5764.
Mar 21 15:44:31 bmhv9 systemd[1]: Started libcontainer container d3b79c6588f80b976ed68b1fc21424ee1459e0273678034e5ee5bf2298b35680.
Mar 21 15:44:32 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:37 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:42 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:44 bmhv9 systemd[1]: Started libcontainer container 94708abb99d4d1299dee031d3134f81130fcfb602bb18d2c5db839da69cd0eac.
Mar 21 15:44:44 bmhv9 systemd[1]: Started libcontainer container 4586913ccc859a4f8ae495f6e23fcb1594e83d9923d3ab2ef33a7ac41c6fd723.
Mar 21 15:44:45 bmhv9 systemd[1]: Started libcontainer container ad8efd7bdc27bf272a06f5c11cce2bbbef8b1abb373d1481e8aa07c4ee297819.
Mar 21 15:44:47 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:44:52 bmhv9 node[1780565]: Waiting to become master
Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
kernel:[73646.799077] NMI watchdog: Watchdog detected hard LOCKUP on cpu 26
Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
kernel:[73647.598937] NMI watchdog: Watchdog detected hard LOCKUP on cpu 27
Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
kernel:[73659.545002] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kworker/10:3:1797192]
Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
kernel:[73659.557003] watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [migration/14:103]
Message from syslogd@bmhv9 at Mar 21 15:47:00 ...
kernel:[73659.561002] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [runc:1799309]
Mar 21 15:44:57 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:02 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:07 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:12 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:16 bmhv9 systemd[1]: Started libcontainer container 61aac3dabe59401178cb9e1b7df09f8a46e14ef055e4fd10a720cb0a269e7da3.
Mar 21 15:45:17 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:22 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:27 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:32 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:38 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:43 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:48 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:53 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:45:58 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:03 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:08 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:13 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:18 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:18 bmhv9 systemd[1]: Started libcontainer container a49a9db20d4aed7f8e06282534dd5c4ea6fec9fd5fc762e2a22faa23dc3e7455.
Mar 21 15:46:23 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:24 bmhv9 systemd[1]: Started libcontainer container 9c87575152ce6f679ea02b2cc6a2331e7be100d93995d35889b40c1783a71a3c.
Mar 21 15:46:28 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:33 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:38 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:43 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:48 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:53 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:46:58 bmhv9 node[1780565]: Waiting to become master
Mar 21 15:47:00 bmhv9 kernel: NMI watchdog: Watchdog detected hard LOCKUP on cpu 26
Mar 21 15:47:00 bmhv9 kernel: Modules linked in: msr xfs xt_recent vhost_net vhost vhost_iotlb tap xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 ip_set xt_CT cls_bpf algif_hash af_alg sch_ingress vxlan ip6_udp_tunne
l udp_tunnel veth xfrm_user xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6table_raw iptable_raw tls nf_conntrack_netlink xt_statistic xt_nat xt_addrtype ipt_REJECT nf_reject_ipv4 xt_tcpudp nf_tables nfnetlink ip_
vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6table_filter ip6table_nat ip6_tables iptable_mangle xt_MASQUERADE xt_conntrack xt_comment iptable_filter xt_mark iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 libcrc32c bpfilter ib_core aufs binfmt_misc ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd ast kvm_amd drm_vram_helper kvm joydev input_leds crct10dif_pclmul drm_ttm_helper
nls_iso8859_1 ghash_clmulni_intel ttm aesni_intel drm_kms_helper crypto_simd cryptd cls_u32 cec rc_core rapl sch_prio fb_sys_fops syscopyarea sysfillrect sysimgblt ccp k10temp
Mar 21 15:47:00 bmhv9 kernel: acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mac_hid tcp_bbr sch_fq_codel br_netfilter bridge stp llc overlay ramoops reed_solomon pstore_blk pstore_zone efi_pstore drm ip_tabl
es x_tables autofs4 hid_generic usbhid hid ixgbe igb nvme xfrm_algo i2c_algo_bit ahci nvme_core mdio crc32_pclmul xhci_pci libahci i2c_piix4 xhci_pci_renesas dca
Mar 21 15:47:00 bmhv9 kernel: CPU: 26 PID: 550435 Comm: virt-handler Not tainted 5.15.0-67-generic #74~20.04.1-Ubuntu
Mar 21 15:47:00 bmhv9 kernel: Hardware name: Supermicro Super Server/H11SSL-i, BIOS 1.3 06/25/2019
Mar 21 15:47:00 bmhv9 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x79/0x230
Mar 21 15:47:00 bmhv9 kernel: Code: 7c f0 0f ba 2b 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 03 30 e4 09 d0 a9 00 01 ff ff 75 56 85 c0 74 0e 8b 03 84 c0 74 08 f3 90 <8b> 03 84 c0 75 f8 b8 01 00 00 00 66 89 03 48
83 c4 08 5b 41 5c 41
Mar 21 15:47:00 bmhv9 kernel: RSP: 0018:ffffb2b046950c20 EFLAGS: 00000002
Mar 21 15:47:00 bmhv9 kernel: RAX: 0000000000700101 RBX: ffff94d8ccefc7d8 RCX: ffff94dee18e7d28
Mar 21 15:47:00 bmhv9 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff94d8ccefc7d8
Mar 21 15:47:00 bmhv9 kernel: RBP: ffffb2b046950c48 R08: 0000000000000002 R09: 0000000000000064
Mar 21 15:47:00 bmhv9 kernel: R10: ffff94dee1c00600 R11: 0000000000000002 R12: ffffb2b046950cb0
Mar 21 15:47:00 bmhv9 kernel: R13: ffff94d8ccefc7d8 R14: ffff94ef7bc48710 R15: 0000000000000001
Mar 21 15:47:00 bmhv9 kernel: FS: 00007f8b78ff9700(0000) GS:ffff94eebfd80000(0000) knlGS:0000000000000000
Mar 21 15:47:00 bmhv9 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 21 15:47:00 bmhv9 kernel: CR2: 00007fed55bcc620 CR3: 0000001048fda000 CR4: 00000000003506e0
Mar 21 15:47:00 bmhv9 kernel: Call Trace:
Mar 21 15:47:00 bmhv9 kernel: <IRQ>
Mar 21 15:47:00 bmhv9 kernel: _raw_spin_lock+0x22/0x30
Mar 21 15:47:00 bmhv9 kernel: take_dentry_name_snapshot+0x20/0x80
Mar 21 15:47:00 bmhv9 kernel: __fsnotify_parent+0x1d6/0x310
Mar 21 15:47:00 bmhv9 kernel: ? native_smp_send_reschedule+0x2a/0x50
Mar 21 15:47:00 bmhv9 kernel: ? mempool_free_slab+0x17/0x20
Mar 21 15:47:00 bmhv9 kernel: __io_complete_rw_common+0x132/0x180
Mar 21 15:47:00 bmhv9 kernel: ? __io_complete_rw_common+0x132/0x180
Mar 21 15:47:00 bmhv9 kernel: io_complete_rw+0x17/0x50
Mar 21 15:47:00 bmhv9 kernel: blkdev_bio_end_io+0x8c/0x110
Mar 21 15:47:00 bmhv9 kernel: bio_endio+0x106/0x170
Mar 21 15:47:00 bmhv9 kernel: blk_update_request+0x249/0x4d0
Mar 21 15:47:00 bmhv9 kernel: blk_mq_end_request+0x21/0x140
Mar 21 15:47:00 bmhv9 kernel: nvme_complete_rq+0x80/0x260 [nvme_core]
Mar 21 15:47:00 bmhv9 kernel: nvme_pci_complete_rq+0x82/0xa0 [nvme]
Mar 21 15:47:00 bmhv9 kernel: nvme_irq+0x15f/0x250 [nvme]
Mar 21 15:47:00 bmhv9 kernel: __handle_irq_event_percpu+0x42/0x170
Mar 21 15:47:00 bmhv9 kernel: handle_irq_event_percpu+0x33/0x80
Mar 21 15:47:00 bmhv9 kernel: handle_irq_event+0x3b/0x60
Mar 21 15:47:00 bmhv9 kernel: handle_edge_irq+0x93/0x1c0
Mar 21 15:47:00 bmhv9 kernel: __common_interrupt+0x52/0xe0
Mar 21 15:47:00 bmhv9 kernel: common_interrupt+0x89/0xa0
Mar 21 15:47:00 bmhv9 kernel: </IRQ>
Mar 21 15:47:00 bmhv9 kernel: <TASK>
Mar 21 15:47:00 bmhv9 kernel: asm_common_interrupt+0x27/0x40
Mar 21 15:47:00 bmhv9 kernel: RIP: 0010:_raw_spin_lock+0x10/0x30
Mar 21 15:47:00 bmhv9 kernel: Code: 02 00 00 e8 f2 d0 34 ff 5d e9 0c bc 28 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 e9 e9 bb 28 00 55 89 c6 48 89 e5 e8 2e
16 3c ff 66 90 5d e9
Mar 21 15:47:00 bmhv9 kernel: RSP: 0018:ffffb2b066a3fb58 EFLAGS: 00000246
Mar 21 15:47:00 bmhv9 kernel: RAX: 0000000000000000 RBX: ffff94d8ccefc788 RCX: 0000000000000009
Mar 21 15:47:00 bmhv9 kernel: RDX: 0000000000000001 RSI: ffffb2b066a3fc90 RDI: ffff94d8ccefc7d8
Mar 21 15:47:00 bmhv9 kernel: RBP: ffffb2b066a3fb98 R08: 0000000000000000 R09: 61c8864680b583eb
Mar 21 15:47:00 bmhv9 kernel: R10: 8080808080807fff R11: 8080808080808080 R12: ffff94d8ccefc7d8
Mar 21 15:47:00 bmhv9 kernel: R13: ffff94dee1c00600 R14: 0000000014ce31d5 R15: ffffb2b066a3fdd8
Mar 21 15:47:00 bmhv9 kernel: ? __d_lookup+0x64/0x150
Mar 21 15:47:00 bmhv9 kernel: lookup_fast+0xbb/0x160
Mar 21 15:47:00 bmhv9 kernel: walk_component+0x43/0x1c0
Mar 21 15:47:00 bmhv9 kernel: ? path_init+0x2b2/0x390
Mar 21 15:47:00 bmhv9 kernel: path_lookupat.isra.0+0x6e/0x150
Mar 21 15:47:00 bmhv9 kernel: filename_lookup+0xcf/0x1a0
Mar 21 15:47:00 bmhv9 kernel: ? path_get+0x27/0x30
Mar 21 15:47:00 bmhv9 kernel: user_path_at_empty+0x3f/0x60
Mar 21 15:47:00 bmhv9 kernel: vfs_statx+0x73/0x110
Mar 21 15:47:00 bmhv9 kernel: __do_sys_newfstatat+0x36/0x70
Mar 21 15:47:00 bmhv9 kernel: ? syscall_exit_to_user_mode+0x27/0x50
Mar 21 15:47:00 bmhv9 kernel: ? syscall_trace_enter.isra.0+0x14e/0x1c0
Mar 21 15:47:00 bmhv9 kernel: __x64_sys_newfstatat+0x1e/0x30
Mar 21 15:47:00 bmhv9 kernel: do_syscall_64+0x5c/0xc0
Mar 21 15:47:00 bmhv9 kernel: ? do_syscall_64+0x69/0xc0
Mar 21 15:47:00 bmhv9 kernel: ? do_syscall_64+0x69/0xc0
Mar 21 15:47:00 bmhv9 kernel: ? irqentry_exit_to_user_mode+0x9/0x20
Mar 21 15:47:00 bmhv9 kernel: ? irqentry_exit+0x1d/0x30
Mar 21 15:47:00 bmhv9 kernel: ? common_interrupt+0x55/0xa0
Mar 21 15:47:00 bmhv9 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xcb
Mar 21 15:47:00 bmhv9 kernel: RIP: 0033:0x47ff2a
Mar 21 15:47:00 bmhv9 kernel: Code: e8 fb b6 fe ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 4c 8b 54 24 28 4c 8b 44 24 30 4c 8b 4c 24 38 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 40 ff ff
ff ff 48 c7 44 24 48
Mar 21 15:47:00 bmhv9 kernel: RSP: 002b:000000c00298db10 EFLAGS: 00000202 ORIG_RAX: 0000000000000106
Mar 21 15:47:00 bmhv9 kernel: RAX: ffffffffffffffda RBX: 000000c000067800 RCX: 000000000047ff2a
Mar 21 15:47:00 bmhv9 kernel: RDX: 000000c003c209f8 RSI: 000000c003c192a0 RDI: ffffffffffffff9c
Mar 21 15:47:00 bmhv9 kernel: RBP: 000000c00298dba0 R08: 0000000000000000 R09: 0000000000000000
Mar 21 15:47:00 bmhv9 kernel: R10: 0000000000000100 R11: 0000000000000202 R12: 000000c003c192a0
Mar 21 15:47:00 bmhv9 kernel: R13: 0000000000000001 R14: 000000c001df11e0 R15: ffffffffffffffff
Mar 21 15:47:00 bmhv9 kernel: </TASK>
Mar 21 15:47:00 bmhv9 kernel: NMI watchdog: Watchdog detected hard LOCKUP on cpu 27
Mar 21 15:47:00 bmhv9 kernel: Modules linked in: msr xfs xt_recent vhost_net vhost vhost_iotlb tap xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 ip_set xt_CT cls_bpf algif_hash af_alg sch_ingress vxlan ip6_udp_tunne
l udp_tunnel veth xfrm_user xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6table_raw iptable_raw tls nf_conntrack_netlink xt_statistic xt_nat xt_addrtype ipt_REJECT nf_reject_ipv4 xt_tcpudp nf_tables nfnetlink ip_
vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6table_filter ip6table_nat ip6_tables iptable_mangle xt_MASQUERADE xt_conntrack xt_comment iptable_filter xt_mark iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 libcrc32c bpfilter ib_core aufs binfmt_misc ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd ast kvm_amd drm_vram_helper kvm joydev input_leds crct10dif_pclmul drm_ttm_helper
nls_iso8859_1 ghash_clmulni_intel ttm aesni_intel drm_kms_helper crypto_simd cryptd cls_u32 cec rc_core rapl sch_prio fb_sys_fops syscopyarea sysfillrect sysimgblt ccp k10temp
Mar 21 15:47:00 bmhv9 kernel: acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mac_hid tcp_bbr sch_fq_codel br_netfilter bridge stp llc overlay ramoops reed_solomon pstore_blk pstore_zone efi_pstore drm ip_tabl
es x_tables autofs4 hid_generic usbhid hid ixgbe igb nvme xfrm_algo i2c_algo_bit ahci nvme_core mdio crc32_pclmul xhci_pci libahci i2c_piix4 xhci_pci_renesas dca
Mar 21 15:47:00 bmhv9 kernel: CPU: 27 PID: 0 Comm: swapper/27 Not tainted 5.15.0-67-generic #74~20.04.1-Ubuntu
Mar 21 15:47:00 bmhv9 kernel: Hardware name: Supermicro Super Server/H11SSL-i, BIOS 1.3 06/25/2019
Mar 21 15:47:00 bmhv9 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x159/0x230
Mar 21 15:47:00 bmhv9 kernel: Code: 5d 41 5e 5d e9 78 a4 ec 00 83 c6 01 c1 e2 10 c1 e6 12 09 f2 89 d0 c1 e8 10 66 87 43 02 89 c1 c1 e1 10 75 53 31 c9 eb 02 f3 90 <8b> 03 66 85 c0 75 f7 89 c6 66 31 f6 39 f2 0f
84 93 00 00 00 c6 03
Mar 21 15:47:00 bmhv9 kernel: RSP: 0018:ffffb2b046984c20 EFLAGS: 00000002
Mar 21 15:47:00 bmhv9 kernel: RAX: 0000000000700101 RBX: ffff94d8ccefc7d8 RCX: 0000000000000000
Mar 21 15:47:00 bmhv9 kernel: RDX: 0000000000700000 RSI: 0000000000700000 RDI: ffff94d8ccefc7d8
Mar 21 15:47:00 bmhv9 kernel: RBP: ffffb2b046984c48 R08: 0000000000000002 R09: 0000000000000064
Mar 21 15:47:00 bmhv9 kernel: R10: ffff94dee1c00600 R11: 0000000000000002 R12: ffff94eebfdf1900
Mar 21 15:47:00 bmhv9 kernel: R13: ffff94d8ccefc7d8 R14: ffff94ef7bc48710 R15: 0000000000000001
Mar 21 15:47:00 bmhv9 kernel: FS: 0000000000000000(0000) GS:ffff94eebfdc0000(0000) knlGS:0000000000000000
Mar 21 15:47:00 bmhv9 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 21 15:47:00 bmhv9 kernel: CR2: 000055cd2403b0e0 CR3: 000000183196e000 CR4: 00000000003506e0
Mar 21 15:47:00 bmhv9 kernel: Call Trace:
Mar 21 15:47:00 bmhv9 kernel: <IRQ>
Mar 21 15:47:00 bmhv9 kernel: _raw_spin_lock+0x22/0x30
Mar 21 15:47:00 bmhv9 kernel: take_dentry_name_snapshot+0x20/0x80
Mar 21 15:47:00 bmhv9 kernel: __fsnotify_parent+0x1d6/0x310
Mar 21 15:47:00 bmhv9 kernel: ? trace_event_raw_event_block_rq_remap+0xb1/0x120
Mar 21 15:47:00 bmhv9 kernel: ? mempool_free_slab+0x17/0x20
Mar 21 15:47:00 bmhv9 kernel: __io_complete_rw_common+0x132/0x180
Mar 21 15:47:00 bmhv9 kernel: ? __io_complete_rw_common+0x132/0x180
Mar 21 15:47:00 bmhv9 kernel: io_complete_rw+0x17/0x50
Mar 21 15:47:00 bmhv9 kernel: blkdev_bio_end_io+0x8c/0x110
Mar 21 15:47:00 bmhv9 kernel: bio_endio+0x106/0x170
Mar 21 15:47:00 bmhv9 kernel: blk_update_request+0x249/0x4d0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment