Article ID: 123248, created on Oct 24, 2014, last review on Oct 24, 2014

  • Applies to:
  • Virtuozzo 6.0

Symptoms

Chunk Server is switching from Active to Inactive and back, e.g:

# pstorage -c mycluster get-event | less
...
12-06-14 09:49:47  MDS WRN CS#1061 is inactive
12-06-14 09:49:48  MDS INF CS#1061 is active
12-06-14 10:49:22  MDS WRN CS#1064 is inactive
12-06-14 10:49:23  MDS INF CS#1064 is active
12-06-14 13:21:02  MDS WRN CS#1063 is inactive
12-06-14 13:21:03  MDS INF CS#1063 is active
12-06-14 15:39:47  MDS WRN CS#1061 is inactive
12-06-14 15:39:48  MDS INF CS#1061 is active
12-06-14 15:56:57  MDS WRN CS#1061 is inactive
12-06-14 15:56:58  MDS INF CS#1061 is active
12-06-14 16:20:34  MDS WRN CS#1061 is inactive
...

The following stack can be found the the corresponding CS log:

...
12-06-14 16:31:35.599 pcs process is inactive for 5002 msecs (0)
12-06-14 16:31:35.599 [<ffffffffa00900ad>] do_get_write_access+0x29d/0x510 [jbd2]
12-06-14 16:31:35.600 [<ffffffffa0090471>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
12-06-14 16:31:35.600 [<ffffffffa00e2718>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
12-06-14 16:31:35.600 [<ffffffffa00b8bc3>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
12-06-14 16:31:35.600 [<ffffffffa00b8c3c>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
12-06-14 16:31:35.600 [<ffffffffa00b8f30>] ext4_dirty_inode+0x40/0x60 [ext4]
12-06-14 16:31:35.600 [<ffffffff811da89b>] __mark_inode_dirty+0x3b/0x190
12-06-14 16:31:35.600 [<ffffffff811c91ba>] file_update_time+0x10a/0x1a0
12-06-14 16:31:35.600 [<ffffffff81135ca4>] __generic_file_write_iter+0x1f4/0x420
12-06-14 16:31:35.600 [<ffffffff81135f55>] __generic_file_aio_write+0x85/0xa0
12-06-14 16:31:35.600 [<ffffffff81135ff8>] generic_file_aio_write+0x88/0x100
12-06-14 16:31:35.600 [<ffffffffa00b21d8>] ext4_file_write+0x58/0x190 [ext4]
12-06-14 16:31:35.600 [<ffffffff811fb64b>] aio_rw_vect_retry+0xbb/0x220
12-06-14 16:31:35.600 [<ffffffff811fde14>] aio_run_iocb+0x64/0x170
12-06-14 16:31:35.600 [<ffffffff811fe85c>] do_io_submit+0x2bc/0x670
12-06-14 16:31:35.600 [<ffffffff811fec20>] sys_io_submit+0x10/0x20
12-06-14 16:31:35.600 [<ffffffff8100b102>] system_call_fastpath+0x16/0x1b
12-06-14 16:31:35.600 [<ffffffffffffffff>] 0xffffffffffffffff
...

Cause

The problem is recognized as product bug with ID PSBM-20411.

Resolution

The fix will be included in one of the future product updates.

Search Words

cs going inactive in circles

active inactive

Chunk server timeouts

blinking

chunkserver inactive

c62e8726973f80975db0531f1ed5c6a2 2897d76d56d2010f4e3a28f864d69223 0dd5b9380c7d4884d77587f3eb0fa8ef

Email subscription for changes to this article
Save as PDF