Skip to content

Instantly share code, notes, and snippets.

@johnp789
Last active January 3, 2016 19:59
Show Gist options
  • Save johnp789/8512045 to your computer and use it in GitHub Desktop.
Save johnp789/8512045 to your computer and use it in GitHub Desktop.
rsync write to F2FS hangs
$ uname -a
Linux storm 3.12.7-2-ARCH #1 SMP PREEMPT Sun Jan 12 13:09:09 CET 2014 x86_64 GNU/Linux
$ ps aux | grep rsync
root 1032 0.0 0.0 60324 3772 pts/0 D 16:23 0:00 rsync -aAX --info=progress2 /home/john/Documents/scratch/root/bin /home/john/Documents/scratch/root/boot /home/john/Documents/scratch/root/dev /home/john/Documents/scratch/root/etc /home/john/Documents/scratch/root/home /home/john/Documents/scratch/root/lib /home/john/Documents/scratch/root/mnt /home/john/Documents/scratch/root/opt /home/john/Documents/scratch/root/proc /home/john/Documents/scratch/root/root /home/john/Documents/scratch/root/run /home/john/Documents/scratch/root/sbin /home/john/Documents/scratch/root/srv /home/john/Documents/scratch/root/sys /home/john/Documents/scratch/root/tmp /home/john/Documents/scratch/root/usr /home/john/Documents/scratch/root/var .
root 1033 0.6 0.0 0 0 pts/0 Z 16:23 0:02 [rsync] <defunct>
john 1430 0.0 0.0 10668 1060 pts/0 S+ 16:29 0:00 grep rsync
in journal:
Jan 19 16:18:37 storm kernel: nvidia: module license 'NVIDIA' taints kernel.
Jan 19 16:18:37 storm kernel: Disabling lock debugging due to kernel taint
[...]
Jan 19 16:26:35 storm kernel: INFO: task kworker/u16:5:109 blocked for more than 120 seconds.
Jan 19 16:26:35 storm kernel: Tainted: P O 3.12.7-2-ARCH #1
Jan 19 16:26:35 storm kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 16:26:35 storm kernel: kworker/u16:5 D 0000000000000246 0 109 2 0x00000000
Jan 19 16:26:35 storm kernel: Workqueue: writeback bdi_writeback_workfn (flush-8:80)
Jan 19 16:26:35 storm kernel: ffff880406a799b0 0000000000000046 00000000000144c0 ffff880406a79fd8
Jan 19 16:26:35 storm kernel: ffff880406a79fd8 00000000000144c0 ffff880406f3f0f0 ffff880037f73000
Jan 19 16:26:35 storm kernel: ffff880406a799a8 ffffffff8128cdad 0000000099ea002f ffff880037ca0840
Jan 19 16:26:35 storm kernel: Call Trace:
Jan 19 16:26:35 storm kernel: [<ffffffff8128cdad>] ? cfq_insert_request+0x1ed/0x560
Jan 19 16:26:35 storm kernel: [<ffffffff8126733e>] ? elv_set_request+0x1e/0x30
Jan 19 16:26:35 storm kernel: [<ffffffff813841c7>] ? put_device+0x17/0x20
Jan 19 16:26:35 storm kernel: [<ffffffffa00cfae9>] ? scsi_request_fn+0xa9/0x4a0 [scsi_mod]
Jan 19 16:26:35 storm kernel: [<ffffffff814f16e9>] schedule+0x29/0x70
Jan 19 16:26:35 storm kernel: [<ffffffff814f1b23>] schedule_preempt_disabled+0x23/0x30
Jan 19 16:26:35 storm kernel: [<ffffffff814f0058>] __mutex_lock_slowpath+0x158/0x3d0
Jan 19 16:26:35 storm kernel: [<ffffffff814f02e2>] mutex_lock+0x12/0x30
Jan 19 16:26:35 storm kernel: [<ffffffffa004f226>] write_checkpoint+0xf6/0xaf0 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffff814f1b82>] ? preempt_schedule_irq+0x52/0x80
Jan 19 16:26:35 storm kernel: [<ffffffffa0051eb0>] ? f2fs_read_data_page+0x20/0x20 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa004ccb9>] f2fs_sync_fs+0xe9/0x100 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa00565fd>] f2fs_write_node_pages+0xdd/0xf0 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffff81140ade>] do_writepages+0x1e/0x40
Jan 19 16:26:35 storm kernel: [<ffffffff811cced0>] __writeback_single_inode+0x40/0x330
Jan 19 16:26:35 storm kernel: [<ffffffff811cdf22>] writeback_sb_inodes+0x262/0x450
Jan 19 16:26:35 storm kernel: [<ffffffff811ce1af>] __writeback_inodes_wb+0x9f/0xd0
Jan 19 16:26:35 storm kernel: [<ffffffff811ce50b>] wb_writeback+0x32b/0x360
Jan 19 16:26:35 storm kernel: [<ffffffff811cedae>] bdi_writeback_workfn+0x32e/0x520
Jan 19 16:26:35 storm kernel: [<ffffffff8107cb67>] process_one_work+0x167/0x450
Jan 19 16:26:35 storm kernel: [<ffffffff8107d561>] worker_thread+0x121/0x3a0
Jan 19 16:26:35 storm kernel: [<ffffffff8107d440>] ? manage_workers.isra.23+0x2a0/0x2a0
Jan 19 16:26:35 storm kernel: [<ffffffff81083f60>] kthread+0xc0/0xd0
Jan 19 16:26:35 storm kernel: [<ffffffff81083ea0>] ? kthread_create_on_node+0x120/0x120
Jan 19 16:26:35 storm kernel: [<ffffffff814fab7c>] ret_from_fork+0x7c/0xb0
Jan 19 16:26:35 storm kernel: [<ffffffff81083ea0>] ? kthread_create_on_node+0x120/0x120
Jan 19 16:26:35 storm kernel: INFO: task rsync:1032 blocked for more than 120 seconds.
Jan 19 16:26:35 storm kernel: Tainted: P O 3.12.7-2-ARCH #1
Jan 19 16:26:35 storm kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 16:26:35 storm kernel: rsync D ffff880382fa1b80 0 1032 1031 0x00000000
Jan 19 16:26:35 storm kernel: ffff880382fa1af0 0000000000000082 00000000000144c0 ffff880382fa1fd8
Jan 19 16:26:35 storm kernel: ffff880382fa1fd8 00000000000144c0 ffff8803fed656e0 0000000000000000
Jan 19 16:26:35 storm kernel: 00000000ffffffff 0000004100000000 ffff88041efeab08 0000000000000010
Jan 19 16:26:35 storm kernel: Call Trace:
Jan 19 16:26:35 storm kernel: [<ffffffff8113d918>] ? __alloc_pages_nodemask+0x158/0xad0
Jan 19 16:26:35 storm kernel: [<ffffffff8101c4c9>] ? read_tsc+0x9/0x20
Jan 19 16:26:35 storm kernel: [<ffffffff811344d0>] ? filemap_fdatawait+0x30/0x30
Jan 19 16:26:35 storm kernel: [<ffffffff814f16e9>] schedule+0x29/0x70
Jan 19 16:26:35 storm kernel: [<ffffffff814f198f>] io_schedule+0x8f/0xe0
Jan 19 16:26:35 storm kernel: [<ffffffff811344de>] sleep_on_page+0xe/0x20
Jan 19 16:26:35 storm kernel: [<ffffffff814ee0fd>] __wait_on_bit_lock+0x5d/0xc0
Jan 19 16:26:35 storm kernel: [<ffffffff8119896e>] ? __mem_cgroup_commit_charge+0xce/0x3f0
Jan 19 16:26:35 storm kernel: [<ffffffff8113461a>] __lock_page+0x6a/0x70
Jan 19 16:26:35 storm kernel: [<ffffffff81084e00>] ? wake_atomic_t_function+0x40/0x40
Jan 19 16:26:35 storm kernel: [<ffffffff81134770>] find_lock_page+0x50/0x70
Jan 19 16:26:35 storm kernel: [<ffffffff811351b4>] find_or_create_page+0x34/0x90
Jan 19 16:26:35 storm kernel: [<ffffffffa00547d9>] get_node_page+0x39/0x170 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa0048924>] update_inode_page+0x24/0x70 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa0054bc8>] sync_inode_page+0xa8/0xc0 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa0055f30>] new_node_page+0x2b0/0x2d0 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa005e555>] f2fs_setxattr+0x475/0x680 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa005eb3a>] f2fs_set_acl+0x2ba/0x370 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa005f2f8>] f2fs_init_acl+0x138/0x160 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa0045a35>] __f2fs_add_link+0x275/0x800 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffffa004992c>] f2fs_mkdir+0x10c/0x190 [f2fs]
Jan 19 16:26:35 storm kernel: [<ffffffff8123ef00>] ? security_inode_rename+0x30/0x40
Jan 19 16:26:35 storm kernel: [<ffffffff811b00b0>] vfs_mkdir+0xe0/0x170
Jan 19 16:26:35 storm kernel: [<ffffffff811b455a>] SyS_mkdirat+0xaa/0xe0
Jan 19 16:26:35 storm kernel: [<ffffffff811b45a9>] SyS_mkdir+0x19/0x20
Jan 19 16:26:35 storm kernel: [<ffffffff814fac2d>] system_call_fastpath+0x1a/0x1f
$ ./f3write /run/media/john/bbbroot/foo
Free space: 13.95 GB
Creating file 1.fff ... OK!
Creating file 2.fff ... OK!
Creating file 3.fff ... OK!
Creating file 4.fff ... OK!
Creating file 5.fff ... OK!
Creating file 6.fff ... OK!
Creating file 7.fff ... OK!
Creating file 8.fff ... OK!
Creating file 9.fff ... OK!
Creating file 10.fff ... OK!
Creating file 11.fff ... OK!
Creating file 12.fff ... OK!
Creating file 13.fff ... OK!
Creating file 14.fff ... OK!
Free space: 107.50 MB
Average writing speed: 5.19 MB/s
$ ./f3read /run/media/john/bbbroot/foo
SECTORS ok/corrupted/changed/overwritten
Validating file 1.fff ... 2097152/ 0/ 0/ 0
Validating file 2.fff ... 2097152/ 0/ 0/ 0
Validating file 3.fff ... 2097152/ 0/ 0/ 0
Validating file 4.fff ... 2097152/ 0/ 0/ 0
Validating file 5.fff ... 2097152/ 0/ 0/ 0
Validating file 6.fff ... 2097152/ 0/ 0/ 0
Validating file 7.fff ... 2097152/ 0/ 0/ 0
Validating file 8.fff ... 2097152/ 0/ 0/ 0
Validating file 9.fff ... 2097152/ 0/ 0/ 0
Validating file 10.fff ... 2097152/ 0/ 0/ 0
Validating file 11.fff ... 2097152/ 0/ 0/ 0
Validating file 12.fff ... 2097152/ 0/ 0/ 0
Validating file 13.fff ... 2097152/ 0/ 0/ 0
Validating file 14.fff ... 1744872/ 0/ 0/ 0
Data OK: 13.83 GB (29007848 sectors)
Data LOST: 0.00 Byte (0 sectors)
Corrupted: 0.00 Byte (0 sectors)
Slightly changed: 0.00 Byte (0 sectors)
Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 25.12 MB/s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment