Bug 1037847

Summary: btrfs goes read only with fs/btrfs/extent-tree.c:2956 btrfs_run_delayed_refs+0x282/0x2e0 BTRFS: Transaction aborted (error -28)
Product: [openSUSE] openSUSE Distribution Reporter: Stefan Priebe <s.priebe>
Component: KernelAssignee: Jeff Mahoney <jeffm>
Status: RESOLVED WONTFIX QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P2 - High CC: fdmanana, s.priebe, tiwai
Version: Leap 42.2   
Target Milestone: ---   
Hardware: x86-64   
OS: All   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Bug Depends on: 1034635    
Bug Blocks:    

Description Stefan Priebe 2017-05-05 19:11:22 UTC
Hello,

while running a git based kernel build from openSUSE-42.2 commit id 328b08be8a2f8a1d09c27e5f8038f9c7446669c3.

I'm no longer able to write more than 1 to 2 hours of data to my disk.

# dmesg:
------------[ cut here ]------------
WARNING: CPU: 5 PID: 6589 at fs/btrfs/extent-tree.c:2956 btrfs_run_delayed_refs+0x282/0x2e0 [btrfs]()
BTRFS: Transaction aborted (error -28)
Modules linked in: netconsole xt_multiport ipt_REJECT nf_reject_ipv4 xt_set iptable_filter ip_tables x_tables ip_set_hash_net ip_set nfnetlink 8021q garp bonding ipmi_devintf x86_pkg_temp_thermal coretemp ast kvm_intel ttm kvm drm_kms_helper irqbypass drm crc32_pclmul ghash_clmulni_intel sb_edac fb_sys_fops syscopyarea sysfillrect sysimgblt edac_core i40e(O) vxlan ip6_udp_tunnel udp_tunnel shpchp wmi ipmi_si ipmi_msghandler button loop btrfs dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid0 multipath linear usbhid raid1 md_mod sg sd_mod xhci_pci ehci_pci xhci_hcd ehci_hcd i2c_i801 ahci igb usbcore i2c_algo_bit usb_common libahci i2c_core ptp pps_core aacraid
CPU: 5 PID: 6589 Comm: btrfs-transacti Tainted: G        W  O     4.4.65+132-ph #1
Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015
 0000000000000000 ffff880a7847bbb8 ffffffff813c23f3 ffff880a7847bc00
 ffffffffc037114d ffff880a7847bbf0 ffffffff81083746 ffff880ff32f23c0
 ffff880cac6c2000 ffff880fb4ee6060 ffffffffffffffff ffff880cac6c2000
Call Trace:
 [<ffffffff81008b45>] try_stack_unwind+0x175/0x190
 [<ffffffff81007d24>] dump_trace+0x64/0x380
 [<ffffffff81008bad>] show_trace_log_lvl+0x4d/0x60
 [<ffffffff81008148>] show_stack_log_lvl+0x108/0x170
 [<ffffffff81008c05>] show_stack+0x25/0x70
 [<ffffffff813c23f3>] dump_stack+0x63/0x90
 [<ffffffff81083746>] warn_slowpath_common+0x86/0xc0
 [<ffffffff810837cc>] warn_slowpath_fmt+0x4c/0x50
 [<ffffffffc02d6322>] btrfs_run_delayed_refs+0x282/0x2e0 [btrfs]
 [<ffffffffc02ebdd8>] create_pending_snapshot.isra.23+0x5b8/0x1010 [btrfs]
 [<ffffffffc02ec8ba>] create_pending_snapshots+0x8a/0xb0 [btrfs]
 [<ffffffffc02ecf11>] btrfs_commit_transaction.part.24+0x371/0xa30 [btrfs]
 [<ffffffffc02ed60a>] btrfs_commit_transaction+0x3a/0x70 [btrfs]
 [<ffffffffc02e78df>] transaction_kthread+0x1ff/0x260 [btrfs]
 [<ffffffff810a0f5b>] kthread+0xeb/0x110
 [<ffffffff816bdf0f>] ret_from_fork+0x3f/0x70
DWARF2 unwinder stuck at ret_from_fork+0x3f/0x70

Leftover inexact backtrace:

 [<ffffffff810a0e70>] ? kthread_park+0x60/0x60
---[ end trace 457d5c0a7792677e ]---
BTRFS: error (device sdc1) in btrfs_run_delayed_refs:2956: errno=-28 No space left
BTRFS info (device sdc1): forced readonly
BTRFS: error (device sdc1) in create_pending_snapshot:1603: errno=-28 No space left
BTRFS warning (device sdc1): Skipping commit of aborted transaction.
BTRFS: error (device sdc1) in cleanup_transaction:1852: errno=-28 No space left
BTRFS error (device sdc1): pending csums is 383705088


I i umount the device i get a new trace:
# umount /backup
# dmesg -t
BTRFS error (device sdc1): cleaner transaction attach returned -30
------------[ cut here ]------------
WARNING: CPU: 2 PID: 12105 at fs/btrfs/extent-tree.c:133 btrfs_put_block_group+0x60/0x70 [btrfs]()
Modules linked in: netconsole xt_multiport ipt_REJECT nf_reject_ipv4 xt_set iptable_filter ip_tables x_tables ip_set_hash_net ip_set nfnetlink 8021q garp bonding ipmi_devintf x86_pkg_temp_thermal coretemp ast kvm_intel ttm kvm drm_kms_helper irqbypass drm crc32_pclmul ghash_clmulni_intel sb_edac fb_sys_fops syscopyarea sysfillrect sysimgblt edac_core i40e(O) vxlan ip6_udp_tunnel udp_tunnel shpchp wmi ipmi_si ipmi_msghandler button loop btrfs dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid0 multipath linear usbhid raid1 md_mod sg sd_mod xhci_pci ehci_pci xhci_hcd ehci_hcd i2c_i801 ahci igb usbcore i2c_algo_bit usb_common libahci i2c_core ptp pps_core aacraid
CPU: 2 PID: 12105 Comm: umount Tainted: G        W  O     4.4.65+132-ph #1
Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015
 0000000000000000 ffff880753a13ce0 ffffffff813c23f3 0000000000000000
 ffffffffc037114d ffff880753a13d18 ffffffff81083746 ffff881030d25000
 ffff881030d25000 ffff88103c3c4088 ffff88103c3c4000 ffff88103c3c4098
Call Trace:
 [<ffffffff81008b45>] try_stack_unwind+0x175/0x190
 [<ffffffff81007d24>] dump_trace+0x64/0x380
 [<ffffffff81008bad>] show_trace_log_lvl+0x4d/0x60
 [<ffffffff81008148>] show_stack_log_lvl+0x108/0x170
 [<ffffffff81008c05>] show_stack+0x25/0x70
 [<ffffffff813c23f3>] dump_stack+0x63/0x90
 [<ffffffff81083746>] warn_slowpath_common+0x86/0xc0
 [<ffffffff8108383a>] warn_slowpath_null+0x1a/0x20
 [<ffffffffc02cd670>] btrfs_put_block_group+0x60/0x70 [btrfs]
 [<ffffffffc02d5358>] btrfs_free_block_groups+0xf8/0x430 [btrfs]
 [<ffffffffc02e7a8d>] close_ctree+0x14d/0x320 [btrfs]
 [<ffffffffc02b6429>] btrfs_put_super+0x19/0x20 [btrfs]
 [<ffffffff811d2aff>] generic_shutdown_super+0x6f/0x100
 [<ffffffff811d2ba2>] kill_anon_super+0x12/0x20
 [<ffffffffc02b71b6>] btrfs_kill_super+0x16/0x90 [btrfs]
 [<ffffffff811d24d3>] deactivate_locked_super+0x43/0x70
 [<ffffffff811d255c>] deactivate_super+0x5c/0x60
 [<ffffffff811ee23f>] cleanup_mnt+0x3f/0x90
 [<ffffffff811ee2d2>] __cleanup_mnt+0x12/0x20
 [<ffffffff8109f51e>] task_work_run+0x7e/0xa0
 [<ffffffff81002383>] exit_to_usermode_loop+0xf3/0x100
 [<ffffffff81002bb4>] syscall_return_slowpath+0xd4/0x130
 [<ffffffff816bdd0c>] int_ret_from_sys_call+0x25/0x8f
DWARF2 unwinder stuck at int_ret_from_sys_call+0x25/0x8f

Leftover inexact backtrace:

---[ end trace 457d5c0a7792677f ]---
------------[ cut here ]------------
WARNING: CPU: 2 PID: 12105 at fs/btrfs/extent-tree.c:9818 btrfs_free_block_groups+0x291/0x430 [btrfs]()
Modules linked in: netconsole xt_multiport ipt_REJECT nf_reject_ipv4 xt_set iptable_filter ip_tables x_tables ip_set_hash_net ip_set nfnetlink 8021q garp bonding ipmi_devintf x86_pkg_temp_thermal coretemp ast kvm_intel ttm kvm drm_kms_helper irqbypass drm crc32_pclmul ghash_clmulni_intel sb_edac fb_sys_fops syscopyarea sysfillrect sysimgblt edac_core i40e(O) vxlan ip6_udp_tunnel udp_tunnel shpchp wmi ipmi_si ipmi_msghandler button loop btrfs dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid0 multipath linear usbhid raid1 md_mod sg sd_mod xhci_pci ehci_pci xhci_hcd ehci_hcd i2c_i801 ahci igb usbcore i2c_algo_bit usb_common libahci i2c_core ptp pps_core aacraid
CPU: 2 PID: 12105 Comm: umount Tainted: G        W  O     4.4.65+132-ph #1
Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015
 0000000000000000 ffff880753a13cf8 ffffffff813c23f3 0000000000000000
 ffffffffc037114d ffff880753a13d30 ffffffff81083746 ffff880f0feb2200
 ffff880f0feb2288 ffff88103c3c4088 ffff88103c3c4000 ffff88103c3c4098
Call Trace:
 [<ffffffff81008b45>] try_stack_unwind+0x175/0x190
 [<ffffffff81007d24>] dump_trace+0x64/0x380
 [<ffffffff81008bad>] show_trace_log_lvl+0x4d/0x60
 [<ffffffff81008148>] show_stack_log_lvl+0x108/0x170
 [<ffffffff81008c05>] show_stack+0x25/0x70
 [<ffffffff813c23f3>] dump_stack+0x63/0x90
 [<ffffffff81083746>] warn_slowpath_common+0x86/0xc0
 [<ffffffff8108383a>] warn_slowpath_null+0x1a/0x20
 [<ffffffffc02d54f1>] btrfs_free_block_groups+0x291/0x430 [btrfs]
 [<ffffffffc02e7a8d>] close_ctree+0x14d/0x320 [btrfs]
 [<ffffffffc02b6429>] btrfs_put_super+0x19/0x20 [btrfs]
 [<ffffffff811d2aff>] generic_shutdown_super+0x6f/0x100
 [<ffffffff811d2ba2>] kill_anon_super+0x12/0x20
 [<ffffffffc02b71b6>] btrfs_kill_super+0x16/0x90 [btrfs]
 [<ffffffff811d24d3>] deactivate_locked_super+0x43/0x70
 [<ffffffff811d255c>] deactivate_super+0x5c/0x60
 [<ffffffff811ee23f>] cleanup_mnt+0x3f/0x90
 [<ffffffff811ee2d2>] __cleanup_mnt+0x12/0x20
 [<ffffffff8109f51e>] task_work_run+0x7e/0xa0
 [<ffffffff81002383>] exit_to_usermode_loop+0xf3/0x100
 [<ffffffff81002bb4>] syscall_return_slowpath+0xd4/0x130
 [<ffffffff816bdd0c>] int_ret_from_sys_call+0x25/0x8f
DWARF2 unwinder stuck at int_ret_from_sys_call+0x25/0x8f

Leftover inexact backtrace:

---[ end trace 457d5c0a77926780 ]---
BTRFS info (device sdc1): space_info 1 has 144388956160 free, is not full
BTRFS info (device sdc1): space_info total=21839908700160, used=21695519477760, pinned=0, reserved=4096, may_use=0, readonly=262144

# btrfs filesystem df /vmbackup/ 
Data, RAID0: total=19.86TiB, used=19.72TiB
System, RAID0: total=64.00MiB, used=1.23MiB
Metadata, RAID0: total=76.00GiB, used=72.62GiB
GlobalReserve, single: total=512.00MiB, used=6.16MiB

# btrfs filesystem usage /vmbackup/
Overall:
    Device size:                  58.20TiB
    Device allocated:             19.94TiB
    Device unallocated:           38.26TiB
    Device missing:                  0.00B
    Used:                         19.79TiB
    Free (estimated):             38.41TiB      (min: 38.41TiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 213.95MiB)

Data,RAID0: Size:19.86TiB, Used:19.72TiB
   /dev/sdc1       4.97TiB
   /dev/sdd1       4.97TiB
   /dev/sde1       4.97TiB
   /dev/sdf1       4.97TiB

Metadata,RAID0: Size:76.00GiB, Used:72.61GiB
   /dev/sdc1      19.00GiB
   /dev/sdd1      19.00GiB
   /dev/sde1      19.00GiB
   /dev/sdf1      19.00GiB

System,RAID0: Size:64.00MiB, Used:1.23MiB
   /dev/sdc1      16.00MiB
   /dev/sdd1      16.00MiB
   /dev/sde1      16.00MiB
   /dev/sdf1      16.00MiB

Unallocated:
   /dev/sdc1       9.57TiB
   /dev/sdd1       9.57TiB
   /dev/sde1       9.57TiB
   /dev/sdf1       9.57TiB


I already tried to clear the space cache but it happens again after some time.

Greets,
Stefan
Comment 1 Tomáš Chvátal 2018-04-17 14:09:14 UTC
This is automated batch bugzilla cleanup.

The openSUSE 42.2 changed to end-of-life (EOL [1]) status. As such
it is no longer maintained, which means that it will not receive any
further security or bug fix updates.
As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
openSUSE, or you can still observe it under openSUSE Leap 15.0, please
feel free to reopen this bug against that version (see the "Version"
component in the bug fields), or alternatively open
a new ticket.

Thank you for reporting this bug and we are sorry it could not be fixed
during the lifetime of the release.

[1] https://en.opensuse.org/Lifetime