135 lines
5.4 KiB
HTML
135 lines
5.4 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
|
||
<html xmlns="http://www.w3.org/1999/xhtml"><head>
|
||
<meta http-equiv="Content-type" content="text/html;charset=UTF-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||
<link rel="stylesheet" href="main.css" type="text/css" />
|
||
<link rel="stylesheet" href="blog.css" type="text/css" />
|
||
<link rel="alternate" type="application/rss+xml" title="Subscribe to this page..." href="feed.rss" />
|
||
<title>Scrubbing my data - BTRFS</title>
|
||
</head><body>
|
||
<div id="divbodyholder">
|
||
<div class="headerholder"><div class="header">
|
||
<div id="title">
|
||
<h1 class="nomargin"><a class="ablack" href="http://zigford.org/index.html">zigford.org</a></h1>
|
||
<div id="description"><a href="about.html">About</a><a href="links.html"> | Links</a><a href="scripts.html"> | Scripts</a><br>Sharing linux/windows scripts and tips</br></div>
|
||
</div></div></div>
|
||
<div id="divbody"><div class="content">
|
||
<!-- entry begin -->
|
||
<h3><a class="ablack" href="scrubbing-my-data---btrfs.html">
|
||
Scrubbing my data - BTRFS
|
||
</a></h3>
|
||
<!-- bashblog_timestamp: #202004232149.01# -->
|
||
<div class="subtitle">April 23, 2020 —
|
||
Jesse Harris
|
||
</div>
|
||
<!-- text begin -->
|
||
|
||
<p>It's no secret I use BTRFS, so I have a fair amount of data stored on this
|
||
filesystem. With most popular filesystems you have no way of knowing if your
|
||
data is the same read as was originally written. A few modern filesystems
|
||
support a function known as scrubbing.</p>
|
||
|
||
<hr />
|
||
|
||
<p>Scrubbing is:</p>
|
||
|
||
<blockquote>
|
||
<p>btrfs scrub is used to scrub a btrfs filesystem, which will read all
|
||
data and metadata blocks from all devices and verify checksums.
|
||
Automatically repair corrupted blocks if there’s a correct copy
|
||
available.</p>
|
||
</blockquote>
|
||
|
||
<p>On the most common filesystems<sup class="footnote-ref" id="fnref-regfilesystems"><a href="#fn-regfilesystems">1</a></sup> data is written and read but
|
||
never validated. Between writing and reading data, sometimes the data can get
|
||
changed. This could occur due to a media fault or firmware bug or user error.
|
||
(dd the wrong volume anyone?)</p>
|
||
|
||
<p>Before this technology was available, it was always a worry that my most
|
||
precious stored family photos and videos could at any time become silently
|
||
corrupted without my knowing!</p>
|
||
|
||
<p>Now thanks to BTRFS (or ZFS if your more into the BSDs) you never have to
|
||
suffer not knowing about data corruption, and as a bonus you can prevent it too.
|
||
(if you have mirrored volumes)</p>
|
||
|
||
<h2>My BTRFS Volumes</h2>
|
||
|
||
<p>I currently have the following volumes in BTRFS:</p>
|
||
|
||
<ul>
|
||
<li>root sd card of the Raspberry Pi 4 hosting this site 32 gigs</li>
|
||
<li>7Tb external spinning disk attached to the RPI4</li>
|
||
<li>256Gb ssd boot volume on my main desktop PC</li>
|
||
<li>2Tb ssd home volume on my main desktop PC</li>
|
||
<li>3Tb + 2Tb external spinning disks connected via USB3 to my main desktop PC</li>
|
||
<li>500Gb nvme luks encrypted volume on precision 5510</li>
|
||
<li>1Tb external luks encrypted backup 2.5" USB2 sometimes attached to Precision
|
||
5510</li>
|
||
</ul>
|
||
|
||
<h2>Backing them up</h2>
|
||
|
||
<p>All of these systems have their root/home volumes backed up at least once via
|
||
BTRFS send and in the case of the desktop PC, it also performs daily snapshots
|
||
with restic to a cloud storage system.</p>
|
||
|
||
<p>RPI:</p>
|
||
|
||
<p>Daily snapshots sent to it's locally attached 7Tb volume
|
||
Occasionally I <code>btrfs send</code> the 7Tb volume to the 3+2 spanned external disks on
|
||
the PC. I'm not too fussed if I loose the 7Tb volume. It is mainly backup and
|
||
some unimportant media.</p>
|
||
|
||
<p>PC:</p>
|
||
|
||
<p>I wrote a <a href="https://github.com/zigford/linux-worktools/blob/master/linux/snapshot">little bash
|
||
script</a>
|
||
to btrfs snapshot and send incrementally backups every hour for the last 24
|
||
hours, day for the last month and monthly until it gets to a % full. These go to
|
||
the 2+3 Tb external disks. Restic performs cloud backups</p>
|
||
|
||
<p>Precision:</p>
|
||
|
||
<p>Thee same bash script on a systemd timer, sends backups to my external disk when
|
||
it's connected</p>
|
||
|
||
<h2>Scrubbing it all</h2>
|
||
|
||
<p>Not long ago I was running my home volumes on mirrored spinning disks. In the
|
||
modern era these were beginning to feel slow and one drive started failing.
|
||
BTRFS scrub would fix errors for me then:</p>
|
||
|
||
<p><img src="images/btrfs1.png" alt="screenshot" /></p>
|
||
|
||
<p>You can see that a few errors were corrected because another copy of the block
|
||
was available. The whole 800+Gb of data took an hour 44 to scrub. Compare that
|
||
to today:</p>
|
||
|
||
<p><img src="images/btrfsscrub.png" alt="screenshot" /></p>
|
||
|
||
<p>In this screenshot the bottom left terminal is a scrub of my new 2Tb ssd.
|
||
Bottom right is 256Gb boot volume.
|
||
Top left is a snapshot of a backup of the 7Tb volume on the 2+3Tb spanning
|
||
disks.
|
||
And top right is the 7Tb volume itself connected to the RPI4.</p>
|
||
|
||
<p>Tags: <a href='tag_btrfs.html'>btrfs</a>, <a href='tag_backup.html'>backup</a></p>
|
||
<div class="footnotes">
|
||
<hr />
|
||
<ol>
|
||
<li id="fn-regfilesystems">
|
||
<p>Ext2, Ext3, Ext4, XFS, NTFS, APFS, HFS+ <a href="#fnref-regfilesystems" class="footnoteBackLink" title="Jump back to footnote 1 in the text.">↩</a></p>
|
||
</li>
|
||
</ol>
|
||
</div>
|
||
|
||
|
||
<!-- text end -->
|
||
<!-- entry end -->
|
||
</div>
|
||
<div id="footer">© <a href="http://twitter.com/zigford_org">Jesse Harris</a> — <a href="mailto:jesse@zigford.org">jesse@zigford.org</a><br/>
|
||
Generated with <a href="https://github.com/cfenollosa/bashblog">bashblog</a>, a single bash script to easily create blogs like this one</div>
|
||
</div></div>
|
||
</body></html>
|