Path: ...!news.nobody.at!news.mb-net.net!open-news-network.org!news.gegeweb.eu!gegeweb.org!news.ortolo.eu!.POSTED.localhost!not-for-mail
From: Tanguy Ortolo <tanguy@ortolo.eu>
Newsgroups: news.software.nntp
Subject: Switching INN 2 storage format
Date: Tue, 11 Feb 2025 15:11:54 -0000 (UTC)
Sender: tanguy@localhost
Message-ID: <vofpbq$pmf$1@herbert.ortolo.eu>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 11 Feb 2025 15:11:54 -0000 (UTC)
Injection-Info: herbert.ortolo.eu; posting-account="tanguy"; posting-host="localhost:::1";
	logging-data="26319"; mail-complaints-to="usenet@ortolo.eu"
Summary: I want to switch INN 2 storage from timecaf to timehash. IIRC,
 this can be done /for new articles/ by /adding/ the new storage backend
 to storage.conf, can it not?
Keywords: news, INN, storage, backend, method, timecaf, timehash,
 switch, migrate, btrfs, filesystem, CoW, copy-on-write
User-Agent: tin/2.6.2-20221225 ("Pittyvaich") (Linux/6.12.9+bpo-amd64 (x86_64))
Cancel-Lock: sha1:loH2kvu7eEMJKdgKpmWcT5oAul8=
Bytes: 3335
Lines: 50

Hello all,

My news server running INN 2 is storing all articles to a timecaf. I am
currently in the process of switching my file systems to btrfs (mainly
to get bitrot detection, see below for more details about that).

I do not expect timecaf and a CoW filesystem such as btrfs to play well
together. Indeed, writing a new small article to a large file should
inevitably fragment it. To avoid that, I disabled copy-on-write for the
timecaf, but that also disables file extent checksuming and therefore
bitrot detection, defeating the main reason I am switching to btrfs in
the first place.

Therefore, I am considering switching to a news storage format more
suitable with a CoW filesystem. I think the best option would be
timehash, any thoughts on that?

As I understand it, switching storage format for /new/ articles can
simply be done by simply adding the new storage backend in first
position in storage.conf, can it not?

I do not plan on migrating existing articles, but simply to wait for
them to expire since there does not seem to exist any simple migration
procedure. But if someone knows a way to do so, I could be interested.
;-)

Thanks for reading!


For those interested, I have two SSDs that are set up in a software
RAID1 with Linux LVM lvmraid(7). This is very flexible and will support
a drive failure… but not bitrot. Indeed, bitrot is detected by the
LVM /scrubbing/ process, but it will not know which drive has the
unaltered data (if any).

Linux LVM RAID has an optional integrity layer that can identify
corrupted data, but while it does fix it on-the-fly by querying the
other drive, it does not report which drive is altering data. And it
disables volume snapshotting.

After doing some research, it seems btrfs does fix all this, since it
maintains data checksums, and its scrubbing process does update
per-drive error counters. 

Btrfshat checksuming works with its copy-on-write design. In practice,
disabling CoW on some files, or even on an entire filesystem, also
disables checksuming. Therefore, I am looking for a way to store news
that would work well with btrfs' copy-on-write. :-)

-- 
Tanguy