Path: ...!weretis.net!feeder9.news.weretis.net!newsfeed.endofthelinebbs.com!.POSTED.47.186.30.161!not-for-mail
From: Nigel Reed <sysop@endofthelinebbs.com>
Newsgroups: news.software.nntp
Subject: Re: Compacting CNFS buffers
Date: Wed, 8 May 2024 00:43:09 -0500
Organization: End Of The Line BBS
Message-ID: <20240508004309.53a4572b@wibble.sysadmininc.com>
References: <20240504210746.79036ffe@wibble.sysadmininc.com>
	<v1cuu2$qiqi$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Injection-Info: www.sysadmininc.com; posting-host="47.186.30.161";
	logging-data="2045923"; mail-complaints-to="abuse@endofthelinebbs.com"
X-Newsreader: Claws Mail 4.2.0git6 (GTK 3.24.33; x86_64-pc-linux-gnu)
Bytes: 4043
Lines: 77

On Tue, 7 May 2024 12:14:26 +0200
Julien =C3=89LIE <iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Nigel,
>=20
> > Has anyone investigated the feasibility of compacting or compressing
> > the cnfs buffer files? =20
>=20
> Some people use ZFS to compress CNFS buffers (cancelled articles are=20
> still present though).  I am not aware of a compaction feature like
> the one you want.

I am using ZFS with CNFS and it does a good job. I also want to use the
server for other purposes so reclaiming any space would be extremely
useful.

> > If you can find where an expired article is on disk and then find
> > the next article, you can just move it on disk and update the
> > pointers to the file. This could be a process that you just kick off
> > or, preferably, something that runs when innd isn't fully occupied
> > using spare cycles or something. =20
> I understand your point; I can add it to the wish list.

That would be good.=20
>=20
> > 1. You are sent a bunch of articles but discover you've left some
> > binary newsgroups in your active file. You put this groups in your
> > expire list and delete rmgroup but you're left with a lot of empty
> > space, never to be used again unless the buffer recycles. =20
>=20
> You may want to configure Cleanfeed to reject binaries (including in=20
> binary groups) so as not to store them and waste space.  Since a few=20
> weeks, NoCeM notices have also been sent for misplaced binaries (in=20
> non-binary groups).

Unfortunately the articles are already in the CFS buffers. My bad for
forgetting to remove some binary groups from the active file. I did not
have cleanfeed running when importing since it's advised to turn off
perl and python filtering.

> > 2. You receive a bunch of googlegroup spam articles that are deleted
> > via NOCEM, however considering there are so many, that leaves a lot
> > of unused space. =20
>=20
> Christoph Biedl implemented a new feature for INN 2.7.2 to store=20
> articles by their Path header field.  It is a new "path" option in=20
> storage.conf.  A typical use case is to store articles from a spammy=20
> site in a small CNFS buffer to avoid overall retention impacts.

I'll look into it, but again, the damage is already done.

>=20
> There's also the delayer program (in the contrib directory before INN=20
> 2.7.2) that you can use to delay articles, and give cancel control=20
> articles and NoCeM messages time to arrive.  For instance, by having
> a frontend instance of innd receiving the articles from all your
> peers and another local instance of innd fed by your frontend with a
> delay except for cancels and NoCeM articles.  The CNFS buffers of
> that second instance will be spam free.
>    https://www.eyrie.org/~eagle/software/inn/docs/delayer.html
>=20

Sounds interesting but, again, I already have a lot of binary articles.
I'm not sure I want to set up a second server. I have a hard enough
time with one :)

I'll hold out hope someone with more knowledge than I also sees the
issue and decides to look into compacting CNFS buffers.

Thanks,
Nigel


--=20
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23