Path: ...!news.snarked.org!xmission!nnrp.xmission!.POSTED.shell.xmission.com!not-for-mail
From: legalize+jeeves@mail.xmission.com (Richard)
Newsgroups: news.software.nntp
Subject: Improving trn: LISTGROUP with no arguments vs. ...?
Date: Mon, 4 Nov 2024 23:16:17 -0000 (UTC)
Organization: multi-cellular, biological
Sender: legalize+jeeves@mail.xmission.com
Message-ID: <vgbkk1$30c1$1@news.xmission.com>
Reply-To: (Richard) legalize+jeeves@mail.xmission.com
Injection-Date: Mon, 4 Nov 2024 23:16:17 -0000 (UTC)
Injection-Info: news.xmission.com; posting-host="shell.xmission.com:2607:fa18:0:beef::4";
	logging-data="98689"; mail-complaints-to="abuse@xmission.com"
X-Reply-Etiquette: No copy by email, please
Mail-Copies-To: never
X-Newsreader: trn 4.0-test77 (Sep 1, 2010)
Originator: legalize@shell.xmission.com (Richard)
Bytes: 2883
Lines: 41

[Please do not mail me a copy of your followup]

Hi All,

Currently trn4 does a LISTGROUP command with no arguments when
entering a newsgroup.  When entering a newsgroup with many messages
(e.g. a large difference between low and high article numbers), this
can be very slow.  (See <https://github.com/LegalizeAdulthood/trn/issues/9>)

LISTGROUP can take low and high article number arguments, and my initial
thought was that trn should issue LISTGROUP with arguments based on
the user's newsrc file to build a list of article numbers representing
unread articles.

Then I wondered about cross-reference headers, which mention articles
by number, and I wondered if the reason trn was fetching all article
numbers was in order to prefetch headers for every article in case it
was mentioned in a cross-reference.

Using strace, I see that trn will start fetching overview data for
(seemingly all the listed) articles while paused for user input.

trn is quite old code and doesn't even use the older NNTP extension,
nevermind the newer extensions one could expect from a more modern
server like inn.

For reference the existing code that takes a long time is here:
<https://github.com/LegalizeAdulthood/trn/blob/a70daddff78d08e0d2c97739009376ad29094905/libtrn/bits.cpp#L252>
It's not the algorithm that is slow per se, but asking inn for all the
article numbers in a group with thousands of articles takes quite a
long time.  I suspect that inn doesn't have any sort of in-memory
cache for this information and instead is directory scanning the spool
directory and reporting back the article numbers one by one.

What's the best alternative to LISTGROUP with no args?

Thanks
-- 
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
            The Terminals Wiki <http://terminals-wiki.org>
     The Computer Graphics Museum <http://computergraphicsmuseum.org>
  Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>