Date: Tue, 10 Sep 2024 22:35:44 +0100
Content-Transfer-Encoding: 7bit
From: D <noreply@mixmin.net>
References: <database-20240909114248@ram.dialup.fu-berlin.de>
Subject: Re: Post DB
Message-Id: <20240910.223544.bd3c64d8@mixmin.net>
Newsgroups: news.software.readers
Path: ...!weretis.net!feeder8.news.weretis.net!news.mixmin.net!news2.arglkargh.de!alphared!sewer!news.dizum.net!not-for-mail
Organization: dizum.com - The Internet Problem Provider
X-Abuse: abuse@dizum.com
Injection-Info: sewer.dizum.com - 2001::1/128
Bytes: 5840
Lines: 109

On 9 Sep 2024 10:44:36 GMT, ram@zedat.fu-berlin.de (Stefan Ram) wrote:
>  I'm not 100% sure I'm barking up the right tree (newsgroup) here,
>  but whatever.
>  So, yesterday I was chewing the fat about how to whip up a database
>  for posts retrieved from newsservers.
>  I'm picturing some program that pulls newsgroups from newsservers
>  and dumps them into a database.
>  In my mind's eye, a post looks something like this, give or take:
>Path: A
>Message-ID: B
>Body: C
>  . But if you snag the same post from a different server, it might
>  look like this:
>Message-ID: B
>Path: D
>Body: C
>  . At first blush, you'd end up with the same body stored multiple
>  times in the database. Talk about a waste of space!
>  To trim the fat, we could rejigger these posts so all the variable
>  stuff is up front:
>Path: A
>Message-ID: B
>Body: C
>  and
>Path: D
>Message-ID: B
>Body: C
>  Now the tail end of both posts is identical, so we can toss that
>  in a separate table at position 0.
>  The posts themselves would then just contain the different parts
>  and a pointer to the shared bit that's only stored once:
>Path: A
>Rest: 0
>Path: D
>Rest: 0
>0:
>Message-ID: B
>Body: C
>  . This way, you could store the same post from multiple newsservers
>  without eating up your hard drive space like it's In-N-Out fries.

p.s.  this feature of 40tude dialog might possibly be useful...

40tude Dialog > Help [F1] > The "Copy/Move selected articles" Window:
>...
>How can I create virtual groups?
>You can't. There is nothing like virtual groups in Dialog. However,
>you can join the articles from two groups into one in Dialog. This
>includes joining the articles from the same newsgroup but different
>servers into one group (in the perspective of Dialog) or folder,
>e.g. to fill missing parts of multipart postings. To do this, you
>can manually Copy/Move articles from one group to another or you
>can use an action rule to automate this. 

40tude Dialog > Help [F1] > Index > Scoring syntax:
>...
>Copy and Move actions
>There is an extended syntax for copy and move action. Say you want to move
>all emails from a mailing list to a separate folder. You can use:
> [email.*]
> !move(%X-Mailing-List%) Header {^(X-Mailing-List:)}
>The section identifier here says that the following rule should be applied
>to emails only.
>The !move action itself looks for a X-Mailing-List: header in the message,
>but instead of copying the message to a folder with a fixed name, it is
>copied to a folder named after the value of the X-Mailing-List header
>field itself. Say, e.g. you receive a message which has a "X-Mailing-List:
>speedboats@yahoogroups.com" header, then this message is moved to the
>"speedboats@yahoogroups.com" folder. By now, you might have noticed that
>this is a very powerful rule for organizing mailing list into folders.
>A variation of this is:
> [email.*]
> !move(@%X-Mailing-List%) Header {^(X-Mailing-List:)}
>Notice the additional "@" in the target folder expression. If this @ is
>present, the value of the header field is simply searched for the
>character @ and only the text before the first found @ is used as the
>target folder. Again, for a message which has a "X-Mailing-List:
>speedboats@yahoogroups.com" header field, the message would be moved to the
>"speedboats" folder, instead of the "speedboats@yahoogroups.com" folder.
>Almost virtual groups
>There are no virtual groups in Dialog, however you can simply copy or move
>articles from one group to another manually or by using an action rule:
> [a.binary.group]
> !move(a.binary.group;NewsserverOne) bytes %>0
>All previous examples used a folder as the target of the copy/move
>operation. This rule uses a newsgroup as the target. Note that the name of
>the target group and the name of the newsserver where this group is from
>is separated by a semicolon.
>Say, you are using two newsservers named "NewsserverOne" and
>"NewsserverTwo". Both newsservers have the group "a.binary.group", but
>while most of the messages are the same in this group on the two servers,
>there are some that are not available on the other server.
>The first line in the preceding sample makes sure that the following rule
>is only applied to the "a.binary.group" newsgroup.
>The rule itself simply says to move all articles (since the condition
>"bytes %>0" is always met) to the group a.binary.group of NewsserverOne.
>If you retrieve message headers for this group from server NewsserverOne,
>the rule is ignored, since all articles go to the correct group anyway. If
>you retrieve message headers from NewsserverTwo for this group though,
>they will not show up in "a.binary.group (NewsserverTwo)", but in
>"a.binary.group (NewsserverOne)".
>The result is that you joined messages from the same group, but different
>newsservers into one group in Dialog. This is especially useful for binary
>groups to fill missing multipart postings.
[end quoted plain text]

see also: news:de.comm.software.40tude-dialog for many detailed articles
(45160 total, from 8/18/2003 to 8/19/2024, on news.blueworldhosting.com)
about scoring, actions, scripting, etc.  this tech group is still active