Path: ...!eternal-september.org!feeder2.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: Phillip Frabott <nntp@fulltermprivacy.com>
Newsgroups: comp.os.linux.misc
Subject: Re: Named pipes vs. Unix sockets
Date: Fri, 22 Nov 2024 17:06:06 -0500
Organization: A noiseless patient Spider
Lines: 41
Message-ID: <vhqv8f$1br87$1@dont-email.me>
References: <Sp-cnSz8UupYQaf6nZ2dnZfqnPednZ2d@earthlink.com>
 <vhf5ts$16rpr$1@dont-email.me> <vhfjtj$19ijm$1@dont-email.me>
 <vhja6j$23f5e$4@dont-email.me> <vhmm4c$hnbj$1@dont-email.me>
 <vhmn2t$hv8i$3@dont-email.me> <vhnikj$me7m$1@dont-email.me>
 <wwvcyio8m9p.fsf@LkoBDZeT.terraraq.uk> <vhorrb$t48o$1@dont-email.me>
 <vhosra$1171f$1@dont-email.me> <lqalg1F7fi9U2@mid.individual.net>
 <vhp8qi$12m83$2@dont-email.me> <lqaoknF8btnU2@mid.individual.net>
 <lqaq6cF8btnU3@mid.individual.net> <lqau7sF8btnU4@mid.individual.net>
 <vhq6ag$176f0$1@dont-email.me> <vhqqu0$1b2t7$6@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Fri, 22 Nov 2024 23:06:08 +0100 (CET)
Injection-Info: dont-email.me; posting-host="e284d08f11627a26d6d133d7fb8219a0";
	logging-data="1436935"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1/seT9ZrdoMUSLUXyeZreNA/V0eLNchV4c="
User-Agent: Mozilla Thunderbird
Cancel-Lock: sha1:o9pkQObGJT9yn2+9KdYg2hDK/ig=
Content-Language: en-US
In-Reply-To: <vhqqu0$1b2t7$6@dont-email.me>
Bytes: 3642

On 11/22/2024 15:52, Lawrence D'Oliveiro wrote:
> On Fri, 22 Nov 2024 10:00:31 -0500, Phillip Frabott wrote:
> 
>> At the moment we have to do 4.6GB per 0.5sec to keep up, and
>> 7GB per 0.5sec to stay ahead. We've done everything we can to squeeze
>> out as much performance from pipes as we could but there just isn't much
>> we can do anymore.
> 
> Sounds like a job for a shared memory section.

That only works if the entire dataset originates on, and results within, 
the same system. Unfortunately this is not the case so it wouldn't be 
possible. This is an intermediate system between the intake and final 
processing systems which are further down the line.

Unless you can recommend a way to do shared memory across different 
machines without extra hardware. Currently we ingest data from a 
(external) /dev (unbuffered data stream) into a process on the machine 
that buffers the data (currently though an unnamed pipe, which as I said 
before is in the process of being replaced with UNIX socket) then that 
data pipes to another process (again, being replaced with UNIX socket 
soon) which converts the data stream from the buffer to block data and 
then categorizes those blocks based on data types and depending on those 
data types sends it to one of 4 /dev devices that leave the machine. The 
bottleneck is not between the first and second process (buffer process 
and stream to block data processor) it's between the intake /dev to the 
intermediate and the intermediate to the 4 outbound /devs. These /dev 
are not networks so there is no TCP data. They are hard lined between 
the /dev machine and the intermediate computer and the intermediate 
computer to the 4 other /dev machines. So how would I share memory 
between /dev machines from the 80s-90s and the intermediate computer 
without using any network or TCP/UDP capabilities?

(If it was not apparent. these are manufacturing machines.)

-- 
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------