Path: ...!eternal-september.org!feeder2.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: Phillip Frabott <nntp@fulltermprivacy.com>
Newsgroups: comp.os.linux.misc
Subject: Re: Named pipes vs. Unix sockets
Date: Fri, 22 Nov 2024 10:00:31 -0500
Organization: A noiseless patient Spider
Lines: 43
Message-ID: <vhq6ag$176f0$1@dont-email.me>
References: <Sp-cnSz8UupYQaf6nZ2dnZfqnPednZ2d@earthlink.com>
 <vhf5ts$16rpr$1@dont-email.me> <vhfjtj$19ijm$1@dont-email.me>
 <vhja6j$23f5e$4@dont-email.me> <vhmm4c$hnbj$1@dont-email.me>
 <vhmn2t$hv8i$3@dont-email.me> <vhnikj$me7m$1@dont-email.me>
 <wwvcyio8m9p.fsf@LkoBDZeT.terraraq.uk> <vhorrb$t48o$1@dont-email.me>
 <vhosra$1171f$1@dont-email.me> <lqalg1F7fi9U2@mid.individual.net>
 <vhp8qi$12m83$2@dont-email.me> <lqaoknF8btnU2@mid.individual.net>
 <lqaq6cF8btnU3@mid.individual.net> <lqau7sF8btnU4@mid.individual.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Fri, 22 Nov 2024 16:00:33 +0100 (CET)
Injection-Info: dont-email.me; posting-host="e284d08f11627a26d6d133d7fb8219a0";
	logging-data="1284576"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX19gVh5jBHUIlgI6qmk9oYuTjeOYOdDCGcg="
User-Agent: Mozilla Thunderbird
Cancel-Lock: sha1:yu8xS3EnhWVOSri8JeA6jxnkHS0=
In-Reply-To: <lqau7sF8btnU4@mid.individual.net>
Content-Language: en-US
Bytes: 3621

On 11/22/2024 03:38, vallor wrote:
> On 22 Nov 2024 07:29:16 GMT, vallor <vallor@cultnix.org> wrote in
> <lqaq6cF8btnU3@mid.individual.net>:
> 
>> However, the speed appears to be limited by dd in my examples -- setting
>> a block size to fill the pipe/packets seems to increase throughput:
>>
>> $ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero
>> count=$[1024*1024*4] bs=1024 | nc -U -N /tmp/socket > /dev/null )
> 
> Realized the bottleneck would be the pipe between dd and nc, so wrote
> a program to connect to /tmp/socket and spew data at it -- it sends
> 46950 212992-byte buffers (9999974400 bytes) in 2.41 seconds.
> (4149366971 bytes/second, or 4.1GB/s).
> 
> (The default "MTU" for a Linux Unix socket connection
> is 212992 bytes.  Default pipe size is 8*512 bytes.)
> 

Yeah, this has been the experience of our testing as well. We can also 
bump the MTU up a bit to get sockets to do better and we've also 
modified some of the socket code to our environment so it's able to do 
around 12GB/s. We think we can tune it further to get about 15GB/s from 
sockets. At the moment we have to do 4.6GB per 0.5sec to keep up, and 
7GB per 0.5sec to stay ahead. We've done everything we can to squeeze 
out as much performance from pipes as we could but there just isn't much 
we can do anymore. But yeah, sockets are way more capable of large data 
intake vs pipes, and this shows as well in a lot of research that's been 
done. We've been testing sockets for about the last year and the results 
have shown we are keeping up better then we are with pipes right now.

And for tribute to pipes, we've been using pipes in this project for the 
last 30 years. That's a pretty great track record for something to have 
been useful for such a long time and not be replaced with something else 
until now. Sadly sockets will likely bottom out in the next 10-15 years 
and we'll have to switch to something else. But for now, it's working.

-- 
Phillip Frabott
----------
- Adam: Is a void really a void if it returns?
- Jack: No, it's just nullspace at that point.
----------