Path: ...!news.misty.com!2.eu.feeder.erje.net!feeder.erje.net!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: David Brown <david.brown@hesbynett.no>
Newsgroups: comp.lang.c
Subject: Re: xxd -i vs DIY Was: C23 thoughts and opinions
Date: Thu, 30 May 2024 15:05:17 +0200
Organization: A noiseless patient Spider
Lines: 102
Message-ID: <v39tie$1mt3k$1@dont-email.me>
References: <v2l828$18v7f$1@dont-email.me>
 <00297443-2fee-48d4-81a0-9ff6ae6481e4@gmail.com>
 <v2lji1$1bbcp$1@dont-email.me> <87msoh5uh6.fsf@nosuchdomain.example.com>
 <f08d2c9f-5c2e-495d-b0bd-3f71bd301432@gmail.com>
 <v2nbp4$1o9h6$1@dont-email.me> <v2ng4n$1p3o2$1@dont-email.me>
 <87y18047jk.fsf@nosuchdomain.example.com>
 <87msoe1xxo.fsf@nosuchdomain.example.com> <v2sh19$2rle2$2@dont-email.me>
 <87ikz11osy.fsf@nosuchdomain.example.com> <v2v59g$3cr0f$1@dont-email.me>
 <20240528144118.00002012@yahoo.com> <v34thr$letg$1@dont-email.me>
 <v385ge$1a8ef$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 30 May 2024 15:05:18 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="fdfefad547a4bfd715e4868508bad851";
	logging-data="1799284"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX19N/cILIoY+9kdKesP2F8c5wS+3R9TILmw="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Cancel-Lock: sha1:JrhCVgSmKPc5G6+EdqtZSiirT5c=
Content-Language: en-GB
In-Reply-To: <v385ge$1a8ef$1@dont-email.me>
Bytes: 5833

On 29/05/2024 23:08, bart wrote:
> On 28/05/2024 16:34, David Brown wrote:
>> On 28/05/2024 13:41, Michael S wrote:
> 
>>> Let's start another round of private parts' measurements turnament!
>>> 'xxd -i' vs DIY
>>>
>>
>> I used 100 MB of random data:
>>
>> dd if=/dev/urandom bs=1M count=100 of=100MB
>>
>> I compiled your code with "gcc-11 -O2 -march=native".
>>
>> I ran everything in a tmpfs filesystem, completely in ram.
>>
>>
>> xxd took 5.4 seconds - that's the baseline.
>>
>> Your simple C code took 4.35 seconds.  Your second program took 0.9 
>> seconds - a big improvement.
>>
>> One line of Python code took 8 seconds :
>>
>> print(", ".join([hex(b) for b in open("100MB", "rb").read()]))
> 
> That one took 90 seconds on my machine (CPython 3.11).
> 
>> A slightly nicer Python program took 14.3 seconds :
>>
>> import sys
>> bs = open(sys.argv[1], "rb").read()
>> xs = "".join([" 0x%02x," % b for b in bs])
>> ln = len(xs)
>> print("\n".join([xs[i : i + 72] for i in range(0, ln, 72)]))
> 
> This one was 104 seconds (128 seconds with PyPy).
> 
> This can't be blamed on the slowness of my storage devices, or moans 
> about Windows, because I know that amount of data (the output is 65% 
> bigger because of using hex format) could be processed in a couple of a 
> seconds using a fast native code program.
> 
> It's just Python being Python.

I have two systems at work with close to identical hardware, both about 
10 years old.  The Windows one has a little bit faster disk, the Linux 
one has more memory, but the processor is the same.  The Windows system 
is Win7 and as old as the machine, while the Linux system was installed 
about 6 years ago.  Both machines have a number of other programs open 
(the Linux machine has vastly more), but none of these are particularly 
demanding when not in direct use.

On the Linux machine, that program took 25 seconds (with python 3.7). 
On the Windows machine, it took 48 seconds (with python 3.8).  In both 
cases, the source binary file was recently written and therefore should 
be in cache, and both the source and destination were on the disk (ssd 
for Windows, hd for Linux).

Python throws all this kind of stuff over to the C code - it is pretty 
good at optimising such list comprehensions.  (But they are obviously 
still slower than carefully written native C code.)  If it were running 
through these loops with the interpreter, it would be orders of 
magnitude slower.

So what I see from this is that my new Linux PC took 14 seconds while my 
old Linux PC took 25 seconds - it makes sense that the new processor is 
something like to 80% faster than the old one for a single-threaded 
calculation.  And Windows (noting that this is Windows 7, not a recent 
version of Windows) doubles that time for some reason.

> 
>> (I have had reason to include a 0.5 MB file in a statically linked 
>> single binary - I'm not sure when you'd need very fast handling of 
>> multi-megabyte embeds.)
> 
> I have played with generating custom executable formats (they can be 
> portable between OSes, and I believe less visible to AV software), but 
> they require a normal small executable to launch them and fix them up.
> 
> To give the illusion of a conventional single executable,  the program 
> needs to be part of that stub file.
> 
> There are a few ways of doing it, like simply concatenating the files, 
> but extracting is slightly awkward. Embedding as data is one way.
> 

Sure.

The typical use I have is for embedded systems where there is a network 
with a master card and a collection of slave devices (or perhaps 
multiple microcontrollers on the same board).  A software update will 
typically involve updating the master board and have that pass on 
updates to the other devices.  So the firmware for the other devices 
will be built into the executable for the master board.

Another use-case is small web servers in program, often for 
installation, monitoring or fault-finding.  There are fixed files such 
as index.html, perhaps a logo, and maybe jquery or other javascript 
library file.