Path: ...!npeer.as286.net!npeer-ng0.as286.net!3.eu.feeder.erje.net!feeder.erje.net!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: bart <bc@freeuk.com>
Newsgroups: comp.lang.c
Subject: Re: xxd -i vs DIY Was: C23 thoughts and opinions
Date: Wed, 29 May 2024 22:08:31 +0100
Organization: A noiseless patient Spider
Lines: 58
Message-ID: <v385ge$1a8ef$1@dont-email.me>
References: <v2l828$18v7f$1@dont-email.me>
 <00297443-2fee-48d4-81a0-9ff6ae6481e4@gmail.com>
 <v2lji1$1bbcp$1@dont-email.me> <87msoh5uh6.fsf@nosuchdomain.example.com>
 <f08d2c9f-5c2e-495d-b0bd-3f71bd301432@gmail.com>
 <v2nbp4$1o9h6$1@dont-email.me> <v2ng4n$1p3o2$1@dont-email.me>
 <87y18047jk.fsf@nosuchdomain.example.com>
 <87msoe1xxo.fsf@nosuchdomain.example.com> <v2sh19$2rle2$2@dont-email.me>
 <87ikz11osy.fsf@nosuchdomain.example.com> <v2v59g$3cr0f$1@dont-email.me>
 <20240528144118.00002012@yahoo.com> <v34thr$letg$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Wed, 29 May 2024 23:08:31 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="f50a0a3313aa9a4c5e71b452f1561318";
	logging-data="1384911"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1/HTl7ks9E0T3SI/tbUAhm7"
User-Agent: Mozilla Thunderbird
Cancel-Lock: sha1:8DGyqsAkdTgfVKAy7J4AgWIX0yo=
Content-Language: en-GB
In-Reply-To: <v34thr$letg$1@dont-email.me>
Bytes: 3514

On 28/05/2024 16:34, David Brown wrote:
> On 28/05/2024 13:41, Michael S wrote:

>> Let's start another round of private parts' measurements turnament!
>> 'xxd -i' vs DIY
>>
> 
> I used 100 MB of random data:
> 
> dd if=/dev/urandom bs=1M count=100 of=100MB
> 
> I compiled your code with "gcc-11 -O2 -march=native".
> 
> I ran everything in a tmpfs filesystem, completely in ram.
> 
> 
> xxd took 5.4 seconds - that's the baseline.
> 
> Your simple C code took 4.35 seconds.  Your second program took 0.9 
> seconds - a big improvement.
> 
> One line of Python code took 8 seconds :
> 
> print(", ".join([hex(b) for b in open("100MB", "rb").read()]))

That one took 90 seconds on my machine (CPython 3.11).

> A slightly nicer Python program took 14.3 seconds :
> 
> import sys
> bs = open(sys.argv[1], "rb").read()
> xs = "".join([" 0x%02x," % b for b in bs])
> ln = len(xs)
> print("\n".join([xs[i : i + 72] for i in range(0, ln, 72)]))

This one was 104 seconds (128 seconds with PyPy).

This can't be blamed on the slowness of my storage devices, or moans 
about Windows, because I know that amount of data (the output is 65% 
bigger because of using hex format) could be processed in a couple of a 
seconds using a fast native code program.

It's just Python being Python.

> (I have had reason to include a 0.5 MB file in a statically linked 
> single binary - I'm not sure when you'd need very fast handling of 
> multi-megabyte embeds.)

I have played with generating custom executable formats (they can be 
portable between OSes, and I believe less visible to AV software), but 
they require a normal small executable to launch them and fix them up.

To give the illusion of a conventional single executable,  the program 
needs to be part of that stub file.

There are a few ways of doing it, like simply concatenating the files, 
but extracting is slightly awkward. Embedding as data is one way.