Path: ...!news.mixmin.net!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: David Brown <david.brown@hesbynett.no>
Newsgroups: comp.lang.c
Subject: Re: xxd -i vs DIY Was: C23 thoughts and opinions
Date: Wed, 29 May 2024 10:18:45 +0200
Organization: A noiseless patient Spider
Lines: 134
Message-ID: <v36od5$12g12$1@dont-email.me>
References: <v2l828$18v7f$1@dont-email.me>
 <00297443-2fee-48d4-81a0-9ff6ae6481e4@gmail.com>
 <v2lji1$1bbcp$1@dont-email.me> <87msoh5uh6.fsf@nosuchdomain.example.com>
 <f08d2c9f-5c2e-495d-b0bd-3f71bd301432@gmail.com>
 <v2nbp4$1o9h6$1@dont-email.me> <v2ng4n$1p3o2$1@dont-email.me>
 <87y18047jk.fsf@nosuchdomain.example.com>
 <87msoe1xxo.fsf@nosuchdomain.example.com> <v2sh19$2rle2$2@dont-email.me>
 <87ikz11osy.fsf@nosuchdomain.example.com> <v2v59g$3cr0f$1@dont-email.me>
 <20240528144118.00002012@yahoo.com> <v34odg$kh7a$1@dont-email.me>
 <20240528185624.00002494@yahoo.com> <v359f1$nknu$1@dont-email.me>
 <20240528232315.00006a58@yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Wed, 29 May 2024 10:18:46 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="2973a68944d5fb399809b749e2ae973b";
	logging-data="1130530"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1/r9VjgulispKAOdWdQ5A+KhQ1WEelp44M="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Cancel-Lock: sha1:T9ZXXPbM0ynyCKT7wgxu2jyREeE=
In-Reply-To: <20240528232315.00006a58@yahoo.com>
Content-Language: en-GB
Bytes: 7094

On 28/05/2024 22:23, Michael S wrote:
> On Tue, 28 May 2024 19:57:38 +0100
> bart <bc@freeuk.com> wrote:
> 
>> On 28/05/2024 16:56, Michael S wrote:
>>> On Tue, 28 May 2024 15:06:40 +0100
>>> bart <bc@freeuk.com> wrote:
>>>    
>>>> On 28/05/2024 12:41, Michael S wrote:
>>>>> On Sun, 26 May 2024 13:09:36 +0200
>>>>> David Brown <david.brown@hesbynett.no> wrote:
>>>>>       
>>>>
>>>> I think you might be missing the point here.
>>>>   
>>>
>>> I don't think so.
>>> I understand your points and agree with just about everything. My
>>> post was off topic, intentionally so.
>>>
>>> If we talk about practicalities, the problems with xxd, if there are
>>> problems at all, are not its speed, but the size of the text file
>>> it produces (~6x the size of original binary) and its availability.
>>> I don't know to which package it belongs in typical Linux or BSD
>>> distributions, but at least on Windows/msys2 it is part of Vim -
>>> rather big package for which, apart from xxd, I have no use at all.
>>>
>>>    
>>
>>
>> OK, I had go with your program. I used a random data file of exactly
>> 100M bytes.
>>
>> Runtimes varied from 4.1 to 5 seconds depending on compiler. The
>> fastest time was with gcc -O3.
>>
> 
> It sounds like your mass storage device is much slower than aging SSD
> on my test machine and ALOT slower than SSD of David Brown.

I didn't use an SSD - I used a tmpfs filesystem, so no disk at all. 
There are still limits from memory bandwidth, of course.

> 
>> I then tried a simple program in my language, which took 10 seconds.
>>
>> I looked more closely at yours, and saw you used a clever method of a
>> table of precalculated stringified numbers.
>>
>> Using a similar table, plus more direct string handling, the fastest
>> timing on mine was 3.1 seconds, with 21 numbers per line. (The 21 was
>> supposed to match your layout, but that turned out to be variable.)
>>
> 
> Yes, I try to get line length almost fixed (77 to 80 characters) and
> make no attempts to control number of entries per line.
> Since you used random generator, a density advantage of my approach is
> smaller than in more typical situations, where 2-digit numbers are more
> common than 3-digit numbers.
> 
> Also, I think that random numbers are close to worst case for branch
> predictor / loop length predictor in my inner loop.

That makes them a good test here.

> Were I thinking about random case upfront, I'd code an inner loop
> differently. I'd always copy 4 octets (comma would be stored in the same
> table). After that I would update outptr by length taken from
> additional table, similarly, but not identically to your method below.
> 
> There exist files that have near-random distribution, e.g. anything
> zipped or anything encrypted, but I would think that we rarely want
> them embedded.
> 

I'd say these are actually quite common cases.

>> Both programs have a trailing comma on the last number, which may be
>> problematical, but also not hard to fix.
>>
> 
> I don't see where (in C) it could be a problem. On the other hand, I can
> imagine situations where absence of trailing comma is inconvinient.
> Now, if your language borrow its array initialization syntax from
> Pascal then trailing commas are indeed undesirable.
> 

For initialising arrays, an extra comma at the end is no issue for C. 
But for more general use, #embed can also be used for things like 
function call or macro parameters, and the extra comma is then a 
problem.  So your program is not a direct alternative to "xxd -i" or 
#embed in those cases.  (I don't think such uses would be common in 
practice.)

>> I then tried xxd under WSL, and that took 28 seconds, real time, with
>> a much larger output (616KB instead of 366KB).
> 
> 616 MB, I suppose.
> Timing is very similar to my measurements. It is obvious that in case
> of xxd, unlike in the rest of our cases, the bottleneck is in CPU rather
> than in HD.
> 
>> But it's using fixed
>> width columns of hex, complete with a '0x' prefix.
>>
>> Below is that program but in my language. I tried transpiling to C,
>> hoping it might be even faster, but it got slower (4.5 seconds with
>> gcc-O3). I don't know why. It would need manual porting to C.
>>
> 
> Why do you measure with gcc -O3 instead of more robust and more popular
> -O2 ? Not that it matters in this particular case, but in general I
> don't think that it is a good idea.
> 

Bart always likes to use "gcc -O3", no matter how often people tell him 
that it should not be assumed to give faster results than "gcc -O2".  It 
often results in a few extra percent of speed, but sometimes works 
against speed (depending on things like cache sizes, branch prediction, 
and other hard to predict factors).  Alternatively, he uses gcc without 
any optimisation and still thinks the speed results are relevant.

> 
> Reading whole file upfront is undoubtly faster than interleaving of
> reads and writes. But by my set of unwritten rules that I imposed on
> myself, it is cheating.
> 

I would expect that for big files, mmap would be the most efficient 
method.  You might also want to use cache pre-loads of later parts of 
the file as you work through it.  But I don't know portable that would 
be, and it would be more effort to write.