Path: ...!news.mixmin.net!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: David Brown <david.brown@hesbynett.no>
Newsgroups: comp.lang.c
Subject: Re: xxd -i vs DIY Was: C23 thoughts and opinions
Date: Tue, 28 May 2024 17:42:15 +0200
Organization: A noiseless patient Spider
Lines: 110
Message-ID: <v34u0n$letg$2@dont-email.me>
References: <v2l828$18v7f$1@dont-email.me>
 <00297443-2fee-48d4-81a0-9ff6ae6481e4@gmail.com>
 <v2lji1$1bbcp$1@dont-email.me> <87msoh5uh6.fsf@nosuchdomain.example.com>
 <f08d2c9f-5c2e-495d-b0bd-3f71bd301432@gmail.com>
 <v2nbp4$1o9h6$1@dont-email.me> <v2ng4n$1p3o2$1@dont-email.me>
 <87y18047jk.fsf@nosuchdomain.example.com>
 <87msoe1xxo.fsf@nosuchdomain.example.com> <v2sh19$2rle2$2@dont-email.me>
 <87ikz11osy.fsf@nosuchdomain.example.com> <v2v59g$3cr0f$1@dont-email.me>
 <20240528144118.00002012@yahoo.com> <v34odg$kh7a$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 28 May 2024 17:42:16 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="d21593105ce034ab606a049d42e9f782";
	logging-data="703408"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX1+vd5xh0S15bwtBfFsWxUM0z5dXYC6VeTc="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Cancel-Lock: sha1:7z1BiARwHMJmdm/equXYLgGE+0E=
In-Reply-To: <v34odg$kh7a$1@dont-email.me>
Content-Language: en-GB
Bytes: 5605

On 28/05/2024 16:06, bart wrote:
> On 28/05/2024 12:41, Michael S wrote:
>> On Sun, 26 May 2024 13:09:36 +0200
>> David Brown <david.brown@hesbynett.no> wrote:
>>
>>>
>>> No, it does /not/.  That's the /whole/ point of #embed, and the main
>>> motivation for its existence.  People have always managed to embed
>>> binary source files into their binary output files - using linker
>>> tricks, or using xxd or other tools (common or specialised) to turn
>>> binary files into initialisers for constant arrays (or structs).
>>> I've done so myself on many projects, all integrated together in
>>> makefiles.
>>>
>>
>> Let's start another round of private parts' measurements turnament!
>> 'xxd -i' vs DIY
>>
>> /c/altera/13.0sp1/quartus/bin64/db_wys.dll is 52 MB file
>>
>> $ time xxd -i < /c/altera/13.0sp1/quartus/bin64/db_wys.dll > xxd.txt
>>
>> real    0m15.288s
>> user    0m15.054s
>> sys     0m0.187s
>>
>> $ time ../quick_xxd/bin_to_list1
>> /c/altera/13.0sp1/quartus/bin64/db_wys.dll > bin_to_list1.txt
>>
>> real    0m8.502s
>> user    0m0.000s
>> sys     0m0.000s
>>
>> $ time ../quick_xxd/bin_to_list
>> /c/altera/13.0sp1/quartus/bin64/db_wys.dll > bin_to_list.txt
>>
>> real    0m1.326s
>> user    0m0.000s
>> sys     0m0.000s
>>
>> bin_to_list probably limited by write speed of SSD that in this
>> particular case is ~9 y.o. and was used rather intensively during these
>> years.
>>
>> bin_to_list1 is DYI written in ~5 min.
>> bin_to_list is DYI written in ~55 min.
>> In post above David Brown mentioned 'other tools (common or
>> specialised)'. I'd like to know what they are and how fast they are.
>>
> 
> I think you might be missing the point here.
> 
> The start point is a possibly large binary data file.
> 
> The end point is to end up with an application whose binary code has 
> embedded that data file. (And which makes that data available inside the 
> C program as a C data structure.)
> 
> Without #embed, one technique (which I've only learnt about this week) 
> is to use a tool called 'xxd' to turn that binary file into C source 
> code which contains an initialised array or whatever.
> 
> But, that isn't the bottleneck. You run that conversion once (or 
> whenever the binary changes), and use the same resulting C code time you 
> build the application. And quite likely, the makefile recognises you 
> don't need to compile it anyway.

Exactly, yes.

(Still, speed tests can be fun as long as you don't pretend they 
actually matter!)

> 
> It is that building process that can be slow if that C source describing 
> the data is large.

It is actually the link step that is typically the bottleneck, as that 
does not easily run in parallel.

When the "data.bin" file changes, your make (or other build system) will 
see the change and use xxd (or whatever) to generate data.c, and then 
compile that to data.o.  It's done once, whenever data.bin changes. 
It's the linking that has to be done at every build of the executable, 
whether "data.bin" changes or not.

> 
> That is what #embed helps to address. At least, if it takes the fast 
> path that has been discussed. But implemented naively, or the fast path 
> is not viable, then it can be just as slow as compiling that 
> xxd-generated C.
> 
> It will at least however have eliminated that xxd step.

Yes - it makes things a little neater and self-contained.  I don't see 
it as a game-changed, but if you want to avoid make, you might like it.

> 
> The only translation going on here might be:
> 
> * Expanding a binary file to text, or tokens (if #embed is done poorly)
> * Parsing that text or tokens into the compiler's internal rep
> 
> But all that is happening inside the compiler.
> 
> It might be that when xxd /is/ used, there might be a faster program to 
> do the same thing, but I've not heard anyone say xxd's speed is a 
> problem, only that it's a nuisance to do.
> 
>