Path: ...!news.mixmin.net!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: bart <bc@freeuk.com>
Newsgroups: comp.lang.c
Subject: Re: xxd -i vs DIY Was: C23 thoughts and opinions
Date: Tue, 28 May 2024 15:06:40 +0100
Organization: A noiseless patient Spider
Lines: 88
Message-ID: <v34odg$kh7a$1@dont-email.me>
References: <v2l828$18v7f$1@dont-email.me>
 <00297443-2fee-48d4-81a0-9ff6ae6481e4@gmail.com>
 <v2lji1$1bbcp$1@dont-email.me> <87msoh5uh6.fsf@nosuchdomain.example.com>
 <f08d2c9f-5c2e-495d-b0bd-3f71bd301432@gmail.com>
 <v2nbp4$1o9h6$1@dont-email.me> <v2ng4n$1p3o2$1@dont-email.me>
 <87y18047jk.fsf@nosuchdomain.example.com>
 <87msoe1xxo.fsf@nosuchdomain.example.com> <v2sh19$2rle2$2@dont-email.me>
 <87ikz11osy.fsf@nosuchdomain.example.com> <v2v59g$3cr0f$1@dont-email.me>
 <20240528144118.00002012@yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Tue, 28 May 2024 16:06:41 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="1d2d1a7652200fbc651c6448e3777044";
	logging-data="673002"; mail-complaints-to="abuse@eternal-september.org";	posting-account="U2FsdGVkX18bZVnASvm2aIr2Aw3DIlAc"
User-Agent: Mozilla Thunderbird
Cancel-Lock: sha1:kcmU2mWHnDoc68hGJrazT1UymuM=
In-Reply-To: <20240528144118.00002012@yahoo.com>
Content-Language: en-GB
Bytes: 4609

On 28/05/2024 12:41, Michael S wrote:
> On Sun, 26 May 2024 13:09:36 +0200
> David Brown <david.brown@hesbynett.no> wrote:
> 
>>
>> No, it does /not/.  That's the /whole/ point of #embed, and the main
>> motivation for its existence.  People have always managed to embed
>> binary source files into their binary output files - using linker
>> tricks, or using xxd or other tools (common or specialised) to turn
>> binary files into initialisers for constant arrays (or structs).
>> I've done so myself on many projects, all integrated together in
>> makefiles.
>>
> 
> Let's start another round of private parts' measurements turnament!
> 'xxd -i' vs DIY
> 
> /c/altera/13.0sp1/quartus/bin64/db_wys.dll is 52 MB file
> 
> $ time xxd -i < /c/altera/13.0sp1/quartus/bin64/db_wys.dll > xxd.txt
> 
> real    0m15.288s
> user    0m15.054s
> sys     0m0.187s
> 
> $ time ../quick_xxd/bin_to_list1
> /c/altera/13.0sp1/quartus/bin64/db_wys.dll > bin_to_list1.txt
> 
> real    0m8.502s
> user    0m0.000s
> sys     0m0.000s
> 
> $ time ../quick_xxd/bin_to_list
> /c/altera/13.0sp1/quartus/bin64/db_wys.dll > bin_to_list.txt
> 
> real    0m1.326s
> user    0m0.000s
> sys     0m0.000s
> 
> bin_to_list probably limited by write speed of SSD that in this
> particular case is ~9 y.o. and was used rather intensively during these
> years.
> 
> bin_to_list1 is DYI written in ~5 min.
> bin_to_list is DYI written in ~55 min.
> In post above David Brown mentioned 'other tools (common or
> specialised)'. I'd like to know what they are and how fast they are.
> 

I think you might be missing the point here.

The start point is a possibly large binary data file.

The end point is to end up with an application whose binary code has 
embedded that data file. (And which makes that data available inside the 
C program as a C data structure.)

Without #embed, one technique (which I've only learnt about this week) 
is to use a tool called 'xxd' to turn that binary file into C source 
code which contains an initialised array or whatever.

But, that isn't the bottleneck. You run that conversion once (or 
whenever the binary changes), and use the same resulting C code time you 
build the application. And quite likely, the makefile recognises you 
don't need to compile it anyway.

It is that building process that can be slow if that C source describing 
the data is large.

That is what #embed helps to address. At least, if it takes the fast 
path that has been discussed. But implemented naively, or the fast path 
is not viable, then it can be just as slow as compiling that 
xxd-generated C.

It will at least however have eliminated that xxd step.

The only translation going on here might be:

* Expanding a binary file to text, or tokens (if #embed is done poorly)
* Parsing that text or tokens into the compiler's internal rep

But all that is happening inside the compiler.

It might be that when xxd /is/ used, there might be a faster program to 
do the same thing, but I've not heard anyone say xxd's speed is a 
problem, only that it's a nuisance to do.