Path: ...!eternal-september.org!feeder3.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail From: Ed Morton Newsgroups: comp.lang.awk Subject: Re: [gawk] Handling variants of CSV input data formats Date: Tue, 27 Aug 2024 06:45:32 -0500 Organization: A noiseless patient Spider Lines: 147 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Date: Tue, 27 Aug 2024 13:45:32 +0200 (CEST) Injection-Info: dont-email.me; posting-host="0da34e7009e7efa5655ea33214084bce"; logging-data="3133207"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/u87TNOUxxOZTxmRPOtmDb" User-Agent: Mozilla Thunderbird Cancel-Lock: sha1:eQrItMMCdwQXP/m+Vy3O9BUpBxk= X-Antivirus-Status: Clean Content-Language: en-US In-Reply-To: X-Antivirus: Avast (VPS 240826-4, 8/26/2024), Outbound message Bytes: 6256 On 8/26/2024 8:39 PM, Janis Papanagnou wrote: > On 27.08.2024 02:49, Ed Morton wrote: >> On 8/26/2024 7:54 AM, Janis Papanagnou wrote: >>> snip> >>> I'd have liked to provide more concrete information here, but I'm at >>> the moment even unable to reproduce Awk's behavior as documented in >>> its manual; I've tried the following command with various locales >>> >>> $ echo 4,321 | LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }' >>> -| 5,321 >>> >>> but always got just 5 as result. >> >> You need to specifically TELL gawk to use your locale to read input >> numbers: >> >> $ echo 4,321 | LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }' >> 5 >> >> $ echo 4,321 | POSIXLY_CORRECT=1 LC_ALL=en_DK.utf-8 gawk '{ print $1 + 1 }' >> 5,321 >> >> $ echo 4,321 | LC_ALL=en_DK.utf-8 gawk -N '{ print $1 + 1 }' >> 5,321 >> >> See >> https://www.gnu.org/software/gawk/manual/gawk.html#Locale-influences-conversions >> for more info on that. > > Thanks. That's actually where I got above example from. > > I've missed that there was an explicit > $ export POSIXLY_CORRECT=1 > set on the very top of these examples. Gee! > > Feels anyway strange that an explicit LC_* setting is ineffective > without the additional POSIXLY_CORRECT variable. And the page also > says: "The POSIX standard says that awk always uses the period as > the decimal point when reading the awk program source code". > So despite POSIX saying that, you have to use a variable named > POSIXLY_CORRECT. - Do I need some more coffee to understand that? POSIXLY_CORRECT=1 (or equivalently `--posix` aka `-P`) affects numbers in the input your script reads (as shown in the previous post) and strings being converted to numbers in your code, it doesn't affect literal numbers in the source code for your script that awk reads. In the source code the decimal separator for a literal number (as opposed to a string being converted to a number) is always `.`. You can't use, say, a comma as the decimal separator in a literal number because a comma already means something in the awk syntax, e.g. `print 4,321` means the same as "print 4 OFS 321`. For example, this code compares a literal number 4.321 to another literal number 4.1 and prints "bigger" because 4.321 is a bigger number than 4.1: $ awk 'BEGIN{ if (4.321 > 4.1) print "bigger"; else print "not" }' bigger while this attempt to use `,` as the decimal separator is a syntax error because `,` has a meaning in awk syntax: $ awk 'BEGIN{ if (4,321 > 4.1) print "bigger"; else print "not" }' awk: cmd. line:1: BEGIN{ if (4,321 > 4.1) print "bigger"; else print "not" } awk: cmd. line:1: ^ syntax error You can't just write the number as a string because then you're doing a string comparison and comparing the collation order of `.` vs `,`, not the values of the numbers: $ awk 'BEGIN{ if ("4,321" > 4.1) print "bigger"; else print "not" }' not You might think you can just add `0` to convert the string to a number: $ awk 'BEGIN{ if (("4,321"+0) > 4.1) print "bigger"; else print "not" }' not but by default in gawk that conversion truncates everything from the `,` on and so you end up comparing the number 4 to the number 4.1. To convert `"4,321"` to the number `4.321` in gawk you need to once again set your locale and tell gawk to use it: $ LC_ALL=en_DK.utf-8 awk 'BEGIN{ if (("4,321"+0) > 4.1) print "bigger"; else print "not" }' not $ LC_ALL=en_DK.utf-8 awk -N 'BEGIN{ if (("4,321"+0) > 4.1) print "bigger"; else print "not" }' bigger $ LC_ALL=en_DK.utf-8 awk -P 'BEGIN{ if (("4,321"+0) > 4.1) print "bigger"; else print "not" }' bigger or in gawk instead of adding 0 you could use `strtonum()` on a string containing that value to convert it to a number. For example: $ LC_ALL=en_DK.utf-8 awk 'BEGIN{ if (strtonum("4,321") > 4.1) print "bigger"; else print "not" }' not $ LC_ALL=en_DK.utf-8 awk -N 'BEGIN{ if (strtonum("4,321") > 4.1) print "bigger"; else print "not" }' bigger $ LC_ALL=en_DK.utf-8 awk -P 'BEGIN{ if (strtonum("4,321") > 4.1) print "bigger"; else print "not" }' awk: cmd. line:1: fatal: function `strtonum' not defined In that last example above we see why `-N` is better than `-P` since they both do what you want with the number but `-P` also disables gawk extensions while `-N` doesn't. > > And I see there's an additional GNU Awk option '--use-lc-numeric'. That's just the long form of `-N`, identical in meaning. > What a mess! > > (I suppose current status can only be explained by the mentioned > forth-and-back during history of various GNU Awk versions.) Right. > > What's worth the LC_* variables if they are ignored (or maybe not). They have their uses but using them for everything by default apparently isn't the way people most frequently want to use awk so you need options to tell gawk when to use them in specific situations. Ed. > > Janis > >> >> Regards, >> >> Ed >