Path: ...!3.eu.feeder.erje.net!feeder.erje.net!fu-berlin.de!uni-berlin.de!not-for-mail
From: 2QdxY4RzWzUUiLuE@potatochowder.com
Newsgroups: comp.lang.python
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60
 GB) from Kenna API
Date: Mon, 30 Sep 2024 20:06:57 -0400
Lines: 42
Message-ID: <mailman.18.1727741223.3018.python-list@python.org>
References: <CA+hg4RiGjXw3am1s=zVLDpcA-VGS+cWNp_YEyzvS+j2MyDE2Cg@mail.gmail.com>
 <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
 <CA+hg4Rhn8iX7rp0uC=MbOi+8g73wQ4y4=uV0dU0jHdDUz3jk4w@mail.gmail.com>
 <CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
 <4XHQPG4LzsznVwM@mail.python.org> <Zvrt0RJe5omaFkQq@anomaly>
 <CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
 <ZvsjI-E8Qtz4rCeL@anomaly> <4XHbxS5jl4znVGD@mail.python.org>
 <CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
 <Zvs9IZ5zxaosKcln@anomaly>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Trace: news.uni-berlin.de oZX7X4MGVv/x6EsI3E5Y1Ql7labU9XQaAML8kabzc9Pg==
Cancel-Lock: sha1:12eNQcVpIUvOGadVbM/Dwhw93MU= sha256:8j85O3ULqmov18aCJpcZc0L3rzVqr+Y4jAM+axzxMMg=
Return-Path: <2QdxY4RzWzUUiLuE@potatochowder.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
 reason="2048-bit key; unprotected key"
 header.d=potatochowder.com header.i=@potatochowder.com
 header.b=W7j3AuLi; dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.011
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'looks': 0.02; 'variable':
 0.05; 'subject:API': 0.07; 'angelico': 0.09; 'characters,': 0.09;
 'dan': 0.09; 'meant': 0.09; 'received:78': 0.09; '2024': 0.16;
 'anyway.': 0.16; 'base.': 0.16; 'decimal': 0.16;
 'from:addr:2qdxy4rzwzuuilue': 0.16; 'from:addr:potatochowder.com':
 0.16; 'iirc,': 0.16; 'integer': 0.16; 'literals': 0.16; 'odd':
 0.16; 'recall': 0.16; 'received:136.243': 0.16; 'received:172.58':
 0.16; 'received:78.46': 0.16; 'received:78.46.172': 0.16;
 'received:www458.your-server.de': 0.16; 'received:your-server.de':
 0.16; 'wrote:': 0.16; 'subject:Help': 0.17; 'grant': 0.17; '16,':
 0.19; 'tue,': 0.19; 'to:addr:python-list': 0.20; 'input': 0.21;
 'written': 0.22; 'received:de': 0.23; 'anything': 0.25; 'chris':
 0.28; 'think': 0.32; "doesn't": 0.32; '(as': 0.32; 'end.': 0.32;
 'python-list': 0.32; 'received:136': 0.32; 'but': 0.32; "i'm":
 0.33; 'subject:for': 0.33; 'header:In-Reply-To:1': 0.34; 'yes,':
 0.35; 'presented': 0.37; 'subject:from': 0.37; "it's": 0.37;
 'hard': 0.37; 'put': 0.38; 'read': 0.38; 'two': 0.39; 'least':
 0.39; 'still': 0.40; 'base': 0.40; 'both': 0.40; 'tell': 0.60;
 '10,': 0.61; 'above': 0.62; 'lower': 0.62; 'between': 0.63;
 'leading': 0.63; 'our': 0.64; 'clear': 0.64; 'look': 0.65;
 'numbers': 0.67; 'right': 0.68; 'matter': 0.68; 'skip:# 10': 0.68;
 'interpreted': 0.69; 'knowing': 0.71; 'subject:Data': 0.71;
 'global': 0.73; 'little': 0.73; 'significant': 0.78; 'admit':
 0.84; 'bind': 0.84; 'handled': 0.84; 'identifies': 0.84; 'six,':
 0.84; 'subject: \n ': 0.84
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
 d=potatochowder.com; s=default2305; h=In-Reply-To:Content-Type:MIME-Version:
 References:Message-ID:Subject:To:From:Date:Sender:Reply-To:Cc:
 Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:
 Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID;
 bh=M5OrN57pRSm7isZdVzSbBYsl+jpXgR8mfmw4JEnhFEA=; b=W7j3AuLiQWTKbUoTYsaBvVvwiB
 yvPF7ofcPZ1y7qMU7R9bNY3jTEJcLnzz4m2Ik+N5B3TbW7aicJ7PGUBsq8jiYsy7ntGFPErpb02Po
 D0E3JDAlpXu+vjccbSYEtGUen3vcOUnjPcCFGdNSCRqvMk1teCPNB/yWkyDl55MUyZwPcPbAATs/U
 7J3hJwoAhPA484kpZtspUTeZxbQwIFiLZKj6dfaoQ/hywmRf+ApTQaieuCQV8fRKtM4Q/ua/s6X+q
 /A3Zw+Wgfz9/PJ/kD8S5tl6KutE116hm0BmNGUaeI2mWCgudqAesFj6Ktib0qfGS5gwNpNPCW/9fX
 YK6aJ1ow==;
Mail-Followup-To: python-list@python.org
Content-Disposition: inline
In-Reply-To: <CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
X-Authenticated-Sender: 2QdxY4RzWzUUiLuE@potatochowder.com
X-Virus-Scanned: Clear (ClamAV 0.103.10/27413/Mon Sep 30 10:48:24 2024)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
 <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
 <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
 <mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <Zvs9IZ5zxaosKcln@anomaly>
X-Mailman-Original-References: <CA+hg4RiGjXw3am1s=zVLDpcA-VGS+cWNp_YEyzvS+j2MyDE2Cg@mail.gmail.com>
 <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
 <CA+hg4Rhn8iX7rp0uC=MbOi+8g73wQ4y4=uV0dU0jHdDUz3jk4w@mail.gmail.com>
 <CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
 <4XHQPG4LzsznVwM@mail.python.org>
 <Zvrt0RJe5omaFkQq@anomaly>
 <CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
 <ZvsjI-E8Qtz4rCeL@anomaly>
 <4XHbxS5jl4znVGD@mail.python.org>
 <CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
Bytes: 7584

On 2024-10-01 at 09:09:07 +1000,
Chris Angelico via Python-list <python-list@python.org> wrote:

> On Tue, 1 Oct 2024 at 08:56, Grant Edwards via Python-list
> <python-list@python.org> wrote:
> >
> > On 2024-09-30, Dan Sommers via Python-list <python-list@python.org> wrote:
> >
> > > In Common Lisp, integers can be written in any integer base from two
> > > to thirty six, inclusive.  So knowing the last digit doesn't tell
> > > you whether an integer is even or odd until you know the base
> > > anyway.
> >
> > I had to think about that for an embarassingly long time before it
> > clicked.
> 
> The only part I'm not clear on is what identifies the base. If you're
> going to write numbers little-endian, it's not that hard to also write
> them with a base indicator before the digits [...]

In Common Lisp, you can write integers as #nnR[digits], where nn is the
decimal representation of the base (possibly without a leading zero),
the # and the R are literal characters, and the digits are written in
the intended base.  So the input #16fFFFF is read as the integer 65535.

You can also set or bind the global variable *read-base* (yes, the
asterisks are part of the name) to an integer between 2 and 36, and then
anything that looks like an integer in that base is interpreted as such
(including literals in programs).  The literals I described above are
still handled correctly no matter the current value of *read-base*.  So
if the value of *read-base* is 16, then the input FFFF is read as the
integer 65535 (as is the input #16rFFFF).

(Pedants may point our details I omitted.  I admit to omitting them.)

IIRC, certain [old 8080 and Z-80?] assemblers used to put the base
indicator at the end.  So 10 meant, well, 10, but 10H meant 16 and 10b
meant 2 (IDK; the capital H and the lower case b both look right to me).

I don't recall numbers written from least significant digit to most
significant digit (big and little endian *storage*, yes, but not the
digits when presented to or read from a human).