Path: ...!npeer.as286.net!npeer-ng0.as286.net!fu-berlin.de!uni-berlin.de!not-for-mail
From: Chris Angelico <rosuav@gmail.com>
Newsgroups: comp.lang.python
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60
 GB) from Kenna API
Date: Tue, 1 Oct 2024 04:46:35 +1000
Lines: 20
Message-ID: <mailman.11.1727722011.3018.python-list@python.org>
References: <CA+hg4RiGjXw3am1s=zVLDpcA-VGS+cWNp_YEyzvS+j2MyDE2Cg@mail.gmail.com>
 <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
 <CA+hg4Rhn8iX7rp0uC=MbOi+8g73wQ4y4=uV0dU0jHdDUz3jk4w@mail.gmail.com>
 <CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
 <4XHQPG4LzsznVwM@mail.python.org> <Zvrt0RJe5omaFkQq@anomaly>
 <CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Trace: news.uni-berlin.de OxbIUFbjuOU//wAtoLJCWw/XMLYTSFJtsljixp9lTxvA==
Cancel-Lock: sha1:P+MtRmTE+6taUmGKV5FWFVZsRdY= sha256:DI2W7d7smkEU0YRnhlNvHU4z9uYgL38t0hfFygue/Ow=
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
 reason="2048-bit key; unprotected key"
 header.d=gmail.com header.i=@gmail.com header.b=IK//9adZ;
 dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.053
X-Spam-Evidence: '*H*': 0.90; '*S*': 0.00; 'subject:API': 0.07; 'dan':
 0.09; 'infinite': 0.09; 'received:mail-lj1-x22e.google.com': 0.09;
 '2024': 0.16; 'chrisa': 0.16; 'from:addr:rosuav': 0.16;
 'from:name:chris angelico': 0.16; 'odd': 0.16; 'parsing': 0.16;
 'wrote:': 0.16; 'subject:Help': 0.17; 'tue,': 0.19; 'to:addr
 :python-list': 0.20; 'url:wiki': 0.23; 'anything': 0.25;
 'examples': 0.25; 'python-list': 0.32; 'message-
 id:@mail.gmail.com': 0.32; 'but': 0.32; 'subject:for': 0.33;
 'there': 0.33; 'particular': 0.33; 'header:In-Reply-To:1': 0.34;
 'received:google.com': 0.34; 'from:addr:gmail.com': 0.35;
 'subject:from': 0.37; "it's": 0.37; 'least': 0.39; 'something':
 0.40; "there's": 0.61; 'sequence': 0.69; 'url-ip:208.80/16': 0.70;
 'url:wikipedia': 0.70; 'subject:Data': 0.71; 'url-ip:208/8': 0.71;
 'significant': 0.78; 'left': 0.83; 'significant,': 0.84; 'subject:
 \n ': 0.84; 'very,': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1727722008; x=1728326808; darn=python.org;
 h=to:subject:message-id:date:from:in-reply-to:references:mime-version
 :from:to:cc:subject:date:message-id:reply-to;
 bh=Z2BRySl0uaX8R8ST8QwPtm/D88srEcLC82Z1GmJfvaY=;
 b=IK//9adZFXbqZvTpWvR5cz12VTQRgeb6DpxtkV4/gl2L1LCn0qwkUWlHraSfGL0xCa
 58NzSYGJD/dW6oMY2PzHXOptgVt+tSNU0bxrBaiWoKC0EeAs3Zh//juJ4uFt386SDyx+
 nG8GEF8BP7/eaSu92v6QdOeyP1tzWoowtmwn0S32JP9GUZZoNc320e93gxY3vR2vdKEO
 NsW81qocbWr1zUxdQ9cJVg8uVWFyovz+/krOfhXNh4kZ+VrWhHyd49NrBRI8IqscscjS
 A47pmV9QtsM421nmFvMuZwFzDCfeNDn2rHjEuTLJZ3tw8ajGqFywOLlyre+oY3sGhxXI
 wS9g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1727722008; x=1728326808;
 h=to:subject:message-id:date:from:in-reply-to:references:mime-version
 :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
 bh=Z2BRySl0uaX8R8ST8QwPtm/D88srEcLC82Z1GmJfvaY=;
 b=dBXyaJhJVtuN3hVDi8ldX+aIGxf/aNPt/0KLqk3qe2kpdWG+X6LctYIKeneRRPdvFw
 UxLJeA11rFS5G1OlQNx2HMF1CfifwkG7urPHR6JMe6EsIBcz5AfnqLD8kv/uohvLGqBS
 zRZRcea10+2gSfTvFbErgNqzS/Q4feTnqfvdhBDvHyhQS636XU5fD2EkVjro/86weII1
 2xaX8F86ZPZa1BMICPgaljBlOo2BApZbJ/gZOAdgHxMwMogsRqZd1tq6KsfCTOqKA2Cu
 UGy75WN4FbreyuQ7AQjwqX+qSry+xrUpU445jBjrLJdHvwrbHoK3gI4782BaR1FCMM2M
 JKAA==
X-Gm-Message-State: AOJu0YwrL1AV7nm6y9lcTUv+4sGr5btS01TyfiRzan6Y+SGAGov9sp6x
 AsQ3INR0Tl+1emQCodmlxcRi5aU8LIy78PswpX7Gzw0/8yWIJApLVpJGFp7C9H2wYa4qL4exYIp
 llQuLWZna9Tt/pAdWz489GB2AB9WNmsCj
X-Google-Smtp-Source: AGHT+IF8bqcT4NKh0vscxq0nij+8JfI+STzBKYBH+mDC7rHPUdKl1jYFER1uGrqFmBWYm33YchOe24xc/EX1DNt5BQ0=
X-Received: by 2002:a2e:be24:0:b0:2fa:d4ef:f222 with SMTP id
 38308e7fff4ca-2fad4eff4b4mr13195291fa.38.1727722007825; Mon, 30 Sep 2024
 11:46:47 -0700 (PDT)
In-Reply-To: <Zvrt0RJe5omaFkQq@anomaly>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
 <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
 <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
 <mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
X-Mailman-Original-References: <CA+hg4RiGjXw3am1s=zVLDpcA-VGS+cWNp_YEyzvS+j2MyDE2Cg@mail.gmail.com>
 <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
 <CA+hg4Rhn8iX7rp0uC=MbOi+8g73wQ4y4=uV0dU0jHdDUz3jk4w@mail.gmail.com>
 <CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
 <4XHQPG4LzsznVwM@mail.python.org> <Zvrt0RJe5omaFkQq@anomaly>
Bytes: 6088

On Tue, 1 Oct 2024 at 04:30, Dan Sommers via Python-list
<python-list@python.org> wrote:
>
> But why do I need to start with the least
> significant digit?

If you start from the most significant, you don't know anything about
the number until you finish parsing it. There's almost nothing you can
say about a number given that it starts with a particular sequence
(since you don't know how MANY digits there are). However, if you know
the LAST digits, you can make certain statements about it (trivial
examples being whether it's odd or even).

It's not very, well, significant. But there's something to it. And it
extends nicely to p-adic numbers, which can have an infinite number of
nonzero digits to the left of the decimal:

https://en.wikipedia.org/wiki/P-adic_number

ChrisA