Article <mailman.35.1729784858.4695.python-list@python.org>

Deutsch English Français Italiano
<mailman.35.1729784858.4695.python-list@python.org>

View for Bookmarking (what is this?)
Look up another Usenet article
Path: ...!fu-berlin.de!uni-berlin.de!not-for-mail
From: MRAB <python@mrabarnett.plus.com>
Newsgroups: comp.lang.python
Subject: Re: shutil.rmtree() fails when used in Fedora (rpm) "mock" environment
Date: Thu, 24 Oct 2024 16:44:30 +0100
Lines: 121
Message-ID: <mailman.35.1729784858.4695.python-list@python.org>
References: <4a13731716200669342338ae409e73ca@posteo.de>
 <CAJQBtg=kCr0VMGXvkBcj4U6CwatC2_G2dstO5pS56_koAXJurA@mail.gmail.com>
 <0c080a7d-92de-4518-ac44-78d97fc9e3bb@mrabarnett.plus.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Cancel-Lock: sha1:uU3AENV3PYre5lWNg4Jyj9MfZe0= sha256:jDVZU46fohpvzsKoEmpbMwzNcdPaRENLk2WQ6BVCxmw=
User-Agent: Mozilla Thunderbird
Content-Language: en-GB
In-Reply-To: <CAJQBtg=kCr0VMGXvkBcj4U6CwatC2_G2dstO5pS56_koAXJurA@mail.gmail.com>
Precedence: list
Bytes: 11331

On 2024-10-24 16:17, Left Right via Python-list wrote:
>  From reading the code where the exception is coming from, this is how
> I interpret the intention of the author: they build a list (not sure
> why they used list, when there's a stack datastructure in Python)
> which they use as a stack, where the elements of the stack are
> 4-tuples, the important part about these tuples is that the first
> element is the operation to be performed by rmtree() has to be one of
> the known filesystem-related functions. The code raising the exception
> checks that it's one of those kinds and if it isn't, crashes.
> 
> There is, however, a problem with testing equality (more strictly,
> identity in this case) between functions.  I.e. it's possible that a
> function isn't identical to itself is, eg. "os" module was somehow
> loaded twice.  I'm not sure if that's a real possibility with how
> Python works... but maybe in some cases, like, multithreaded
> environments it could happen...
> 
> To investigate this, I'd edit the file with the assertion and make it
> print the actual value found in os.lstat and func.  My guess is that
> they are both somehow "lstat", but with different memory addresses.
> 
The stack is created on line 760 with os.lstat and entries are appended 
on lines 677 (os.rmdir), 679 (os.close) and 689 (os.lstat).

'func' is popped off the stack on line 651 and check in the following lines.

I can't see anywhere else where something else is put onto the stack or 
an entry is replaced.

> On Thu, Oct 24, 2024 at 4:06 PM Christian Buhtz via Python-list
> <python-list@python.org> wrote:
>>
>> Hello,
>> I am upstream maintainer of "Back In Time" [1] investigating an issue a
>> distro maintainer from Fedora reported [2] to me.
>>
>> On one hand Fedora seems to use a tool called "mock" to build packages
>> in a chroot environment.
>> On the other hand the test suite of "Back In Time" does read and write
>> to the real file system.
>> One test fails because a temporary directory is cleaned up using
>> shutil.rmtree(). Please see the output below.
>>
>> I am not familiar with Fedora and "mock". So I am not able to reproduce
>> this on my own.
>> It seems the Fedora maintainer also has no clue how to solve it or why
>> it happens.
>>
>> Can you please have a look (especially at the line "assert func is
>> os.lstat").
>> Maybe you have an idea what is the intention behind this error raised by
>> an "assert" statement inside "shutil.rmtree()".
>>
>> Thanks in advance,
>> Christian Buhtz
>>
>> [1] -- <https://github.com/bit-team/backintime>
>> [2] -- <https://github.com/bit-team/backintime/issues/1911>
>>
>> __________________________ General.test_ctor_defaults
>> __________________________
>> self = <test.test_uniquenessset.General testMethod=test_ctor_defaults>
>>      def test_ctor_defaults(self):
>>          """Default values in constructor."""
>> >       with TemporaryDirectory(prefix='bit.') as temp_name:
>> test/test_uniquenessset.py:47:
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>> _ _ _ _
>> /usr/lib64/python3.13/tempfile.py:946: in __exit__
>>      self.cleanup()
>> /usr/lib64/python3.13/tempfile.py:950: in cleanup
>>      self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
>> /usr/lib64/python3.13/tempfile.py:930: in _rmtree
>>      _shutil.rmtree(name, onexc=onexc)
>> /usr/lib64/python3.13/shutil.py:763: in rmtree
>>      _rmtree_safe_fd(stack, onexc)
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>> _ _ _ _
>> stack = []
>> onexc = <function TemporaryDirectory._rmtree.<locals>.onexc at
>> 0xffffb39bc860>
>>      def _rmtree_safe_fd(stack, onexc):
>>          # Each stack item has four elements:
>>          # * func: The first operation to perform: os.lstat, os.close or
>> os.rmdir.
>>          #   Walking a directory starts with an os.lstat() to detect
>> symlinks; in
>>          #   this case, func is updated before subsequent operations and
>> passed to
>>          #   onexc() if an error occurs.
>>          # * dirfd: Open file descriptor, or None if we're processing the
>> top-level
>>          #   directory given to rmtree() and the user didn't supply
>> dir_fd.
>>          # * path: Path of file to operate upon. This is passed to
>> onexc() if an
>>          #   error occurs.
>>          # * orig_entry: os.DirEntry, or None if we're processing the
========== REMAINDER OF ARTICLE TRUNCATED ==========