Path: ...!weretis.net!feeder9.news.weretis.net!i2pn.org!i2pn2.org!.POSTED!not-for-mail
From: fir <fir@grunge.pl>
Newsgroups: comp.lang.c
Subject: Re: program to remove duplicates
Date: Sun, 22 Sep 2024 00:18:09 +0200
Organization: i2pn2 (i2pn.org)
Message-ID: <8630bec343aec589a6cdc42bb19dae28120ceabf@i2pn2.org>
References: <ecb505e80df00f96c99d813c534177115f3d2b15@i2pn2.org> <vcnfbi$1ocq6$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Sat, 21 Sep 2024 22:18:10 -0000 (UTC)
Injection-Info: i2pn2.org;
	logging-data="2932740"; mail-complaints-to="usenet@i2pn2.org";
	posting-account="+ydHcGjgSeBt3Wz3WTfKefUptpAWaXduqfw5xdfsuS0";
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0 SeaMonkey/2.24
X-Spam-Checker-Version: SpamAssassin 4.0.0
In-Reply-To: <vcnfbi$1ocq6$1@dont-email.me>
Bytes: 2359
Lines: 42

Chris M. Thomasson wrote:
> On 9/21/2024 11:53 AM, fir wrote:
>>
>>
>> i think if to write a simple comandline program
>> that remove duplicates in a given folder
> [...]
>
> Not sure if this will help you or not... ;^o
>
> Fwiw, I have to sort and remove duplicates in this experimental locking
> system that I called the multex. Here is the C++ code I used to do it. I
> sort and then remove any duplicates, so say a threads local lock set was:
>
> 31, 59, 69, 31, 4, 1, 1, 5
>
> would become:
>
> 1, 4, 5, 31, 59, 69
>
> this ensures no deadlocks. As for the algorithm for removing duplicates,
> well, there are more than one. Actually, I don't know what one my C++
> impl is using right now.
>
> https://groups.google.com/g/comp.lang.c++/c/sV4WC_cBb9Q/m/Ti8LFyH4CgAJ
>
> // Deadlock free baby!
> void ensure_locking_order()
> {
>    // sort and remove duplicates
>
>    std::sort(m_lock_idxs.begin(), m_lock_idxs.end());
>
>    m_lock_idxs.erase(std::unique(m_lock_idxs.begin(),
>      m_lock_idxs.end()), m_lock_idxs.end());
> }
>
> Using the std C++ template lib.

im not sure what you talking about but i write on finding file 
duplicates (by binary contents not by name).. it is disk thing and i 
dont think mutexes are needed - you just need to read all files in 
folder and compare it byte by byte to other files in folder of the same size