Path: news.eternal-september.org!eternal-september.org!.POSTED!not-for-mail From: Don Y Newsgroups: sci.electronics.design Subject: Re: "RESET" Date: Tue, 27 May 2025 17:40:44 -0700 Organization: A noiseless patient Spider Lines: 146 Message-ID: <1015m2h$2snv2$1@dont-email.me> References: <100thgs$v8cm$1@dont-email.me> <10159t3$2q2ds$1@dont-email.me> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Date: Wed, 28 May 2025 02:40:51 +0200 (CEST) Injection-Info: dont-email.me; posting-host="7f426fe14b63144f55aceb62bb3f7a1c"; logging-data="3039202"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18JmrtqdctL6PMfS+eJ3krx" User-Agent: Mozilla Thunderbird Cancel-Lock: sha1:RPIynET2LJZTrPgxRNRQ75/ePO0= In-Reply-To: Content-Language: en-US On 5/27/2025 4:13 PM, Joe Gwinn wrote: > On Tue, 27 May 2025 14:13:02 -0700, Don Y > wrote: > >> On 5/25/2025 12:33 PM, Joe Gwinn wrote: >>> Exactly. I recall a customer wanting us to verify all possible paths >>> through a bit of air traffic control radar software, about 100,000 >>> lines of plain C. Roughly one in five executable line was an IF >>> statement, which is 20,000 IF statements. So there are 2^20000 = >>> 10^6020 such paths. >> >> And probably 99.9% of them are superfluous. > > [snip] > > The problem is that you have no way to know which cases are > irrelevant. And practical hardware will have many things able to > retain state. The design of the specification becomes important. The *implementor* can further refine the test cases based on other potential vulnerabilities that his implementation may create. E.g., if you implement a multiply routine with repeated additions of the multiplicand, the case where the multiplier is zero can be buggy -- because you may have examined the "remaining multiplier" in a place that assumes at least one iteration of the multiplicand's processing. Or, a negative multiplier (how do you run a loop some negative number of times?) >> Experience teaches you to construct your code so that testability is >> enhanced. Instead of waiting until it seems to be "done" and then >> trying to reassure yourself that it works as intended -- usually by >> throwing EXPECTED conditions at it and hoping for the expected >> results (that's not testing). > > It is not usually the expected that causes trouble: It ain't what you > don't know that matters, it what you know that ain't so that's the > problem. Which is why it is so easy to prove code doesn't work! You identify what the likely assumptions were and then challenge each of those. People treat assumptions as so ingrained that they never actually consider their validity: Will this value always be non-zero? will the datalogger always be powered on? will "spare" memory always be available? Attack one and watch the code flail about -- because it's expectations have been defied. >> You need 2^20000 if there are 2^20000 distinct outcomes (leaf nodes) in >> your code. I strongly doubt that to be the case. > > So assume only 1000 IF statements, so it's 2^1000 or 10^600 or so. > You'll still run out of lifetime. If you have 1000 if statements in a module, then your module is at least 16 pages long -- containing no OTHER code. This is why you make small modules so their complete behavior can be "groked" without having ludicrous levels of complexity. if (x==0) { // } else if (x==1) { // } else if (x==2) { // } else if (x==3) { // } else { // } Four conditionals. 6 test cases for 100% coverage (<0, 0, 1, 2, 3, >=4) Not 16. I.e., assuming complexity correlates with a geometric weighing of the number of conditionals is specious reasoning. >>> The testing campaign will have only scratched the surface when the Sun >>> runs out of hydrogen and goes supernova. Tomorrow's problem. >> >> How do you test an electronic circuit? Let's impose an infinite number of >> discrete voltages on each of the input signals and verify the correct >> outputs for each? (Do you deliberately verify all ranges of signal values >> and frequencies? Or, just say "operation outside of these conditions is >> indeterminate"?) > > You do all the tests for required behavior - does it meet stated > requirements. But, you only test the *boundary* conditions. I.e., if something is suposed to work over a range of 0 - 12 volts, you test 0, 12, something in between and then < 0 and > 12. You don't test 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.50000000000001, 0.5000000000002, etc. Because you know the circuit is well-behaved BETWEEN these conditions. A UNIX pathname can be thousands of characters long. Do you test: a b c d ... z A ... Z 0 ... 9 aa ab ... az ... Of course not! Yet, a test case like: /0/1/2/3/4/../../../../../../../../../../../../../../../.. could tickle a bug in the implementation. UNLESS, the implementor took a shortcut that exposed some buried condition that needs to be exposed to testing. E.g., if he routed the signal through two different paths based on whether or not it exceeded 5 volts. The specification likely won't have anticipated such an "optimization" so wouldn't have drawn attention to "5V" as a particularly important case. The implementor/designer has to understand how his implementation could potentially break in ways that the specification couldn't catch. > Then you random probe it for weeks and see what goes Bang! > > One form of this is Fuzzing. > > . You still rely on "luck" to stumble on bugs. You need good specifications, "bite size" modules (where complexity is limited) and implementors who are aware of testing IN their design choices -- so they don't meet the published specifications (and likely test cases) but still fail, miserably when their internal quirks are tickled.