Quality Assurance

The software development experience we’ve accumulated over the many years unfortunately tells us that

• Every program contains bugs
• If a program does not contain bugs, the algorithm that it implements contains them
• If neither the program nor the algorithm contains bugs, no one needs the program (almost always)

Quality Assurance #2

However, everything is relative:

• A bug in a website might not really hurt anybody
• A bug in rocket launch calculations can result in terrible consequences and expenses (Ariane 5)
• A bug in a radiation-treatment device can lead to deaths (Therac-25)

Ariane 5

“On June 4, 1996, the engines of the very first Ariane 5 rocket caught fire, and it began to move away from the coast of French Guiana. After 37 seconds, the rocket flipped 90 degrees in the wrong direction, and less than two seconds later, aerodynamic forces tore the boosters off the main stage at an altitude of 4 km. This triggered a self-destruct mechanism, and the spacecraft turned into a giant fireball of liquid hydrogen. The catastrophic launch cost approximately $370 million. The launch of Ariane 5 is widely recognized as one of the most expensive software failures in history.”

The reason for the crash? A single incorrect conversion of a 64-bit floating point number to a 16-bit integer in the rocket’s software.

Therac-25

Several people were irradiated with lethal doses of radiation when being treated on a Therac-25 radiation therapy machine. Even after the bugs had supposedly been fixed, problems persisted and several more people tragically died.

The problem arose because the configurations for two interconnected systems (irradiation mode and dosage) were polled at different frequencies, and if you made the changes too fast, they were not registered.

Quality Assurance: history

60s: Do exhaustive testing
Programs were smaller, we started out by trying to cover all possible paths in the code and all possible input combinations, but pretty soon we found out this was impossible.

Early 70s: Show that the program works correctly
Testing was understood as a “demonstration of correctness”, but we quickly came to the realization that this was not feasible.

Late 70s: Show that the program does not work correctly
The goal was changed to “demonstration of incorrectness” - finding bugs. This is feasible (if your program fails with a bug, it has a bug), and goal became the priority.

80s: Prevent defects throughout development
Defect detection -> defect prevention. Testing principles applied not only to the compiled version of the program, but to all stages of its development, including the design, implementation, architecture, and the tests themselves. Automated testing emerges.

Quality Assurance: goals

• Correct behavior of the product in all conditions
• Compliance with the requirements
• Information about the current state of the product
• Error prevention and detection
• Development cost reduction

Our goals when testing software are not limited to “finding bugs”. Testing is not simply about ensuring that the software meets functional requirements.

We can also use testing to check non-functional requirements, such as security, performance, or scalability. If we integrate testing into our development process, it can actively help us with defect prevention. By catching errors early, we can significantly decrease development costs.

Testing: principles

• Testing demonstrates the presence of defects, but it does not prove their absence
• The earlier the better
• The absence of bugs is not an absolute goal
• Many more

Pareto principle: 20% of your program components are responsible for 80% of the bugs. This means that you should be zeroing in on these “buggy” components by running risk analyses and getting feedback, among other techniques

Pesticide paradox: “Every method you use to prevent or find bugs leaves a residue of more subtle bugs against which those methods are ineffectual" - if you see a decrease in testing effectiveness, introduce new testing methods (or tweak your existing methods)

Goal: Not perfection, but achieving level where it is just good enough to be used by people

Testing: types

• Functional – Checking the behavior given in the specifications
• Load – Simulating a real load (for example, a certain number of users on the server)
• Stress – Checking the system’s operation under abnormal conditions (for example, a power outage or a huge number of operations per second)
• Configuration – Checking software using different system configurations
• Regression – Making sure new changes did not break anything that had worked previously
• Others - Security, Compliance, ...

Testing: levels

• Unit testing – Testing components separately (checking modules, classes, functions)
• Integration testing – Checking the interaction of components and program modules
• System testing – Checking the entire system
• Acceptance testing – Verifying system compliance with all the client requirements

Unit testing

For each non-trivial function, their own tests are written that check that the method works correctly:

• Frequent launch expected ⇒ should run fast
• One test ⇒ one use case

Related Methodology: Test-Driven Development (TDD)

Unit testing in Kotlin

The JUnit5 framework is the most popular way to test Java and Kotlin programs.

class MyTests {
    @Test
    @DisplayName("Check if the calculator works correctly")
    fun testCalculator() {
        Assertions.assertEquals(
            3, 
            myCalculator(1, 2, "+"), 
            "Assertion error message"
        )
    }
}

Unit testing in Kotlin #2

class MyParametrizedTests {
    companion object {
        @JvmStatic
        fun calculatorInputs() = listOf(
            Arguments.of(1, 2, "+", 3),
            Arguments.of(0, 5, "+", 5),
        )
    }

    @ParameterizedTest
    @MethodSource("calculatorInputs")
    fun testCalculator(a: Int, b: Int, op: String, expected: Int) {
        Assertions.assertEquals(expected, myCalculator(a, b, op), "Assertion error message")
    }
}

Unit testing in Kotlin #3

There are many annotations for tests customization

• @BeforeEach – Methods with this annotation are run before each test
• @AfterEach – Methods with this annotation are run after each test
  
  
• @BeforeAll – Methods with this annotation are run before all tests in the class
• @AfterAll – Methods with this annotation are run after all tests in the class

Setup/Teardown

Code quality

Often, testing includes checking not only the correctness of the functionality but also the quality of the code itself.

Static code analyzers such as detekt, ktlint, and diktat exist to help you avoid having to do this manually.

The build of static analyzers should also be green.