Software Verification

Verifying Software as a Medical Device (SaMD) requires a balance between automated and manual testing.

Automated testing assesses software quality early: unit, integration and system tests run quickly and repeatably, and continuous validation across the software lifecycle supports regulatory compliance such as IEC 62304.

Manual testing covers usability and exploratory testing, where human judgment is needed to evaluate how clinicians and patients interact with the software. It also covers edge-case validation, where the system must handle unexpected inputs and real-world complexities that automated tests can overlook.

Overview

P4SaMD lists all the tests planned for a version of the software system in three tabs:

• All tests: the individual tests applicable to the current Software Version. The list includes tests associated to the current and to other system versions, as long as they are not deprecated.
• Test suites: the available Test Suites, with navigation into suite details. Test Suites collect tests to make grouping and execution easier.
• Executions: all test executions, with access to execution details and reports. An execution can refer to a single test or to multiple tests, through the execution of Test Suites.

Tests are defined in the integrated ALM, where you create, update and delete them. The P4SaMD table reflects any change made inside the ALM. AI-powered evaluation features help you assess the quality and compliance of the tests.

All tests

For each test the following information is provided:

• Title: the unique identifier (ID or key) and title of the test;
• Suggestions: a list of suggestions generated by P4SaMD (for example if a test has never been executed or is not linked to a requirement);
• Quality: the latest AI evaluation result. If the test is modified after the evaluation, the result is no longer displayed and must be requested again. See the icon legend below.
• Type: the type of test, like integration or system;
• Execution Mode: if the test is executed automatically or manually;
• Test Suite: if the test is part of an automated Test Suite;
• Latest Execution: when the last test execution was performed and its outcome. The outcome can refer to the overall test (e.g. passed or failed for manual tests) or to the test cases underneath it (e.g. count of successful, failed or skipped test cases). If no execution of a manual test is available for the current system version, Not executed is displayed.
• Software Items: the number of software items associated to the test;
• Requirements: the number of requirements verified by the test.

The icons in the Quality column indicate the evaluation status of each test. For further details, see the AI evaluation section.

Evaluation	Icon
Missing
Very low quality
Low quality
High quality
Very high quality

Actions

The last column of the table provides the following actions:

• Link to Implementation: click the arrow icon to open the implementation of the test. Available only for tests with the Implementation Link field populated.
• Add to Test Suite: click the list icon to add the individual test to an existing Test Suite or create a new one. This opens the Add to Test Suite modal. Available only for software versions that are not yet released.
• Link Software Items: click the link icon to link a software item to the test, or to unlink an already associated one. The linked software items are displayed in the drawer under the Traceability tab.

Selecting and Adding Multiple Tests to a Test Suite

For software versions that are not yet released, a checkbox column in the table lets you select one or more tests and add them to a Test Suite in bulk.

After the first test is selected, you can only select tests with the same Execution Mode (automatic or manual). Tests with a different execution mode are disabled.

Once one or more tests are selected, a floating action bar appears at the bottom of the screen with the number of selected tests and an Add to test suite button. It opens the Add to Test Suite modal.

Add Tests to Test Suite

The Add to Test Suite modal adds the selected tests to a Test Suite, whether you picked them individually via the row action or in bulk via checkbox selection. The modal provides two options through separate tabs:

Add to Existing Suite

In this tab you can search for and select one or more existing Test Suites as the destination. Only Test Suites matching the execution mode of the selected tests are shown.

For each selected destination suite, the modal displays the suite name and the number of tests currently included. If any of the selected tests are already present in a destination suite, a warning banner reports that duplicate tests have been detected and will be ignored.

Create New Suite

In this tab you can create a new Test Suite and add the selected tests to it. The execution mode is determined by the selected tests and cannot be changed. A name for the new suite is required. The selected tests are listed at the bottom of the form as confirmation.

Drawer

Clicking on a table row opens a drawer with detailed information about the selected test, organized into four tabs: Details, Traceability, Suggestions and Executions.

To navigate between the linked entities (requirements and software items), select them under the Traceability section in the detailed view.

To browse back to previous entities, open the history menu at the top of the detailed view and select the entity of interest.

Details

In addition to the information displayed in the table, this tab shows:

• Description: A paragraph describing the test including the steps and phases.
• Implementation Link: Define the link related to the implementation of the test.

Traceability

This tab shows the linked issues grouped by:

• Software Items
• Requirements

Suggestions

This tab shows the suggestions related to the test. For further details, see Insight & Suggestions.

Executions

This tab lists all test executions, from the most recent to the least.

For each test execution, the following information is available:

• the user who executed the test;
• when the test was executed;
• the test outcome of selected test suite (passed, failed, etc.)
• Download report if available, download the report for the specific execution.

AI evaluation

danger

When performing an evaluation using AI, Assignee, Reporter and Approver information are not provided to AI. Test title, description and other information, including related requirements and software items, are shared with AI service.

Do not insert any personal or sensitive information in the AI-process fields. For more details about third-party organizations privacy and security measures, see the [FAQ section][faq-data-sharing].

Information generated by AI may be inaccurate or misleading. Always verify it, and never base an assumption or a decision on it alone.

Each test can be evaluated through the AI-powered evaluation features. The results are displayed both in the test details drawer and in a dedicated column of the test table.

To assess a test, hover on the icon under the Quality column in the corresponding table row and click the Get evaluation button.

The assessment usually takes around a minute. It runs in the background, so you can keep working on P4SaMD and come back to check the progress at any time.

info

If the test is modified after an evaluation has been performed, the evaluation result is no longer displayed and the Quality icon reverts to Missing. Request a new evaluation to obtain up-to-date results.

After the evaluation is completed, the icon on the table takes a different color depending on the overall rating. Hover on it to see a preview of the results.

The rating provides an overall score, which aggregates four scores on specific criteria:

• Clarity and Specificity: if the test is clear, detailed and unambiguous;
• Traceability: if the test is uniquely identified and is linked to requirements and software items;
• Testability and Verification: if the test is written in a way that is easy to execute and replicate;
• Appropriateness: if the test is appropriate according to IEC 62304.

Select the row and open the Suggestions tab in the modal on the right side of the page to see detailed information about the evaluation.

At the top you can see a suggested description, an example of how you could rewrite your test description to address its main weaknesses.

You can also check how the test scored on each criterion above, including the specific areas of strength and weakness.

Test Suites

In this tab you can create, update and delete Test Suites. The displayed Test Suites refer to the current system version and group multiple tests of the same type. A Test Suite is either automatic or manual, and the related tests must be coherent to the defined type.

For each Test Suite, the following information is displayed:

• Title: Name of the Test Suite.
• Execution Mode: Automatic or manual, strictly tied to the Execution Mode of the associated tests.
• Tests: Number of tests included in the suite.
• Last Execution: Status and date of the last execution.

Test Suites Management

Creating Test Suites

• Add suite: Create a new Test Suite, containing one or more tests. NB. Test suites can be created for software versions that are not yet released.
• Add tests from All Tests tab: Tests can also be added to new or existing Test Suites directly from the All tests tab using the Add to Test Suite feature.

Running Test Suites

• Run all: Run all automated Test Suites that are configured with an External Test Executor.
• Run Test Suite: Execute a specific automated Test Suite when its API Trigger is configured. Multiple Test Suites can be selected and executed simultaneously.
• Manual executions: Must be handled in the ALM tool, where testers initiate and update execution information.

Other Actions

• View Details: Click on a row to access detailed information about the selected Test Suite.
• Edit Test Suite: Modify Test Suite properties such as the title.
• Delete Test Suite: Remove a specific Test Suite without deleting the associated tests.

Test Suite Details

Clicking on a row displays the Test Suite details in three tabs:

Tests Tab

Shows all tests included in the Test Suite with their individual properties and status.

From this section you can perform the following actions:

• Edit test suite: change the basic test suite information (name and execution mode).
• Run test suite: execute the automated Test Suite when its API Trigger is configured. Progress information is available in the Test Suites section.

Executions Tab

Displays chronological executions of the specific Test Suite, with options to download corresponding reports when available.

External Test Executor Tab

Configure the external service required for automatic test suite execution and run the automated test suite. Progress information is available in the Test Suites section.

Configuration Requirements:

• P4SaMD requires a jobId to trigger the executor correctly
• Include the {{@jobId}} placeholder in at least one of: Endpoint URL, Payload, Header or Query parameters
• Without proper jobId configuration, the executor cannot run

Endpoint URL and Query Parameters:

Enter the endpoint URL in full (including query parameters) or build it incrementally using the dedicated Query Parameters table (recommended solution). Any query parameter included in the saved endpoint URL is parsed and added to the table, where individual values can be edited or removed without touching the full URL string.

Endpoint Validation:

After the configuration is saved, P4SaMD performs a reachability check on the configured endpoint URL to verify it is correctly set up. The check issues a lightweight request to the endpoint without triggering an actual job execution. A status badge is displayed next to the endpoint URL in the configuration card:

Badge	Meaning
Checking…	The reachability check is in progress.
Reachable	The endpoint responded successfully.
Unreachable	The endpoint could not be reached or returned an error.

The validation badge is reset whenever the configuration modal is reopened, and a fresh check is run each time the configuration is saved successfully.

Automatic Test Suite Execution

Execution Flow

1. Initialization: User triggers execution → new job created with assigned JobId
2. Running: jobId inserted into configured location → External Test Executor runs → job updates based on response
3. Results: External executor sends updates via webhook → execution status is updated → results are processed
4. Reports: JUnit format reports are automatically processed and displayed; other received formats are saved and are downloadable

Configuration Prerequisites

• External test executor must be configured
• {{@jobId}} placeholder must be set in test suite details
• Without proper configuration, P4SaMD cannot trigger or run test suites

Webhook Integration

External Test Executors update job status via webhook:

https://{{domain}}p4samd/webhook/update-run-test-suite/:jobId

Supported Parameters:

Name	Type	Description
outcome	string	Updates the status of the job execution
result	string	Final test results (if report not in JUnit format)
message	string	Execution description
junitXml	string	JUnit format report (automatically processed)
reportFile	file	Report file (downloadable, not auto-processed)

Executions

This tab lists all test executions sorted by date, newest first. Each execution is a single procedure in which the user selects one or more tests, executes them and can then access the outcomes and reports. NB. The selection is allowed just for automatic Test Suites, hence the executions and their reports refer to automatic tests only. Manual tests are performed by human testers and need to be handled in the integrated ALM tool. The related report can be produced through the templating system: see the reference Documentation engine section.

For each execution, the following pieces of information are available:

• Date and time: When the execution was performed.
• Note: Show an icon if there are some notes.
• Test suites: Number of Test Suites involved in the execution, or the title if only one.
• Outcome: Result of the execution of all Test Suites included in that execution (passed, failed, etc.).
• Executed by: User who performed the execution.
• Download report: Download a ZIP archive containing a summary of the execution and all available individual reports from that execution.

Execution Reports

Each execution provides a downloadable ZIP archive containing:

• README.md: A summary file with execution details, including:
  • Execution date and executor name
  • Overview of all Test Suites included in the execution
  • Final result for each Test Suite (passed, failed, etc.)
  • References to corresponding report files when available
• Individual Reports: All available Test Suite reports from that execution

Detail

Clicking on a row opens a new page showing the execution details: the execution date, the user who performed the execution, and any associated notes. The view also lists all Test Suites involved in the execution along with their related tests. You can download a report for the Test Suite execution, which includes a summary file and all available individual Test Suite reports as attachments.