DO Qualification Kit

Simulink Test
Test Cases and Procedures
R2015b, September 2015

How to Contact MathWorks

Latest news:

www.mathworks.com

Sales and services:

www.mathworks.com/sales_and_services

User community:

www.mathworks.com/matlabcentral

Technical support:

www.mathworks.com/support/contact_us

Phone:

508-647-7000

The MathWorks, Inc.

3 Apple Hill Drive
Natick, MA 01760-2098
DO Qualification Kit: Simulink Test Test Cases and Procedures

COPYRIGHT 2015 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used or copied only under
the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written
consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the
federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees
that this software or documentation qualifies as commercial computer software or commercial computer software documentation
as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and
conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,
reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or
other entity acquiring for or through the federal government)and shall supersede any conflicting contractual terms or conditions.
If this License fails to meet the governments needs or is inconsistent in any respect with federal procurement law, the
government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a
list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective
holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more
information.

Revision History
September 2015

New for DO Qualification Kit Version 3.0 (Applies to Release 2015b)

Contents
1
2
3
4
5

Introduction ...................................................................................................................................... 1-1

Test Cases and Procedures ............................................................................................................... 2-1
Traceability Between Operational Requirements and Test Cases and Procedures........................... 3-1
Executing Test Cases and Procedures .............................................................................................. 4-1
Test Results ...................................................................................................................................... 5-1

vi

1 Introduction
This document comprises test procedures (Reference DO-330 Sections 10.3.3.c) for the
following capabilities of the Simulink Test verification tool:

Comparing simulation results or executable object code test results against expected results
to determine pass/fail simulation or test status.
Generating reports that contain simulation or test results, including pass/fail simulation or
test status, and requirements tracing.

This document is intended for use in the DO-330 tool qualification process for TQL-5 tools.
See also the DO Qualification Kit User's Guide, R2015b.

1-2

2 Test Cases and Procedures

Simulink Test Manager test files (*. mldatx), containing positive and negative test cases for
baseline, simulation, and equivalence tests, exercise the qualifying Simulink Test capabilities.
The test files invoke simulation of test models (*.slx), evaluate the simulation results according
to the pass/fail criteria specified in the test files (using *.mat files with expected results for
baseline test cases), and generate test reports. The test files run in a batch mode, and output test
reports are compared with expected results using Report Generator setup file. The qualification
test report summarizes the outcome of the comparison. Figure 1 shows testing workflow.
Microsoft Windows platform with Microsoft Word 2013 or higher supports automated
comparison of .docx report formats. See Simulink Test Tool Operational Requirements and
Executing Test Cases and Procedures in this document.

Figure 1: Test Workflow

Requirements-based test cases and procedures have been developed in:

Simulink Test Tool Operational Requirements
To access the tool operational requirements document, on the MATLAB command line, type
qualkitdo to open the Artifacts Explorer. The document is in Simulink Test.
The aggregated test procedure, qualkitdo_sltest_rs.rpt, is a MATLAB Report
Generator file. Executing the file:

Runs the test sub-procedures in batch mode.

Evaluates the qualification test outcome.
Generates a summary qualification report.

The file is located in:

matlabroot/toolbox/qualkits/do/sltest/
The test cases, associated test sub-procedures, and expected results are provided as:

Simulink test models with embedded test harness *.slx

Simulink Test Manager test files *.mldatx
MAT-files containing test vectors for baseline tests *.mat
Pre-generated test reports in all supported formats *.zip, *.pdf and *.docx

The files are located in:

matlabroot/toolbox/qualkits/do/sltest/tests/
The expected results are located in:
matlabroot/toolbox/qualkits/do/sltest/expected_results/

2-2

3 Traceability Between Operational

Requirements and Test Cases
and Procedures
A list of the test files and associations between them is provided in the traceability matrix.
Bi-directional traceability between the Tool Operation Requirements and test cases and
procedures is provided in a traceability matrix:
qualkitdo_sltest_trace.xlsx
The traceability matrix is located in:
matlabroot/toolbox/qualkits/do/sltest/

3-2

4 Executing Test Cases and

Procedures
In the scope of Simulink Test Tool qualification process, the applicant will:

Review the Tool Operational Requirements for applicability to the project under
consideration and select the relevant test procedures to be run in the installed users
environment (activity 6.2.1.aa of DO-330)
Configure and execute the test procedures in the installed user environment
Evaluate the generated test reports are correct (activities 6.2.1.b and 6.2.1.bb of DO330)
Configure shipped test data and generated test reports in a configuration management
system (activities 7.2.7 of the of DO-330)

Note:

To run the test cases and procedures, copy the folder

matlabroot/toolbox/qualkits/do/sltest and its subfolders to a location
where you have write access, and then run the test procedures from there.

Before you execute the test procedures, start a new MATLAB session (the Report
Generator setup file runs MATLAB code in global workspace and might interfere
with data existing in the global workspace).

To execute the test procedure you must have the following licenses:
o
o
o
o
o

MATLAB
Simulink
Simulink Test
MATLAB Report Generator
Simulink Report Generator

A Fixed-Point Designer license is required to qualify comparison of fixed-point data

types.

Execute the Simulink Test qualification test procedure:

1.

Specify the Simulink Test report format to be qualified using the

qualkitdoCtrl.sltest.reportFormat variable in the base workspace:

At the MATLAB command line, enter

-

qualkitdoCtrl.sltest.reportFormat = {'pdf'}; to generate

test reports in PDF format. PDF formats are not supported on Linux
platforms.

qualkitdoCtrl.sltest.reportFormat = {'zip'}; to generate

test reports in HTML format compressed to zip archive.

qualkitdoCtrl.sltest.reportFormat = {'docx'} to generate

test reports in Microsoft Word .docx format.

Note: If a report format is not specified, by default, the HTML format will be used.

2.

Navigate to the folder where you copied the content of

matlabroot/toolbox/qualkits/do/sltest folder

Run the qualification test:

Using the GUI:

-

Double-click the qualkitdo_sltest_rs.rpt file to open the Report

Explorer

Select File > Report. Simulink Report Generator creates the summary report
and the model level reports

At the MATLAB command line, enter

report ('qualkitdo_sltest_rs.rpt')
Simulink Report Generator sequentially runs the Simulink Test Manager test files and
stores the generated Simulink Test reports in:
sltest/outputs/

4-2

After running the tests, Simulink Report Generator compares the generated Simulink
Test reports to the expected results (pre-generated reports that ship with the DO
Qualification Kit) and generates the summary qualification report. The summary
qualification report contains information about test environment, executed test files, and
qualification results.
Note: Automated comparison is supported for HTML and Microsoft Word .docx
report formats, depending on the operational environment. If automated comparison is
not supported for selected formats, manually compare the reports to the expected
results. Automated report evaluation is supported in the following environments:
Platform

Report Format

Linux

HTML

Windows

3.

(32-bit and 64-bit)

HTML, Microsoft Word .docx

After Simulink Report Generator has finished running the aggregate test procedure, verify
that there are no warning or error messages in the Simulink Report Generator message list
window. Review the summary qualification report. For more information, see Test Results.
If you do not have a Fixed-Point Designer license, the test file tFP.mldatx, containing
test cases for fixed-point data types, will not run. If you do not need to qualify comparison
of fixed-point data types, you can ignore this warning message.

4.

In your environment, manually verify the requirement SLT_TIR_010:

a.

Using Simulink Test Manager GUI, create a baseline test case for the Simulink
model mTIR010.slx, stored in the sltest/outputs/ folder.

b.

Using the Simulink Test Manager GUI, capture the outputs of the model and store
it as a MAT-file.

c.

Set the captured MAT-file as a baseline criteria with 0 absolute and relative
tolerances.

d.

Run created test case. Ensure that it passes.

Note: Simulink Test does not provide an API to capture the simulation outputs.
This capability must be tested manually.

4-3

5 Test Results
Executing the aggregate test procedure file qualkitdo_sltest_rs.rpt generates and
stores the following data in /sltest/outputs/:

Summary qualification report

qualkitdo_simulinktest_qualificationreport.html

Simulink Test reports in the specified format. The list of reports will be included in
summary qualification report.

The applicant will:

1.

Review the test results:

If automated comparison of the reports is supported in your environment, review the

generated summary qualification report
qualkitdo_simulinktest_qualificationreport.html stored in the
sltest/outputs/ folder:
-

Check that the information regarding test environment is correct and the list of
generated test reports includes all selected formats.

In the Qualification Test Results section of the report, check that the
qualification status is evaluated as PASS.
Note 1: If the qualification status is evaluated as FAIL, manually review the
content of the differing test reports to assess the impact of the mismatch with
respect to the software life cycle activities performed using the tool.
Note 2: Automation comparison capability is verified in your environment using
the pre-generated reports shipped with the DO Qualification Kit. A list of
corresponding test cases is provided in the CMP page of the traceability matrix
qualkitdo_sltest_trace.xlsx

If automatic comparison is not supported in your environment:

-

Review the summary qualification report

qualkitdo_simulinktest_qualificationreport.html in the
sltest/outputs/ folder. Check that information regarding test environment
is correct and the list of generated test reports includes all selected formats.

Compare each Simulink Test report file, listed in the section Qualification Test
Results of the summary qualification report, to the pre-generated version of the
report with the same file name provided in sltest/expected_results/.
Note: When comparing the reports dates, time stamps and test duration time may
differ between actual and expected results.

2.

If you are not qualifying comparison of fixed-point data types, the summary report will
include a relevant note in the Qualification Test Outcome section.

Configure the content of the folder comprising the test cases, procedures and results in a
configuration management system.

5-2