Complete Automotive Tests in a Box

Complete Automotive Tests in a Box (CATBox) is an open source package that provides the frameworks and tools required to streamline and execute automotive tests with minimal configuration. CATBox is scalable and compatible with the tools, infrastructure, and tests you use to test and validate your Android Automotive OS (AAOS) implementations. As a result, you can provide high-quality products to the marketplace.

CATBox design and implementation

The CATBox test suite provides the tools, frameworks, and tests needed to execute functional and performance tests on an automotive target device and while using a companion device.

The following image illustrates a high level design of a CATBox package.

CATBox

Figure 1. CATBox package

Tradefed

Tradefed is an open source continuous testing framework used to execute tests on Android devices. For details, see Trade Federation Overview.

Automotive test framework

CATBox is based on an Automotive test framework called Spectatio, which provides APIs for testing various apps on automotive devices. It is built on top of UI Automator, an open source testing framework, that provides a set of APIs to build UI tests that interact with user and system applications.

CATBox test suite downloads

Android 14

Android Automotive 14 is the release of the development milestone code-named U. Use the following links to download CATBox test suites for Android 14:

Android 12

Android 12 is the release of the development milestone code-named S. Use the following links to download CATBox test suites for Android 12:

Android 11

Android 11 is the release of the development milestone code-named R. Use the following links to download CATBox test suites for Android 11:

Build CATBox

Use the catbox build command once the Android AOSP code is available on the local workstation.

To build the CATBox package locally:

Select the Device Target based on the device architecture and run the following command:
```
lunch <target>
```
To build CATBox, run the following command:
```
make catbox
```
This command creates the android-catbox.zip file in the /out/host/linux-x86/catbox directory, which can then be used to run tests.

Set up the environment

The environment setup required to execute CATBox tests is similar to Setting up CTS. To set up the environment, complete each of the following tasks:

Install ADB and AAPT.
Install JDK. See Java Development Kit for Ubuntu.
Increase memory limit. See Storage requirements.

Set up and run CATBox

Use the Tradefed test framework to run the tests on Android devices. Before you set up CATBox, see the Trade Federation Overview.

To set up and run CATBox, see the sections below.

Set up the device

Before running a test plan, complete the following steps to set up your device:

Ensure that you have installed the recent versions of both the Android Debug Bridge (adb) and the Android Asset Packaging Tool (AAPT), as well as added the location of these tools to the system path of your machine. For details, see ADB and AAPT.
Connect at least one device and prepare the Device Under Test (DUT) as follows:
1. To factory reset the automotive device, go to Settings > System > Reset Options > Erase all data.
2. Wait for the device to automatically restart.
3. Make sure adb is enabled. To confirm the device is available, run the following command:
```
adb devices
```
4. To enable root, run the following command:
```
adb -s DEVICE_SERIAL root
```
  Replace DEVICE_SERIAL with the Serial ID for the automotive device.
5. Connect to Wi-Fi.

Extract the CATBox package

After setting up the device, unzip the CATBox package and go to the extracted folder:

unzip android-catbox.zip
cd android-catbox

Execute tests in CATBox

To run CATBox with the test plans in the CATBox package:

./tools/catbox-tradefed run commandAndExit TEST_PLAN_NAME --serial DEVICE_SERIAL

Replace DEVICE_SERIAL with the Serial ID for the automotive device. If you have multiple devices connected to the host machine, use DEVICE_SERIAL to distinguish them. Replace TEST_PLAN_NAME with the name of the test plan you want to run.

To get the most complete list of available plans, use the command line:

./tools/catbox-tradefed list plans | grep -i catbox

Types of test plans

The following table displays the available functional test plans.

Functional test plans	Description
`catbox-functional`	Functional tests.
`catbox-functional-app-info-setting`	App info setting functional tests.
`catbox-functional-admin-user-grant-permissions`	Admin user grant permissions functional tests.
`catbox-functional-app-info-setting-ui-elements`	App info setting functional tests.
`catbox-functional-appgrid`	App grid functional tests.
`catbox-functional-base`	List of functional tests.
`catbox-functional-bluetooth-audio`	Bluetooth audio Functional tests.
`catbox-functional-bluetooth-palette`	Status bar functional tests.
`catbox-functional-bluetooth-tests`	Bluetooth tests using Mobly.
`catbox-functional-brightness-palette`	Status bar functional tests.
`catbox-functional-btmoped`	Bluetooth Moped-based functional tests.
`catbox-functional-date-time-setting`	Date and time setting functional tests.
`catbox-functional-dial`	Dial functional tests.
`catbox-functional-display-setting`	Display setting functional tests.
`catbox-functional-driving-optimized-apps`	UX restriction functional tests.
`catbox-functional-enable-developers-option`	Enable developers option functional tests.
`catbox-functional-home`	Home functional tests.
`catbox-functional-lock-screen`	Lock screen functional tests.
`catbox-functional-mediacenter`	Media Center functional tests.
`catbox-functional-microphone-recent-apps`	Microphone setting functional tests.
`catbox-functional-microphone-setting`	Microphone setting functional tests.
`catbox-functional-multiuser`	Multiuser functional tests.
`catbox-functional-multiuser-system-user`	Multiuser functional tests.
`catbox-functional-navigation-bar`	Navigation bar functional tests.
`catbox-functional-network-palette`	Status bar functional tests.
`catbox-functional-network-setting`	Network setting functional tests.
`catbox-functional-notification`	Notification functional tests.
`catbox-functional-privacy-permission-manager`	Privacy App Permission setting functional tests.
`catbox-functional-privacy-setting`	Microphone setting functional tests.
`catbox-functional-privacy-setting-ui-elements`	Privacy setting functional tests.
`catbox-functional-profile-icon`	Profile Icon Bar functional tests.
`catbox-functional-profiles-icon-list`	Profiles icon list tests.
`catbox-functional-security-setting`	Security setting functional tests.
`catbox-functional-setting`	Setting functional tests.
`catbox-functional-settings-location`	Setting location Tests.
`catbox-functional-sound-setting`	Sound setting functional tests.
`catbox-functional-status-bar`	Status bar functional tests.
`catbox-functional-storage-setting`	Storage setting functional tests.
`catbox-functional-system-setting`	System setting functional tests.
`catbox-functional-systemui`	System UI tests.
`catbox-functional-ux-restriction`	UX restriction functional tests.
`catbox-functional-ux-restriction-facet-bar`	UX restriction functional tests.

The following table lists CATBox performance test plans.

Performance test plans	Description
`catbox-performance-cold-app-start-up-dialer`	Cold app start-up performance test for Dialer and Phone.
`catbox-performance-cold-app-start-up-mediacenter`	Cold app start-up performance test for Media Center.
`catbox-performance-cold-app-start-up-settings`	Cold app start-up performance test for settings.
`catbox-performance-create-and-switch-to-new-guest`	Measure latency to switch to a new guest.
`catbox-performance-create-and-switch-to-new-user`	Measure latency to switch to a new non-admin user.
`catbox-performance-hot-app-start-up-dialer`	Hot app start-up performance test for Dialer and Phone.
`catbox-performance-hot-app-start-up-mediacenter`	Hot app start-up performance test for Media Center.
`catbox-performance-hot-app-start-up-settings`	Hot app start-up performance test for Settings.
`catbox-performance-jank-appgrid`	Performance test for Appgrid.
`catbox-performance-jank-contact-list`	Performance test for Contacts.
`catbox-performance-jank-media`	Performance tests for Media.
`catbox-performance-jank-media-switch-playback`	Performance test for media switch playback.
`catbox-performance-jank-notifications`	Performance test for Notifications.
`catbox-performance-jank-settings`	Performance test for Settings.
`catbox-performance-switch-to-existing-user`	Measure latency to switch to a existing user.

Run the multidevice host side test

The execution of multidevice host side tests uses CATBox to run the test in one session. For example, BTDiscoveryTest:

make catbox
./tools/catbox-tradefed run commandAndExit catbox-functional-bluetooth-tests --{phone}serial <serial> --{auto}serial <serial> --mobly-host:mobly-par-file-name BTDiscoveryTest

CATBox installs the apk and then the test is run in the correct Python execution environment.

Review CATBox test results

Results of functional and performance test runs are saved in android-catbox/results/latest.

Result type	Location
Test results	`test_result.xml`
Failure tests details	`test_result_failures_suite.html`
Performance metrics results	`/report-log-files/CatboxPerformanceTests.reportlog.json`
Functional and performance test runs	`/android-catbox/logs/latest`