imageThe Firmware Test Suite (fwts) is a Linux-based tool that automates firmware validation.  It is sponsored by Canonical Inc., the fine folks behind the Ubuntu Linux distribution.  Fwts is open-source and new versions are released monthly.  Fwts is comprised of 83 tests (as of version 16.03.00) which attempt to validate the following BIOS subsystems:

  • ACPI
  • CPU configuration
  • PCI/PCIe configuration
  • power management
  • security (e.g. TPM2 testing)
  • various other Legacy and UEFI BIOS interfaces

Most of the tests are ACPI-focused.  While this test suite runs on Linux, it validates OS-agnostic interfaces; therefore, it is equally helpful for making sure your firmware will work well with Windows too.  The caliber of these tests is so high that the UEFI Board of Directors has recommended that fwts be used as the ACPI Self-Certification test.

What Comes with Fwts?

You have two options when downloading and using the fwts: 1) the “standard” fwts package and 2) the fwts live image.

Standard Package

What I’m calling the “standard” package contains all the source code, build scripts, libraries, readme files, and packaging collateral that make up the fwts.  It is the ultimate in customization.  You can edit it, build it yourself, and run the tests from a Linux command-line in either batch or interactive mode.

To get started, read the README that comes with the standard package.  There is a quick start guide to building and installing the fwts, some help on command-line options, and a little bit about interpreting the results.

There is also a README_SOURCE.txt that discusses coding conventions of the fwts project and offers some more explanation of the project’s dependencies.

Links to the standard packages:

http://fwts.ubuntu.com/release/

Live Image

The corollary to the standard package is the live image.  What the standard package is to configurability, the live image is to simplicity.  It is a bootable image you burn to a USB flash drive.  Upon booting to it, there is a friendly GUI that greets you and helps you specify the tests you want to run.  You don’t need a Linux installation in order to run the tests—nothing could be easier.

The tests generate a results.log file on the USB flash drive which you can take to another computer and analyze.  Beautiful instructions for using the live image can be found here.

Links to the live images:

http://fwts.ubuntu.com/fwts-live/

Preparing to Run Fwts

I’m testing today with the Fwts Live Image, version 16.03.00.  By the way, there is extra meaning to the version number:  the first two digits represent the year, and the second two represent the month.  The first thing I need to do is burn the live image to a USB flash drive.  From Windows, I used the excellent freeware tool Rufus to burn the live image:

image

With that complete, I did a UEFI boot to the USB flash drive and arrived at the main menu, ready to run the tests.  Note that you do have to do a UEFI boot, as the image does not have a Legacy/MBR style boot sector.  Main menu:

image

Test Results

I ran all 83 tests and it took less than five minutes for the tests to complete.  Afterward, I moved the USB flash drive to another computer to examine the results.  All test results are stored in directory \fwts\<date>\<time>.  You’ll find the following logs:

  • cpuinfo.log – the kind of information you’d expect from a utility like CPU-Z:

image

  • acpidump.log – self explanatory:

image

  • dmesg.log – debug output from the tests
  • lspci.log – list of PCI devices and their configurations
  • results.log – a verbose log of all test results in simple text format
  • results.html – a verbose log of all test results in HTML format

Brief screenshot of results.html:

image

One of the neatest parts of the results.log file is the ADVICE lines.  These are notes from the fwts developers to the users of the fwts giving tips on what particular test results might mean.  An example:

Kernel message: [ 0.173780] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150619/hwxface-580)

ADVICE: The exception comes from kernel cannot find _S1 namespace object that contains the register values for the sleep state when kernel would like to setup all the sleep state information. This means that the kernel does not know how to enter the S1 sleep state, however, it should not be a problem if the S1 sleep state isn't supported intentionally.

With this “advice”, the developer reading through the log can better understand the nature of the failures.  The results.log ends with a nice summary of test pass/fail results:

image

With these results in hand, a BIOS developer can dig into the specific failures and start debugging.

Resources

There are excellent support resources for better understanding fwts and for implementing fwts into your development cycle.  Here are some of them:

Fwts wiki

https://wiki.ubuntu.com/FirmwareTestSuite

introductory slide deck

http://www.uefi.org/sites/default/files/resources/fwts_uefi_0920_2013.pdf

This is a nice introductory presentation by Alex Hung, a BIOS engineer at Canonical.

Fwts reference guide

https://wiki.ubuntu.com/Kernel/Reference/fwts

The Reference Guide is the detailed documentation for the fwts.  What’s particularly remarkable about it is that there is a help page for every test that goes into details about what the test is testing and expected results.  Highly recommended!

Contribute to Fwts

Launchpad sites

There are Launchpad sites for fwts development (including a live image-specific site) which provide bug tracking and interaction with the fwts development team.

https://launchpad.net/fwts

https://launchpad.net/fwts-live

Source

Git tree: http://kernel.ubuntu.com/git/hwe/fwts.git

Mailing Lists

fwts-announce:  Formal announcements for the FWTS and the FWTS-Live projects.

fwts-devel:  Patches, discussions about the FWTS and the FWTS-Live projects.

Conclusion

I hope this introduction helped you better understand fwts, and also helped you get started using it more quickly.  I was impressed by the support it has received by the community and the maturity it has achieved.

Post a Comment

  1. Thanks for the Firmware Test Suite write-up. Much appreciated!

    ReplyDelete

Be sure to select an account profile (e.g. Google, OpenID, etc.) before typing your comment!