SFO15-TR6-A: Openstack Testing on ARM64
SFO15-TR6-A: Openstack Testing on ARM64 Presented by Clark Laughlin
Date Wednesday 23 September 2015
Event SFO15
Clark Laughlin
Agenda 1. Introduction 2. Progress since HKG15 3. Current testing effort a. Why aren’t we running an Openstack 3rd-party CI setup? b. openstack.linaro.org & current CI c. results
4. Issues and Next Steps 5. Questions?
Openstack Hopefully know what Openstack is. If not… http://www.openstack.org/software/
Devstack A set of scripts for deploying an Openstack developer environment. Makes many assumptions, and deploys a working, non-production Openstack deployment on a single node.
http://docs.openstack.org/developer/devstack/
Tempest Integration test suite for verifying an existing Openstack deployment ● API tests, scenarios (no benchmarking) ● Running as part of a Devstack deployment is much easier -- Devstack generates the (mostly) proper Tempest configuration http://docs.openstack.org/developer/tempest/
Since HKG15...
Since HKG15... March 2015 Distro/Version
September 2015
Ubuntu/Trusty
Trusty
Wily
713
1521
1522
470 (66%)
1431 (94%)
1432 (94%)
Fail
51 (7%)
23 (1.5%)
23 (1.5%)
Skip
192 (27%)
5 (0.3%)
5 (0.3%)
No analysis done
62
63
Tests Run Pass
Ignored Not tested
cinder, neutron
Since HKG15... enabled cinder, neutron
enabled missing CONFIG_* switched to kernel 4.1.x
virtio-scsi, cirros
Current Testing - Why Not What We Originally Planned? We did go through the process of creating an Openstack 3rd-Party CI system as we had planned and presented at HKG15. So why aren’t we using it?
Current Testing - Why Not What We Originally Planned? ● Requires more ARM64 nodes than we had available to scale ● Not enough personnel to support… we wanted lunch breaks and sleep ● Openstack’s third-party CI processes are not geared towards platform testing, which is what we wanted to accomplish
So, what ARE we doing? ● Nightly Tempest runs on ARM64 systems ○ Jenkins Matrix Job: ■ branch: master, stable/kilo (latest released stable) ■ os distro: Ubuntu ■ os version: trusty, vivid, wily ○ Devstack & Tempest deployed and run by LAVA job
So, what ARE we doing? Results reported publicly:
http://openstack.linaro.org
Openstack.linaro.org: Overall Summary
Openstack.linaro.org: Historical Graph
Openstack.linaro.org: Single Test
Failing Tests ● Depending on the test combination, ~19-28 tests currently failing ● Many failures we have eliminated have been due to Tempest configuration or the cloud image we were using ● Causes of remaining failures have not been root-caused yet
Skipped Tests ● Expected: ○ Depending on the test combination, ~60-90 tests are expected to be skipped for various reasons ○ Various reasons: ■ subsystem needed by test not installed, i.e. ironic, nova-network ■ tests disabled due to known Openstack bugs ■ feature disabled, i.e. VNC console
Skipped Tests ● Unexpected: ○ Currently, only about 5 tests are being skipped ○ 3 are due to Tempest configuration settings that still need to be changed ○ 2 (EC2 tests) are due to the cloud image we are using not having ami/aki/ari manifest
Issues And Next Steps Issue
Next Step(s)
Publicising Results - since we are not running as an official 3rd-party CI, our results are not readily visible to developers
Continue work with Openstack CI and Third-Party Testing Teams to determine the best way to share results
Hardware - currently only utilizing APM Mustang boards
Two HP Moonshot enclosures with 40 m400 cartridges are now available. Work to integrate these into the test environment. Will you provide your hardware to include in our test environment?
Issues And Next Steps #2 Issue
Next Step(s)
Testing is not against a production deployment with HA/redundancy/failover -everything is running on a single node
This is standard practice for normal Openstack CI. However, we are looking into testing against more production-quality deployments.
Only testing on Ubuntu
Incorporate additional OS distros / versions ● ●
Failure count still > 0
Debian CentOS
Continue to root-cause remaining failures
Presentation Finished.
Presentation Finished. Any questions?
Any Questions?
Date Wednesday 23 September 2015
Event SFO15
Clark Laughlin
Agenda 1. Introduction 2. Progress since HKG15 3. Current testing effort a. Why aren’t we running an Openstack 3rd-party CI setup? b. openstack.linaro.org & current CI c. results
4. Issues and Next Steps 5. Questions?
Openstack Hopefully know what Openstack is. If not… http://www.openstack.org/software/
Devstack A set of scripts for deploying an Openstack developer environment. Makes many assumptions, and deploys a working, non-production Openstack deployment on a single node.
http://docs.openstack.org/developer/devstack/
Tempest Integration test suite for verifying an existing Openstack deployment ● API tests, scenarios (no benchmarking) ● Running as part of a Devstack deployment is much easier -- Devstack generates the (mostly) proper Tempest configuration http://docs.openstack.org/developer/tempest/
Since HKG15...
Since HKG15... March 2015 Distro/Version
September 2015
Ubuntu/Trusty
Trusty
Wily
713
1521
1522
470 (66%)
1431 (94%)
1432 (94%)
Fail
51 (7%)
23 (1.5%)
23 (1.5%)
Skip
192 (27%)
5 (0.3%)
5 (0.3%)
No analysis done
62
63
Tests Run Pass
Ignored Not tested
cinder, neutron
Since HKG15... enabled cinder, neutron
enabled missing CONFIG_* switched to kernel 4.1.x
virtio-scsi, cirros
Current Testing - Why Not What We Originally Planned? We did go through the process of creating an Openstack 3rd-Party CI system as we had planned and presented at HKG15. So why aren’t we using it?
Current Testing - Why Not What We Originally Planned? ● Requires more ARM64 nodes than we had available to scale ● Not enough personnel to support… we wanted lunch breaks and sleep ● Openstack’s third-party CI processes are not geared towards platform testing, which is what we wanted to accomplish
So, what ARE we doing? ● Nightly Tempest runs on ARM64 systems ○ Jenkins Matrix Job: ■ branch: master, stable/kilo (latest released stable) ■ os distro: Ubuntu ■ os version: trusty, vivid, wily ○ Devstack & Tempest deployed and run by LAVA job
So, what ARE we doing? Results reported publicly:
http://openstack.linaro.org
Openstack.linaro.org: Overall Summary
Openstack.linaro.org: Historical Graph
Openstack.linaro.org: Single Test
Failing Tests ● Depending on the test combination, ~19-28 tests currently failing ● Many failures we have eliminated have been due to Tempest configuration or the cloud image we were using ● Causes of remaining failures have not been root-caused yet
Skipped Tests ● Expected: ○ Depending on the test combination, ~60-90 tests are expected to be skipped for various reasons ○ Various reasons: ■ subsystem needed by test not installed, i.e. ironic, nova-network ■ tests disabled due to known Openstack bugs ■ feature disabled, i.e. VNC console
Skipped Tests ● Unexpected: ○ Currently, only about 5 tests are being skipped ○ 3 are due to Tempest configuration settings that still need to be changed ○ 2 (EC2 tests) are due to the cloud image we are using not having ami/aki/ari manifest
Issues And Next Steps Issue
Next Step(s)
Publicising Results - since we are not running as an official 3rd-party CI, our results are not readily visible to developers
Continue work with Openstack CI and Third-Party Testing Teams to determine the best way to share results
Hardware - currently only utilizing APM Mustang boards
Two HP Moonshot enclosures with 40 m400 cartridges are now available. Work to integrate these into the test environment. Will you provide your hardware to include in our test environment?
Issues And Next Steps #2 Issue
Next Step(s)
Testing is not against a production deployment with HA/redundancy/failover -everything is running on a single node
This is standard practice for normal Openstack CI. However, we are looking into testing against more production-quality deployments.
Only testing on Ubuntu
Incorporate additional OS distros / versions ● ●
Failure count still > 0
Debian CentOS
Continue to root-cause remaining failures
Presentation Finished.
Presentation Finished. Any questions?
Any Questions?
