In this manner we provide a test strategy which fosters test driven pipelines with a result that meets our guaranteed Quality.
Each phase or Quality Gate will be described as follows.
|Phase / Quality Gate||Definition||tool|
|Component||an entity which may also be a building block which is tested atomically. This is not really a test but an entity|
|Template Testing||major part relies on templates that will processed during the deployment. It is mainly verification, and enforcement of custom policies on Kubernetes objects||validation with conftest|
|Smoke Testing||single component test. Basic health checks (availability and responsiveness).||helmfile-test, goss, terratest|
|Component Testing||Single component test with a set of application specific functional tests. Assess possible functional degradation. Can be reused in other testing approaches (performance, compatibility, integration)||terratest|
|Integration Testing||Consideration of multiple building blocks (or components). This Test approach will be utilized only for components which are integrated with other components (e.g monitoring or logging). Application specific tests, targeting the integrated functionality have to be initiated upfront in order to assess that integrations work as expected. Can be reused in other testing approaches.||terratest|
|System testing||Extends the integration testing approach to the fullest by including all building blocks. All components will be deployed into the same environment with all available component and integration tests being commenced. This would reduce the risk of overlooking unforeseen side-effects or hazards||terratest|
|Compatibility Component Testing||This Test approach is responsible for detecting possible breaking changes due to in-compatibility. Known as 'upgrade path testing'. This guarantees that deployed components seamlessly operate on previous state when doing drop-in upgrades. Those breaking changes can only be detected on major releases of the corresponding building block||gitlab+terratest|
|Compatibility integration Testing||This is basically integration Testing with the additional parameter of a matrix (or vector set) of component versions. All integration test suites would be commenced against every combination of component versions. This would reduce the risk of component upgrades that would possibly require upgrades of other components as well.||gitlab+terratest|
|Performance Testing||Otherwise known as load Testing, this approach makes use of existing functional test suites which are scaled up and out to assess performance indicators of a given component or point of integration. It would make sense to utilize vendor-provided benchmark suites if they exist.||gitlab with vendor-provided (benchmark suite from provider, e.g sysbench for mysql) visualization in planing.|
|Resiliency Testing||Otherwise known as chaos Testing. This would normally commenced by a dedicated solution to induce error conditions into the infrastructure in order to observe things like fail-over, redundancy and recovery with certain building blocks (at least those that are capable and configured with HA functionality).||ChaosMonkey et al., chaosmesh, kraken|
|Test reporting||Extends the integration Testing approach to the fullest by including all building blocks. All components will be deployed into the same environment with all available component and integration tests being commenced. This would reduce the risk of overlooking unforeseen side-effects or hazards||terratest|
upcoming feature grey text.
component Testing mostly relies on the vendor Testing. As the vendor/provider/project of a building block will most likely test their components by themselves.