lfn-cnti/testsuite v2.0.0 · lfn-cnti/testsuite-action v1.0.0
A test result is only useful if you can trust it and act on it. In version 2.0.0 of the CNTi Test Suite, a passed test tells you what was checked, and a failed test tells you what was found and how to fix it.
The release brings a suite with a clear purpose, results you can check, and a real 5G core to prove it on. With it comes a GitHub Action that puts the suite into your pipeline in one step.
Every result says what it found
This is output from a run of version 2.0.0 against free5GC, an open source 5G core:
Every failed test names the resources it found and carries a remediation. The text comes from the test’s entry in the documentation, which is built into the binary, so the results and the documentation always agree.
A passing test shows its work as well:
The same care went into the rest:
- Security findings name the field and the container.
- Chaos experiments report the step and the probes of the LitmusChaos run.
Proven on a real 5G core
free5GC is our reference CNF. We run the essential tests against it every night. When a test and a real network function disagree, we find out which of the two is right, and the suite gets better either way.
The example lives in example-cnfs/free5gc, with the configuration we use, ready for you to run.
One step in your pipeline: the GitHub Action
Together with the suite we release testsuite-action 1.0.0. It runs the suite against your CNF in a GitHub Actions job, so every pull request to your Helm chart is checked. For a chart in your repository, one step is enough:
The action creates a three-node kind cluster, installs the suite, deploys your CNF and runs the cert tests. No cluster of your own is needed. When the job is done you have:
- a summary of the run on the job page, and an annotation for every failed test;
- the results file as an artifact;
- outputs for the steps that follow, such as the number of essential tests passed;
- a badge for your README, if you want one.
The badge shows the result of the last run on your default branch, for example cert | 17/19. It is published to a branch of your own repository and needs no secrets.
Anything the suite can install works with the action as well: charts from a repository or an OCI registry, plain manifests, several deployments, or an image built in the same job. It can also run other tests than cert, and use a cluster you provide.
A focused suite
The suite asks one clear question: does this network function follow cloud native practices on Kubernetes? Everything in it serves that question. The platform, RAN and 5G functional suites have been retired, so that every test left is one we stand behind.
Seven categories, 67 tests, 19 of them essential. The essential tests are the ones cert judges.
Tests that measure what they claim
We went through the tests one by one and asked of each: does it observe the thing it judges? Many were reworked.
- sig_term_handled sends the signal and checks that the processes have stopped. It judges the outcome, for every pod and every container.
- zombie_handled finds zombie processes on every node, whatever the cgroup version.
- The process tests say what they checked when they pass: the pod, the container and the processes they looked at.
- specialized_init_system reports every pod it found at fault, not only the first.
- Tracing asks the Jaeger API whether your pods appear in real traces.
- cni_compatible is a fast static check of your manifests for coupling to one CNI plugin.
- alpha_k8s_apis is a static check of your manifests for alpha API versions.
The suite also sees more of your CNF: custom resource definitions that an operator installs are discovered and included in what the tests read.
Where a test cannot measure, it reports skipped and says why. A pass means that something was measured.
A new CLI that behaves like a first-class Unix tool
The binary now carries the name of the project: cnf-testsuite has become cnti-testsuite, and the install line and the environment variables follow it.
Every option is GNU-style and validated before anything runs. Where the old grammar is used, the suite tells you what to type instead:
Exit codes mean something: 0 the run met its objective, 1 it did not, 2 a test errored, 64 the command line was wrong. Logs go to stderr and results to stdout. Bash and zsh completion comes with the binary.
cnti/results/latest.yml always points at the newest run, and a run writes only inside the cnti/ directory. The results file has a versioned schema, so tools that read it know what to expect.
Current tools, and a way to stay current
LitmusChaos 3.31 · Kubescape 4.0 · Kyverno 1.19 · Helm 4 · Crystal 1.19.1
Every pinned version is watched by Renovate. setup installs only what your tests use: the chaos tests bring Litmus, the security tests bring Kubescape.
Upgrading
Your results will be more precise, so some numbers will move. The result itself tells you why.
- Look at the skipped count first. It says where a test could not measure.
- Scripts that use the old key=value grammar get exit code 64 and a message with the new form.
- The binary, the environment variables and the results directory have new names.
What comes next
We have many ideas for the releases to come. Some examples:
- better integration with AI tools;
- more reference workloads;
- improved tests beyond the essential ones;
- new tests;
- a fresh look at which tests are essential and what cert requires.
If you maintain a network function or have an idea of your own, we would like to hear from you.
Try it
Start with CoreDNS for speed, then try free5GC to see a real network function judged. Or skip the cluster and add the action to your repository.
Questions, ideas or a counterexample: the tracker · #lfn-cnti-discussion on LFN Zulip.
