🧪 Testing

📜 Table of Contents

1. Running the tests
2. What is covered
3. Mocks
4. Writing a test
5. Manual verification against a real cluster
6. Chainsaw multi-node RWX E2E
7. CSI sanity
8. What CI runs

▶ Running the tests

mise run test

That is go test ./... -cover -coverprofile=coverage.out -outputdir=. -coverpkg=./.... Note -coverpkg=./..., which measures coverage across every package rather than only the package under test, so the number reflects how much of the driver the whole suite exercises.

# Coverage as HTML
mise run cover

# One package
go test ./internal/driver/...

# One test, verbose
go test ./internal/driver/ -run TestCreateVolume -v

# With the race detector
go test -race ./...

The whole suite is hermetic: no network, no cluster, no Linode token. If a test needs any of those, it is in the wrong place.

🎯 What is covered

Test file	Subject
internal/driver/controllerserver_test.go	Every controller RPC, including error paths
internal/driver/controller_helpers_test.go	Parameter parsing, handle formats, label normalization, idempotency comparisons, error mapping
internal/driver/nodeserver_test.go	Node RPC validation, locking, stats
internal/driver/nodeserver_helpers_test.go	Mount option assembly, mount-point setup
internal/driver/metadata_test.go	Node identity, workload kubeconfig selection, retryable region discovery, and concurrent region-cache reuse
internal/driver/identityserver_test.go	Plugin info, capabilities, probe readiness
internal/driver/driver_test.go	Setup, role validation, capability advertisement
pkg/linode-client/client_test.go	Client construction and configuration
pkg/mount-manager/safe_mounter_test.go	The mounter wrapper
tests/sanity/sanity_test.go	csi-sanity against the real servers over fakes. See CSI sanity

Focused boundary tests cover composite handles, label truncation, HTTP-to-gRPC error mapping, and region discovery that retries failures but caches successful results across concurrent requests.

🎭 Mocks

Three interfaces are mocked with gomock, generated into mocks/:

Mock	Generated from
mocks/mock_linodeclient.go	pkg/linode-client/client.go
mocks/mock_safe-mounter.go	pkg/mount-manager/safe_mounter.go
mocks/mock_filesystem.go	pkg/filesystem/filesystem.go

mise run gen-mock

Regenerate whenever you change one of those interfaces, and commit the result. mise run ci runs gen-mock before vet, so stale mocks surface as a compile error in CI rather than as a confusing test failure. Never edit a file in mocks/ by hand.

mocks/mock_metadata.go also exists and is checked in but is not produced by gen-mock; leave it alone unless you are changing the metadata interfaces deliberately.

The mock client is what makes the controller tests possible: they assert on the exact sequence of Linode API calls, which is the only way to pin down behavior like “publish makes no API write when the Linode is already in the ACL”.

✍ Writing a test

The existing tests are table-driven with gomock expectations. The shape to follow:

func TestSomething(t *testing.T) {
  t.Parallel()
  
  for _, tt := range []struct {
  name    string
  setup   func(*mocks.MockLinodeClient)
  wantErr codes.Code
  }{
    {
      name: "success",
      setup: func(c *mocks.MockLinodeClient) {
      c.EXPECT().GetNFSSpace(gomock.Any(), 42).Return(&linodego.NFSSpace{ID: 42}, nil)
      },
    },
    {
      name: "space not found",
      setup: func(c *mocks.MockLinodeClient) {
      c.EXPECT().GetNFSSpace(gomock.Any(), 42).Return(nil, &linodego.Error{Code: 404})
      },
      wantErr: codes.NotFound,
    },
  } {
    t.Run(tt.name, func(t *testing.T) {
      t.Parallel()
      // ...
    })
  }
}

Two conventions that matter here:

• Assert on the gRPC code, not the message string. The codes are the contract with the sidecars; the messages are not.
• Cover the idempotent path. Every operation that can be retried has a “second call finds the existing resource” path, and that path is where the real bugs live. A new RPC without such a test is incomplete.

Run mise run lint before pushing. golangci-lint run --fix will reformat as it goes, so run it before you finalize a diff rather than after.

🔬 Manual verification against a real cluster

The unit tests cannot tell you whether a mount works. For that, use the development cluster and walk the full lifecycle.

export KUBECONFIG="$(pwd)/nfs-csi-driver-dev-kubeconfig"

1. The driver registered.

kubectl -n kube-system get pods -l role=csi-linode
kubectl get csidrivers linodenfs.csi.linode.com
kubectl get csinodes -o 'custom-columns=NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'

Every node must list the driver. A node that does not will never mount anything.

2. Provisioning.

kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linode-nfs-test
provisioner: linodenfs.csi.linode.com
parameters:
  linodenfs.csi.linode.com/space-id: "42"
reclaimPolicy: Delete
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes: [ReadWriteMany]
  storageClassName: linode-nfs-test
  resources:
    requests:
      storage: 10Gi
EOF

kubectl wait --for=jsonpath='{.status.phase}'=Bound pvc/test-pvc --timeout=10m

The filesystem activation wait has a five-minute timeout; this is not the expected provisioning duration. The driver reads the Space access policy only for its mTLS mode; it does not validate, wait on, or update it. Make sure the policy is enabled, active, and admits the test cluster’s VPC before this step, or the PVC binds and the mount in the next step fails.

3. Mounting, and the multi-node case that matters most.

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-writers
spec:
  replicas: 3
  selector:
    matchLabels: {app: test-writers}
  template:
    metadata:
      labels: {app: test-writers}
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels: {app: test-writers}
              topologyKey: kubernetes.io/hostname
      containers:
        - name: writer
          image: busybox
          command: [/bin/sh, -c, 'echo "$(hostname) $(date)" >> /data/shared.log; sleep 86400']
          volumeMounts: [{name: data, mountPath: /data}]
      volumes:
        - name: data
          persistentVolumeClaim: {claimName: test-pvc}
EOF

kubectl rollout status deploy/test-writers --timeout=5m
kubectl exec deploy/test-writers -- cat /data/shared.log

The anti-affinity forces one replica per node, so shared.log containing three distinct hostnames proves that ReadWriteMany across nodes actually works. That is the single most valuable manual check, because it is exactly what unit tests cannot cover.

4. Volume stats.

kubectl exec deploy/test-writers -- df -h /data

5. Cleanup, in order.

kubectl delete deploy test-writers
kubectl delete pvc test-pvc
kubectl delete sc linode-nfs-test

Then confirm the backend filesystem is actually gone, since a leak here is invisible from Kubernetes:

curl -sS -H "Authorization: Bearer ${LINODE_API_TOKEN}" \
  https://api.linode.com/v4beta/nfs/spaces/42/filesystems

Things worth deliberately breaking while you are in here, because each one exercises a distinct code path that is easy to regress:

Scenario	Expected
A StorageClass with neither space parameter	PVC stays Pending with InvalidArgument in the events
A space-id that does not exist	NotFound
volumeMode: Block	InvalidArgument: only mount volume capabilities are supported
A dataSource of kind: PersistentVolumeClaim	InvalidArgument: unsupported volume content source
Deleting the PVC while a pod still uses it	The PVC waits on pvc-protection until the pod is gone
Restarting the node DaemonSet with a pod mounted	The pod keeps working; the mount lives in the host namespace
Two pods on the same node sharing the PVC	Both start without serializing; publish locks the path, not the volume

🪚 Chainsaw multi-node RWX E2E

This local Chainsaw suite verifies a real RWX mount across three nodes. Run it only against an authorized cluster with the driver already installed, at least three schedulable nodes, a pre-created Space, and a StorageClass that points to that Space. Set the standard KUBECONFIG environment variable and E2E_STORAGE_CLASS to the existing class name. Chainsaw creates a random ephemeral namespace for each test and cleans it up afterward. The suite does not install the driver or create, modify, or delete the shared StorageClass or Space.

Set the kubeconfig path and existing StorageClass name:

KUBECONFIG=/path/to/test.kubeconfig E2E_STORAGE_CLASS=<storage-class-name> mise run e2e-test

The task requires both variables. The test creates a PVC and three labeled writer pods in a Chainsaw-managed ephemeral namespace and checks that all unique markers are visible through the shared file. Its finally step removes the marker file before PVC cleanup so a Delete-policy StorageClass can reclaim the test filesystem. Chainsaw cleans up only the test-created resources and namespace.

🧷 CSI sanity

csi-sanity runs against the real driver in tests/sanity, with no cluster and no Linode account:

just sanity

That is go test -race -count=1 -v ./tests/sanity -run '^TestCSISanity$' -timeout 15m. It is also an ordinary Go test, so mise run test and go test ./... pick it up, which means CI runs it on every PR as part of mise run ci.

How it is wired, in tests/sanity/sanity_test.go:

1. Three fakes stand in for everything external: fake_sanity_linode_client.go implements the LinodeClient interface over in-memory spaces, filesystems, access policies, and snapshots; fake_sanity_metadata.go fakes the instance metadata service and the Kubernetes node lookups; fake_sanity_mounter.go fakes the mounter so nothing is actually mounted.
2. driver.NewTestServices builds the identity, controller, and node servers over those fakes, and the real NewNonBlockingGRPCServer serves all three on one Unix socket in a t.TempDir().
3. The test waits for the socket to accept a gRPC connection, then hands the endpoint to csisanity.Test with space-id: "1" as the volume parameters and a custom IDGen that produces {space}/{filesystem} handles, since the suite’s default identifiers are not valid handles for this driver.

Because the whole thing is in-process and hermetic, a sanity failure is a real protocol-conformance bug rather than an environment problem. Two consequences when you change the driver:

• Advertising a new capability turns on more sanity tests. capabilities.go is what the suite reads to decide which RPCs to exercise, so adding a capability without implementing it fails here first.
• A new backend call needs the fake to implement it. Each fake carries a compile-time interface assertion (var _ linodeclient.LinodeClient = (*fakeSanityLinodeClient)(nil) and the equivalents for the metadata and mounter fakes), so adding a method to one of those interfaces without updating the fake breaks the build rather than the test run. What sanity catches is the layer above that: a fake that compiles but returns something the CSI spec does not allow.

🤖 What CI runs

Workflow	Trigger	What it does
ci.yml	Push to main, every PR	mise run ci: fmt, gen-mock, vet, lint, test, build
helm.yml	PRs touching charts/, deploy/kubernetes/, the hack scripts, justfile, or mise.toml; GitHub release events	Sets chart metadata via mise run set-chart-version, lints the chart, verifies Kustomize, and publishes chart packages and the Helm index.
actionlint.yml	Workflow changes	Lints the workflow files.
image-build-push.yml	See the workflow	Builds and publishes images.
release.yml	Tags	Builds the image and attaches release artifacts.

mise run ci locally is the same command CI runs, so a green local run means a green pipeline.

Every workflow runs step-security/harden-runner, but policies vary by job. ci.yml, the helm.yml lint job, and autoupdate-gh-pages.yml use egress-policy: block with explicit allowlists; the helm.yml chart-release job and other workflows use audit. For block-mode jobs, downloads from new hosts fail unless added to the allowlist. Check that workflow’s allowed-endpoints before suspecting the code.

ci.yml does not run verify-kustomize; helm.yml does for matching chart-related PRs and published product releases.

📚 Related pages

• Development Setup
• Contributing