VPC

This guide covers how VPC is used with CAPL clusters. By default, CAPL clusters are provisioned within VPC.

Default configuration

Each linode within a cluster gets provisioned with two interfaces:

1. eth0 (connected to VPC, for pod-to-pod traffic and public traffic)
2. eth1 (for nodebalancer traffic)

Key facts about VPC network configuration:

1. VPCs are provisioned with a private subnet 10.0.0.0/8.
2. All pod-to-pod communication happens over the VPC interface (eth0).
3. We assign a pod CIDR of range 10.192.0.0/10 for pod-to-pod communication.
4. By default, cilium is configured with native routing
5. Kubernetes host-scope IPAM mode is used to assign pod CIDRs to nodes. We run linode CCM with route-controller enabled which automatically adds/updates routes within VPC when pod cidrs are added/updated by k8s. This enables pod-to-pod traffic to be routable within the VPC.
6. kube-proxy is disabled by default.

Configuring the VPC interface

In order to configure the VPC interface beyond the default above, an explicit interface can be configured in the LinodeMachineTemplate. When the LinodeMachine controller find an interface with purpose: vpc it will automatically inject the SubnetID from the VPCRef.

Example template where the VPC interface is not the primary interface

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: LinodeMachineTemplate
metadata:
  name: test-cluster-md-0
  namespace: default
spec:
  template:
    spec:
      region: "us-mia"
      type: "g6-standard-4"
      image: linode/ubuntu22.04
      interfaces:
      - purpose: vpc
        primary: false
      - purpose: public
        primary: true

How VPC is provisioned

A VPC is tied to a region. CAPL generates LinodeVPC manifest which contains the VPC name, region and subnet information. By defult, VPC name is set to cluster name but can be overwritten by specifying relevant environment variable.

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: LinodeVPC
metadata:
  name: ${VPC_NAME:=${CLUSTER_NAME}}
  labels:
    cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME}
spec:
  region: ${LINODE_REGION}
  subnets:
    - ipv4: 10.0.0.0/8
      label: default

Reference to LinodeVPC object is added to LinodeCluster object which then uses the specified VPC to provision resources.

Additional Configuration

By default, the VPC will use the subnet with the default label for deploying clusters. To modify this behavior, set the SUBNET_NAME environment variable to match the label of the subnet to be used. Make sure the subnet is set up in the LinodeVPC manifest.

Additionally, the VPC_NETWORK_CIDR and K8S_CLUSTER_CIDR environment variables can be used to change which CIDR blocks are used by the VPC and its clusters. VPC_NETWORK_CIDR designates the range used by the VPC, while K8S_CLUSTER_CIDR designates the range used by clusters for nodes. The K8S_CLUSTER_CIDR should be within the VPC_NETWORK_CIDR.

Troubleshooting

If pod-to-pod connectivity is failing

If a pod can't ping pod ips on different node, check and make sure pod CIDRs are added to ip_ranges of VPC interface.

curl --header 'Authorization: Bearer $LINODE_API_TOKEN' -X GET https://api.linode.com/v4/linode/instances/${LINODEID}/configs | jq .data[0].interfaces[].ip_ranges

Running cilium connectivity tests

One can also run cilium connectivity tests to make sure networking works fine within VPC. Follow the steps defined in cilium e2e tests guide to install cilium binary, set the KUBECONFIG variable and then run cilium connectivity tests.