LKE multi-node NodePool scale-up (Draft)

This document summarizes the core assumptions, invariants, and high-level flow for scaling existing LKE node pools in LKE mode, optimized for fast review and feedback.

Goals

• Maintain a strict 1:1 mapping between a Karpenter NodeClaim and a Linode VM.
• Set nodeClaim.status.providerID during CloudProvider.Create().
• Support many concurrent Create() calls without double-claiming a VM or racing pool mutations.

Terminology

• NodePool (Karpenter): Kubernetes resource karpenter.sh/NodePool.
• NodePool (LKE): Linode LKE node pool (linodego.LKENodePool).
• NodeClaim: Kubernetes resource karpenter.sh/NodeClaim.
• Instance: Linode VM (linodego.Instance), identified by instanceID.

Core model and invariants

• Pool mapping: exactly 1 LKE pool per (Karpenter NodePool name, instanceType).
• Pool discovery: always list pools and filter client-side (server-side tag filtering is unreliable for LKE pools).
• Determinism: if multiple claimable instances exist, claim the first instanceID in the list.
• Tag ownership:
  • Karpenter writes provider-managed tags to LKE pools through the LKE API and manages identity tags directly on Linode instances.
  • karpenter.sh/nodeclaim:<nodeClaimName> is instance-scoped only.
  • karpenter.sh/nodeclaim is the authoritative orchestration signal for claim ownership: instances without any karpenter.sh/nodeclaim:* tag are treated as unclaimed and can only be claimed by writing karpenter.sh/nodeclaim:<nodeClaimName>.
  • Pool-scoped tags may be written via LKE pool APIs. Do not change or remove Karpenter-managed tags outside the provider; pool and instance discovery and claim ownership rely on them.

Pool mapping diagram

flowchart LR
    KNP_A["Karpenter NodePool: nodepool-a"]
    KNP_B["Karpenter NodePool: nodepool-b"]

    LKE_A_T1["LKE NodePool (LKE pool)\nkey=(nodepool-a, g6-standard-2)\nType: g6-standard-2"]
    LKE_A_T2["LKE NodePool (LKE pool)\nkey=(nodepool-a, g6-standard-4)\nType: g6-standard-4"]
    LKE_B_T1["LKE NodePool (LKE pool)\nkey=(nodepool-b, g6-standard-2)\nType: g6-standard-2"]

    KNP_A -->|"instanceType=g6-standard-2"| LKE_A_T1
    KNP_A -->|"instanceType=g6-standard-4"| LKE_A_T2

    KNP_B -->|"instanceType=g6-standard-2"| LKE_B_T1

Tier-specific assumptions

• Enterprise tier
  • LKE pool tags propagate to underlying Linode instance tags.
  • Linode automatically applies nodepool=<poolID> and lke<clusterID> tags to instances.
  • LKE node pool Linodes information can be delayed (30-60s); enumerate membership via the Linode Instance API filtered by nodepool=<poolID>, lke<clusterID>, and karpenter.k8s.linode/lke-managed:true, then client-side filter by instanceType.
  • Instance Label is used as the node ID (critical for early discovery before pool.Linodes is populated).
• Standard tier
  • LKE pool tags do not propagate to instances.
  • Pool membership discovered via GetLKENodePool(poolID).Linodes, then GetInstance for each node. This is because the nodepool=<poolID> tag is not applied to instances in Standard tier.
  • Required identity tags (pool tags + nodeclaim tag) are applied directly to the instance at claim time via UpdateInstance.

Concurrency model

• Assume a single active writer (leader election).
• Use an in-process keyed mutex keyed by (karpenterNodePoolName, instanceType).
• The mutex protects:
  • pool lookup/create
  • pool count mutations
  • candidate selection and instance claiming (karpenter.sh/nodeclaim:<name>)
  • node-level delete + orphan GC mutations for that pool key

High-level Create() flow

Inputs:

• karpenterNodePoolName := nodeClaim.labels[karpenter.sh/nodepool]
• instanceType := resolved single instance type
• poolKey := (karpenterNodePoolName, instanceType)

Step 1: Fast idempotency

• Query all instance pages for utils.NodeClaimTag(nodeClaim.Name) (karpenter.sh/nodeclaim=<normalizedName>).
• If one or more instances match, return the first instance in the API response.
• If multiple instances match, log the ownership tag, match count, and selected instance ID. Do not request more capacity.
• After the NodeClaim records the selected provider ID, instance garbage collection can remove the other workers. Ownership tags do not prevent this cleanup.

Step 2: Claim-or-scale loop (bounded by DefaultCreateDeadline)

Repeat until deadline:

• Lock mutex for poolKey.
• Repeat the ownership lookup before pool discovery, candidate claiming, or scaling. This recovers a worker when an earlier ownership update succeeded but its response was lost.
• Return a matching worker immediately. If the lookup fails, do not treat the failure as missing capacity.
• Find or create the LKE pool for (nodepoolName, instanceType).
• Determine claimable instances:
  • Standard: pool membership from GetLKENodePool(poolID).Linodes.
    • Iterate all nodes; for each with InstanceID != 0, call GetInstance to check tags.
    • If any node has InstanceID == 0, return ErrNodesProvisioning after checking all nodes (allows claiming already-ready nodes first). InstanceID == 0 indicates that the instance is being created but not yet ready and should be retried.
  • Enterprise: pool membership from Instance API filtered by:
    • nodepool=<poolID> (Linode auto-tag), lke<clusterID> (Linode auto-tag), karpenter.k8s.linode/lke-managed:true.
    • Then client-side filter by instanceType.
• Claimability gate: only consider instances that do not already have any karpenter.sh/nodeclaim:* tag.
• If a claimable instance exists:
  • Write karpenter.sh/nodeclaim:<nodeClaimName> to the instance.
  • (Standard) also write required identity tags to the instance.
  • Unlock mutex and return providerID.
• If no claimable instance exists:
  • scale the pool by incrementing pool.Count only if this Create() call has not already created or scaled a pool.
• Unlock.
• Retry until deadline with brief sleeps:
  • Preserve the successful create/scale flags across attempts, so delayed instance discovery does not request extra nodes.
  • Sleep for the configured retry delay (default: 2 seconds).

On timeout: return a retryable CreateError (reason NodePoolProvisioning) so Karpenter requeues.

Enterprise Create() sequence diagram

sequenceDiagram
    autonumber
    actor NC as NodeClaim Controller
    participant LKE as LKE Provider
    participant INST as Linode Instance API
    participant POOL as LKE NodePool API
    participant M as Keyed Mutex

    NC->>LKE: Create(nodeClaim)
    LKE->>INST: List all pages matching ownership tag
    alt ownership found
        LKE->>LKE: Select first instance and log duplicates
        LKE-->>NC: Return selected worker
    else ownership missing
        loop until create deadline
            LKE->>M: Lock(poolKey)
            LKE->>INST: Repeat ownership lookup
            alt ownership found
                LKE->>LKE: Select first instance and log duplicates
                LKE->>M: Unlock(poolKey)
                LKE-->>NC: Return selected worker
            else ownership missing
                LKE->>POOL: Find or create pool
                LKE->>INST: Discover unclaimed workers
                alt unclaimed worker found
                    LKE->>INST: Apply ownership tag
                    LKE->>INST: Verify ownership tag
                else no unclaimed worker
                    opt No acknowledged create or scale in this call
                        LKE->>POOL: Scale existing pool by one
                    end
                end
                LKE->>M: Unlock(poolKey)
                Note over LKE: Return verified worker or retry with capacity flags intact
            else ownership lookup failed
                LKE->>M: Unlock(poolKey)
                Note over LKE: Retry transient errors without capacity changes
            end
        end
    end

Error policy

• Plan unavailable:
  • Karpenter marks a plan unavailable in the cluster’s region only when LKE identifies that plan as unavailable during new pool creation.
• Retryable errors:
  • If Karpenter does not find an assignable instance before DefaultCreateDeadline expires, it returns an error.
  • Karpenter retries while it waits for Linode API data to become consistent.
  • Karpenter retries API errors with HTTP status 429 or 5xx.
• Pool and instance matching:
  • If more than one pool matches a node pool name and instance type, Karpenter uses the first pool.
  • If more than one instance matches a NodeClaim, Karpenter selects the first instance in the API response and logs the duplicate ownership. It leaves the other instances for garbage collection.
  • Karpenter does not check whether an instance has multiple NodeClaim tags.

Plan availability

• Region availability check:
  • The offering provider marks an offering available only when GET /regions/{region}/availability reports the plan as available.
  • The instance type controller refreshes this data every 5 minutes.
  • A plan that sells out between refreshes stays selectable until the next refresh.
• Scale-up errors:
  • Karpenter does not treat an HTTP 400 on pool scale-up as insufficient capacity. These errors also come from validation, networking, and request limits, and the response does not show which cause applies.
  • Karpenter reports the error as a creation error. This error does not trigger the NodeClaim deletion that an insufficient capacity error causes.
• Plan unavailable on pool creation:
  • If the LKE API rejects a new pool because the plan is unavailable in the region, Karpenter returns an insufficient capacity error. Karpenter then deletes the NodeClaim and selects another offering.
  • Karpenter logs pool create rejected with an unrecognized 400 when a pool creation 400 does not match this error.

API call volume and scalability concerns

This section calls out Linode API call volume by function, the key multipliers (N+1, polling, and controller requeues), and why this design can be rate-limit bound under concurrency.

Relevant rate limits and client behavior (external)

• Linode API rate limits (per-user / token):
  • GET operations returning paginated collections: 200 requests / minute (default).
  • All other operations: 1600 requests / minute (default).
  • Create a Linode: 20 requests / 15 seconds (specific override).
  • Source: https://techdocs.akamai.com/linode-api/reference/rate-limits
• linodego retry behavior:
  • The Go SDK is configured to retry and to sleep based on the Retry-After header when provided.
  • Reference implementation: https://github.com/linode/linodego/blob/main/retries.go

Per-function API call “budgets” (current behavior)

Budgets below are per single invocation, excluding extra attempts from transient errors (429/5xx) and excluding Karpenter/controller requeues. Under contention, actual totals can be higher.

LKE provider (pkg/providers/lke/lke.go)

• Create()
  • Fast idempotency (Step 1):
    • ListInstances via utils.LookupInstanceByTag (pkg/utils/utils.go) for karpenter.sh/nodeclaim:<nodeClaimName>.
  • Claim-or-scale loop (Step 2, bounded by DefaultCreateDeadline):
    • Ownership recovery: ListInstances via LookupInstanceByTag reads all matching pages at the start of each locked attempt. A match returns before pool or worker mutations.
    • Pool discovery:
      • ListLKENodePools in findOrCreatePool().
    • Standard-tier claim path:
      • GetLKENodePool in findClaimableInstanceStandard().
      • N× GetInstance (N+1 pattern) while iterating pool.Linodes to find a claimable instance.
      • UpdateInstance in claimInstance() to write karpenter.sh/nodeclaim:<name> (+ identity tags).
      • Read-after-write:
        • verifyTagsApplied() polls GetInstance (up to ~5 reads over ~5s).
    • Enterprise-tier claim path:
      • ListInstances in findClaimableInstanceEnterprise() filtered by nodepool=<poolID>, lke<clusterID>, karpenter.k8s.linode/lke-managed:true, then client-side filter by instanceType.
      • UpdateInstance in claimInstance() to write karpenter.sh/nodeclaim:<name> (+ pool tags).
      • verifyTagsApplied() polls GetInstance (up to ~5 reads over ~5s).
    • Scale path:
      • UpdateLKENodePool when no claimable instance exists.
  • Multipliers to watch:
    • Standard tier is inherently O(pool size) for the GetInstance scan.
    • The loop repeats within the deadline, so the above can repeat multiple times per Create() call.
• Get()
  • Cache hit: 0 Linode API calls.
  • Cache miss:
    • GetInstance
• List()
  • ListInstances (managed tag filtered; Enterprise also filters by lke<clusterID>).
• Delete()
  • ListLKENodePools via findLKENodePoolFromLinodeInstanceID() (scans all pools to find instance).
  • Either DeleteLKENodePoolNode (common) or DeleteLKENodePool (when last node).
• CreateTags()
  • GetInstance (merge/dedupe)
  • UpdateInstance

Instance provider (pkg/providers/instance/instance.go)

• Create() → CreateInstance
• Get() → cache hit 0 / miss GetInstance
• List() → ListInstances
• Delete() → GetInstance (forced via SkipCache) + optionally DeleteInstance
• CreateTags() → CreateTag once per tag (+ time.Sleep(1s) between tags)

Instance types / offerings

File: pkg/providers/instancetype/offering/offering.go

• createOfferings() can call GetType inside a loop over regions (i.e., instanceTypes × regions bursts on cache miss/invalidation).

Cross-layer amplifiers (where “one Create()” becomes many calls)

• CloudProvider orchestration (pkg/cloudprovider/cloudprovider.go)
  • CloudProvider.Create() calls instanceTypeProvider.List() and then nodeProvider.Create().
  • CloudProvider.List() calls nodeProvider.List() and then resolves instance types and node class.
• Tagging controller forces freshness (pkg/controllers/nodeclaim/tagging/controller.go)
  • Uses nodeProvider.Get(..., instance.SkipCache), which bypasses provider caches and increases read pressure.
• Garbage collection controller (pkg/controllers/nodeclaim/garbagecollection/controller.go)
  • Calls cloudProvider.List() and can fan out deletes (up to 100 workers), which can create bursts of Get()/Delete() API calls.

Why this may not scale

• Key Create() paths rely on list-then-filter, per-node GetInstance scans, and read-after-write polling to implement determinism and avoid double-claim.
• Under concurrency (many NodeClaims) and retries (requeues on retryable errors), these patterns can drive call rates into the GET paginated collection limit (200/min) quickly, especially when ListInstances/ListLKENodePools are in tight loops.
• Without additional server-side API surfaces (e.g., an atomic “claim next unclaimed node in pool” or a pool membership endpoint that includes claim state) the design remains fundamentally rate-limit/latency bound.

Delete + orphan capacity (summary)

• Delete must be node-level (delete node from pool), deleting the pool only when removing the last node.
• Orphan capacity can occur if scaling succeeds but claiming fails; orphan GC will remove unclaimed capacity after a fixed TTL (5m).

Open questions for review

• Is DefaultCreateDeadline a reasonable upper bound in Create() before returning a retryable error?
• Are the invariants and “hard error” cases correct, or should any be relaxed/strengthened?
• Is the orphan TTL (5m) the right tradeoff for cost vs safety?