ModelPool
inference.llmkube.dev / v1alpha1
apiVersion: inference.llmkube.dev/v1alpha1
kind: ModelPool
metadata:
name: example
apiVersion
string
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
object
spec object required
spec defines the desired state of ModelPool
default
string
Default names the member to warm on a cold pool (before any request has
picked an owner). When empty, the first request to arrive selects the
owner. Must match one of the members' InferenceServiceRef names.
gpu
integer
GPU is the number of GPUs the shared slot occupies. Informational for the
pool invariant (members declare their own resource requests); recorded so
operators can see the slot size at a glance.
format:
int32minimum:
0members []object required
Members lists the InferenceServices that share the pool's GPU slot. Each
member is an ordinary single-model InferenceService with its own image,
context size, and runtime.
minItems:
1inferenceServiceRef object required
InferenceServiceRef references the member InferenceService by name. The
InferenceService must live in the same namespace as the ModelPool.
name
string
Name of the referent.
This field is effectively required, but due to backwards compatibility is
allowed to be empty. Instances of this type with an empty value here are
almost certainly wrong.
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
nodeSelector
object
NodeSelector pins the shared slot to a node (or node class). All members
are expected to schedule onto the same node so they contend for the same
device memory. Mirrors a standard Kubernetes node selector.
swapBudget
string
SwapBudget bounds how long the router holds a cross-model request open
while it drains the incumbent and cold-loads the target member. It is
deliberately decoupled from the router's response-header (generation)
timeout: a large model can take minutes to load without forcing operators
to loosen the per-request generation cap for every request. When the
budget elapses before the target becomes resident, the held request is
failed with 503 + Retry-After (the caller can retry; the now-warm member
serves the next request). Defaults to 300s.
swapPolicy
string
SwapPolicy selects how the slot owner is chosen when demand crosses model
boundaries. "sticky" (default) keeps the incumbent until a different
member is requested. It is the only policy in v1; a priority-based reclaim
policy is a planned follow-up.
enum:
stickystatus object
status defines the observed state of ModelPool
conditions []object
Conditions represent the current state of the ModelPool resource.
lastTransitionTime
string required
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format:
date-time
message
string required
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength:
32768
observedGeneration
integer
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format:
int64minimum:
0
reason
string required
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
pattern:
^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$minLength:
1maxLength:
1024
status
string required
status of the condition, one of True, False, Unknown.
enum:
True, False, Unknown
type
string required
type of condition in CamelCase or in foo.example.com/CamelCase.
pattern:
^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$maxLength:
316members []object
Members reports the observed state of each pool member.
name
string required
Name is the member InferenceService name.
phase
string
Phase mirrors the member InferenceService status.phase.
resident
boolean
Resident is true when this member currently owns the shared GPU slot.
observedGeneration
integer
ObservedGeneration is the spec generation the status reflects.
format:
int64
pendingMember
string
PendingMember is the member being loaded during a swap, or empty when the
pool is not mid-swap.
phase
string
Phase is the pool lifecycle phase: Pending, Ready, Swapping, or Degraded.
enum:
Pending, Ready, Swapping, Degraded
residentMember
string
ResidentMember is the member that currently owns the shared GPU slot, or
empty when the pool is cold (no member resident).
No matches. Try .spec.default for an exact path