[KServe] 01. Serving Runtime

KServe 시리즈의 글입니다.

KServe에서 제공하는 서빙 환경을 제어하기 위해 띄우는 CRD 이다.

Serving Runtime이란?

KServe에서는 Serving Runtime이라는 serving 환경을 제어하기위한 k8s CRD(Custom Resource Definition)로 ServingRuntime(namespace 범위), ClusterServingRuntimes(클러스터 범위) 두 가지를 제공함

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: example-runtime
spec:
  supportedModelFormats:
    - name: example-format
      version: "1"
      autoSelect: true
  containers:
    - name: kserve-container
      image: examplemodelserver:latest
      args:
        - --model_name={{.Name}}
        - --model_dir=/mnt/models
        - --http_port=8080
      env:
        - name: PREDICT_PROBA
          value: "True"
      resources:
          requests:
            cpu: 2
            memory: 4Gi
          limits:
            cpu: 4
            memory: 8Gi

요런 모양으로 생겼으며, 보기와 같이 env, request등 컨테이너에 필요한 펙 설정을 할 수 있다.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: example-sklearn-isvc
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://bucket/sklearn/model.joblib
      runtime: example-runtime # 생성한 Serving Runtime 이름을 입력

이런식으로 InferenceService를 띄울때 특정 ServingRuntime을 물고 띄울 수 있다.

Spec Attribute

기본적으로 걍 k8s pod spec에 붙이는거 지원하는듯하다 거기에 더해서 model format에대한 새로운 attribute만 익히면 될듯함.

Attribute	Description
`multiModel`	Whether this ServingRuntime is ModelMesh-compatible and intended for multi-model usage (as opposed to KServe single-model serving). Defaults to false
`disabled`	Disables this runtime
`containers`	List of containers associated with the runtime
`containers[ ].image`	The container image for the current container
`containers[ ].command`	Executable command found in the provided image
`containers[ ].args`	List of command line arguments as strings
`containers[ ].resources`	Kubernetes [limits or requests](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits)
`containers[ ].env`	List of environment variables to pass to the container
`containers[ ].imagePullPolicy`	The container image pull policy
`containers[ ].workingDir`	The working directory for current container
`containers[ ].livenessProbe`	Probe for checking container liveness
`containers[ ].readinessProbe`	Probe for checking container readiness
`supportedModelFormats`	List of model types supported by the current runtime
`supportedModelFormats[ ].name`	Name of the model format
`supportedModelFormats[ ].version`	Version of the model format. Used in validating that a predictor is supported by a runtime. It is recommended to include only the major version here, for example "1" rather than "1.15.4"
`storageHelper.disabled`	Disables the storage helper
`nodeSelector`	Influence Kubernetes scheduling to [assign pods to nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)
`affinity`	Influence Kubernetes scheduling to [assign pods to nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
`tolerations`	Allow pods to be scheduled onto nodes [with matching taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration)