Upgrade Notes¶
This page describes the considerations and precautions when upgrading baize to a new version.
Upgrading from v0.24.1 (or earlier) to v0.25.1¶
Baize v0.25.1 refactors the data space dependency component dataset. Please follow the steps below to perform a manual upgrade.
Baize v0.25.1 also refactors the scheduling-aware component kueue. Please follow the steps below to perform a manual upgrade.
Upgrade dataset¶
The CRD used by baize changes from datasets.dataset.baize.io to datasets.dataset.baizeai.io. The following sections explain how to perform a smooth upgrade while preserving existing data.
- Old CRD apiVersion: datasets.dataset.baize.io
- New CRD apiVersion: datasets.dataset.baizeai.io
Breaking Changes Details¶
-
Both baize and hydra will use the BaizeAI/dataset project to reconcile datasets.
-
The baize project no longer includes code related to the dataset controller.
-
BaizeAI/dataset will be deployed as a Helm application in the
dataset-systemnamespace. -
The CRs of
datasets.dataset.baizeai.ioare distinguished by the label with the key "app.kubernetes.io/part-of", indicating whether they are created by baize or hydra.
Upgrade Steps¶
-
Upgrade the baize
apiserverin the Global cluster. After the upgrade, datasets will temporarily disappear from the UI because theapiserverqueries the new version of the CR. -
Stop the baize-agent-cluster-controller and baize-agent-jobset-controller deployments, and keep kueue-controller-manager running.
Note
Do not upgrade
baize-agentin worker clusters at this stage to avoid deleting the old CRD. -
Install the Helm application of BaizeAI/dataset in the worker cluster. After the installation is complete, stop the only deployment in the dataset-system namespace.
-
At this point, two versions of the CRD coexist in the worker cluster. Do not create any new datasets, either through the UI or via API calls.
-
For datasets that support the preheating type (non-PVC and non-NFS datasets), run the following script. The first parameter is the namespace, and the second parameter is the dataset name.
Note
This script deletes the dataset-datasetname-round-1 job. If the logs of this job and its corresponding pod are important, save them beforehand.
#!/bin/bash # Check the arguments if [ $# -ne 2 ]; then echo "Usage: $0 <namespace> <dataset-name>" echo "Example: $0 default my-dataset" exit 1 fi NAMESPACE="$1" DATASET_NAME="$2" PVC_NAME="$DATASET_NAME" JOB_NAME=dataset-"${DATASET_NAME}-round-1" echo "Namespace: $NAMESPACE" echo "Dataset Name: $DATASET_NAME" echo "PVC Name: $PVC_NAME" echo "Job Name: $JOB_NAME" echo "" # ========== Step 0: Get the Dataset information and determine the type ========== echo "Step 0: Checking Dataset source type..." DATASET_TYPE=$(kubectl get dataset.dataset.baize.io "$DATASET_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.source.type}' 2>/dev/null) if [ -z "$DATASET_TYPE" ]; then echo "Error: Failed to get dataset source type" exit 1 fi echo "Dataset source type: $DATASET_TYPE" # Determine whether a Job needs to be created SKIP_JOB=false if [ "$DATASET_TYPE" = "NFS" ] || [ "$DATASET_TYPE" = "PVC" ]; then SKIP_JOB=true echo "Type is NFS or PVC, will skip Job creation" fi echo "" # ========== Step 1: Update the Dataset ========== echo "Step 1: Updating Dataset..." # Build the jq command and decide whether to add rcloneOp based on the type if [ "$DATASET_TYPE" = "S3" ] || [ "$DATASET_TYPE" = "HTTP" ]; then echo "Type is S3 or HTTP, setting rcloneOp to syncMode" kubectl get dataset.dataset.baize.io "$DATASET_NAME" -n "$NAMESPACE" -o json | jq ' del( .metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.managedFields, .metadata.generation, .metadata.selfLink ) | .apiVersion="dataset.baizeai.io/v1alpha1" | .spec.dataSyncRound=1 | .status.phase="Ready" | .status.inProcessingRound=1 | .metadata.labels["app.kubernetes.io/part-of"] = "baize" | .spec.source.options.syncMode = .spec.source.options.rcloneOp | del(.spec.source.options.rcloneOp) ' | kubectl apply -f - else kubectl get dataset.dataset.baize.io "$DATASET_NAME" -n "$NAMESPACE" -o json | jq ' del( .metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.managedFields, .metadata.generation, .metadata.selfLink ) | .apiVersion="dataset.baizeai.io/v1alpha1" | .spec.dataSyncRound=1 | .status.phase="Ready" | .status.inProcessingRound=1 | .metadata.labels["app.kubernetes.io/part-of"] = "baize" ' | kubectl apply -f - fi if [ $? -ne 0 ]; then echo "Error: Failed to update dataset" exit 1 fi echo "Dataset updated successfully" echo "" # If the type is PVC, finish right after the first step if [ "$DATASET_TYPE" = "PVC" ]; then echo "Type is PVC, only updating Dataset. Done." exit 0 fi # Wait for the Dataset to be created sleep 10 # ========== Step 2: Get the Dataset UID ========== echo "Step 2: Getting Dataset UID..." DATASET_UID=$(kubectl get dataset.dataset.baizeai.io "$DATASET_NAME" -n "$NAMESPACE" -o jsonpath='{.metadata.uid}') if [ -z "$DATASET_UID" ]; then echo "Error: Failed to get dataset UID" exit 1 fi echo "Dataset UID: $DATASET_UID" echo "" # ========== Step 3: Update the PVC ========== echo "Step 3: Checking PVC before update..." echo "Current PVC ownerReferences:" kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o json | jq '.metadata.ownerReferences // "null"' echo "" echo "Step 3: Updating PVC..." kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o json | jq --arg uid "$DATASET_UID" --arg name "$DATASET_NAME" ' del(.metadata.ownerReferences) | .metadata.ownerReferences = [{ "apiVersion": "dataset.baizeai.io/v1alpha1", "kind": "Dataset", "name": $name, "uid": $uid, "controller": true, "blockOwnerDeletion": true }] | .metadata.labels["baize.io/dataset-name"] = $name | del(.metadata.labels["dataset.baize.io/name"]) ' | kubectl replace -f - if [ $? -ne 0 ]; then echo "Error: Failed to update PVC" exit 1 fi echo "PVC updated successfully" echo "" # ========== Step 3 (additional): if the type is NFS, update the PV ========== if [ "$DATASET_TYPE" = "NFS" ]; then echo "Step 3.5: Getting PV name from PVC..." PV_NAME=$(kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.volumeName}') if [ -z "$PV_NAME" ]; then echo "Error: Failed to get PV name from PVC" exit 1 fi echo "PV Name: $PV_NAME" echo "" echo "Step 3.5: Updating PV..." kubectl get pv "$PV_NAME" -o json | jq --arg uid "$DATASET_UID" --arg name "$DATASET_NAME" ' del(.metadata.ownerReferences) | .metadata.ownerReferences = [{ "apiVersion": "dataset.baizeai.io/v1alpha1", "kind": "Dataset", "name": $name, "uid": $uid, "controller": true, "blockOwnerDeletion": true }] | .metadata.labels["baize.io/dataset-name"] = $name | del(.metadata.labels["dataset.baize.io/name"]) ' | kubectl replace -f - if [ $? -ne 0 ]; then echo "Error: Failed to update PV" exit 1 fi echo "PV updated successfully" echo "" fi # ========== Step 4: Create the Job (conditional) ========== if [ "$SKIP_JOB" = true ]; then echo "Step 4: Skipping Job creation (type is $DATASET_TYPE)" else echo "Step 4: Creating Job..." kubectl delete job "$JOB_NAME" -n "$NAMESPACE" 2>/dev/null || true cat <<-EOF | kubectl apply -f - apiVersion: batch/v1 kind: Job metadata: name: $JOB_NAME namespace: $NAMESPACE ownerReferences: - apiVersion: dataset.baizeai.io/v1alpha1 kind: Dataset name: $DATASET_NAME uid: $DATASET_UID controller: true blockOwnerDeletion: true spec: template: spec: containers: - name: main image: 10.6.178.191:5000/demo/busybox:latest command: ["sh", "-c", "sleep 1"] resources: requests: memory: "16Mi" cpu: "10m" limits: memory: "32Mi" cpu: "50m" restartPolicy: Never backoffLimit: 2 EOF if [ $? -ne 0 ]; then echo "Error: Failed to create Job" exit 1 fi echo "Job created successfully" fi echo "" # ========== Verify the results ========== echo "=== Verification ===" echo "Dataset UID: $(kubectl get dataset "$DATASET_NAME" -n "$NAMESPACE" -o jsonpath='{.metadata.uid}' 2>/dev/null || echo 'N/A')" echo "PVC Owner UID: $(kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o jsonpath='{.metadata.ownerReferences[0].uid}' 2>/dev/null || echo 'N/A')" if [ "$SKIP_JOB" = false ]; then echo "Job Status:" kubectl get job "$JOB_NAME" -n "$NAMESPACE" 2>/dev/null || echo "Job not found" fi echo "" echo "Success! All resources created/updated." -
After all datasets have been processed using this script, you can upgrade
baize-agent. -
After confirming that
baize-agenthas been successfully upgraded, you can delete the old CRs and CRD (depending on your requirements, you may also choose to keep them). -
Restore the deployment in dataset-system and verify that the status of the Data Space entries in the UI is normal.
Note
If the dataset type is NFS, the script modifies the labels and
ownerReferencesof both the PVC and its corresponding PV. For other dataset types, only the PVC is modified.
Upgrade kueue¶
Starting from Baize v0.25.1, the kueue component is decoupled from baize-agent and deployed as a standalone plugin (addon) in the kueue-system namespace. The purpose is to allow both baize and hydra to share the same kueue component. The following sections describe the upgrade procedure in detail.
-
After upgrading to v0.25.1, execute the following script:
Note
Ensure that all workloads have been admitted before running this script.
#!/bin/bash set -e # ========================================== # Environment variable configuration section (verify against your actual situation) # ========================================== OLD_NAMESPACE="baize-system" OLD_RELEASE="baize-agent" NEW_NAMESPACE="kueue-system" NEW_RELEASE="kueue" BACKUP_DIR="./kueue-migration-backup" VISIBILITY_ROLE_NAME="kueue-visibility-server-auth-reader" VISIBILITY_ROLE_NAMESPACE="kube-system" annotate_release() { local resource="$1" kubectl annotate "$resource" \ meta.helm.sh/release-name="$NEW_RELEASE" \ meta.helm.sh/release-namespace="$NEW_NAMESPACE" \ --overwrite } annotate_release_in_namespace() { local kind="$1" local name="$2" local namespace="$3" kubectl annotate "$kind" "$name" -n "$namespace" \ meta.helm.sh/release-name="$NEW_RELEASE" \ meta.helm.sh/release-namespace="$NEW_NAMESPACE" \ --overwrite } annotate_release_ignore_error() { local resource="$1" kubectl annotate "$resource" \ meta.helm.sh/release-name="$NEW_RELEASE" \ meta.helm.sh/release-namespace="$NEW_NAMESPACE" \ --overwrite 2>/dev/null || true } annotate_release_in_namespace_ignore_error() { local kind="$1" local name="$2" local namespace="$3" kubectl annotate "$kind" "$name" -n "$namespace" \ meta.helm.sh/release-name="$NEW_RELEASE" \ meta.helm.sh/release-namespace="$NEW_NAMESPACE" \ --overwrite 2>/dev/null || true } annotate_keep_policy() { local resource="$1" kubectl annotate "$resource" helm.sh/resource-policy=keep --overwrite } echo "🚀 Start: transferring Kueue resource ownership and stopping the old controller..." # 1. Data backup echo "📦 [1/4] Backing up the Kueue CRDs and instance data in the current cluster..." mkdir -p "$BACKUP_DIR" kubectl get crd -o yaml | grep -A 10 "kueue.x-k8s.io" > "$BACKUP_DIR/kueue-crds-backup.yaml" || true kubectl get clusterqueue,localqueue,resourceflavor,workload,topology -A -o yaml > "$BACKUP_DIR/kueue-data-backup.yaml" 2>/dev/null || true echo "✅ Backup completed and saved in the ${BACKUP_DIR} directory." # 2. Transfer the CRD ownership and add deletion protection echo "🔀 [2/4] Transferring the Helm ownership of cluster-level CRDs to the new release (${NEW_RELEASE})..." CRDS=$(kubectl get crd -o name | grep 'kueue.x-k8s.io' || true) if [ -n "$CRDS" ]; then for crd in $CRDS; do annotate_release "$crd" annotate_keep_policy "$crd" echo " - Taken over: $crd" done else echo "⚠️ No Kueue CRD found. Please confirm the cluster status." fi # Batch modify the ownership annotations of v1beta1 and v1beta2 for version in v1beta1 v1beta2; do annotate_release "apiservice/${version}.visibility.kueue.x-k8s.io" echo "✅ Ownership of APIService ${version} transferred successfully" done # 3. Transfer the ownership of ClusterRole and Binding echo "🔀 [3/4] Transferring the ownership of ClusterRole and ClusterRoleBinding..." ROLES=$(kubectl get clusterrole,clusterrolebinding -o name | grep 'kueue' || true) if [ -n "$ROLES" ]; then for role in $ROLES; do annotate_release "$role" done fi # 4. Gracefully silence the control plane echo "⏸️ [4/4] Stopping the old Kueue controller and cleaning up Webhooks..." kubectl scale deployment -l app.kubernetes.io/name=kueue -n "$OLD_NAMESPACE" --replicas=0 || echo "⚠️ Old Deployment not found, it may already be stopped." kubectl delete validatingwebhookconfigurations,mutatingwebhookconfigurations -l app.kubernetes.io/name=kueue --ignore-not-found annotate_release_in_namespace rolebinding "$VISIBILITY_ROLE_NAME" "$VISIBILITY_ROLE_NAMESPACE" annotate_release_in_namespace_ignore_error role "$VISIBILITY_ROLE_NAME" "$VISIBILITY_ROLE_NAMESPACE" # Add the keep policy to all ClusterRoles and ClusterRoleBindings whose names contain kueue for res in $(kubectl get clusterrole,clusterrolebinding,serviceaccount -o name | grep kueue); do annotate_keep_policy "$res" echo "🛡️ Deletion protection added for $res" done echo "" echo "🎉 Done! The old controller in the cluster is now silent, and the CRDs are safely isolated." echo "👉 Existing Jobs are not affected, but newly submitted Jobs will be Pending briefly." echo "Please immediately proceed to: install kueue and upgrade baize-agent." -
Navigate to the worker cluster's Helm Apps -> Helm Templates page, locate the kueue addon, and install it.
-
Navigate to the worker cluster's Helm Apps -> Helm Apps page, locate baize-agent , and perform an update.
After completing the preceding steps, the kueue component can be used normally.