Edit

Share via

Troubleshooting ClusterResourcePlacement API in Azure Kubernetes Fleet Manager

  • Article

This troubleshooting guide helps you resolve ClusterResourcePlacement API object-related issues when you use Azure Kubernetes Fleet Manager. Troubleshooting these errors on the hub cluster requires knowledge of the following objects:

  • ClusterResourceSnapshot
  • ClusterSchedulingPolicySnapshot
  • ClusterResourceBinding
  • Work

For more details about each object, see the API reference.

Complete progression of the ClusterResourcePlacement deployment

Understanding the progression and status of the ClusterResourcePlacement custom resource is crucial for diagnosing and identifying failures. You can view the status of the ClusterResourcePlacement custom resource by using the following command:

kubectl describe clusterresourceplacement <name>

For more information, see Use the ClusterResourcePlacement API to propagate resources to member clusters.

The complete progression of ClusterResourcePlacement is as follows:

  1. ClusterResourcePlacementScheduled: Indicates that a resource has been scheduled for placement.

    If false, see How to troubleshoot when the ClusterResourcePlacementScheduled condition status is false.

  2. ClusterResourcePlacementRolloutStarted: Indicates that the rollout process has begun.

    If false, see How to troubleshoot when the ClusterResourcePlacementRolloutStarted condition status is false.

  3. ClusterResourcePlacementOverridden: Indicates that the resource has been overridden.

    If false, see How to troubleshoot when the ClusterResourcePlacementOverridden condition status is false.

  4. ClusterResourcePlacementWorkSynchronized: Indicates that work objects were synchronized.

    If false, see How to troubleshoot when the ClusterResourcePlacementWorkSynchronized condition status is false.

  5. ClusterResourcePlacementApplied: Indicates the resource was applied.

    If false, see How to troubleshoot when the ClusterResourcePlacementApplied condition status is false.

  6. ClusterResourcePlacementAvailable: Indicates that the resource is available.

    If false, see How to troubleshoot when the ClusterResourcePlacementAvailable condition status is false.

FAQ

How can I debug if some clusters aren't selected as expected?

Check the status of ClusterSchedulingPolicySnapshot to determine which clusters were selected and the reason for their selection.

How can I debug if a selected cluster doesn't have the expected resources or if ClusterResourcePlacement doesn't pick up the latest changes?

  1. Check whether the ClusterResourcePlacementRolloutStarted condition in the ClusterResourcePlacement status is set to true or false.
  2. Check whether the ClusterResourcePlacementApplied condition is set to unknown, false, or true.
  3. Check the placementStatuses section in the ClusterResourcePlacement status for the particular cluster. The FailedPlacements section should provide reasons for any resource application failures.

How can I find and verify the latest ClusterSchedulingPolicySnapshot for a ClusterResourcePlacement deployment?

To find the latest ClusterSchedulingPolicySnapshot for the ClusterResourcePlacement API deployment, run the following command:

kubectl get clusterschedulingpolicysnapshot -l kubernetes-fleet.io/is-latest-snapshot=true,kubernetes-fleet.io/parent-CRP={CRPName}

Note

In this command, replace {CRPName} with your ClusterResourcePlacement name.

Then, compare the ClusterSchedulingPolicySnapshot with the ClusterResourcePlacement policy to make sure that they match, excluding the numberOfClusters field from the `ClusterResourcePlacement' spec.

If the placement type is PickN, check whether the number of clusters that's requested in the ClusterResourcePlacement policy matches the value of the number-of-clusters label.

How can I find the latest ClusterResourceBinding resource?

The following command lists all ClusterResourceBindings instances that are associated with ClusterResourcePlacement:

Kubectl get clusterresourcebinding -l kubernetes-fleet.io/parent-CRP={CRPName}

Note

In this command, replace {CRPName} with your ClusterResourcePlacement name.

Example

  1. Run the following command to view the status of the ClusterResourcePlacement deployment. In this case, the ClusterResourcePlacement name is test-crp.

    kubectl describe clusterresourceplacement test-crp

  2. Here's an example output. From the placementStatuses section of the test-crp status, notice that it has distributed resources to two member clusters and, therefore, has two ClusterResourceBindings instances:

    status:
  conditions:
  - lastTransitionTime: "2023-11-23T00:49:29Z"
    ...
  placementStatuses:
  - clusterName: kind-cluster-1
    conditions:
      ...
      type: ResourceApplied
  - clusterName: kind-cluster-2
    conditions:
      ...
      reason: ApplySucceeded
      status: "True"
      type: ResourceApplied

  3. To get the ClusterResourceBindings value, run the following command:

    kubectl get clusterresourcebinding -l kubernetes-fleet.io/parent-CRP=test-crp 

    NAME                               WORKCREATED   RESOURCESAPPLIED   AGE
test-crp-kind-cluster-1-be990c3e   True          True               33s
test-crp-kind-cluster-2-ec4d953c   True          True               33s

    The output lists all ClusterResourceBindings instances that are associated with test-crp. The ClusterResourceBinding resource name uses the following format:

    {CRPName}-{clusterName}-{suffix}

How can I find the latest ClusterResourceSnapshot resource?

To find the latest ClusterResourceSnapshot resource, run the following command:

kubectl get clusterresourcesnapshot -l kubernetes-fleet.io/is-latest-snapshot=true,kubernetes-fleet.io/parent-CRP={CRPName}

Note

In this command, replace {CRPName} with your ClusterResourcePlacement name.

How can I find the correct work resource that's associated with ClusterResourcePlacement?

To find the correct work resource, follow these steps:

  1. Identify the member cluster namespace and the ClusterResourcePlacement name. The format for the namespace is fleet-member-{clusterName}.

  2. To get the work resource, run the following command:

    kubectl get work -n fleet-member-{clusterName} -l kubernetes-fleet.io/parent-CRP={CRPName}

    Note

    In this command, replace {clusterName} and {CRPName} with the names that you identified in the first step.

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.