Manage emergency access to a bare metal machine using the az networkcloud cluster baremetalmachinekeyset

Caution

Please note this process is used in emergency situations when all other troubleshooting options using Azure have been exhausted. Any write or edit actions executed on the BMM node(s) will require users to 'reimage' in order to restore Microsoft support to the impacted BMM node(s). Please note that SSH access to these bare metal machines is restricted to users managed via this method from the specified jump host list.

There are rare situations where a user needs to investigate & resolve issues with a bare metal machine and all other ways via Azure are exhausted. Azure Operator Nexus provides the az networkcloud cluster baremetalmachinekeyset command so users can manage SSH access to these bare metal machines. On keyset creation, users are validated against Microsoft Entra ID for proper authorization by cross referencing the User Principal Name provided for a user against the supplied Microsoft Entra Group ID --azure-group-id <Entra Group ID>.

Users in a keyset are validated every four hours, and also when any changes are made to any keyset. Each user's status is then set to "Active" or "Invalid." Invalid users remain in the keyset but their keys are removed from all hosts and they aren't allowed access. Reasons for a user being invalid are:

  • The user's User Principal Name isn't a member of the given Entra group (if specified)
  • The given Entra group (if specified) doesn't exist (in which case all users in the keyset are invalid)
  • The keyset is expired (in which case all users in the keyset are invalid)

Note

There is currently a transitional period where specifying User Principal Names is optional. In a future release, it will become mandatory and Microsoft Entra ID validation will be enforced for all users. Users are encouraged to add User Principal Names to their keysets before the transitional period ends (planned for July 2024) to avoid keysets being invalidated. Note that if any User Principal Names are added to a keyset, even if they are not added for all users, Microsoft Entra ID validation will be enabled, and this will result in the entire keyset being invalidated if the Group ID specified is not valid.

The keyset and each individual user also have detailed status messages communicating other information:

  • The keyset's detailedStatusMessage tells you whether the keyset is expired, and other information about problems encountered while updating the keyset across the cluster.
  • The user's statusMessage tells you whether the user is active or invalid, and a list of machines that aren't yet updated to the user's latest active/invalid state. In each case, causes of problems are included if known.

When the command runs, it executes on each bare metal machine in the Cluster with an active Kubernetes node. There's a reconciliation process that runs periodically that retries the command on any bare metal machine that wasn't available at the time of the original command. Also, any bare metal machine that returns to the cluster via an az networkcloud baremetalmachine reimage or az networkcloud baremetalmachine replace command (see BareMetal functions) sends a signal causing any active keysets to be sent to the machine as soon as it returns to the cluster. Multiple commands execute in the order received.

There's no limit to the number of users in a group.

Caution

Notes for jump host IP addresses

  • The keyset create/update process adds the jump host IP addresses to the IP tables for each machine in the Cluster. The IP tables update restricts SSH access to be allowed only from those jump hosts.
  • It's important to specify the Cluster facing IP addresses for the jump hosts. These IP addresses might be different than the public facing IP address used to access the jump host.
  • While at least one keyset is defined, ssh access is allowed from any jump host in any keyset. For example, if keyset A specifies jump host A and keyset B specifies jump host B, users in either keyset can use either jump host A or B.
  • While no keysets are defined, ssh access is allowed from any jump host that has network connectivity to the machines.

Prerequisites

  • Install the latest version of the appropriate CLI extensions.
  • The on-premises Cluster must have connectivity to Azure.
  • Get the Resource Group name for the Cluster resource.
  • The process applies keysets to all running bare metal machines.
  • The added users must be part of a Microsoft Entra group. For more information, see How to Manage Groups.
  • To restrict access for managing keysets, create a custom role. For more information, see Azure Custom Roles. In this instance, add or exclude permissions for Microsoft.NetworkCloud/clusters/bareMetalMachineKeySets. The options are /read, /write, and /delete.

Note

When bare metal machine access is created, modified, or deleted via the commands described in this article, a background process delivers those changes to the machines. This process is paused during Operator Nexus software upgrades. If an upgrade is known to be in progress, you can use the --no-wait option with the command to prevent the command prompt from waiting for the process to complete.

Creating a bare metal machine keyset

The baremetalmachinekeyset create command creates SSH access to the bare metal machine in a Cluster for a group of users.

The command syntax is:

az networkcloud cluster baremetalmachinekeyset create \
  --name "<bare metal machine Keyset Name>" \
  --extended-location name="<Extended Location ARM ID>" \
    type="CustomLocation" \
  --location "<Azure Region>" \
  --azure-group-id "<Azure Group ID>" \
  --expiration "<Expiration Timestamp>" \
  --jump-hosts-allowed "<List of jump server IP addresses>" \
  --os-group-name "<Name of the Operating System Group>" \
  --privilege-level "<"Standard" or "Superuser">" \
  --user-list '[{"description":"<User List Description>","azureUserName":"<User Name>",\
    "sshPublicKey":{"keyData":"<SSH Public Key>"}, \
    "userPrincipalName":""}]', \
  --tags key1="<Key Value>" key2="<Key Value>" \
  --cluster-name "<Cluster Name>" \
  --resource-group "<cluster_RG>"

Create Arguments

  --azure-group-id                            [Required] : The object ID of Azure Active Directory
                                                           group that all users in the list must
                                                           be in for access to be granted. Users
                                                           that are not in the group do not have
                                                           access.
  --bare-metal-machine-key-set-name --name -n [Required] : The name of the bare metal machine key
                                                           set.
  --cluster-name                              [Required] : The name of the cluster.
  --expiration                                [Required] : The date and time after which the users
                                                           in this key set are removed from
                                                           the bare metal machines. The maximum
                                                           expiration date is a year from creation
                                                           date. Format is: "YYYY-MM-DDTHH:MM:SS.000Z".
  --extended-location                         [Required] : The extended location of the cluster
                                                           associated with the resource.
    Usage: --extended-location name=XX type=XX
      name: Required. The resource ID of the extended location on which the resource is created.
      type: Required. The extended location type: "CustomLocation".
  --jump-hosts-allowed                        [Required] : The list of IP addresses of jump hosts
                                                           with management network access from
                                                           which a login is be allowed for the
                                                           users. Supports IPv4 or IPv6 addresses.
  --privilege-level                           [Required] : The access level allowed for the users
                                                           in this key set.  Allowed values:
                                                           "Standard" or "Superuser".
  --resource-group -g                         [Required] : Name of cluster resource group. Optional if
                                                           configuring the default group using `az
                                                           configure --defaults group=<name>`.
  --user-list                                 [Required] : The unique list of permitted users.
    Usage: --user-list azure-user-name=XX description=XX key-data=XX
      azure-user-name: Required. User name used to login to the server.
      description: The free-form description for this user.
      key-data: Required. The public ssh key of the user.
      userPrincipalName: Optional. The User Principal Name of the User.

      Multiple users can be specified by using more than one --user-list argument.
  --os-group-name                                        : The name of the group that users are assigned
                                                           to on the operating system of the machines.
  --tags                                                 : Space-separated tags: key[=value]
                                                           [key[=value] ...]. Use '' to clear
                                                           existing tags.
  --location -l                                          : Azure Region. Values from: `az account
                                                           list-locations`. You can configure the
                                                           default location using `az configure
                                                           --defaults location=<location>`.
  --no-wait                                              : Do not wait for the long-running
                                                           operation to finish.

Global Azure CLI arguments (applicable to all commands)

  --debug                                                : Increase logging verbosity to show all
                                                           debug logs.
  --help -h                                              : Show this help message and exit.
  --only-show-errors                                     : Only show errors, suppressing warnings.
  --output -o                                            : Output format.  Allowed values: json,
                                                           jsonc, none, table, tsv, yaml, yamlc.
                                                           Default: json.
  --query                                                : JMESPath query string. See
                                                           http://jmespath.org/ for more
                                                           information and examples.
  --subscription                              [Required] : Name or ID of subscription. Optional if
                                                           configuring the default subscription
                                                           using `az account set -s NAME_OR_ID`.
  --verbose                                              : Increase logging verbosity. Use --debug
                                                           for full debug logs.

This example creates a new keyset with two users that have standard access from two jump hosts.

az networkcloud cluster baremetalmachinekeyset create \
  --name "bareMetalMachineKeySetName" \
  --extended-location name="/subscriptions/subscriptionId/resourceGroups/cluster_RG/providers/Microsoft.ExtendedLocation/customLocations/clusterExtendedLocationName" \
    type="CustomLocation" \
  --location "eastus" \
  --azure-group-id "f110271b-XXXX-4163-9b99-214d91660f0e" \
  --expiration "2022-12-31T23:59:59.008Z" \
  --jump-hosts-allowed "192.0.2.1" "192.0.2.5" \
  --os-group-name "standardAccessGroup" \
  --privilege-level "Standard" \
  --user-list '[{"description":"Needs access for troubleshooting as a part of the support team","azureUserName":"userABC", "sshPublicKey":{"keyData":"ssh-rsa  AAtsE3njSONzDYRIZv/WLjVuMfrUSByHp+jfaaOLHTIIB4fJvo6dQUZxE20w2iDHV3tEkmnTo84eba97VMueQD6OzJPEyWZMRpz8UYWOd0IXeRqiFu1lawNblZhwNT/ojNZfpB3af/YDzwQCZgTcTRyNNhL4o/blKUmug0daSsSXISTRnIDpcf5qytjs1XoyYyJMvzLL59mhAyb3p/cD+Y3/s3WhAx+l0XOKpzXnblrv9d3q4c2tWmm/SyFqthaqd0= admin@vm"},"userPrincipalName":"example@contoso.com"},\
  {"description":"Needs access for troubleshooting as a part of the support team","azureUserName":"userXYZ","sshPublicKey":{"keyData":"ssh-rsa  AAtsE3njSONzDYRIZv/WLjVuMfrUSByHp+jfaaOLHTIIB4fJvo6dQUZxE20w2iDHV3tEkmnTo84eba97VMueQD6OzJPEyWZMRpz8UYWOd0IXeRqiFu1lawNblZhwNT/ojNZfpB3af/YDzwQCZgTcTRyNNhL4o/blKUmug0daSsSXTSTRnIDpcf5qytjs1XoyYyJMvzLL59mhAyb3p/cD+Y3/s3WhAx+l0XOKpzXnblrv9d3q4c2tWmm/SyFqthaqd0= admin@vm"}, "userPrincipalName":"example@contoso.com"}]' \
  --tags key1="myvalue1" key2="myvalue2" \
  --cluster-name "clusterName"
  --resource-group "cluster_RG"

For assistance in creating the --user-list structure, see Azure CLI Shorthand.

Deleting a bare metal machine keyset

The baremetalmachinekeyset delete command removes SSH access to the bare metal machine for a group of users. All members of the group no longer have SSH access to any of the bare metal machines in the Cluster.

The command syntax is:

az networkcloud cluster baremetalmachinekeyset delete \
  --name "<bare metal machine Keyset Name>" \
  --cluster-name "<Cluster Name>" \
  --resource-group "<cluster_RG>"

Delete Arguments

    --bare-metal-machine-key-set-name --name -n [Required] : The name of the bare metal machine key set to be
                                                             deleted.
    --cluster-name                              [Required] : The name of the cluster.
    --resource-group -g                         [Required] : Name of cluster resource group. Optional if configuring the
                                                             default group using `az configure --defaults
                                                             group=<name>`.
    --no-wait                                              : Do not wait for the long-running operation to
                                                             finish.
    --yes -y                                               : Do not prompt for confirmation.

This example removes the "bareMetalMachineKeysetName" keyset group in the "clusterName" Cluster.

az networkcloud cluster baremetalmachinekeyset delete \
  --name "bareMetalMachineKeySetName" \
  --cluster-name "clusterName" \
  --resource-group "cluster_RG"

Updating a Bare Metal Machine Keyset

The baremetalmachinekeyset update command allows users to make changes to an existing keyset group.

The command syntax is:

az networkcloud cluster baremetalmachinekeyset update \
  --name "<bare metal machine Keyset Name>" \
  --jump-hosts-allowed "<List of jump server IP addresses>" \
  --privilege-level "<"Standard" or "Superuser">" \
  --user-list '[{"description":"<User List Description>","azureUserName":"<User Name>",\
   "sshPublicKey":{"keyData":"<SSH Public Key>"}, \
   "userPrincipalName":""}]', \
  --tags key1="<Key Value>" key2="<Key Value> "\
  --cluster-name "<Cluster Name>" \
  --resource-group "<cluster_RG>"

Update Arguments

  --bare-metal-machine-key-set-name --name -n [Required] : The name of the bare metal machine key set.
  --cluster-name                              [Required] : The name of the cluster.
  --expiration                                           : The date and time after which the users
                                                           in this key set are removed from
                                                           the bare metal machines. The maximum
                                                           expiration date is a year from creation
                                                           date. Format is: "YYYY-MM-DDTHH:MM:SS.000Z".
  --jump-hosts-allowed                                   : The list of IP addresses of jump hosts
                                                           with management network access from
                                                           which a login is allowed for the
                                                           users. Supports IPv4 or IPv6 addresses.
  --privilege-level                                      : The access level allowed for the users
                                                           in this key set.  Allowed values:
                                                           "Standard" or "Superuser".
  --user-list                                            : The unique list of permitted users.
    Usage: --user-list azure-user-name=XX description=XX key-data=XX
      azure-user-name: Required. User name used to login to the server.
      description: The free-form description for this user.
      key-data: Required. The public SSH key of the user.
      userPrincipalName: Optional. The User Principal Name of the User.

      Multiple users can be specified by using more than one --user-list argument.
  --resource-group -g                         [Required] : Name of cluster resource group. Optional if
                                                           configuring the default group using `az
                                                           configure --defaults group=<name>`.
  --tags                                                 : Space-separated tags: key[=value]
                                                           [key[=value] ...]. Use '' to clear
                                                           existing tags.
  --no-wait                                              : Do not wait for the long-running
                                                           operation to finish.

This example adds two new users to the "baremetalMachineKeySetName" group and changes the expiry time for the group.

az networkcloud cluster baremetalmachinekeyset update \
  --name "bareMetalMachineKeySetName" \
 --expiration "2023-12-31T23:59:59.008Z" \
  --user-list '[{"description":"Needs access for troubleshooting as a part of the support team",\
  "azureUserName":"userABC", \
  "sshPublicKey":{"keyData":"ssh-rsa  AAtsE3njSONzDYRIZv/WLjVuMfrUSByHp+jfaaOLHTIIB4fJvo6dQUZxE20w2iDHV3tEkmnTo84eba97VMueQD6OzJPEyWZMRpz8UYWOd0IXeRqiFu1lawNblZhwNT/ojNZfpB3af/YDzwQCZgTcTRyNNhL4o/blKUmug0daSsSXISTRnIDpcf5qytjs1XoyYyJMvzLL59mhAyb3p/cD+Y3/s3WhAx+l0XOKpzXnblrv9d3q4c2tWmm/SyFqthaqd0= admin@vm"}, \
  "userPrincipalName":"example@contoso.com"},\
  {"description":"Needs access for troubleshooting as a part of the support team",\
    "azureUserName":"userXYZ", \
    "sshPublicKey":{"keyData":"ssh-rsa  AAtsE3njSONzDYRIZv/WLjVuMfrUSByHp+jfaaOLHTIIB4fJvo6dQUZxE20w2iDHV3tEkmnTo84eba97VMueQD6OzJPEyWZMRpz8UYWOd0IXeRqiFu1lawNblZhwNT/ojNZfpB3af/YDzwQCZgTcTRyNNhL4o/blKUmug0daSsSXTSTRnIDpcf5qytjs1XoyYyJMvzLL59mhAyb3p/cD+Y3/s3WhAx+l0XOKpzXnblrv9d3q4c2tWmm/SyFqthaqd0= admin@vm"}, \
    "userPrincipalName":"example@contoso.com"}]' \
   --cluster-name "clusterName" \
  --resource-group "cluster_RG"

Listing Bare Metal Machine Keysets

The baremetalmachinekeyset list command allows users to see the existing keyset groups in a Cluster.

The command syntax is:

az networkcloud cluster baremetalmachinekeyset list \
  --cluster-name "<Cluster Name>" \
  --resource-group "<cluster_RG>"

List Arguments

  --cluster-name                              [Required] : The name of the cluster.
  --resource-group -g                         [Required] : Name of cluster resource group. Optional if
                                                           configuring the default group using `az
                                                           configure --defaults group=<name>`.

Show Bare Metal Machine Keyset Details

The baremetalmachinekeyset show command allows users to see the details of an existing keyset group in a Cluster.

The command syntax is:

az networkcloud cluster baremetalmachinekeyset show \
  --cluster-name "<Cluster Name>" \
  --resource-group "<cluster_RG>"

Show Arguments

  --bare-metal-machine-key-set-name --name -n [Required] : The name of the bare metal machine key
                                                           set.
  --cluster-name                              [Required] : The name of the cluster.
  --resource-group -g                         [Required] : Name of cluster resource group. You can
                                                           configure the default group using `az
                                                           configure --defaults group=<name>`.