Create a MIG with GPU VMs

This document describes how to create a managed instance group (MIG) with virtual machine (VM) instances that have attached GPUs. It describes how to add GPU VMs all at once in the group using resize requests. To create a resize request in a MIG that can use the features and services available from Cluster Director, see Create a MIG and a resize request in the AI Hypercomputer documentation.

Using a resize request improves obtainability of GPU VMs in a MIG. In the request, specify the number of GPU VMs and a duration for which you want to run those VMs. Dynamic Workload Scheduler (DWS), the underlying scheduler mechanism, schedules resize requests created across Compute Engine based on requested durations and resource availability. When the resources become available, the MIG automatically creates the VMs.

If your job running on these VMs finishes earlier than the requested duration, you can delete those VMs. Otherwise, the MIG automatically deletes the VMs when the duration ends.

You can also read about other basic scenarios for creating a MIG.

Before you begin

  • To make sure that you have sufficient GPU quota for the resources you're requesting, check your GPU quota.
  • To understand quota consumption, read GPU VMs and preemptible allocation quotas.
  • If you haven't already, then set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

    Select the tab for how you plan to use the samples on this page:

    Console

    When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

    gcloud

    1. After installing the Google Cloud CLI, initialize it by running the following command: 

      gcloudinit

      If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    2. Set a default region and zone.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      After installing the Google Cloud CLI, initialize it by running the following command: 

      gcloudinit

      If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Limitations

Review the limitations for creating a resize request in a MIG.

Create a MIG and add GPU VMs all at once

To create a MIG and add GPU VMs all at once in the group, do the following:

  1. Create an instance template, which is required in order to create a MIG. The MIG creates each VM in the group based on the instance template. In the template, specify the configuration for GPU VMs and additional configurations required for using resize requests.

    For more information about instance templates, see About instance templates.

  2. Create a MIG and a resize request to add GPU VMs all at once.

Create an instance template

Create an instance template as described in this section, and then use the template to create a MIG.

Console

  1. Go to the Instance templates page.

    Go to Instance templates

  2. Click Create instance template.

  3. In the Name field, enter a name for the instance template.

  4. In the Machine configuration section, do the following:

    1. Click the GPUs tab.
    2. In the GPU type list, select the GPU type.
    3. In the Number of GPUs list, select the number of GPUs.
    4. In the Machine type list, select a machine type.

  5. Optional: To change the default value boot disk type or image, in the Boot disk section, click Change. Then, follow the prompts to change the boot disk.

  6. Expand the Advanced options section, and then do the following:

    1. Expand the Management section.
    2. In the Reservations list, select Don't use.

  7. Click Create.

gcloud

Create an instance template using the instance-templates create command.

The following command creates a global instance template based on a Deep Learning VM image.

 gcloud compute instance-templates create INSTANCE_TEMPLATE_NAME \ --machine-type=MACHINE_TYPE \ --image-project=deeplearning-platform-release \ --image-family=common-cu121 \ --maintenance-policy=TERMINATE \ --reservation-affinity=none

Replace the following:

  • INSTANCE_TEMPLATE_NAME: the name of the instance template.
  • MACHINE_TYPE: a machine type that supports GPUs. If you specify an N1 machine type, then include the --accelerator flag to specify the number and type of GPUs to attach to your VMs.

REST

Create an instance template by making a POST request to the instanceTemplates.insert method.

The following request creates a global instance template based on a Deep Learning VM image.

 POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/instanceTemplates { "name": "INSTANCE_TEMPLATE_NAME", "properties": { "machineType": "MACHINE_TYPE", "disks": [ { "boot": true, "mode": "READ_WRITE", "type": "PERSISTENT", "initializeParams": { "sourceImage": "projects/deeplearning-platform-release/global/images/family/common-cu121" } } ], "networkInterfaces": [ { "accessConfigs": [ { "name": "external-IP", "type": "ONE_TO_ONE_NAT" } ], "network": "global/networks/default" } ], "reservationAffinity": { "consumeReservationType": "NO_RESERVATION" }, "scheduling": { "onHostMaintenance": "TERMINATE" } } }

Replace the following:

  • PROJECT_ID: the ID of the project in which you want to create the MIG.
  • INSTANCE_TEMPLATE_NAME: the name of the instance template.
  • MACHINE_TYPE: a machine type that supports GPUs. If you specify an N1 machine type, then include the guestAccelerators field to specify the number and type of GPUs to attach to your VMs.

Create a MIG and add GPU VMs all at once

Create a MIG as described in this section. To use a resize request in the MIG, you must not configure autoscaling and must turn off repairs.

Console

  1. Go to the Instance groups page.

    Go to Instance groups

  2. Click Create instance group.

  3. Enter the following fields:

    1. Name: Enter a name for the group.
    2. Instance template: Select the instance template that you configured in the previous section.
    3. In the Location section:
      1. Select Single zone.
      2. In the Region and Zone lists, select the location where you want to create the group.

    The Number of instances field will be enabled for editing after you delete the autoscaling configuration and turn off repairs in the next steps.

  4. Delete the autoscaling configuration as follows:

    1. In the Autoscaling section, click the Autoscaling mode list, and then click Delete autoscaling configuration.
    2. In the Delete autoscaling configuration? dialog, click Delete.

  5. Turn off repairs in the MIG as follows:

    In the VM instance lifecycle section, set the Default action on failure field to No action.

  6. Specify the number of GPU VMs and the run duration for creating a resize request:

    1. In the Number of instances field, enter the number of GPU VMs that you want to create all at once.

      The number of instances field on the Create instance group page.

    2. Select the Use resize request to create VMs all at once checkbox. The fields to enter the run duration appear.

    3. In the Requested run duration field and Unit list, specify the run duration of the GPU VMs. The duration must be between 1 hour and 7 days.

  7. Click Create.

gcloud

  1. Create a zonal MIG using the instance-groups managed create command.

     gcloud compute instance-groups managed create INSTANCE_GROUP_NAME \ --template=INSTANCE_TEMPLATE_NAME \ --size=0 \ --zone=ZONE \ --default-action-on-vm-failure=do_nothing

  2. In the MIG, create a resize request using the instance-groups managed resize-requests create command. Specify the number of GPU VMs that you want and the duration for which you want to run those VMs.

     gcloud compute instance-groups managed resize-requests create INSTANCE_GROUP_NAME \ --resize-request=RESIZE_REQUEST_NAME \ --resize-by=COUNT \ --requested-run-duration=RUN_DURATION\ --zone=ZONE

Replace the following:

  • INSTANCE_GROUP_NAME: the name of the MIG.
  • INSTANCE_TEMPLATE_NAME: the name of the instance template for GPU VMs.
  • ZONE: one of the zones available for Compute Engine.
  • RESIZE_REQUEST_NAME: the name of the resize request.
  • COUNT: the number of VMs to add all at once in the group.
  • RUN_DURATION: the duration you want the requested VMs to run. The value must be formatted as the number of days, hours, minutes, or seconds followed by d, h, m, and s respectively. For example, specify 30m for 30 minutes or 1d2h3m4s for 1 day, 2 hours, 3 minutes, and 4 seconds. The value must be between 10 minutes and 7 days.

REST

  1. Create a zonal MIG by making a POST request to the instanceGroupManagers.insert method.

     POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers { "versions": [ { "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME" } ], "name": "INSTANCE_GROUP_NAME", "targetSize": 0, "instanceLifecyclePolicy": { "defaultActionOnFailure": "DO_NOTHING" } }

  2. In the MIG, create a resize request by making a POST request to the instanceGroupManagerResizeRequests.insert method. In the request body, specify the number of GPU VMs that you want to create all at once and the duration that you want to run those VMs.

     POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/INSTANCE_GROUP_NAME/resizeRequests { "name": "RESIZE_REQUEST_NAME", "resizeBy": COUNT, "requestedRunDuration": { "seconds": "RUN_DURATION" } }

Replace the following:

  • PROJECT_ID: the ID of the project in which you want to create the MIG.
  • INSTANCE_GROUP_NAME: the name of the MIG.
  • INSTANCE_TEMPLATE_NAME: the name of the instance template for GPU VMs.
  • ZONE: one of the zones available for Compute Engine.
  • RESIZE_REQUEST_NAME: the name of the resize request.
  • COUNT: the number of VMs to add all at once in the group.
  • RUN_DURATION: the duration, in seconds, you want the requested VMs to run before the MIG automatically deletes them. The value must be between 600, which is 600 seconds (10 minutes), and 604800, which is 604,800 seconds (7 days).

The resize request that you create stays in the ACCEPTED state until the MIG creates all the requested GPU VMs. After all GPU VMs are created in the group, the state of the request changes to SUCCEEDED.

What's next