Sunday, November 15, 2026 Independent reporting on artificial intelligence Newsletter
AI News Beats
Guides & Insights

How to size a Google G4 inference pilot before reserving capacity

G4 support entered Cluster Director preview on November 13, making workload sizing and capacity choices more important than a headline GPU count.

Rows of equipment cabinets line a data center corridor
Photo: Brett Sayles / Pexels

Google Cloud added preview support for G4 machines in Cluster Director on November 13, describing them as an option for single-host AI inference and model tuning, alongside graphics and video workloads. This is an expansion of Cluster Director support, not an announcement that G4 hardware itself has just launched.

For a team considering the new option, the useful first step is a bounded pilot rather than a large reservation. This guide explains how to define that pilot and compare its operating requirements. It does not report independent benchmark results or promise that G4 will be the cheapest choice for a particular application.

Define one workload precisely

Write down the model, its numerical precision, typical input length, maximum output length and expected simultaneous requests. Add the response-time target users actually need. A document-processing batch and an interactive assistant can use the same model while placing very different demands on a server.

Choose representative examples before running the pilot. Include ordinary requests, long inputs and a burst of concurrent requests. Keep the examples fixed across configurations so an apparent improvement is not just the result of giving one machine easier work.

Separate inference from tuning. Inference uses existing weights to generate outputs; tuning changes weights or related parameters through additional training. Do not assume that a model fitting into memory for inference establishes that the same configuration can support the planned tuning process.

Distinguish GPU memory from system memory

Google’s supported-machines documentation lists G4 configurations with one, two, four or eight Nvidia RTX PRO 6000 Blackwell Server Edition GPUs. The listed single-GPU machine has 96 GB of GPU memory and 180 GB of instance memory. Those are different pools, not one combined amount available interchangeably to the model.

Record peak memory use during the longest realistic requests, not only immediately after loading the model. Leave room for the serving software and request state. A configuration that loads successfully but fails when several users arrive is not a successful capacity test.

The same Google documentation requires N2 login nodes and Hyperdisk Balanced boot disks for G4 compute nodes in Cluster Director. It also describes attached local SSD storage. Include the surrounding infrastructure in the pilot design rather than treating the GPU count as the complete specification.

Measure useful output and waiting time

Track both completed requests and how long users wait. Record the time until an answer starts, the time until it finishes and how those measurements change under load. An average alone can hide a small group of unusually slow requests that make an interactive product frustrating.

Record failures separately: out-of-memory errors, timeouts, invalid outputs and interrupted jobs. Keep the model and generation settings stable when comparing runs. If you change precision or output limits to improve throughput, label that change and check whether answer quality still meets the application’s requirements.

For a batch process, use completed acceptable jobs as the denominator. For an interactive service, include idle time required to keep capacity ready. These are different operating patterns, so a single headline measure such as tokens per second cannot decide the deployment on its own.

Choose a capacity arrangement deliberately

Google’s consumption-options guide distinguishes on-demand capacity, reservations, Spot and Flex-start. Its G4 entries include these options, but they differ in availability, interruption risk and billing. Spot capacity can be preempted; Flex-start is for a limited run duration and a workload that can wait for resources.

A restartable offline job may be able to accept interruption in exchange for different economics. A customer-facing service needs a plan for maintaining availability. Test checkpointing and restart behavior before treating interruption-tolerant capacity as an ordinary substitute for a continuously available server.

Reservations deserve a separate budget review. Google’s guide says standard pricing applies while an on-demand reservation exists, subject to relevant commitments. Avoid confusing a stopped experiment with a completed cleanup. Assign someone to inspect the resources and reservations remaining after the test.

Make the decision from the pilot record

Prepare a concise result sheet showing the configuration, test inputs, successful work completed, response-time distribution, failures and estimated full operating cost. Include storage, supporting nodes and the effort required to operate the service, not just the compute line item.

Finally, treat preview status as a real deployment constraint. Google’s documentation says pre-general-availability features may have limited support. The decision should state whether the measured benefit justifies that limitation, what remains untested and which existing configuration can take over if the pilot cannot progress safely.

Stay on top of AI

Get the stories that matter in artificial intelligence, straight to your inbox.

Free. One email with the stories that matter. Unsubscribe any time. See our privacy policy.