Solutions · High Performance Computing

High performance computing built around the research window.

Scalable compute clusters, R&D workload orchestration and advanced simulation estates. We size the machine against elapsed time for your real jobs, because a run that lands after the deadline is worth nothing at any FLOP rate.

Send us an input deck See indicative tiers

The constraint we design around first

The window, not the FLOPS.

Time-to-result is a chain of five terms, and raw compute improves only one of them. A job queues, stages its input, solves, writes output, and then waits for a person to interpret it. Peak double-precision throughput touches the middle term alone.

On most estates we audit, queue wait is the largest single term. A six-hour solve that sits four days behind a badly bounded job array is a four-day result, and no node purchase fixes it.

We will tell you when the answer is not a bigger cluster. Roughly a third of the profiling engagements we run end with a scheduler policy change, an IO rework or a solver setting, not a quote. That advice is the same price as the quote.

Queue wait: Set by policy and job mix, not by hardware. The first term we measure and usually the cheapest to recover.
Data staging: Moving a case onto fast scratch. Often single-threaded and metadata-bound, so it never appears in a FLOPS figure.
Solver wall-clock: The only term a faster node shortens, and rarely the majority of the window once the other four are measured.
Post-processing: Reduction and visualisation, commonly memory-bound and serial. A visualisation partition removes the download entirely.
Re-runs: A failed case doubles the window. Short debug queues and input validation cost far less than more nodes.

Cluster design

Four decisions set the ceiling.

Node topology

Most simulation codes are limited by memory bandwidth per core, not by core count. We choose the node shape from a STREAM measurement and your solver's own timers, then keep every memory channel populated.

NUMA-aware layout: one MPI rank per NUMA domain, threads pinned inside it, sub-NUMA clustering enabled only when the code benefits
Every channel filled. A half-populated board can cost more bandwidth than a generation of clock speed
Thin nodes for domain-decomposed solvers, a small fat-node tier for implicit FEA assembly and large in-memory meshes
Node-local NVMe scratch for solver spill and checkpoints, keeping that traffic off the parallel filesystem
Separate login and build nodes, stateless provisioning with Warewulf or xCAT, so an image rebuild is routine rather than an outage

Node block diagram — dual-socket NUMA domains, memory channels, HCA placement and local NVMe scratch — 1200×900

CPU and GPU partitions

A GPU partition is only worth buying for codes with a maintained accelerator path. We check that path against your actual physics models before anything is quoted, and we say so when it does not exist.

GPU-ready today: GROMACS, AMBER, LAMMPS with KOKKOS, NAMD, most deep-learning frameworks, and parts of the commercial CFD stack
Frequently not: in-house Fortran, implicit FEA with sparse direct solvers, and any commercial solver whose GPU path excludes the models you run
Licence-aware scheduling for Fluent, Abaqus and LS-DYNA, so jobs wait for a token rather than starting and failing
Generation-partitioned queues once a cluster spans two CPU families, so an MPI job never straddles microarchitectures

cpu: Exclusive nodes, topology-aware placement, the default home for MPI-bound solvers.
gpu: Accelerated codes and training runs. GPU-count and MPS policy per job, not per node.
himem: A small tier for implicit assembly, large meshes and genome-scale in-memory steps.
debug: Tight walltime, high priority, capped per user. A five-minute test never queues behind a five-day run.
viz: Remote GPU desktops with VirtualGL, so post-processing happens beside the data.
preempt: Low-priority, requeue-able work that soaks idle capacity between scheduled jobs.

Interconnect fabric and blocking ratio

Tightly coupled codes are limited by small-message latency, not by bandwidth. The fabric choice follows the message-size histogram we take from your jobs, and the blocking ratio follows the size of the jobs that will run on it.

InfiniBand where collectives dominate: adaptive routing, a managed subnet, and in-network reduction for allreduce-heavy solvers
RoCEv2 where the team already runs Ethernet and can hold a lossless configuration — PFC and ECN discipline is the whole job
GPUDirect RDMA between accelerators and storage, so the host CPU stops copying data it never reads
Slurm switch topology declared, so a 128-rank job is packed inside one leaf instead of scattered across the spine
A separate management and IPMI network, because out-of-band access must survive a fabric problem

Non-blocking: Warranted when single jobs routinely span the whole cluster, or when allreduce sits in the inner loop.
2:1 spine taper: Defensible when the job mix is mostly single-leaf and the scheduler is topology-aware. It buys nodes instead of switches.
Island design: Non-blocking inside an island, tapered between. The common shape for a shared research facility.
How we decide: Message-size and collective-frequency traces from your codes, then a cost comparison against the nodes the switches would displace.

Fabric diagram — two-level fat tree with leaf islands, spine taper and storage attachment — 1200×900

Workload orchestration

A poor scheduler wastes more capacity than any hardware choice.

Partitions that match the hardware truth

One partition per genuine hardware difference, and no more. Walltime ceilings per partition, exclusive allocation for MPI jobs, shared allocation for serial work. When a partition exists for political rather than physical reasons, it fragments the machine and hides idle nodes.

Fair-share instead of first-come

A pure queue order lets one group with a large job array starve every other project for a fortnight. Multifactor priority with a tuned fair-share decay half-life restores the balance an allocation committee thought it had agreed, and it does so without anybody filing a ticket.

QoS tiers with honest limits

Express, normal, long and preempt-able tiers, each with its own priority weight, walltime and running-resource cap. Group limits on concurrently running resources stop a single account from occupying the cluster, while a tight express tier keeps interactive debugging usable during the working day.

Backfill that actually fills

The backfill scheduler needs a window long enough to see the gaps and a resolution fine enough to place work in them. It also needs honest walltimes: a user who always requests the maximum makes their own jobs unschedulable. We publish wait times by requested walltime so the incentive becomes visible.

Preemption for opportunistic capacity

A low-priority tier that requeues on preemption turns idle nodes into completed parameter sweeps. The prerequisite is checkpointing, either native to the code or through a transparent checkpoint layer. Without it, preemption destroys work rather than recovering capacity.

Accounting that ends the argument

The accounting database, per-project charge rates by partition, and a monthly report of consumption, wait time and completion rate. When allocation is disputed at a governance meeting, the discussion is about the data rather than about who complained most recently.

Simulation domains

Each discipline stresses a different part of the machine.

Computational fluid dynamics

OpenFOAM and Ansys Fluent are memory-bandwidth bound at the cell level and latency bound at the halo. Scaling holds while each rank keeps enough cells to hide its own neighbour exchange, then falls off a cliff.

We find the cells-per-rank floor for your mesh, then set the default rank count under it
Decomposition method and rank ordering matched to the fabric topology
Licence tokens modelled as a schedulable resource for the commercial stack

Architecture driver — memory bandwidth, then small-message latency

Finite element analysis

Implicit solvers in Abaqus and Ansys Mechanical are limited by memory capacity and by scratch IO when the sparse factorisation spills. Explicit codes such as LS-DYNA scale further but are sensitive to interconnect latency.

Fat nodes with fast local NVMe usually beat a larger count of thin nodes
Scratch sized from the real factorisation footprint, not from a rule of thumb
Memory-per-core requested honestly, so the scheduler can pack the rest of the node

Architecture driver — memory capacity and scratch write throughput

Molecular dynamics

GROMACS, LAMMPS and NAMD do very little work per timestep, so the step rate is governed by latency and by the balance between host and accelerator. Ensembles of independent replicas are a throughput problem rather than a scaling one.

CPU-to-GPU ratio tuned per code, with long-range electrostatics offloaded where it pays
Replica ensembles run as job arrays, which keeps them inside the backfill window
Trajectory write frequency reviewed early, since it silently dominates the filesystem

Architecture driver — MPI latency and host-accelerator balance

Climate and weather

WRF and coupled climate models run against a fixed delivery hour, which makes runtime determinism a requirement rather than a preference. Output dominates: NetCDF and HDF5 writes at every reporting interval, from every rank.

Parallel filesystem striping and collective IO tuned for many-writer output
Reserved capacity and a dedicated QoS for operational forecast windows
Restart cadence set so a node failure costs one interval, not the whole run

Architecture driver — IO pattern and deterministic wall-clock

High-throughput parallel computing

Embarrassingly parallel is a compliment.

Parameter sweeps, Monte Carlo campaigns, replica ensembles, per-sample bioinformatics stages and per-trace seismic processing have no inter-task communication at all. They belong in job arrays and in the preempt-able tier, where they convert idle nodes into finished work.

A decomposed solver is the opposite. Every timestep ends in a synchronising collective, so the slowest rank sets the pace for all of them. Strong scaling measures a fixed problem on more ranks and eventually stops improving. Weak scaling grows the problem with the ranks, and is the honest number to quote.

The ceiling is usually allreduce. Krylov solvers perform a global reduction every iteration, so cost becomes latency multiplied by iteration count and no extra bandwidth helps. We cut rank counts with hybrid MPI and OpenMP, reorder ranks to match the fabric, and use in-network reduction where the fabric supports it.

Past the knee, more nodes cost money and return nothing. We will name the rank count where your case stops scaling, and often the right answer is four cases at 256 ranks rather than one at 1024.

512Ranks at which the reference CFD case holds 76% parallel efficiency

2.4×Throughput gain from moving ensemble work into the preempt-able tier

31%Allreduce time removed by hybrid ranks and topology-aware placement

Chart — speed-up and parallel efficiency against rank count, with the knee marked — 1200×900

Benchmarking

Linpack measures Linpack.

HPL rewards exactly the machine attributes that matter least to a bandwidth-bound solver. It is still useful as a stress test. Driving a new cluster to its thermal and electrical limits during acceptance is the only claim we make for it.

The proxies that correlate with real work are less flattering. HPCG for sparse, memory-bound behaviour. STREAM for bandwidth per socket. The OSU micro-benchmarks for point-to-point latency and allreduce at scale. IOR and mdtest for filesystem throughput and metadata rate. Then your own input deck, which outranks all of them.

Primary measure: Wall-clock time-to-result on your named cases, run end to end including staging and output.
Scaling: Parallel efficiency against rank count, with the knee identified and documented.
Queue behaviour: Wait time at the median and the 95th percentile, replayed against your real job mix.
Delivery rate: Completed jobs per project per week, which is what a research group actually experiences.
Energy: Kilowatt-hours per completed run, reported so efficiency claims can be checked.
Filesystem: Metadata operations per second under concurrent load, not peak sequential throughput.

We would rather hand over a cluster that looks slower on a benchmark table and finishes your Friday job on Thursday.

Principal Engineer HPC practice, Cloud Natives — attribution pending approval

Acceptance is a document, not a demonstration. Before a cluster is handed over we agree the cases, the input data, the rank counts and the pass criteria in writing. The report states the method beside every number, so your own team can reproduce it after we leave.

Sizing and delivery

Three shapes, and the sequence that builds them.

Indicative cluster tiers

Indicative only. Bands are illustrative and are confirmed against your workload profile before any proposal.
Attribute	Departmental	Faculty or institute	National scale
Compute nodes	8–16 nodes, one rack	48–128 nodes, several racks	300+ nodes across islands
CPU partition	Dual-socket, all memory channels populated	As departmental, plus a high-memory tier	Multiple generations, partitioned by microarchitecture
Accelerator partition	1–2 nodes, only if a GPU code path exists	8–24 GPU nodes with in-node peer links	Dedicated GPU island with GPUDirect RDMA
Fabric	100 GbE with RoCEv2, single leaf	InfiniBand HDR, two-level fat tree	InfiniBand NDR, topology-aware placement
Blocking ratio	Non-blocking within the leaf	2:1 taper at the spine	Non-blocking inside islands, tapered between
Parallel filesystem	NFS over RDMA on all-flash	BeeGFS or Lustre, flash metadata tier	Lustre with separate metadata and capacity tiers
Node-local scratch	2 TB NVMe per node	4 TB NVMe per node	8 TB NVMe per node, plus a burst tier
Cooling	Contained air in an existing room	Air with rear-door heat exchangers	Direct liquid cooling on dense nodes
Rack power envelope	Roughly 8–15 kW	Roughly 15–35 kW	40 kW and above
Scheduling	Slurm, two partitions, simple fair-share	Slurm with QoS tiers, preemption and accounting	Slurm with declared topology and federation
What we commit to	Time-to-result on named cases	Plus queue-wait targets per project	Plus energy per completed run

Deployment sequence

Workload profiling — two to three weeks

We take your real input decks and run them. The code's own timers, hardware counters and MPI traces give us the split between compute, communication and IO, plus the rank count where each case stops scaling.

Architecture and costed sizing

Node shape, partition layout, fabric and blocking ratio, filesystem and cooling strategy, each justified by a measurement from the previous stage. You receive the evidence alongside the bill of materials, including the options we rejected.

Facility and power readiness

Floor loading, power distribution, containment, chilled water and any secondary loop. This stage frequently changes the design, because the building sets a ceiling that no procurement decision can lift.

Build, burn-in and acceptance

Provisioning, firmware baselines, then a deliberate thermal and electrical stress run. STREAM, OSU and IOR establish the node and fabric baseline, we hunt outlier nodes, and only then do we run the agreed acceptance cases.

Scheduler policy and user onboarding

Partitions, QoS tiers, fair-share weights and accounting. A module environment built with Spack or EasyBuild and presented through Lmod. Container support with Apptainer, then training for the research groups who will use it daily.

Operate, report and re-tune quarterly

Utilisation, wait time and completion reporting each month. A quarterly queue policy review with your allocation committee, firmware and software currency, and a capacity forecast delivered before the next funding round rather than after it.

Straight answers

What procurement actually asks.

Start with a profile

Bring the run that never finishes in time. We profile before we quote.

Give us an input deck, the current wall-clock time and the date it has to meet. You get a measured breakdown of queue, staging, solve and output, and a direct answer about whether hardware is the problem.

Book a profiling session All solutions

HPC practice enquiries: hello@cloudnatives.example
Direct line: +61 0 0000 0000
What to send: Input deck, current rank count and wall-clock, and the deadline

Profiling is done on our bench under a mutual non-disclosure agreement.

Infrastructure for workloads that refuse to wait.

Bespoke architecture, sector by sector.