GPU basics for software engineers — Architecture & Trade‑offs — Practical Guide (Nov 17, 2025)

GPU basics for software engineers — Architecture & Trade‑offs

body { font-family: Arial, sans-serif; line-height: 1.6; max-width: 720px; margin: 2em auto; }
h2, h3 { color: #004080; }
pre { background: #f4f4f4; padding: 1em; border-left: 4px solid #008080; overflow-x: auto; }
.audience { font-style: italic; color: #666; margin-bottom: 1em; }
.social { margin-top: 2em; font-weight: bold; color: #005500; }
ul { margin-left: 1.5em; }

GPU basics for software engineers — Architecture & Trade‑offs

Level: Intermediate (familiar with CPU software, starting GPU programming)

As of November 17, 2025

Introduction

For software engineers venturing into parallel computing, understanding the fundamental architecture and design trade‑offs of GPUs is critical. This article focuses on GPU architecture essentials, common programming models, and practical trade‑offs to consider when leveraging GPUs for high-performance workloads as of late 2025.

Prerequisites

Before diving in, ensure you have:

Basic familiarity with CPU programming and concurrency concepts.
Understanding of parallelism (data-level and thread-level).
Access to a supported GPU development environment (e.g. CUDA, OpenCL, or Vulkan compute).

If you are new to GPU hardware specifics or parallel programming, consider preliminary tutorials on SIMD/SIMT principles and GPU programming models.

GPU architecture fundamentals

Key architectural components

GPUs are designed to execute thousands of lightweight threads concurrently, optimising throughput over latency. The main components to be aware of:

Streaming Multiprocessors (SMs) / Compute Units (CUs): GPU cores are grouped into these units. Each SM or CU contains multiple ALUs (arithmetic logic units) able to execute instructions simultaneously.
Warps / Wavefronts: Threads are grouped in fixed-size batches (usually 32 for NVIDIA, 64 for AMD) called warps or wavefronts which execute in lockstep (SIMT—single instruction, multiple threads).
Memory hierarchy: Includes on-chip registers, shared/local memory, L1 and L2 caches, and global memory (device DRAM). High bandwidth but higher latency for global memory access.
Instruction throughput: Optimised for arithmetic and data processing with high flop rates but relatively low cache sizes compared to CPUs.

Trade-offs inherent in architecture

The design choices behind GPUs serve specific workloads efficiently but come with trade-offs:

Throughput vs latency: GPUs prioritise throughput by running many threads concurrently, which means individual latency is often higher than on CPUs.
Branch divergence: When threads within a warp take different code paths (branches), execution serialises those paths, lowering efficiency.
Memory hierarchy impact: Efficient use of shared memory and coalesced global memory access patterns dramatically affects performance.
Power and thermal constraints: Modern GPUs balance clock speed against power delivery and thermal limits, influencing achievable performance and scalability.

Hands-on steps: Writing efficient GPU code

1. Setup your environment

Using NVIDIA CUDA as an example (version 12.x stable in 2025), install the CUDA Toolkit and appropriate GPU drivers. For AMD GPUs, the ROCm platform or HIP offers a similar ecosystem. Vulkan compute shaders and OpenCL remain viable cross-vendor options with distinct API designs and portability characteristics.

2. Simple CUDA kernel example

A standard GPU kernel executes many threads; here is a minimal vector addition kernel in CUDA:

// Vector addition: C[i] = A[i] + B[i]
__global__ void vecAdd(const float* A, const float* B, float* C, int N) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < N) {
        C[i] = A[i] + B[i];
    }
}

3. Launch configuration matters

Choosing the right number of blocks and threads per block affects utilisation and latency hiding. Example launch:

int N = 1 << 20; // size: 1 million elements
int threadsPerBlock = 256;
int blocks = (N + threadsPerBlock - 1) / threadsPerBlock;
vecAdd<<<blocks, threadsPerBlock>>>(d_A, d_B, d_C, N);

Use multiples of warp size (32 threads) to maximise efficiency and avoid underutilised threads.

Common pitfalls

Ignoring memory coalescing: Accessing global memory in non-coalesced patterns causes memory transaction inefficiencies.
Branch divergence within warps: Excessive divergence reduces parallelism and wastes cycles.
Over-using shared memory: Excessive shared memory usage per block reduces number of concurrently active blocks, hurting occupancy.
Not considering occupancy vs register usage: Too many registers per thread limit active warps.
Mixing CPU and GPU synchronisation incompatibly: Host-side threading and device kernels require proper synchronisation primitives.

Validation

Validate your GPU code by:

Comparing GPU outputs against well-tested CPU implementations.
Using vendor profiling tools like nvprof / NVIDIA Nsight Compute or AMD ROCm’s profiler to analyse kernel execution, memory usage, and identify bottlenecks.
Checking warp execution efficiency metrics to detect divergence.
Ensuring no illegal memory accesses via tools such as CUDA’s cuda-memcheck or Vulkan validation layers.

Checklist / TL;DR

Understand GPU architecture: SMs/CUs, warps/wavefronts, and memory hierarchy.
Write highly data-parallel code suitable for SIMT execution.
Minimise branch divergence and optimise for memory coalescing.
Balance threads per block and resource usage to maximise occupancy.
Use profiling tools to validate performance and correctness.
Choose CUDA for NVIDIA GPUs, ROCm/HIP for AMD, or portable APIs like Vulkan/OpenCL depending on cross-vendor needs and language preferences.

When to choose X vs Y

CUDA vs ROCm/HIP: CUDA is mature and highly supported on NVIDIA hardware with broad libraries and ecosystem. ROCm/HIP offers interoperability and portability on AMD GPUs and supports some NVIDIA devices partially but requires careful validation.

OpenCL vs Vulkan compute: OpenCL has broad vendor support but is considered legacy by some; Vulkan compute shaders give more low-level control and integration with graphics pipelines but have a steeper API learning curve.