# hpcnotes.eu > HPC notes and wiki: high-performance computing and performance engineering, from CPU caches, SIMD and compilers to MPI, OpenMP, CUDA and Slurm clusters. Canonical site: https://hpcnotes.eu. Cite each note's HTML URL, not this combined file. --- title: "Argmin with SIMD" summary: "Argmin with SIMD: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-algorithms order: 7 tags: hpc draft: false --- Horizontal min, then a lane index. Horizontal is the slow part. --- title: "Integer Factorization" summary: "Integer Factorization: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-algorithms order: 3 tags: hpc draft: false --- Pollard, ECM, and knowing when to stop. 64-bit is not a research paper. --- title: "Binary GCD" summary: "Binary GCD: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-algorithms order: 1 tags: hpc draft: false --- Shifts instead of division. The binary variant is the one the core likes. --- title: "Optimizing Logistic Regression" summary: "Optimizing Logistic Regression: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-algorithms order: 5 tags: hpc draft: false --- The bottleneck is the data walk, not the sigmoid. Layout first. --- title: "Matrix Multiplication" summary: "Matrix Multiplication: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-algorithms order: 20 tags: hpc draft: false --- Blocking, packing, and the microkernel. For-for-for is the baseline you are here to kill. --- title: "Prefix Sum with SIMD" summary: "Prefix Sum with SIMD: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-algorithms order: 8 tags: hpc draft: false --- Scan inside the vector, then across. The serial chain is the enemy. --- title: "Reading Decimal Integers" summary: "Reading Decimal Integers: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-algorithms order: 10 tags: hpc draft: false --- SWAR, SIMD, and never scanf in the hot path. --- title: "Sorting" summary: "Sorting: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-algorithms order: 6 tags: hpc draft: false --- Keys, payload, and the branchy partition. SIMD sort is a different algorithm. --- title: "Algorithms Case Studies" summary: "Algorithms Case Studies: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 11 tags: hpc draft: false --- GCD, scan, matmul, parse, sort: the same tricks, applied until the kernel is limited by the machine, not the textbook. --- title: "Assembly Language" summary: "Assembly Language: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-architecture order: 1 tags: hpc draft: false --- Read the listing before you argue with the compiler. If you cannot find the loop, you are not in the kernel. --- title: "Functions and Recursion" summary: "Functions and Recursion: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-architecture order: 3 tags: hpc draft: false --- Calls are ABI, spills, and lost inlining. Recursion is a stack, not a metaphor. --- title: "Indirect Branching" summary: "Indirect Branching: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-architecture order: 4 tags: hpc draft: false --- Indirect jumps are where predictors go to die. Tables and visitors pay this tax. --- title: "Interrupts and System Calls" summary: "Interrupts and System Calls: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-architecture order: 9 tags: hpc draft: false --- Leaving user mode is a fence. Batch syscalls; do not sample the kernel in the inner loop. --- title: "Instruction Set Architectures" summary: "Instruction Set Architectures: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-architecture order: -1 tags: hpc draft: false --- Registers, addressing, and what the core actually retires. The C abstract machine is a courtesy. --- title: "Machine Code Layout" summary: "Machine Code Layout: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-architecture order: 10 tags: hpc draft: false --- I-cache and jump alignment. The order of functions is part of the algorithm. --- title: "Loops and Conditionals" summary: "Loops and Conditionals: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-architecture order: 2 tags: hpc draft: false --- A mispredicted branch is a pipeline flush. Flatten the hot path; keep the rare path cold. --- title: "Computer Architecture" summary: "Computer Architecture: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 2 tags: hpc draft: false --- What the ISA, the assembler, and the call stack actually do to a tight loop. Read this before blaming the compiler. --- title: "Bit Manipulation" summary: "Bit Manipulation: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 7 tags: hpc draft: false --- Isolate, count, and scatter bits without a loop if the ISA has the op. --- title: "Data Compression" summary: "Data Compression: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 8 tags: hpc draft: false --- Entropy in the cache line. Compress for bandwidth, not for the archive. --- title: "Integer Division" summary: "Integer Division: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 6 tags: hpc draft: false --- Still expensive. Reciprocals, magic numbers, or do not divide. --- title: "Rounding Errors" summary: "Rounding Errors: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 2 tags: hpc draft: false --- The error is the algorithm. If you need bits, say how many, and where they die. --- title: "Floating-Point Numbers" summary: "Floating-Point Numbers: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 1 tags: hpc draft: false --- Rounding, associativity, and why the parallel reduction does not match the serial one. --- title: "IEEE 754 Floats" summary: "IEEE 754 Floats: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 2 tags: hpc draft: false --- NaN, denormals, and FTZ. Denormals in the hot path are a performance bug. --- title: "Integer Numbers" summary: "Integer Numbers: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 5 tags: hpc draft: false --- Wrapping, widening, and the multiply that becomes a shift. Know the width. --- title: "Newton's Method" summary: "Newton's Method: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 3 tags: hpc draft: false --- Iterate until the bits stop changing. The first guess is the whole trick. --- title: "Fast Inverse Square Root" summary: "Fast Inverse Square Root: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-arithmetic order: 4 tags: hpc draft: false --- A constant, a shift, and a Newton step. History, then the instruction that replaced it. --- title: "Arithmetic" summary: "Arithmetic: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 6 tags: hpc draft: false --- Floats, integers, and the shortcuts that are legal on this microarchitecture. Division is still the expensive one. --- title: "Non-Zero-Cost Abstractions" summary: "Non-Zero-Cost Abstractions: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-compilation order: 7 tags: hpc draft: false --- Some layers vanish; some become calls. Read the assembly of the abstraction. --- title: "Arithmetic Optimizations" summary: "Arithmetic Optimizations: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-compilation order: 10 tags: hpc draft: false --- Strength reduction, lea, and the multiply the compiler already rewrote. Do not fight it blindly. --- title: "Contract Programming" summary: "Contract Programming: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-compilation order: 6 tags: hpc draft: false --- restrict, noexcept, and the promises the optimiser is allowed to believe. --- title: "Flags and Targets" summary: "Flags and Targets: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-compilation order: 2 tags: hpc draft: false --- -O3 is not a target. Name the CPU, the ISA, and the ABI you built for. --- title: "What Compilers Can and Can't Do" summary: "What Compilers Can and Can't Do: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-compilation order: 10 tags: hpc draft: false --- It will not invent a better layout. Aliasing and the math library still need a human. --- title: "Precomputation" summary: "Precomputation: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-compilation order: 8 tags: hpc draft: false --- Move work to compile time or to startup. The hot path should load, not reconstruct. --- title: "Situational Optimizations" summary: "Situational Optimizations: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-compilation order: 3 tags: hpc draft: false --- A flag that helps the kernel can wreck the rest. Measure the binary you ship. --- title: "Stages of Compilation" summary: "Stages of Compilation: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-compilation order: 1 tags: hpc draft: false --- Preprocess, compile, assemble, link. Know which stage ate your intrinsic. --- title: "Compilation" summary: "Compilation: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 4 tags: hpc draft: false --- Flags, contracts, and what the compiler will never see. If you cannot name the pass, you are guessing. --- title: "Modern Hardware" summary: "Modern Hardware: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-complexity order: 1 tags: hpc draft: false --- Clock stopped being the whole story. Density, parallel issue, and the memory wall are the model now. --- title: "Programming Languages" summary: "Programming Languages: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-complexity order: 2 tags: hpc draft: false --- Pick the language that lets you see loads, stores, and the ABI. Abstraction is allowed; opacity is not. --- title: "When to Optimize" summary: "When to Optimize: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-complexity order: 4 tags: hpc draft: false --- If it is not on the critical path, leave it. The rest of this note assumes you already checked. --- title: "Models of Computation" summary: "Models of Computation: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-complexity order: 3 tags: hpc draft: false --- RAM, I/O, and the realistic cost of a cache miss. Use the model that matches the bottleneck you measured. --- title: "Complexity Models" summary: "Complexity Models: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 1 tags: hpc draft: false --- Asymptotics still matter, but they are not a runtime. This section is the map of when the machine disagrees with the textbook. --- title: "Alignment and Packing" summary: "Alignment and Packing: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 8 tags: hpc draft: false --- Split a line, pay twice. Align the hot struct; pack the cold one. --- title: "AoS and SoA" summary: "AoS and SoA: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 13 tags: hpc draft: false --- The loop dictates the layout. Gather is the cost of getting it wrong. --- title: "Cache Associativity" summary: "Cache Associativity: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 11 tags: hpc draft: false --- Conflict misses on a power-of-two stride. Pad, or change the layout. --- title: "Memory Bandwidth" summary: "Memory Bandwidth: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 1 tags: hpc draft: false --- Bytes per cycle from DRAM. If you are under the roof, it is not compute. --- title: "Cache Lines" summary: "Cache Lines: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 3 tags: hpc draft: false --- 64 bytes, usually. False sharing is two writers on one line. --- title: "Memory Latency" summary: "Memory Latency: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 2 tags: hpc draft: false --- The round trip you cannot hide with more cores. Prefetch or MLP. --- title: "Memory-Level Parallelism" summary: "Memory-Level Parallelism: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 5 tags: hpc draft: false --- Several misses in flight. Independent loads are the other SIMD. --- title: "Memory Paging" summary: "Memory Paging: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 12 tags: hpc draft: false --- 4K vs 2M. The TLB is a cache; treat it like one. --- title: "Pointer Alternatives" summary: "Pointer Alternatives: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 10 tags: hpc draft: false --- Indices, packed ids, and arrays of structs that actually sit together. --- title: "Prefetching" summary: "Prefetching: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 6 tags: hpc draft: false --- Hardware first, software when the pattern is obvious and the lookahead is known. --- title: "Memory Sharing" summary: "Memory Sharing: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-cpu-cache order: 3 tags: hpc draft: false --- Who owns the line. Coherence is a protocol, not a feeling. --- title: "RAM & CPU Caches" summary: "RAM & CPU Caches: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 9 tags: hpc draft: false --- Bandwidth, latency, lines, associativity, prefetch. Most 'CPU bound' codes are waiting on RAM. --- title: "Search Trees" summary: "Search Trees: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-data-structures order: 3 tags: hpc draft: false --- std::set is a baseline, not a design. Layout and prefetch beat rotations in the probe path. --- title: "Binary Search" summary: "Binary Search: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-data-structures order: 1 tags: hpc draft: false --- Branchless, Eytzinger, and the cache line you actually hit. --- title: "Bitmaps" summary: "Bitmaps: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-data-structures order: 6 tags: hpc draft: false --- Bits as the data structure. Rank and select if you need them, not as decoration. --- title: "Probabilistic Filters" summary: "Probabilistic Filters: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-data-structures order: 10 tags: hpc draft: false --- False positives on purpose. Size the filter for the miss cost you measured. --- title: "Hash Tables" summary: "Hash Tables: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-data-structures order: 8 tags: hpc draft: false --- Open addressing, load factor, and the probe that stays in cache. --- title: "Static B-Trees" summary: "Static B-Trees: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-data-structures order: 2 tags: hpc draft: false --- A search tree laid out for cache, not for malloc. Build once, probe many. --- title: "Segment Trees" summary: "Segment Trees: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-data-structures order: 4 tags: hpc draft: false --- Bottom-up, iterative, Fenwick when you only need prefix. The tree is an array. --- title: "Data Structures Case Studies" summary: "Data Structures Case Studies: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 12 tags: hpc draft: false --- Search, trees, hashes, filters. Layout beats pointer chasing; the case studies are there to prove it. --- title: "Actor Model" summary: "Actor Model: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-distributed order: 4 tags: hpc draft: false --- Messages, not shared objects. The mailbox is the queueing theory. --- title: "Distributed Algorithms" summary: "Distributed Algorithms: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-distributed order: 2 tags: hpc draft: false --- Communication is the complexity. Count bytes and rounds before you count flops. --- title: "Cloud Computing" summary: "Cloud Computing: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-distributed order: 5 tags: hpc draft: false --- Someone else's cluster. The API is a scheduler; the bill is the profiler. --- title: "MapReduce" summary: "MapReduce: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-distributed order: 3 tags: hpc draft: false --- Shuffle, reduce, and the filesystem in between. The algorithm is the data movement. --- title: "Message Passing Interface" summary: "Message Passing Interface: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-distributed order: 1 tags: hpc draft: false --- Ranks, communicators, collectives. The cluster's calling convention. --- title: "Distributed Computing" summary: "Distributed Computing: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 200 tags: hpc draft: false --- MPI, actors, MapReduce, the cloud. Another nesting level, for notes that do not live on one node. --- title: "Memory Hierarchy" summary: "Memory Hierarchy: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 1 tags: hpc draft: false --- Registers, caches, DRAM, disk. Each step is a different algorithm. --- title: "List Ranking" summary: "List Ranking: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 5 tags: hpc draft: false --- Pointer chasing off-chip. Batch the jumps or lose. --- title: "Spatial and Temporal Locality" summary: "Spatial and Temporal Locality: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 8 tags: hpc draft: false --- Touch a line, finish with it. Then come back, not in a random walk. --- title: "Memory Management" summary: "Memory Management: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 99 tags: hpc draft: false --- Allocators in the hot path are a policy. Pool, or do not allocate. --- title: "External Memory Model" summary: "External Memory Model: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 3 tags: hpc draft: false --- I/Os, not FLOPs. Sort and scan for the disk you actually have. --- title: "Cache-Oblivious Algorithms" summary: "Cache-Oblivious Algorithms: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 7 tags: hpc draft: false --- The layout that works without knowing the cache size. Recursive blocking. --- title: "Eviction Policies" summary: "Eviction Policies: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 6 tags: hpc draft: false --- LRU is a story. Measure the miss curve on your working set. --- title: "External Sorting" summary: "External Sorting: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 4 tags: hpc draft: false --- Runs on disk, merge in RAM. The fan-in is the parameter. --- title: "Sublinear Algorithms" summary: "Sublinear Algorithms: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 10 tags: hpc draft: false --- Read less than the input. Streaming and sampling when the data will not fit. --- title: "Virtual Memory" summary: "Virtual Memory: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-external-memory order: 2 tags: hpc draft: false --- Pages, TLBs, and huge pages. TLB misses look like RAM slowness. --- title: "External Memory" summary: "External Memory: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 8 tags: hpc draft: false --- The disk and the page table are part of the algorithm. If it does not fit in cache, the model changes. --- title: "Cryptography" summary: "Cryptography: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 7 tags: hpc draft: false --- Constant time or it is a leak. HPC codes that hash still have this duty. --- title: "Error Correction" summary: "Error Correction: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 6 tags: hpc draft: false --- Detect, correct, and do not pretend ECC is free on the data path. --- title: "Extended Euclidean Algorithm" summary: "Extended Euclidean Algorithm: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 3 tags: hpc draft: false --- GCD and the inverse in one pass. Watch the signs. --- title: "Binary Exponentiation" summary: "Binary Exponentiation: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 2 tags: hpc draft: false --- Square and multiply. The addition chain is the schedule. --- title: "Finite Fields" summary: "Finite Fields: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 5 tags: hpc draft: false --- Prime fields and the reductions that belong in the inner loop. --- title: "Hashing" summary: "Hashing: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 6 tags: hpc draft: false --- Avalanche, multiply, and the table that thrashes if the mix is weak. --- title: "Modular Arithmetic" summary: "Modular Arithmetic: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 1 tags: hpc draft: false --- Reduce often, or once, with a wide intermediate. Overflow is the bug. --- title: "Montgomery Multiplication" summary: "Montgomery Multiplication: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 4 tags: hpc draft: false --- Stay in Montgomery form until the end. The conversion is the tax. --- title: "Random Number Generation" summary: "Random Number Generation: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-number-theory order: 7 tags: hpc draft: false --- A generator with a known period. Do not call /dev/urandom in the kernel. --- title: "Number Theory" summary: "Number Theory: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 7 tags: hpc draft: false --- Modular inverse, Montgomery, hashing, RNG: the integer algorithms that show up in HPC codes whether you asked for them or not. --- title: "OpenMP" summary: "OpenMP: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel-algorithms order: 0 tags: hpc draft: false --- pragmas, reductions, and the schedule you actually wanted. Check the assembly of the loop. --- title: "Parallel Algorithms" summary: "Parallel Algorithms: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel order: 4 tags: hpc draft: false --- Shared-memory loops and reductions. OpenMP is the portable on-ramp, not the whole story. --- title: "Event-driven Concurrency" summary: "Event-driven Concurrency: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel-concurrency order: 4 tags: hpc draft: false --- One thread, many callbacks. The state machine is the program. --- title: "Fibers" summary: "Fibers: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel-concurrency order: 3 tags: hpc draft: false --- User-scheduled stacks. Cheap context, still a stack, still a data race if you share. --- title: "Processes" summary: "Processes: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel-concurrency order: 1 tags: hpc draft: false --- Address spaces. Isolation is the feature; the TLB and the copy are the cost. --- title: "Threads" summary: "Threads: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel-concurrency order: 2 tags: hpc draft: false --- Shared memory, shared bugs. Pin them; name the sharing. --- title: "Concurrency" summary: "Concurrency: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel order: 1 tags: hpc draft: false --- Processes, threads, fibers, events. Same cores, different ways to lose your place in the memory model. --- title: "CUDA" summary: "CUDA: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel-gpu order: 0 tags: hpc draft: false --- Blocks, warps, occupancy. A kernel that saturates memory is done; more math will not help. --- title: "GPU Programming" summary: "GPU Programming: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel order: 5 tags: hpc draft: false --- A kernel is a grid. Occupancy and the memory path first, then the math. --- title: "Threading Runtimes" summary: "Threading Runtimes: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel order: 6 tags: hpc draft: false --- Work stealing, pools, and the oversubscription that kills the cache. --- title: "Mutual Exclusion" summary: "Mutual Exclusion: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel-synchronization order: 1 tags: hpc draft: false --- A lock is a memory order. Hold it for the line, not for the function. --- title: "Synchronization Primitives" summary: "Synchronization Primitives: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-parallel order: 2 tags: hpc draft: false --- Mutual exclusion and friends. If you can avoid sharing, do; if you cannot, name the primitive. --- title: "Parallel Computing" summary: "Parallel Computing: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 100 tags: hpc draft: false --- Threads, fibers, locks, then a GPU. Nested under this section on purpose, to check sub-notes and sub-sub-notes. --- title: "The Cost of Branching" summary: "The Cost of Branching: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-pipelining order: 2 tags: hpc draft: false --- A branch is cheap until it is not. Profile the mispredicts, then decide on predication. --- title: "Branchless Programming" summary: "Branchless Programming: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-pipelining order: 3 tags: hpc draft: false --- CMOV, masks, and arithmetic selects. Use them where the predictor loses, not everywhere. --- title: "Pipeline Hazards" summary: "Pipeline Hazards: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-pipelining order: 1 tags: hpc draft: false --- Data, control, structural. Name the stall before you rewrite the loop. --- title: "Theoretical Performance Limits" summary: "Theoretical Performance Limits: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-pipelining order: 5 tags: hpc draft: false --- Roofline, peak FLOP, and the bound you will not beat. Stop when you are on the roof. --- title: "Instruction Scheduling" summary: "Instruction Scheduling: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-pipelining order: 4 tags: hpc draft: false --- The compiler reorders. So does the core. Write for dependencies, not for source order. --- title: "Instruction Tables" summary: "Instruction Tables: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-pipelining order: 3 tags: hpc draft: false --- Latency, throughput, ports. The PDF is the spec; your core is the measurement. --- title: "Throughput Computing" summary: "Throughput Computing: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-pipelining order: 4 tags: hpc draft: false --- Keep the ports busy. Independent work is the real parallelism on one thread. --- title: "Instruction-Level Parallelism" summary: "Instruction-Level Parallelism: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 3 tags: hpc draft: false --- The core is a pipeline, not a calculator. Hazards, branches, and issue width decide the cycle count you thought was arithmetic. --- title: "Preface" summary: "Preface: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: -1 tags: hpc draft: false --- Why these notes exist, who they are for, and what they refuse to be: a rewrite of an algorithms textbook with a faster Big-O. --- title: "Benchmarking" summary: "Benchmarking: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-profiling order: 6 tags: hpc draft: false --- A timer around the kernel, pinned, warmed, and repeated. Anything else is a story. --- title: "Statistical Profiling" summary: "Statistical Profiling: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-profiling order: 2 tags: hpc draft: false --- Sample the stacks. Cheap enough to leave on; noisy enough that you need a long run. --- title: "Instrumentation" summary: "Instrumentation: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-profiling order: 1 tags: hpc draft: false --- Counters in the code. Heavy, exact, and useless if you leave them in production by accident. --- title: "Machine Code Analyzers" summary: "Machine Code Analyzers: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-profiling order: 4 tags: hpc draft: false --- uops, ports, frontend. Static analysis of the loop you already isolated. --- title: "Getting Accurate Results" summary: "Getting Accurate Results: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-profiling order: 10 tags: hpc draft: false --- Noise, turbo, and the other socket. Pin, disable boost, and report the spread. --- title: "Program Simulation" summary: "Program Simulation: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-profiling order: 3 tags: hpc draft: false --- When the counters lie, simulate. Cachegrind is slow and honest. --- title: "Profiling" summary: "Profiling: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 5 tags: hpc draft: false --- Measure first. Instrumentation, sampling, and a benchmark that survives noise are the only honest start. --- title: "Auto-Vectorization and SPMD" summary: "Auto-Vectorization and SPMD: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-simd order: 10 tags: hpc draft: false --- The compiler will vectorize the obvious. The rest is a rewrite or an intrinsic. --- title: "Intrinsics and Vector Types" summary: "Intrinsics and Vector Types: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-simd order: 1 tags: hpc draft: false --- The types the compiler will not invent. Use them after the scalar kernel is right. --- title: "Masking and Blending" summary: "Masking and Blending: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-simd order: 4 tags: hpc draft: false --- Lanes that should not run still cost a line. Masks beat branches in the vector loop. --- title: "Moving Data" summary: "Moving Data: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-simd order: 2 tags: hpc draft: false --- Load, store, broadcast. The shuffle is often the real kernel. --- title: "Reductions" summary: "Reductions: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-simd order: 3 tags: hpc draft: false --- Tree, pairwise, and the float order you signed up for. --- title: "In-Register Shuffles" summary: "In-Register Shuffles: a page of the HPC note." date: 2026-09-11 author: hpcnotes kind: note parent: hpc-simd order: 6 tags: hpc draft: false --- Permute, then compute. Crossing lanes is the tax on a bad layout. --- title: "SIMD Parallelism" summary: "SIMD Parallelism: a section of HPC, with nested pages under it." date: 2026-09-11 author: hpcnotes kind: note parent: hpc order: 10 tags: hpc draft: false --- Vectors, masks, shuffles, and the day auto-vectorization gives up. Intrinsics after you know the layout. --- title: HPC summary: A nested performance-engineering note: sections, sub-notes, and sub-sub-notes, walked in order. date: 2026-09-11 author: hpcnotes kind: note order: 0 tags: hpc draft: false --- Performance engineering from the model down to the metal, then out to the node and the cluster. The tree below is the whole note: a section, then sub-notes, then sub-sub-notes. Open a chapter to work inside it. Previous and Next walk the document in order. For the shared-machine habits these pages assume, start with [Working on a shared cluster](/notes/working-on-a-cluster). --- title: Working on a shared cluster summary: Login, modules, a job that finishes, then a first GPU kernel. One note, four pages. date: 2026-09-08 author: hpcnotes kind: note tags: batch, slurm --- A cluster is a waiting room, a contract for software, a scheduler, and only then an accelerator. These notes stay in that order. The [login image](/notes/login-nodes-are-not-compute) is not a workstation. The [modulefile](/notes/modules-are-a-contract) is the stack you actually have. A job that finishes is mostly walltime, cwd, and the launcher. A GPU run starts with a device you can see, not the full application. The four-rank cartoon is the shape of an `srun`, not a screenshot from a centre. ![Four ranks on a mesh, each with a neighbour exchange](/media/ranks.svg) Slurm's [sbatch](https://slurm.schedmd.com/sbatch.html) is the usual door. The first page is [login-node etiquette](/notes/login-nodes-are-not-compute). --- title: What a note can carry summary: Code, diagrams, maths, figures, gifs. Write the note as a Markdown file; the site renders the rest. date: 2026-09-02 author: hpcnotes tags: documentation --- A note is a `.md` file with a short header. The body is ordinary Markdown: headings, lists, tables, quotes, code, diagrams, maths, images, and gifs. Nest them as in LaTeX: section, subsection, subsubsection. > Walltime is a contract. The login node is not a workstation. ## Code Fence a block and name the language. Copy sits on the block. Long lines scroll inside the fence; they do not push the column. ### Fences ```bash #!/bin/bash #SBATCH --job-name=ping #SBATCH --ntasks=4 #SBATCH --time=00:10:00 srun ./a.out ``` ### Inline names Inline names stay in a code span: `srun`, `MPI_Init`, `$SLURM_NTASKS`. ```cpp #include int main(int argc, char** argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } ``` #### A rank that only starts That `MPI_Init` / `MPI_Finalize` pair is a subsubsection: enough to compile, not a solver. ## Diagrams A `mermaid` fence becomes a figure. Flowcharts, sequence charts, and the rest of the Mermaid dialects are fair game. ### Flow ```mermaid flowchart LR login[Login] --> sbatch[sbatch] sbatch --> alloc[Allocation] alloc --> srun[srun] srun --> ranks[Ranks] ``` ### Sequence ```mermaid sequenceDiagram participant User participant Slurm participant Rank User->>Slurm: sbatch run.sh Slurm->>Rank: srun ./a.out Rank-->>User: stdout + sacct ``` ## Maths ### Inline Inline TeX sits in dollar signs: a naive GEMM is $2n^3$ flops. ### Display Displayed expressions use a `math` fence, `$$…$$`, or `\[…\]`. ```math E(p) = \frac{T_1}{p \, T_p} ``` $$ T_p = T_{\text{comp}} + T_{\text{comm}} + T_{\text{I/O}} $$ ##### Paragraph Strong scaling is the ratio of serial time to $p$ times parallel time. If $E(p)$ falls over while the solver is "MPI bound", time the I/O. ## Figures and gifs A Markdown image is a figure. Put files in `public/media/` and link them with `/media/…`. An `https://` URL to a `.png`, `.jpg`, `.gif`, `.webp`, or `.avif` is also a figure. Still rasters are compressed on the fly. Click a still for a lightbox. GIFs animate in place and in the lightbox. A `.mp4` or `.webm` is a video. ![Four ranks on a mesh, each with a neighbour exchange](/media/ranks.svg) ![Four ranks on a mesh, as a compressed raster](/media/ranks.png) ![A pulse you should not run on the login node](/media/heartbeat.gif) ## Links A Markdown link is a link. Internal pages use a path (`/notes/…`). External pages use `https://…` and open in a new tab. Bare URLs also become links: https://slurm.schedmd.com/sbatch.html. ## Tables | Probe | What it answers | | --- | --- | | `sacct -j` | TIMEOUT, OOM, or a traceback | | `nvidia-smi` | Whether the allocation saw a GPU | | A timer around VTK | Whether the "MPI problem" is output | None of this replaces the centre handbook. It is just what a short note is allowed to hold. For the habit behind the gif, start with [Login nodes are not compute](/notes/login-nodes-are-not-compute). --- title: Why site-independent HPC notes still matter summary: Every centre has a hostname. None of them should have to re-explain MPI ranks from scratch. date: 2026-08-24 author: hpcnotes tags: documentation, portability --- Campus clusters, national machines, and cloud partitions disagree about almost every proper noun: the login host, the GPU partition, the scratch purge. They agree about almost every concept: login nodes are not compute, collectives have to match, and walltime is a contract. That split is the brief for these notes — dated, site-independent, and meant to sit beside a centre's own handbook. ## What belongs in a centre's docs - Hostnames, jump hosts, and MFA - Partition names, QoS, and fairshare - Module names and the blessed compiler/MPI pair - Scratch retention and how to call the helpdesk If a sentence contains a string a user must type, it is probably local. ## What belongs in a portable note - Why you should not run MPI on the login node - How to size walltime from a short representative run - What a data race looks like in OpenMP - How to read a Lustre stripe without becoming a storage admin If you maintain a centre handbook, the highest-leverage change is often a dozen links to notes like these, not a new CMS. --- title: Anatomy of a Slurm job that actually finishes summary: The difference between a script that queued and a result you can publish is usually walltime, cwd, and the launcher — not the solver. date: 2026-08-11 author: hpcnotes tags: slurm, batch parent: working-on-a-cluster order: 3 --- A surprising number of "the cluster is broken" tickets are job scripts that never had a chance: wrong directory, stdout path that does not exist, `mpirun` that ignored the allocation, walltime copied from a colleague's 48-hour production run into a compile test. ## The five lines that matter 1. `#SBATCH --output=` pointing at a directory that **exists**. 2. An explicit `cd` to scratch. 3. `module purge` then a known-good stack. 4. `srun` (or the site launcher), not a naked `mpirun`. 5. A walltime you measured, plus a margin, not a round number from folklore. ## cwd is not the submission directory you remember If you `sbatch ../scripts/run.slurm` from a random folder, relative paths inside the script follow Slurm's policy, not your intuition. Either `cd "$SLURM_SUBMIT_DIR"` **or** `cd` to an absolute scratch path. Pick one and stop mixing. ## Launchers `mpirun -np 32 ./a.out` inside an `sbatch` allocation *sometimes* works. On a multi-rail InfiniBand machine with cgroups, it sometimes spreads ranks onto cores you do not own, or onto the login image. `srun ./a.out` asks the daemon that owns the job to start the ranks. That is the difference between a lucky workstation habit and a cluster program. The flags themselves are in the [sbatch](https://slurm.schedmd.com/sbatch.html) man page. ## Arrays instead of a shell for-loop `for i in $(seq 0 999); do sbatch run.sh $i; done` makes the scheduler, the filesystem, and the support team sad. Job arrays exist. Use `%20` if you must cap concurrency. ## After it dies `sacct -j JOBID` before you change the code. TIMEOUT, OOM, and a Python traceback in the log are different jobs. Fix the matching resource, not all of them at once. --- title: Measure before you optimise summary: A CFD mini-app that was 'MPI bound' until we turned the VTK dumps off. Performance engineering as a habit, not a rewrite. date: 2026-07-28 author: hpcnotes tags: performance, io, mpi --- The first profile on a production mesh is allowed to be embarrassing. The unforgivable part is rewriting the flux kernel before that profile exists. ## A short true-to-life sequence We had a finite-volume code that "did not scale past 16 nodes". The MPI traces looked busy. Halo exchanges were the obvious villain. We timed three runs on the same 8 nodes: | Variant | Walltime | | --- | --- | | Production, full VTK every step | 41 min | | VTK every 50 steps | 18 min | | No VTK, same compute | 11 min | The halo exchanges were fine. The parallel filesystem was taking a metadata beating from per-rank VTK. The "MPI problem" was I/O. ## What we changed Not the Riemann solver. A single writer rank, larger intervals, and a binary format. Scaling to 64 nodes started to look like a solver again, at which point a real MPI issue (an unnecessary `Allreduce` in a residual check) was visible. ## Rules we kept - One change per experiment - The same mesh and the same node type - A timer around **output**, **exchange**, and **update**, not only the whole executable - No optimisation sprint during a conference deadline --- title: From login node to first GPU kernel summary: A checklist for the first CUDA or HIP allocation: visibility, a tiny kernel, then the application module — in that order. date: 2026-06-16 author: hpcnotes tags: gpu, slurm parent: working-on-a-cluster order: 4 --- People skip from "I have a GPU partition" to "I built the full application" and then spend a day debugging a solver that never saw a device. ## Get an allocation that includes a GPU ```bash salloc -N 1 --gpus=1 -t 00:20:00 -p gpu-debug nvidia-smi # or rocm-smi echo $CUDA_VISIBLE_DEVICES ``` If this fails, **stop**. Compiling will not fix the scheduler. ## Run a kernel that is not your code A 20-line vector add, built with the site CUDA/HIP module, launched with `srun`. Confirm the device name matches the partition you requested. ## Then load the application Now `module load gromacs-gpu` (or whatever you actually wanted). Check that the binary is the GPU build. Run one timestep. Watch `nvidia-smi` in a second SSH **inside the allocation** if the site allows `srun --overlap`, or dump utilisation at the end of the job. ## Only then scale Multi-GPU and multi-node GPU-aware MPI are where site stacks differ. They are not the first experiment. --- title: Login nodes are not compute summary: If the shared login image is swapping, it is because someone compiled the universe there. Don't. date: 2025-11-04 author: hpcnotes tags: etiquette, batch parent: working-on-a-cluster order: 1 --- A login node is a waiting room with a compiler. It is not a 128-core workstation that happens to have SSH. ## What belongs on login - Editing job scripts - `sbatch`, `squeue`, `module spider` - A tiny configure test, if the site asks for it ## What does not - `make -j` on a full application - Python that reads the whole mesh into RAM - `mpirun` "just to see" ## Move the work `salloc` for an interactive compile. A short serial job for a configure. Scratch, not `$HOME`, for anything that will be written again in ten minutes. If you need a rule you can shout across the office: if it would make your laptop fan spin, it does not belong on login. The gif of that mistake lives in [What a note can carry](/notes/what-a-note-can-carry). --- title: Environment modules are a contract summary: The modulefile is how the machine tells you which compiler, MPI, and CUDA belong together. Mixing stacks by hand is how you get a weekend of undefined references. date: 2024-09-18 author: hpcnotes tags: modules, portability parent: working-on-a-cluster order: 2 --- A cluster is not a laptop with extra cores. The software you can see is a set of blessed combinations, and the modulefile is the contract for one of them. ## Load a stack, not a souvenir `module load gcc openmpi cuda` from three different generations is how you get a binary that links and a job that dies in `MPI_Init`. Load the site's metapackage, or a versioned toolchain the handbook names. Then stop. ## `module purge` is not optional Login profiles accumulate. Yesterday's PETSc is still in `LD_LIBRARY_PATH` when today's GROMACS starts. Purge, then load, in the job script, every time. ## Record what you loaded `module list` in the job log is cheaper than reconstructing a toolchain from memory six months later, after the default GCC has moved. If the site offers `spack` or EasyBuild generated modules, pin the hash or the version, not "the default". ## What not to do Do not `export PATH=$HOME/local/bin:$PATH` and then file a ticket that "MPI is broken". Your overlay is now the software stack. The helpdesk cannot see it.