Parallel source code is the problem of keeping cause, timing, ownership, locality, and failure readable when execution spreads across workers, devices, regions, or machines.

The issue is not only how to run more operations. The issue is how a builder can still understand, change, and support the program after one sequential execution path is no longer enough.

Problem noteOriginally posted 2022-05-12; expanded here into the actual parallel-source-code question.

Included: causality, timing, locality, ownership, failure, the support burden of long-lived code, and the link to multi-device scaling.

Article focus: the source-code problem behind parallel execution: locality, causality, timing, ownership, failure, and scale.

The Question

Hardware can provide more lanes: cores, GPUs, FPGAs, accelerators, workers, services, regions, and machines. That does not by itself give the programmer a coherent program.

The hard question is source-level structure. Can one program describe parallel work while preserving the reasons a future maintainer needs: what caused what, when data is valid, which place owns a value, where failure can happen, and what has to remain local?

What Must Stay Readable

Causality

Which result depends on which inputs, messages, clocks, memory states, or completed tasks.

Timing

When a value is ready, when it expires, and whether a later stage can safely observe it.

Locality

Which work belongs near which memory, device, region, data source, or physical boundary.

Ownership

Which worker, device, actor, process, or region is allowed to mutate a piece of state.

Failure

What happens when one lane stalls, one worker retries, one device disappears, or one message arrives late.

Why It Matters

Single-path source code remains popular because it is supportable. A person can read it, step through it, patch it, and explain the order of events.

Parallel systems often lose that property. The source becomes a collection of callbacks, kernels, queues, jobs, actors, services, promises, locks, message handlers, and deployment assumptions. The runtime may be fast while the program becomes harder to reason about.

That is the problem this note is pointing at. More computation should not automatically mean less control over the program's causal shape.

Scale Boundary

As single execution paths stop carrying the whole workload, parallel computation becomes a central software problem rather than a niche accelerator topic.

The goal is not to claim a language or runtime solves the limits of hardware scaling. The goal is to name the programming burden created by those limits: large-scale computation still needs source code that a human team can understand and support.

Original Post Context

The original post asked whether general-purpose parallel compute can have one orchestrated source form that remains easy to reason about.

That is still the useful question: not just how to launch parallel work, but how to preserve the structure that lets future programmers maintain it.

Original Link

The post linked an NVIDIA GTC session as the immediate context for the question: GTC Spring 2021 session S31880.