nanoGPT C# Port

nanoGPT C# Port is a spike that answered a single question: is TorchSharp a viable path to GPU-accelerated GPT training and inference in C#, without dropping to Python? It ports Andrej Karpathy's nanoGPT — the full architecture, tokenizer, causal self-attention, transformer blocks, weight tying, GPT-2-paper initialization — and the port itself came together with far less friction than I expected: TorchSharp's API mapped onto the Python/PyTorch reference directly enough that two of nanoGPT's Python-specific workarounds turned out to have no equivalent need at all.

Partway through, the spike surfaced two real defects in how .NET's garbage collector interacts with native GPU tensors. The first left roughly 128 undisposed intermediate tensors per training-loop call, and under sustained churn the GC finalizer thread's native disposal raced with the main thread's active libtorch calls and reliably segfaulted the process — confirmed in dmesg as a SIGSEGV inside libLibTorchSharp.so. The second, found only after fixing the first, was subtler: the model's own forward-pass intermediates were never disposed either, and the GC has no visibility into GPU memory pressure, so it collected far too late and reliably exhausted GPU memory within a few hundred iterations.

Both were fixed the same way — stop relying on the GC and dispose explicitly, using TorchSharp's NewDisposeScope() / MoveToOuterDisposeScope() mechanism so every tensor created inside a forward pass is cleaned up at scope exit except the ones the caller actually needs.

With both fixed, a real training run validated the whole port end to end: a 10.67M-parameter model trained on tiny Shakespeare, GPU-accelerated on an RTX 5090 in WSL2, with loss dropping from 3.88 to 1.63 in under two minutes — roughly 56x faster than the same run on CPU. The result is parked as a validated, reusable starting point for any future .NET-hosted GPU training work, documented in ADR-052, dispose-scope discipline included.

flowchart TD
    Port["Port nanoGPT\n(tokenizer, attention,\ntransformer blocks)"] --> Train["Train on GPU\n(RTX 5090, WSL2)"]
    Train --> Loss["Loss 3.88 -> 1.63"]
    Train --> Defects["Two GC/native-tensor\ndefects surfaced"]
    Defects --> Fix["Fixed via explicit\ndispose-scoping"]
    Fix --> Parked["Parked as reusable\nstarting point (ADR-052)"]