Porting nanoGPT to C#: What Two Segfaults Taught Me About TorchSharp

A parking-lot idea I flagged high-priority for myself back on July 3rd read, in its entirety: "C# port of Karpathy's microgpt." Not much of a spec. No target .NET version, no decision on where it would live, no decision on CPU versus GPU, not even a confirmed source repo — I'd been thinking of Andrej Karpathy's minimal-GPT lineage generically, and it turned out he doesn't actually have a repo literally named "microgpt." The spike used karpathy/nanoGPT, MIT-licensed, as the real reference, since it ships the exact character-level toy training configuration (config/train_shakespeare_char.py) I wanted to validate against. I'll come back to that naming mismatch at the end, because today, over two weeks later, I finally fixed it.

What I want to walk through here isn't "I ported GPT to C#," which on its own is a fairly shallow claim. It's what porting a from-scratch GPU training loop into a runtime it wasn't designed for actually costs you — told through the two real defects that spike surfaced, both of which would have bitten anyone doing serious TorchSharp work in this monorepo eventually. I'd rather you hit them here, in a blog post, than in dmesg at 11pm.

Why This Was Worth a Spike

I scoped this as a four-phase spike (spike/microgpt-csharp-port, now folded into project documentation)) with three genuinely open questions:

1. Does GPU-accelerated tensor computation work at all from C#, in this monorepo's actual dev environment — WSL2, with an RTX 5090 behind it?
2. Can nanoGPT's architecture and training loop be faithfully ported to TorchSharp, or does the API surface diverge enough from PyTorch that you end up fighting the library more than learning from the port?
3. Is the result a genuinely reusable library, or an educational curiosity I'd park and forget?

I decided upfront this would be a real class library — NanoGPT.Core plus a NanoGPT.Cli harness and a test project — not a single-purpose console app like the handful of other .NET tools already in this repo (tools/md-to-docx, tools/md-to-pptx, tools/infographics/DrawioExporter). Those are all fine as one-shot CLIs. This was the first .NET project in this monorepo built to be a library someone might actually import later, which meant I couldn't cut corners on the parts that only matter for reuse — proper separation between the model code and the CLI harness, tests that check tensor shapes rather than just "it ran."

The Port Itself Went Better Than I Expected

I went in assuming TorchSharp's API would diverge enough from PyTorch's that the port would be mostly translation work with a few ugly workarounds. That's not what happened. The full nanoGPT architecture — the tokenizer, causal self-attention built on scaled_dot_product_attention, transformer blocks, weight tying between the token embedding and the output head, GPT-2-paper scaled residual initialization — mapped onto TorchSharp directly enough that two of nanoGPT's own Python-specific workarounds turned out to have no equivalent need at all: a manual-attention fallback nanoGPT keeps around for pre-2.0 PyTorch, and a custom LayerNorm implementation nanoGPT wrote just to support bias=False. TorchSharp's built-ins already cover both, so the C# version of those two pieces is genuinely simpler than the Python original, not just differently spelled.

That's a real result worth sitting with for a second: when I set out on this, "faithful port with minimal friction" was the optimistic outcome, not the expected one. It held up.

GPU execution was the bigger unknown going in, and it also came out clean: TorchSharp plus CUDA runs without drama on this environment's RTX 5090 (Blackwell, sm_120), including through WSL2's GPU passthrough — which is exactly the kind of driver/passthrough chain I've had go sideways on me before for reasons that had nothing to do with the actual workload.

The Part That Matters More: Two Real Defects

Here's where the spike stopped being a straightforward "yes, this works" story and became one worth writing down in detail. Phase 3's smaller-scale smoke tests hadn't caught either of these — they only showed up once I ran a real, full-corpus training loop for hundreds of consecutive iterations. That's the honest lesson underneath this whole post: a shape-correctness unit test proves your tensors are the right size. It proves nothing about whether the process survives running for real.

Defect 1: undisposed intermediate tensors, and a segfault with no error message

The first full training attempt on the real tiny Shakespeare corpus died silently after printing exactly one evaluation line — no exception, no stack trace, no checkpoint saved. dmesg told the real story: a SIGSEGV inside libLibTorchSharp.so, on a .NET TP Worker thread, not the main thread.

The cause was in Trainer.GetBatch. The original implementation built a training batch by constructing one native Tensor per row — 64 input rows plus 64 target rows, 128 native tensors per call — via individual tensor(...) calls, then stack-ing them into the final batch tensor. Every one of those 128 per-row intermediates went undisposed, left for the .NET garbage collector to clean up eventually. Under sustained training-loop churn, that "eventually" turned into a race: the GC's finalizer thread would reclaim a batch of those intermediates at the same moment the main thread was mid-flight on an active libtorch call, and the finalizer thread's native disposal corrupted state the main thread was actively using. That's the segfault.

There's a retroactive correction buried in this one, too. Phase 3's own summary had attributed some intermittent multi-second stalls, observed while benchmarking, to "GPU contention from outside the WSL2 guest's visibility." That explanation was wrong. Those stalls were almost certainly the same disposal race quietly degrading performance before eventually crashing outright — not an external GPU contention issue at all. I left Phase 3's summary as-is rather than rewrite history, and corrected the record in Phase 4's summary and in the ADR instead. I'd rather show you where I got something wrong and fixed the record than quietly edit an earlier doc to look right in hindsight.

The fix was straightforward once the cause was clear: build each batch as two flat managed long[] arrays, then construct exactly two native tensors per call — one for inputs, one for targets — instead of 128. That's not just correct, it's faster too: no per-row native allocation, no stack call. While I was auditing for the same shape of bug elsewhere, I found — and fixed — an identical pattern in GptModel's autoregressive generation loop, which was reassigning its running sequence tensor each iteration without disposing the previous value.

Defect 2: the garbage collector can't bound GPU memory, full stop

With defect 1 fixed, the retry got further — and then failed with a clean, catchable CUDA error: out of memory inside the loss-estimation step. Progress, in the sense that this was an exception instead of a crash. But it was the same underlying problem in a different location: GptModel.forward's internal intermediate activations — token and positional embeddings, each transformer block's output as it got reassigned into x, the final layer-norm output — were never disposed either, again left to the GC.

The structural issue here is worth naming plainly, because it isn't specific to this port: the .NET GC schedules collections based on managed heap pressure. A TorchSharp Tensor is a small managed wrapper object sitting on top of potentially enormous unmanaged GPU memory. The GC has zero visibility into that unmanaged pressure, so it collects far too late relative to how fast GPU memory actually fills up. A few hundred training iterations was enough to exhaust it.

TorchSharp has a purpose-built answer for exactly this, and it's the one real API concept anyone doing training-loop work in this library needs to internalize: torch.NewDisposeScope() around a forward pass, with Tensor.MoveToOuterDisposeScope() called on the handful of tensors the caller actually needs to keep — in this case, the returned logits and loss. Everything else created inside that scope gets disposed automatically the moment the scope exits. From GptModel.cs:

using var scope = NewDisposeScope();
// ... embeddings, transformer blocks, final layer norm all happen here ...
return (logits.MoveToOuterDisposeScope(), loss.MoveToOuterDisposeScope());

The one thing I wanted to confirm before trusting this pattern: does it break backward()? It doesn't. PyTorch's (and TorchSharp's) autograd engine holds its own references to any tensor still needed for the backward pass through its saved-tensors mechanism, entirely independent of the C# wrapper's lifetime. That's actually exactly how normal Python PyTorch training loops already work — local variables going out of scope after forward() returns doesn't break backward() there either. TorchSharp's dispose-scope mechanism is doing explicitly, for a language with deterministic disposal semantics, what Python's own scoping does implicitly.

I made a deliberate call not to force a strict test-first discipline on either of these two fixes. Both are concurrency and native-memory races that only manifest under sustained real-workload churn — hundreds of training iterations — not the kind of thing you can turn into a small, deterministic, fast-running unit test without essentially faking the failure mode. I fixed both directly, with clear before/after evidence instead: a crash log, then an OOM log, then a clean 500-step run with neither. That's a documented exception to a standing discipline, not a shortcut I'm glossing over.

What the Validated Run Actually Showed

With both defects fixed, a full 500-step training run on the real tiny Shakespeare corpus — a 10.67M-parameter model, matching nanoGPT's own shakespeare_char configuration — completed cleanly in 99 seconds:

No further crashes, no memory errors, a clean monotonic loss decrease. Generation from the resulting checkpoint (prompt "First Citizen:", 200 tokens, temperature 0.8, top-k 40) isn't fluent — this is 500 of a typical 5,000-iteration nanoGPT run — but it's clearly recognizable as Shakespeare-shaped text: character names in caps followed by a colon, plausible word and punctuation structure. That's the real confirmation I was after — not that the numbers looked right in isolation, but that the tokenizer, model, training loop, and sampling path are all correctly wired together end to end, not just individually shape-correct.

I also ran a direct GPU-versus-CPU benchmark on identical hardware, 20 steps each: 199.7ms per step on GPU versus 11,160.4ms per step on CPU — GPU is roughly 56x faster for this model. That's the number that answers the original open question about whether GPU acceleration from .NET was even worth the setup complexity. It was.

Where I Landed

I recorded all of this in the project documentation with a recommendation to park the spike as complete rather than keep investing — the architecture and training loop are proven correct and GPU-accelerated, and if a future need arises for local, GPU-accelerated model training or inference from .NET in this monorepo, this port is a validated starting point rather than a from-scratch effort. I want to be honest about what "park" means here, though, not let it sound more finished than it is: optimizer state isn't checkpointed, so resuming an interrupted training run isn't supported, and there's no mixed precision, torch.compile, or distributed training support. This targets single-GPU, correctness-first training — not throughput at scale. Fine for a spike. Not something I'd hand someone as a production training pipeline today.

The one thing I'd tell anyone else doing TorchSharp work, in this monorepo or otherwise: use NewDisposeScope()/MoveToOuterDisposeScope() around every forward pass by default. Don't wait to discover you need it through a segfault in dmesg.

And the small coda I promised at the top: the project lived at tools/microgpt/ and MicroGpt.* for the two-plus weeks since the spike closed, even though Karpathy never had a project literally called "microgpt" — the name was slightly wrong from the day I chose it. The blog's own project card for this work has said "nanoGPT C# Port" since it was published, which meant the code and its own public description had quietly disagreed with each other the whole time. Today I renamed the tooling — tools/microgpt/ to tools/nano-gpt/, MicroGpt.* to NanoGPT.* — to catch the code up to what I'd already told you it was called. Small fix, but the kind of drift I'd rather close the same day I notice it than leave sitting.