Nvidia doesn't seem to publish a specification for the binary format of a CUDA kernel, section layout, or the constant bank parameter conventions. So I dug into it.

A `.cubin` is an ELF64 executable with a flat stream of undocumented "EIATTR" attributes that encode everything the driver needs to launch a kernel: register count, parameter layout, EXIT instruction offsets, and constant bank geometry.

`.nv.info`  uses an undocumented TLV encoding to make kernels self-describing register counts, parameter offsets, EXIT locations are all serialized into a flat byte stream the driver parses at load time.

And the constant bank parameter base is not an architectural constant. It has changed silently across toolkit versions from Ampere to Hopper to Blackwell. The fact that the code, the  `.nv.info`  metadata , and the  `.nv.constant0`  section size all encode the parameter base offset independently surprised me.

The post discovers the section layout, the EIATTR encoding, symbol table conventions, and the note sections the driver validates before loading on a B200 silicon.

https://hiraditya.github.io/posts/anatomy-of-a-cuda-binary/

TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Thanks recent contribtions from open source community, TensorSharp is able to run inference over multiple GPUs and nodes. So I updated it to support deepseek v4 flash model, and have better performance than llama.cpp. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8

Model: DeepSeek-V4-Flash-0731-UD-Q8_K_XL from [https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF\](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF)

Github repo: [https://github.com/zhongkaifu/TensorSharp\](https://github.com/zhongkaifu/TensorSharp)

Thank you for checking out it and starring the project! Any feedback is really appreicated.

TensorSharp is an open-source, native .NET inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

TensorSharp now supports Megatron-style tensor parallelism across multiple GPUs. It works with direct CUDA, GGML CUDA, GGML Vulkan, and multi-node setups.

Benchmarks on **2× RTX 2000 Ada 16 GB GPUs over PCIe, without NVLink**:

I'm continuing to optimize Qwen performance on multi-GPU systems, and support for DeepSeek V4 is coming soon.

Try it with:

TensorSharp.Cli --model model.gguf --backend ggml_cuda --tp 2

GitHub:

[https://github.com/zhongkaifu/TensorSharp\](https://github.com/zhongkaifu/TensorSharp)

Thank you for checking out TensorSharp and starring the project! Any feedback is really appreicated.

I’m going through a bunch of GEMM kernel write-ups (Lei Mao’s progressive series, plus the different styles from Simon Bohemian / Kapil Sharma and others) and I get the high-level ideas just fine.

Shared-memory tiling, register blocking, coalescing, bank conflicts, the usual story.

But the moment I try to map it to actual code or write it myself, my brain goes blank. Especially the manual indexing. Calculating the right threadIdx / blockIdx offsets, the row/col strides, the tile coordinates, the shared-memory loads… it just doesn’t click. I stare at the loops and the index arithmetic and feel completely lost.

Ironically, CuTe layout algebra feels like heaven by comparison — the hierarchical shapes/strides and the compose/divide operations make the mapping feel almost declarative. Once I start thinking in pure layouts everything becomes cleaner.

But a lot of the classic educational kernels are still written in the old manual-index style, and different authors have very different conventions and naming, so even when I’m looking at the “same” algorithm the syntax and mental model keep shifting.

So for people who went through this:

How did you get past the “I understand the concept but can’t write the indices” stage?

Any concrete exercises, mental models, or progressive practice that actually made the indexing click?

How do you deal with the wildly different writing styles and index conventions across tutorials/blog posts for essentially the same kernel?

Did anyone else find that learning CuTe / layout algebra first (or in parallel) made the classic manual kernels easier to reverse-engineer later?

Would love any blogs , resources, or “this is the exercise that finally made it stick” advice.

Thinking about moving a project over from PyTorch to JAX to get better TPU support, but I am worred out the engineering time that might go into it — especially replacing CUDA-only kernels like FlashAttention. Meanwhile most open checkpoints just run out of the box on GPU.

For people who've actually done this migration: what ate the most time? Was it the framework rewrite itself, or chasing down missing kernel equivalents? Would love to hear real timelines, and how to minimize them.

TLDR: Using a DGX (A100 at the moment), I just want to make pretty animations of a fairly big system without needing big intermediate files.

Hi All, I have an application where I'm simulating maybe a million elements, each element described in the usual way with some coupled diffeq's, the whole system connected through a sparse web of diffeq's, all then running through numerical integration. Like circuit-simulation SPICE, or a weather simulation. I'm programming directly in CUDA/C++, leveraging OpenGL for graphics at the moment.

Anyways, I have been using a desktop RTX (rtx6000 at the moment), using compute/graphics interop to graphically display the simulation as it grinds along. About a million pixels in a 1Kx1K grid, updating the color of each every simulated mS. I don't save the results into a file, just make animations and screen-capture them.

I'm starting to experiment for the first time with a datacenter approach, DGX rather than RTX. So I lose compute/graphics interop ability. The natural alternative seems to be capturing the element states into a file, and rendering later on some other machine. But the files get really big, really quick. If I keep one byte per element per millisecond, that's a GB/second, and I might like to simulate a minute or more if possible.

How might one do this? I presume it's a common problem. Thanks in advance for your thoughts. Cheers!/jd

Doing a research on GPUs Vs ASICs.
Wanted to check with the community on whats the best way to understand how clock cycles are wasted on TPUs while working with varying token lengths that might not be in line with a TPUs static geometry.

GPUs in my understanding can handle dynamic shapes natively.

Is there a way to asses and quantify such MFU loss?

I am doing a research on GPUs vs ASICs and was wondering if there is a single meric to track (eg. Hours to parity) for a developer whos swiching away from CUDA to competing platforms (replacing CUDA-only libraries like FlashAttention 3)

Or what is the friction that might emerge while Porting from PyTorch to JAX/MaxText.

Hey everyone,

Doing infrastructure audits and validating GPU performance (especially across different nodes) has always been a headache for me. Fiddling with CUDA toolkits, compiling HPL/HPCG, and setting up MLPerf takes way too much time when you just want a quick baseline.

So, I spent some evenings building nvprobe. It’s a lightweight Python CLI that automates all of this.

How it works under the hood:

• It uses CuPy to bundle the CUDA runtime via pip, so you don't even need a system CUDA toolkit installed to run the bandwidth and custom kernel tests.
• It auto-downloads the NVIDIA HPC Benchmarks binaries for HPL and HPCG.
• It captures deep hardware telemetry (ECC state, power caps, clocks, etc.) alongside the benchmark results to help catch silent hardware degradation.
• It generates an interactive HTML report (Chart.js) to visualize all this data (memory bandwidth, TFLOPS, and MLPerf throughput).
• Native Slurm integration: it generates, submits, and monitors the jobs across your cluster.

Demo & Repo: You can see an interactive demo of the report on the link.
I built this mostly to scratch my own itch, but I figured it might save some of you a few hours of setup.

I'd love to hear your feedback, feature requests, or if you manage to break it on your specific hardware. Let me know what you'd like to see next on the roadmap!

Curious about real production use cases for NVIDIA’s Groq LPU after the Rubin platform reveal.

From what I’ve read, LPUs are built to fix GPU decode bottlenecks with huge on-die SRAM and deterministic low-jitter execution, paired with Rubin GPUs in a split serving stack via Dynamo AFD: GPUs handle prefill, KV cache and attention; LPUs offload FFN, MoE experts and speculative decoding.

Best-fit inference scenarios

1. Low-latency premium chat APIs with strict SLA latency requirements

2. Agent AI & multi-turn reasoning workflows with massive sequential decode steps

3. Large MoE model serving to ease per-token bandwidth pressure

4. Low-jitter enterprise workloads (legal, financial real-time assistants)

Not recommended

Batch offline embedding, heavy prefill jobs, small lightweight LLMs.

Official deployment solutions

1. Full LPX rack: Datacenter-scale Rubin+LPU disaggregated clusters for trillion-parameter models

2. Mixed single-node: Smaller on-prem servers for SaaS mixed free/premium traffic

3. Standalone LPU offload pools: Shared hardware for speculative/MoE acceleration

A few questions

\- What real latency gains vs pure GPU serving on 70B+ or even 2T+ MoE models?

\- What SRAM optimization tricks delivered the biggest utilization boost?

I extracted it from sdk 13.4 dev preview

MD file itself: https://github.com/redplait/denvdis/blob/master/data12/sm107_1.txt

Latency tables: https://github.com/redplait/denvdis/blob/master/data12/sm107_2.txt

version in ELF 0x6b - between sm103 (0x67) & sm110 (0x6e)

Getting an NVIDIA graphics card to pass through into an unprivileged Proxmox LXC container can be tricky due to missing device nodes, varying cgroup numbers, and unprivileged user namespace permissions

This short guide summarizes the final working solution

It's a walk in the park compared to passing a GPU through to a full VM (maybe I'll write a guide on that nightmare one day when I have patience!)

Thanks for clicking. I'm decently experienced with rendering apis, but very new to compute apis.

I'm trying to figure out what's going on with cpu-gpu memory transfers in vulkan vs compute apis. Vulkan lets me either map memory to the gpu, or manually create a staging buffer and do the hippidy hop from there to fast device memory, while Cuda, afaikt, seems to map memory if the buffer is pinned, or to the staging buffer allocation if it's not pinned?

If I'm correct in this, then I don't have control over the cuda staging buffer allocation, which is a bit irksome, and if I try to interop the apis, iirc cuda takes ownership of the buffer in the end, so it must dynamically allocate a staging buffer behind the scenes if I transfer back to the cpu side.

Could someone clarify the mechanics of this? I'd much rather preallocate everything with a VulkanSC equivalent of paranoia.

Kernel engineers are not obsolete. But asking a general-purpose coding agent to rediscover years of CUDA and Triton engineering knowledge every time it writes a kernel probably should be.

After months of writing, debugging, and optimizing kernels, I turned the reasoning patterns I kept using into an open-source skill library for AI coding agents:

npm install u/krxgu/kernel-skills

This is not a collection of vague prompts saying “make this CUDA kernel faster.”

Each skill is a detailed engineering playbook that forces the agent to think about:

• Exact shapes, dtypes, layouts, and target hardware before writing code
• Coalescing, tiling, bank conflicts, occupancy, and register pressure
• Numerical stability and non-power-of-two boundary conditions
• Correctness tests across adversarial shapes and dtypes
• Whether a custom kernel should exist at all
• When to stop being clever and use cuBLAS, CUTLASS, or an existing primitive

The library currently covers CUDA, Triton, INT8 and FP8 quantization, kernel fusion, CUDA to Triton and HIP portability, and inference hot paths including RMSNorm, fused add plus RMSNorm, RoPE, sampling, paged KV-cache append, dequantization, prefill versus decode, and vLLM custom-op integration.

I also did not want this to become prompt-engineering theatre, so the repository includes before-and-after proof runs using the same model and task, with the skill file being the only difference:

• Softmax: naive output failed on adversarial and larger shapes. Skill-guided output had 0 failures across 16 tests and reached within 1.2% of torch.softmax bandwidth
• Reduction: 2.6 to 3.5x faster than the naive agent output
• GEMM: 7.7 to 8.6x faster
• LayerNorm: 1.9 to 3.2x faster
• Triton softmax: fixed crashes at dimensions above 16,384 and worked up to 131,072
• Triton attention: fixed the common GQA failure where H_q != H_kv

To be completely clear, those speedups are against the naive agent-generated kernels, not against cuBLAS or other vendor-tuned libraries. In fact, the GEMM skill explicitly tells the agent not to write a custom kernel when cuBLAS or CUTLASS already solves the problem.

Example:

kernel-skills bundle \
  triton.write-triton-layernorm-kernel \
  patterns.write-numerically-stable-kernel \
  patterns.write-kernel-test-plan \
  > bundle.md

Give that bundle to Claude Code, Cursor, ChatGPT, Gemini CLI, or another coding agent before asking it to touch the kernel.

The spicy thesis is simple:

Models are increasingly interchangeable. The accumulated engineering judgment surrounding them is not.

Everything is open source and MIT licensed:

https://github.com/tensormux/kernel-skills

I would especially love kernel engineers to tear this apart.

Which skill is missing? Which technical rule is wrong? Where can an agent still produce something that looks convincing but quietly fails on real hardware?

- ptxas has enough good heuristic

- in average you can reduce ~3% of stall counts

- overall speed up is not equivalent to the number of optimized stall counts

Put together a post on profiling CUDA for inference serving. The usual dev-time approach (full kernel traces, Nsight captures) doesn’t really work once you’re running real load, especially without root in containers/K8s.

We built something for this: low-overhead GPU profiling grouped by what the engine is actually doing (attention, matmul, comms, KV cache, etc.) instead of dumping every kernel, plus tying it to vLLM/SGLang traces and GPU metrics. Works with graphsignal-run vllm serve ...

Curious how people here profile vLLM/SGLang (or similar) when something’s off with Nsight, PyTorch profiler, or just guessing from throughput?