Skip to content

Radar

Daily bookmarks: relevant news, facts and discoveries.

Engineering ·

Free guide on scaling LLMs across TPUs and GPUs

Radar: a free online book that explains TPUs, GPUs and the parallelism used in training and inference of LLMs at large scale, with a focus on cost and memory.

Engineering ·

Peter Bloem's tutorial: transformers from scratch

A blog post by Peter Bloem that explains the internal components of the transformer from basic principles, focusing on attention. Shared as study material on the architecture.

Engineering ·

Kinetic: Rust inference engine that processes the prompt before Enter

Radar: open-source inference engine in Rust (Apache 2.0) that, according to the post, starts processing a pasted prompt before Enter. The reported test on a laptop with an RTX 3060 shows 0.13 s versus 10.95 s for another local engine. The numbers are the author's.

Engineering ·

V4.1 Flash model on a consumer GPU with a local serving stack

A author's report on running a V4.1 Flash model locally on a consumer GPU at about 20 tokens per second, with the API layer separated from the inference engine. The speed figures and the claim of no quality loss are the author's own, with no independent evaluation.

Engineering ·

Modal GPU Glossary: GPU computing terms

Modal published a GPU terms glossary, shared as a GPU Glossary reference for reading CUDA documentation and profiler outputs.

Guides ·

Legal LLM fine-tuning with LoRA and the hold-out set lesson

An author reports fine-tuning a Llama for a legal task with LoRA and Unsloth notebooks. The model memorized the training data, performed worse than the base model on new examples, and the lesson is to validate with a hold-out set.

Get the essentials

Privacy, AI engineering and security in your inbox.

At most one digest per day, when there is news. Cancel at any time. We use your account's verified email.

Privacy

Looking for something else?DocumentationContact