Understanding Flash Attention: Writing the Algorithm from Scratch in Triton
Why is Flash Attention so fast? Find out how Flash Attention works. Afterward, we'll polish our understanding by writing a GPU kernel of the algorithm in Triton.
Alex Dremov
AI RESEARCHER • EPFLI'm a Machine Learning Researcher and Engineer. Here I write posts on the intersection of deep learning theory and efficient machine learning
Featured
MACHINE LEARNING, PAPER · 8 MONTHS AGO · 13 MIN READ
Rethinking Quantization-Aware Training: Why Your QAT Length is Probably Wrong
Training quantized neural networks involves a fundamental trade-off: how should you divide your compute budget between full-precision pretraining and quantization-aware training?
Read the Article →Speed Up PyTorch With Custom Kernels. But It Gets Progressively Darker
It's all about making your models run faster, from flicking a magic “compile” switch to writing your own custom GPU code. In each step, we’ll implement an innocent softmax function, but things are about to get dark by the end.
Simple Ways to Speed Up Your PyTorch Model Training
If all machine learning engineers want one thing, it's faster model training — maybe after good test metrics.
History
All Posts2025
2024
MAY 2024
MACHINE LEARNING, CODE
12 MIN READSimple Ways to Speed Up Your PyTorch Model Training
MACHINE LEARNING, CODE
12 MIN READSimple Ways to Speed Up Your PyTorch Model Training
2023
JUN 2023
IOS & SWIFT
5 MIN READSwift Actors — Common Problems and Tips
IOS & SWIFT
5 MIN READSwift Actors — Common Problems and Tips
MAR 2023
CODE, MACHINE LEARNING
3 MIN READI Contributed to PyTorch. Here's What I Learned
CODE, MACHINE LEARNING
3 MIN READI Contributed to PyTorch. Here's What I Learned
FEB 2023
IOS & SWIFT
5 MIN READConquer Data Races with Swift Actors
IOS & SWIFT
5 MIN READConquer Data Races with Swift Actors
JAN 2023
IOS & SWIFT
8 MIN READDive into Swift's Memory Management
IOS & SWIFT
8 MIN READDive into Swift's Memory Management
2022
DEC 2022
IOS & SWIFT
7 MIN READData Binding in SwiftUI: Tips, Tricks, and Best Practices
IOS & SWIFT
7 MIN READData Binding in SwiftUI: Tips, Tricks, and Best Practices
OCT 2022
IOS & SWIFT
7 MIN READiOS App As a Microservice. Using SwiftUI in Modular App
IOS & SWIFT
7 MIN READiOS App As a Microservice. Using SwiftUI in Modular App
OCT 2022
IOS & SWIFT
7 MIN READiOS App As a Microservice. Modularize Your App With Tuist
IOS & SWIFT
7 MIN READiOS App As a Microservice. Modularize Your App With Tuist
SEP 2022
IOS & SWIFT
7 MIN READiOS App As a Microservice. Build Robust App Architecture
IOS & SWIFT
7 MIN READiOS App As a Microservice. Build Robust App Architecture
AUG 2022
IOS & SWIFT
5 MIN READExploring SwiftUI Layout Protocol | Creating Custom Layout
IOS & SWIFT
5 MIN READExploring SwiftUI Layout Protocol | Creating Custom Layout
JUL 2022
IOS & SWIFT
6 MIN READSwiftUI Navigation Is a Mess. Here’s What You Can Do
IOS & SWIFT
6 MIN READSwiftUI Navigation Is a Mess. Here’s What You Can Do
JUL 2022
ALGORITHMS
2 MIN READSuffix Automaton and Rickroll Lyrics Graph
ALGORITHMS
2 MIN READSuffix Automaton and Rickroll Lyrics Graph
MAY 2022
IOS & SWIFT
4 MIN READUsing Threads in Swift
IOS & SWIFT
4 MIN READUsing Threads in Swift
MAY 2022
IOS & SWIFT, ALGORITHMS
7 MIN READSwiftUI Advanced Animation: Morphing Shapes
IOS & SWIFT, ALGORITHMS
7 MIN READSwiftUI Advanced Animation: Morphing Shapes
APR 2022
IOS & SWIFT
6 MIN READNew Package: Look at Swift Async Algorithms
IOS & SWIFT
6 MIN READNew Package: Look at Swift Async Algorithms
APR 2022
ALGORITHMS, CODE
12 MIN READTreap: The Easiest Search Tree (Explained)
ALGORITHMS, CODE
12 MIN READTreap: The Easiest Search Tree (Explained)
APR 2022
IOS & SWIFT
3 MIN READType Placeholders: New Swift 5.6 Feature
IOS & SWIFT
3 MIN READType Placeholders: New Swift 5.6 Feature
APR 2022
IOS & SWIFT, CODE
8 MIN READQuick Guide to Async Await in Swift
IOS & SWIFT, CODE
8 MIN READQuick Guide to Async Await in Swift
APR 2022
IOS & SWIFT
4 MIN READTop 7 Subtle Swift Features
IOS & SWIFT
4 MIN READTop 7 Subtle Swift Features
2021
AUG 2021
TOOLS
2 MIN READNote-taking apps
TOOLS
2 MIN READNote-taking apps
APR 2021
CODE
15 MIN READThe Mystery of Mach-O Object Structure
CODE
15 MIN READThe Mystery of Mach-O Object Structure
2020
NOV 2020
ALGORITHMS, CODE
10 MIN READSkip List Indexation and kth Maximum
ALGORITHMS, CODE
10 MIN READSkip List Indexation and kth Maximum
MAY 2020
MACHINE LEARNING
8 MIN READHow Deep Neural Networks Work
MACHINE LEARNING
8 MIN READHow Deep Neural Networks Work