More Efficient Multi-Head Attention Implementations
- mha-implementations.ipynb contains and compares different implementations of multi-head attention
Summary
The figures below summarize the performance benchmarks (lower is better).
Forward pass only
<a href="https://github.com/rasbt/LLMs-from-scratch/blob/ef9fbc2f52bf9bbb6769f43e6d726bbba4aed03e/ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb">
</a>
Forward and backward pass
<a href="https://github.com/rasbt/LLMs-from-scratch/blob/ef9fbc2f52bf9bbb6769f43e6d726bbba4aed03e/ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb">
</a>
Forward and backward pass after compilation
<a href="https://github.com/rasbt/LLMs-from-scratch/blob/ef9fbc2f52bf9bbb6769f43e6d726bbba4aed03e/ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb">
</a>