# More Efficient Multi-Head Attention Implementations

- [mha-implementations.ipynb](mha-implementations.ipynb) contains and compares different implementations of multi-head attention



### Summary

The figures below summarize the performance benchmarks (lower is better).


 
#### Forward pass only

<a href="mha-implementations.ipynb"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/mha-benchmark/1_forward-only.webp?1" width="500px"></a>

&nbsp;
#### Forward and backward pass

<a href="mha-implementations.ipynb"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/mha-benchmark/2_forward-and-backward.webp?1" width="500px"></a>

&nbsp;
#### Forward and backward pass after compilation

<a href="mha-implementations.ipynb"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/mha-benchmark/3_forward-and-backward-compiled.webp?1" width="500px"></a>