Index
Neighborhood Attention / NA
- Attention
- 画像処理における Attention
- yhayato1320.hatenablog.com
改善アルゴリズム
Dilated Neighborhood Attention / DiNA / 2022
- Dilated Neighborhood Attention Transformer
- [2022]
- arxiv.org
https://t.co/uNtSy3jS1d
— akira (@AkiraTOSEI) November 6, 2022
dilated Convのような視野の取り方をすることにより、計算量を削減しつつ視野を広げるDilated Neighborhood Attention (DiNA) を提案。Swinのように窓を重ね合わせなくても窓間の相互作用を折り込める。物体検知などでSwin, ConvNext を超える結果。 pic.twitter.com/ze89ubUlLA
causal NA / Fused Neighborhood Attention (FNA) / 2024
- Faster Neighborhood Attention: Reducing the O(n2) Cost of Self Attention at the Threadblock Level
- [2024]
- arxiv.org
strided NA / 2025
- Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light
- [2025]
- arxiv.org
参考
Neighborhood Attention Transformer
NATTEN
- NATTEN is an open-source project dedicated to providing infrastructure for multi-dimensional sparse attention methods, specifically NA.
- github.com
Web サイト
Post
Attention can now be applied in sliding-windows efficiently and flexibly with NA/DiNA, similar to what (dilated) Convolutions do to FC layers. They preserve locality, capture global context, and expand RFs exponentially, all with linear complexity!
— Humphrey Shi (@humphrey_shi) October 1, 2022
PwCode: https://t.co/yL3wjYPZWQ pic.twitter.com/GWVfIsJhNf
