We review why some transformer models use a bias in attention and how ALiBi helps with long context. The provided sources focus on significant advancements in computational biology, specifically the evolution of the AlphaFold series for predicting 3D biomolecular structures. AlphaFold 2 revolutionized the field by using the Evoformer and attention mechanisms to interpret evolutionary and geometric data with near-experimental accuracy. Building on this, AlphaFold 3 expanded capabilities to include complexes with ligands and nucleic acids using an atom-level diffusion module. To further refine these models, HelixFold-S1 introduces a contact-guided sampling strategy that prioritizes likely binding sites to improve structural diversity and accuracy. Additionally, technical papers describe architectural components like ALiBi for handling long sequences and Swin Transformer's shifted windows. Together, these texts illustrate a shift toward more efficient, targeted sampling and integrated deep learning frameworks for complex molecular modeling.Sources:August 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowshttps://arxiv.org/pdf/2103.14030April 2022 - (ALiBi)TRAIN SHORT, TEST LONG: ATTENTION WITH LINEAR
BIASES ENABLES INPUT LENGTH EXTRAPOLATIONhttps://arxiv.org/pdf/2108.12409April 2022:Swin Transformer V2: Scaling Up Capacity and Resolutionhttps://arxiv.org/pdf/2111.09883

Ver todos los episodios