Most neural network taxonomy articles try to do too much in one list. They put CNNs next to Transformers, diffusion, cross-attention, mixture of experts, and skip connections as if those terms live at the same level.

They do not.

The useful split is simple: architectures define the dominant computation pattern. Mechanisms modify how that computation is routed, conditioned, stabilized, or scaled.

This page is the map. It is not trying to be the entire book. Use it to orient yourself, then follow the branch that matches the problem in front of you.

The Taxonomy Axis

The weakest way to classify neural networks is by input type.

"Image model" could mean a CNN classifier, a Vision Transformer, a diffusion model, a masked autoencoder, or an autoregressive image-token model. "Text model" could mean an LSTM, an encoder-only Transformer, a decoder-only LLM, a retrieval system, or a multimodal model that happens to emit text.

Use case has the same problem. A speech system can use convolutional front-ends, recurrent layers, Transformer encoders, sequence-to-sequence decoding, or diffusion-style generation. If you group by use case, every mature system becomes miscellaneous. Very official miscellaneous, but still miscellaneous.

A better axis is how information moves through the model.

A CNN makes a structural bet that local patterns matter and useful features can be built hierarchically. An RNN or state space model makes a structural bet that state should evolve through order. A Transformer makes a structural bet that elements should dynamically attend to other elements. A diffusion-based generative system makes a training and sampling bet: generation can be learned as iterative denoising, while the denoising backbone may still be a U-Net, Transformer, or another architecture.

That axis is not perfect. It is useful because it keeps one engineering question stable across domains: what computation pattern is doing the main work?

Architecture Families

The map has four major families.

Spatial / structured architectures compute representations by exploiting locality, hierarchy, or explicit neighborhoods. MLPs are the simple feedforward vector case; CNNs, ResNet-style networks, U-Nets, and GNNs are the stronger structured cases. They are useful when the input has stable structure: grids, images, graphs, parts, neighborhoods, and reusable local patterns.

Sequential / temporal architectures model ordered data by maintaining state, applying causal filters, or learning dynamics across time. RNNs, LSTMs, GRUs, TCNs, S4, and Mamba-style selective state space models belong here. They matter when order, streaming, long context, or efficient sequence processing is central.

Attention-based architectures compute representations by dynamically weighting interactions between elements. Encoder-only Transformers, decoder-only Transformers, encoder-decoder Transformers, Vision Transformers, and sparse attention systems belong here. They are the default when flexible interaction matters and the compute budget can support it.

Generative / latent systems learn data distributions so they can reconstruct, sample, transform, or generate examples. VAEs, GANs, diffusion models, flow matching systems, and autoregressive generators belong here, but this branch is about the objective and sampling process as much as the backbone. They matter when the model must produce or transform samples, not only classify or embed them.

The deeper articles should unpack each branch separately. This hub keeps the distinction visible without turning every family into a lecture.

Mechanisms vs. Architectures

This is the distinction most guides blur.

An architecture defines the dominant computation pattern: convolutional hierarchy, recurrent state update, graph message passing, attention over tokens, or feedforward mixing. Objectives and generative processes such as autoregressive factorization, latent compression, or denoising describe how the model is trained or sampled; they can sit on top of different backbones.

A mechanism is a reusable technique inside an architecture. It changes routing, conditioning, memory, optimization, scaling, or stability. It does not define the whole family by itself.

Cross-attention is a mechanism. It lets one representation condition another. It can appear in translation, diffusion U-Nets, multimodal assistants, or vision-language systems.

Mixture of experts is a mechanism. It routes tokens or examples through selected expert subnetworks. Most public examples use MoE inside Transformer blocks. The surrounding architecture is still usually a Transformer.

Positional encoding is a mechanism. Transformers need some way to represent order or location. That can be absolute, relative, rotary, learned, or domain-specific.

Residual connections, normalization, and gating are mechanisms too. They help deep systems preserve signal, stabilize training, control memory, and route information. They are not architecture families just because they are important.

Once you separate these levels, model descriptions get much cleaner. You stop asking whether a model is "MoE or Transformer." You ask whether it is a Transformer using MoE routing in some blocks.

Hybrid Systems

Real systems are hybrids. The clean taxonomy is not meant to pretend otherwise. It is meant to make hybrids easier to describe.

Stable Diffusion is a latent diffusion generative system. It operates in a compressed latent space, uses a denoising backbone with spatial structure, and conditions on text through mechanisms such as cross-attention.

Whisper is an attention-based speech model. Audio becomes log-Mel spectrogram features, those features are processed before Transformer blocks, and a decoder emits text tokens autoregressively.

AlphaFold 2 uses Evoformer blocks over sequence/MSA and pair representations, combines attention with pairwise triangle updates, then uses a structure module with geometric reasoning. It is not just a protein model; it is a composition of sequence, pairwise, geometric, and attention mechanisms.

Sora-style video generation systems are useful to describe as diffusion transformer systems when public descriptions support that framing. The generative process is diffusion-like, while the denoising backbone reasons over visual or latent patches across space and time.

Mixtral-style models show how MoE fits into the picture. The architecture is Transformer-based; MoE changes how feedforward capacity is allocated.

The lesson is not "everything is a Transformer now." The lesson is that modern models combine computation paradigms and mechanisms when the problem demands it.

How To Choose

Architecture choice starts with the shape of the problem, not the leaderboard.

Ask three questions first:

  • What structure does the input already have?
  • What kind of output must the model produce?
  • What are the constraints on data, latency, context length, and compute?

Then choose the simplest family whose structural bias matches the job.

Use feedforward or spatial models when local or relational structure is meaningful and stable. Use sequential or temporal models when order, streaming, or efficient long-context behavior matters. Use attention-based models when flexible interaction between elements is the core need. Use generative or latent models when the task is to sample, reconstruct, edit, or transform a distribution.

Mechanisms come later. Add MoE when dense capacity is the bottleneck and sparse activation makes economic sense. Add cross-attention when one representation must condition another. Add sparse attention when full attention is the limiting cost, not because the phrase looks expensive enough to invoice.

Good architecture selection is boring in the best way: match the structure, measure the bottleneck, then add complexity only where it pays rent.

Reading Path

This hub should grow into a tree of focused Labs:

The clickable map above is the navigation layer for that tree. Child nodes now open focused pages instead of pushing readers down this same hub.

References