Vision Transformers
are tokens too.

You have already learned how transformers process text by turning token sequences into embedding matrices. The same mechanism works on images and audio. The only hurdle is converting pixels and acoustic waveforms into sequences of vectors. Once converted, the attention engine cannot tell the difference.

A transformer requires a flat sequence of tokens. The central idea of ViT: treat a photograph like a sentence by slicing it into square patches and feeding those patches as 'tokens'.

A standard 224×224 image cut into 16×16 chunks yields exactly 196 patches. Each patch is flattened into a 768-number vector (16×16 pixels × 3 color channels), then passed through a learned linear projection into the model's embedding width — 768 dimensions in ViT-Base specifically — before reaching the encoder.

224×224 image → 768 Array Flatten → Transformer
Engine

After passing 196 image patches through the Transformer blocks, the system outputs 196 vectors. To classify an image as exactly a "Dog", we prepend a special `[CLS]` token onto the front of the image sequence. Because of self-attention, this single token attends to all 196 patches, effectively summarizing the whole image.

[CLS] P₁ P₂ P₃ … P₁₉₆ [CLS] attends to all patches across every layer.

Because the transformer sees only a flat sequence, each patch embedding is stamped with a positional embedding that records where it sat in the original 2D grid.

A CNN restricts its vision, sliding a small 3×3 window across the grid. It must understand local edges before looking globally. Data efficient, but it struggles to relate objects on opposite sides of a picture.

ViT drops that constraint. The very first attention layer can relate a patch in the top-left to a patch in the bottom-right in a single step.

DeiT (2021) — Distilled ViTs that train competitively without Google-scale datasets.

Swin Transformer (2021) — Microsoft's sliding-window hierarchical variant, strong on object detection.

Masked Autoencoders (MAE) — 2022. Mask out 75% of image patches and train the network to reconstruct them. This is self-supervised pretraining: the reconstruction task itself is discarded, and the learned features are fine-tuned on downstream tasks — not zero-shot classification.

ViT-22B (2023) — Google's 22-billion-parameter vision model.

CLIP runs two independent encoders: a text transformer and an image transformer. Given an image of a dog and the caption "A fluffy dog", the contrastive objective pulls both representations into the same numerical neighborhood and pushes apart pairs that don't match.

ViT Encoder
Image → Dot Product Intersection Map ← GPT Encoder
Text

Text-to-image systems used CLIP in different ways across eras. DALL-E 1 (2021) was autoregressive — a discrete VAE plus a transformer, with no CLIP conditioning at all. DALL-E 2 (2022, "unCLIP") generated images from CLIP embeddings predicted from text. The image of a frozen CLIP encoder steering diffusion describes 2021-era CLIP-guided diffusion — a historical technique. DALL-E 3 and modern models instead condition the denoiser on T5 or LLM text encoders via cross-attention.

Show CLIP a new photo and ask it to compare the image embedding against 1,000 written class descriptions. It clusters the photo to the closest one. Zero-shot CLIP matched ResNet-50's ~76% ImageNet accuracy — matching a 2015 supervised baseline without any fine-tuning was the achievement.

Standard speech files are sampled at 16kHz, so a 30-second excerpt contains 480,000 individual samples. Feeding nearly half a million data points into standard $O(N^2)$ Transformer attention is intractable — memory grows quadratically with sequence length.

1. Raw Waveform 480k Samples → 2. Log-Mel Spectrogram 80 Bins × 3k Frames (Tractable!)

A spectrogram graphs frequency magnitude over time. Warping it with a "Mel" filter (a scale modeled on human inner-ear sensitivity) compresses the raw waveform into a textured visual 'image' that standard vision-style networks can process.

Just like BERT masks out words, Wav2Vec 2.0 ingests roughly 53,000 hours of unlabelled speech (the Libri-Light corpus) and masks spans of the audio. It uses attention to predict what the missing sound must be from the audio surrounding it.

After pretraining on tens of thousands of hours of unlabelled speech, the model can be fine-tuned for transcription with as little as 10 minutes of labelled transcripts.

Later iterations like WavLM inject noise and overlapping speakers during masking, baking denoising and speaker-separation ability directly into the pretrained model.

OpenAI took the opposite route from Wav2Vec. Whisper trains on 680,000 hours of (audio, transcript) pairs scraped from the internet, using a standard encoder-decoder with cross-attention. It handles language detection, translation, and transcription across 99 languages — but it is not a streaming model: it processes audio in fixed 30-second log-Mel windows.

Log-Mel
Spectrogram Ingests ↑ Encoder
Loop ×N Cross-Attn → Decoder
Auto-regressor Writes ↑ "The AI
Transcribes…"

Whisper's known flaw is its reliance on language modelling. Because the decoder is a language model trying to 'read' the audio, when it hits a stretch of silence or static the auto-regressor often guesses a plausible sentence ending — hallucinating fluent prose over empty air.

Omni-modal architectures like Gemini 1.5 and GPT-4o discard separate isolated brains. (Claude 3.5 Sonnet, from the same era, had vision but not native audio.) When every picture can be encoded into a patch sequence and every audio wave converted into discrete acoustic tokens, you pipe all of them into the same Transformer engine side by side.

By bypassing rigid translation chains (Speech-To-Text → Text Engine → Text-to-Speech), the GPT-4o native omni-model responds to audio in about 320ms on average (232ms at minimum) — close enough to human conversational timing to support natural interruptions.

Sensory Modality Hardware Filter Step Flagship Models Engine Output Type Pure Text BPE Tokenization → Dict Lookup GPT, LLaMA, Claude Auto-Regressive Generative Photographic Square Grid Splitting → Arrays ViT, CLIP, Swin Deep Dimensional Matching Acoustic Audio Log-Mel Convolution → Frame Arrays Whisper, Wav2Vec 2.0 Explicit Transcription

> ViT — Dosovitskiy et al., (arXiv:2010.11929)
> MAE — He et al., Meta 2022 (arXiv:2111.06377)
> CLIP — Radford et al., OpenAI 2021 (arXiv:2103.00020)
> Whisper — Radford et al., OpenAI 2022 (arXiv:2212.04356)