ImageGPT (iGPT) applies the autoregressive GPT architecture directly to image generation by treating images as sequences of pixels or color clusters, demonstrating that language model approaches can learn strong visual representations without any vision-specific architectural components.

Architecture Overview

ImageGPT treats an image as a 1D sequence of pixels, scanned in raster order (left to right, top to bottom). Since raw RGB values would create a vocabulary of 256³ = 16.7M colors, the image palette is first reduced to 512 colors using k-means clustering on pixel values from the training set. Each pixel becomes a single token from this 512-color vocabulary.

The sequence of color tokens is processed by a standard GPT decoder-only Transformer with causal (autoregressive) attention. The model predicts the next pixel's color given all previous pixels, trained with the standard cross-entropy language modeling objective. Images are typically downsampled to 32×32, 48×48, or 64×64 before tokenization, producing sequences of 1024, 2304, or 4096 tokens.

For representation learning, the hidden states from the middle or final layers of the pretrained model serve as image features. A linear probe or fine-tuning head on these features achieves competitive image classification accuracy, demonstrating that autoregressive pretraining learns semantically meaningful visual features.

Key Innovations

  • Unified sequence modeling: Applying the exact same GPT architecture and training objective to images proved that autoregressive sequence modeling generalizes beyond language
  • Color quantization: K-means clustering of pixel colors reduces vocabulary to a manageable 512 tokens while preserving visual quality
  • Unsupervised representation learning: Features learned through next-pixel prediction are competitive with supervised and contrastive methods for downstream classification
  • Simplicity: No convolutions, no vision-specific inductive biases, no data augmentation—pure sequence modeling on pixels

Common Use Cases

Unconditional image generation, image completion (given partial images, generate the rest), unsupervised visual representation learning, and as a proof-of-concept for applying language model paradigms to vision tasks.

Notable Variants & Sizes

iGPT-S (76M, 24 layers, 512-dim), iGPT-M (455M, 36 layers, 1024-dim), iGPT-L (1.4B, 48 layers, 1536-dim), iGPT-XL (6.8B, 60 layers, 3072-dim). Conceptual successors include DALL-E 1 (which used a dVAE for tokenization instead of k-means), Parti (ViT-VQGAN tokens + Transformer), and Llamagen. The ImageGPT approach demonstrated principles later adopted by visual tokenizer + autoregressive model pipelines.

Technical Details

iGPT-L: 48 layers, 24 attention heads, 1536-dim embeddings, 8× MLP expansion (12288 inner dim). Input: 48×48 or 64×64 images → 512-color k-means quantization → 2304 or 4096 token sequences. Vocabulary: 512 colors (k-means centroids computed on ImageNet pixels). Training: autoregressive cross-entropy loss, Adam (lr=2e-4 to 5e-5), linear warmup, cosine decay, batch size 64, trained on ImageNet (without labels). Feature extraction: average-pool hidden states from layer n/2 for linear probe. Results: iGPT-L achieves 65.2% top-1 on ImageNet with linear probe, 72.0% with fine-tuning at 48×48 resolution—competitive with SimCLR and other self-supervised methods. Sampling uses temperature scaling and top-k for image generation.