Few-Shot Learning

Few-shot learning refers to providing a language model with a handful of examples (typically 2-10) in the prompt to demonstrate the desired task format and behavior. Unlike fine-tuning, which modifies the model's weights, few-shot learning works entirely through the prompt — the model uses the examples as context to infer the pattern and apply it to new inputs. This technique was popularized by GPT-3's landmark paper, which showed that large models could learn new tasks from just a few examples.

For example, if you want a model to classify customer emails by sentiment, you might include three examples in your prompt: one positive email labeled "positive," one negative labeled "negative," and one neutral labeled "neutral," followed by the email you want classified. The model learns the labeling pattern from these examples and applies it to the new email. This approach works remarkably well across a wide range of tasks, from classification and extraction to translation and reformatting.

Few-shot learning is powerful because it requires no training data pipeline, no compute for fine-tuning, and can be set up in minutes. It works best when the examples are representative of the task, clearly formatted, and diverse enough to cover edge cases. The quality and selection of examples matters more than quantity — three carefully chosen examples often outperform ten random ones. Including examples that demonstrate boundary cases or potential failure modes is particularly effective.

The main limitation of few-shot learning is context window consumption. Each example takes up tokens, leaving fewer tokens for the actual input and output. For complex tasks requiring many long examples, this can be restrictive. Few-shot learning also cannot teach a model truly new knowledge — only how to apply its existing knowledge in a specific format. For tasks requiring specialized domain knowledge or consistent behavior across thousands of calls, fine-tuning may be more appropriate and cost-effective in the long run.