Federated Learning: Training Models Without Collecting the DataPosted 2026-07-16. Hospitals accumulate patient records; phones accumulate typing habits. Yet pooling that data in one place is often impossible — regulation forbids it, moving it is expensive, and the people who own it would rather not hand it over. Federated learning flips the usual assumption on its head: instead of bringing the data to the model, it sends the model to the data, trains locally, and collects only the changes. The basic ideaThe workhorse is FedAvg, proposed by McMahan et al. in 2017. The loop is simpler than it sounds:
Repeat, round after round. The raw data never leaves the device, and the server sees only a summary of what was learned. Google training next-word prediction on mobile keyboards this way is the textbook example. The hard partsIt sounds tidy, but running it surfaces problems that centralized training never had. Statistical heterogeneity. Every client's data is distributed differently (non-IID) — one user may hold examples from only a couple of classes. Plain averaging then converges slowly or drifts in the wrong direction. Methods like FedProx and SCAFFOLD keep local updates from wandering too far, aiming squarely at this. Communication cost. Every round ships a model back and forth, so the larger the model, the sooner communication becomes the bottleneck. A steady line of work sparsifies or low-rank-compresses the updates, and tries to cut the number of rounds outright. System heterogeneity. Devices differ in compute and network, and some simply drop out mid-training. Synchronous rounds are held hostage by the slowest participant, so asynchronous and semi-asynchronous schemes step in — where the open question is how to weigh updates that arrive late (staleness). I have looked at this problem myself — Staleness Aware Semi-asynchronous Federated Learning. Privacy and security. Not shipping the data does not make you safe. Gradient-inversion attacks can reconstruct much of the original input from the updates alone, and, in the other direction, a malicious participant can tamper with its update to plant a backdoor (poisoning). Recent directionsA few threads stand out right now. Meeting large models. The most active area. Sending an LLM back and forth is a non-starter for both communication and compute, so the field has settled on federating only a parameter-efficient slice — LoRA adapters and the like (PEFT). When just a few percent of the weights travel, the communication burden collapses. Stronger privacy guarantees. Moving past “I did not send the data, so I am fine,” differential privacy (DP) adds calibrated noise to the updates, and secure aggregation lets the server see only the sum rather than any individual update — both are now close to standard practice. The catch is always the trade-off between privacy and accuracy. I proposed folding de-identification into the client-selection step to push that balance a little further — Lightweight Multi-Layered De-Identification Architecture. Robustness. Byzantine-robust aggregation that survives faulty or adversarial participants, together with detecting and removing planted backdoors, keeps advancing. Federated unlearning. When a user asks to be dropped from training, their contribution has to be erased from the model — and you cannot simply retrain from scratch on data you no longer hold. Doing this efficiently in the federated setting (federated unlearning) is drawing fresh attention. Wrapping upFederated learning starts from a single idea — move the training, not the data — but making it useful means untangling heterogeneity, communication, privacy, and security all at once. For me that knot — designing large-scale distributed systems that stay both performant and secure — is the most interesting part |