How small can a model get

Most of my research time over the past several months has gone into one question: how small can a model get before it stops being trustworthy enough to hand a real decision to. The answer I'm building lives in a fine-tuned Qwen2.5 family, at 0.5B, 1.5B, and 3B parameters, doing travel risk assessment for the OSINT pipeline behind the Apps page on this site.

The model has to do two different jobs well. The first is classification: given retrieved evidence about a destination, decide where it falls across five separate risk dimensions, armed conflict, civil unrest, health, natural disaster, and crime or infrastructure, each checked against its own ground truth source rather than one blended score. UCDP covers conflict, ACLED and GDELT cover unrest, WHO and CDC cover health, GDACS covers disasters. The second job is tool orchestration. Rather than answer from whatever it already has, the model has to recognize when the evidence in front of it is thin and call back out to the OSINT retrieval pipeline for another pass, using smolagents to make that call instead of guessing.

The training set behind this, Travel_Risk_Data on Hugging Face, is 50,535 records built from three sources: 50,218 country-year conflict records from my own Civil War Prediction dataset, 227 consolidated per-country risk reports pulled from UK FCDO travel advice, and 90 Q&A pairs sourced live from the US State Department's advisory API. Every record follows the same Context: ... / Analysis: ... format no matter which source it came from, split 90/10 train and test with a fixed seed so results are reproducible. It's MIT licensed and public on Hugging Face if you want to fine-tune against it yourself.

I trained these with QLoRA and iterated the pipeline on HuggingFace Jobs and Modal, tracking runs with Trackio. Nothing comes back as free text. Every output is constrained JSON with a confidence score and the evidence IDs behind each claim, and anything above a set confidence threshold gets routed to a human instead of letting the model call it on its own. A tool sitting this close to a security decision has to show its work, or I don't trust it.

Hallucination detection was the hardest part to get right. String matching against the source text caught the easy cases and missed everything subtle, so I replaced it with a three-model judge ensemble that cross-checks each generated claim against the retrieval it's supposedly grounded in. I'm writing this up as a paper now, targeting AAAI's IAAI track or EMNLP's industry track, with a public benchmark to go alongside it on Hugging Face. Firms like International SOS already sell travel security advisory as a product; nobody's published a benchmark for it. That's the gap I'm trying to close.

I've also built two versions of a local deep research agent. The first runs on LangGraph, Tavily, and trafilatura as a five-node pipeline with a cross-check step before synthesis, because most research agents I've used skip straight from retrieval to answer and inherit whatever the first source got wrong. The second runs on Ollama with Brave Search, ACLED, and RSS feeds, split into a supervisor and specialist architecture modeled on how incident command actually divides work during an emergency response.

Conflict prediction is the research thread I've kept the longest. I maintain a dataset and a fine-tuned model under the ConflictAI organization on Hugging Face, and most of what else I build, the Qwen risk models, the Conflict-Risk fine-tunes, comes out of the same interest: anomaly detection and black swan modeling for the kind of event a standard risk score misses until it's already happened.