A 149M Text Encoder as the First Tier for Phone Agents: Pre-Registered Routing to Frontier Models

Paper: paper/main.pdf (preprint: doi:10.5281/zenodo.23818333). Model: Milo Pointer on Hugging Face.

All Claude API calls behind the AndroidControl results cost $86.75 in total (Batch prices), and each training run of the model takes about 6.4 hours on one NVIDIA T4.

• A 149M text-only model scores 86.1% on AndroidControl-Low under the protocol screenshot agents report, above the 85.2% reported for OS-Atlas-7B (step sets differ slightly).
• Small model + frontier beats the frontier alone. The small model answers about 70% of steps on the device and hands the rest to Claude Opus 5.5. In a pre-registered confirmatory test routing beat Opus alone by 2.8 points (p = 0.003); pooled over all 1,650 test steps the gain is 1.5 points (95% CI [+0.5, +2.4]), at under a third of Opus's API cost.
• What did not work is documented too. About 40 registered experiments, negative ones included, with timestamps (paper/PREREGISTRATION.md).

A 149M-parameter local model answers the phone-agent steps it is confident about and hands the rest to a frontier model. On 1,000 held-out AndroidControl steps (offline, one step at a time, pre-registered):

A further registered test on 650 steps from test episodes used in no earlier sample, with a learned hand-off rule frozen beforehand, found routing to Opus more accurate than Opus alone: 599 vs 581 (+2.8 points, 95% CI [+1.0, +4.5], p = 0.003), with 73.7% of steps answered locally (python bench/ac_confirm_score.py, needs numpy).

Limits: offline single steps with a human-written step instruction; a comparable cascade lost on live AndroidWorld tasks; the small model alone is not better than Opus given the same examples; on a Pixel 6a CPU the model takes a median 3.4 s per step. Details in paper/main.pdf.

Follow-up, goal only (registered 2026-10-30, after the paper): a 1.5B planner (Qwen2.5-1.5B, LoRA, trained on the dataset's human step instructions) writes the instruction for the same executor. On 1,804 in-domain steps: human instruction 90.2%, goal alone 19.3%, planner 53.0%. The planner helps (+33.7 points, p < 1e-100) but does not come within 10 points of human instructions (registered margin missed by a wide gap), and the executor's confidence does not catch wrong plans. Details: training/kaggle_planner/RESULT.md.

./reproduce.sh                       # standard library only
pip install pytest numpy && pytest -q tests   # asserts every headline number and registered outcome

Re-scoring on AndroidControl-Curated labels and re-running Claude need the AndroidControl text file (training/ac/androidcontrol_text.jsonl.gz, built from the public AndroidControl release by training/kaggle_ac/androidcontrol_text.ipynb), pip install -r requirements.txt and, for Claude, an Anthropic API key in ANTHROPIC_API_KEY (the Claude arms cost money).

Scripts that run the model need the weights next to milo_pointer.py:

hf download coolnow/milo-pointer --local-dir release/milo-pointer

The clean retrain used by the shortlist test (bench/sl_prepare.py) is on the same model page under clean_r3/; copy clean_r3/pointer_clean_r3.pt to training/ac/pointer_clean_r3/. bench/ptr_latency.py reads the training checkpoint training/ac/pointer_v2/pointer.pt and bench/pointer_eval_local.py takes one as an argument; neither checkpoint is shipped, and the released model.safetensors holds the same weights.

from milo_pointer import MiloPointer
m = MiloPointer.load("release/milo-pointer")   # folder with model.safetensors (see above)
a = m.predict(goal, instruction, elements, width, height, last_step="new step")
if a["defer"]:
    ...  # send this step to your frontier agent

Code: Apache 2.0. Model: Apache 2.0, fine-tuned from ModernBERT-base (Apache 2.0) on AndroidControl (Apache 2.0).

@misc{talapalli2026firsttier,
  title  = {A 149M Text Encoder as the First Tier for Phone Agents: Pre-Registered Routing to Frontier Models},
  author = {Talapalli, Chaitanya},
  note   = {ORCID 0009-0000-8811-3349},
  year   = {2026},
  doi    = {10.5281/zenodo.23818333},
  url    = {https://doi.org/10.5281/zenodo.23818333}
}