TinyModel/plan.txt at main · HyperlinksSpace/TinyModel · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
>>> Actual

Open checks (product / infra):
- Hub-published TinyModel vs same checkpoint evaluated locally: metrics and smoke parity.
- Generative path (e.g. local SmolLM2) vs encoder/RAG backbone (`HyperlinksSpace/TinyModel1`): different roles — document when to compare which.

How to validate now (fast, repeatable):

1) Hub vs local parity (encoder lane)
   - Train/eval local checkpoint with a fixed seed:
     `python scripts/train_tinymodel1_classifier.py --output-dir artifacts/parity-smoke --max-train-samples 120 --max-eval-samples 80 --epochs 1 --batch-size 8 --seed 42`
   - Run one-command parity report:
     `python scripts/parity_check_hub_vs_local.py --local-model artifacts/parity-smoke --hub-model HyperlinksSpace/TinyModel1 --output .tmp/parity-check/hub-vs-local.json`
   - Read `.tmp/parity-check/hub-vs-local.json` (`top_label_match_rate`, average probability deltas) + `eval_report.json` and decide if the Hub model needs a refresh.

2) Generative vs encoder role split (Universal Brain lane)
   - Run generation smoke with local instruct model:
     `python scripts/horizon2_generative.py --task reformulate --max-new-tokens 96`
     (More modes: `python scripts/horizon2_generative.py -h`.)
   - Run encoder/RAG checks:
     `python scripts/horizon1_route_then_retrieve.py --verify --model artifacts/parity-smoke`
   - Use `texts/model-output-improvement-guide.md` to map observed symptom -> script/flag change.

See also:
- `texts/model-output-improvement-guide.md`
- `texts/universal-brain-session-improvement-plan.txt`
- `texts/horizon1-short-term-handbook.md`
- `texts/horizon2-handbook.md`

Maintainer focus: treat checks (1) Hub-vs-local parity and (2) generative vs encoder role validation as the current exit gates before scaling data or publishing a Hub refresh.

>>> Old plan (to think about):

Short-term plan — start now (aligned with Phase A: harden the encoder lane)

1) Ship and verify the pipeline — mostly implemented
   - Kaggle → Hugging Face workflow was hardened and validated through multiple failure classes (auth, kernel conflict, status parsing, retries), and now reaches train/eval/publish path reliably.
   - Remaining routine work: periodic re-runs for regression checks and verifying Space/model release pair per version.

2) Tighten evaluation (before scaling data or model size) — implemented
   - `scripts/train_tinymodel1_classifier.py` now reports accuracy, macro/weighted F1, per-class F1, and writes `eval_report.json` (confusion matrix + reproducibility block).
   - How split + seed work: `texts/eval-reproducibility.md`.
   - Instant test (fast CPU run, ~30s): `python scripts/train_tinymodel1_classifier.py --output-dir artifacts/eval-smoke --max-train-samples 120 --max-eval-samples 80 --epochs 1 --batch-size 8 --seed 42` then open `artifacts/eval-smoke/eval_report.json`.

3) Second dataset or second task (same encoder family) — implemented
   - Hub [`emotion`](https://huggingface.co/datasets/emotion) wired via `scripts/train_tinymodel1_emotion.py` (preset over `train_tinymodel1_classifier.py`). README: section **Second reference dataset (Emotion)**.
   - Instant test: `python scripts/train_tinymodel1_emotion.py --output-dir artifacts/emotion-smoke --max-train-samples 200 --max-eval-samples 100 --epochs 1 --batch-size 8 --seed 42` then open `artifacts/emotion-smoke/eval_report.json`.

4) Embeddings smoke test (product-shaped) — implemented
   - `scripts/embeddings_smoke_test.py` exercises classify / similarity / retrieve (triage-style copy). README: **Embeddings smoke test**.
   - Test: train `artifacts/eval-smoke` then `python scripts/embeddings_smoke_test.py --model artifacts/eval-smoke` (or `--model HyperlinksSpace/TinyModel1`).

5) Optional quick win: pretrained encoder fine-tune — implemented
   - `scripts/finetune_pretrained_classifier.py` (default `distilbert-base-uncased`). Compare `eval_report.json` to scratch runs with same `--seed` and caps. README: **Pretrained encoder fine-tune**.
   - Test: command block in README (`artifacts/finetune-smoke`).

6) Data hygiene (lightweight) — implemented
   - `texts/labeling-and-data-hygiene.md` (label guide template, versioning, leakage). README cross-link under **Custom labels and data hygiene**.

Out of scope for this short list — do not start yet until (1)–(3) are stable: decoder LLM training, RAG production stack, or multimodal — those build on eval + serving discipline from the steps above.