Jersey Planning Scraper

Production-grade Python scraper for Government of Jersey planning application detail pages (gov.je), built to be resilient against WebForms markup quirks and minor frontend changes.

This project demonstrates practical scraping engineering skills:

• Session/cookie warm-up handling for gated pages.
• Robust label-driven extraction (instead of brittle positional selectors).
• Multi-strategy coordinate extraction from URLs, inline scripts, and map handlers.
• Date normalization and explicit null handling.
• CLI workflow for single references and batch processing.
• Fixture-based tests without live-network dependence.

• Accepts either:
  • a planning reference (example: S/2025/10312)
  • or a full detail page URL
• Extracts all key planning fields into normalized JSON.
• Resolves the location map URL when available.
• Extracts coordinates in priority order:
  1. map URL query parameters (lat/lng or x/y)
  2. location-specific map scripts/handlers
  3. relevant script blocks in page HTML
• Date normalization:
  • n/a, empty, or placeholder values -> null
  • real date values -> YYYY-MM-DD
• Built-in retries with backoff for transient HTTP failures.
• Batch mode with optional delay to support polite scraping.

The scraper returns:

• reference
• property
• description
• location_map_url
• coordinates:
  • {"lat": float, "lng": float} or
  • {"x": int, "y": int, "projection": "unknown"} or
  • null
• applicant
• agent
• type
• status
• officer_responsible
• constraints
• validated_date
• advertised_date
• end_publicity_date
• site_visited_date
• committee_meeting_date
• decision_date
• appeal_lodged_date
• appeal_hearing_date
• appeal_decision
• appeal_decision_date
• source_url
• fetched_at (UTC ISO-8601 timestamp)

.
├─ pyproject.toml
├─ src/
│  └─ jersey_planning_scraper/
│     ├─ __init__.py
│     ├─ cli.py
│     ├─ exceptions.py
│     ├─ models.py
│     ├─ scraper.py
│     └─ utils.py
└─ tests/
   ├─ conftest.py
   ├─ test_scraper.py
   └─ fixtures/

Python 3.10+ is required.

python -m venv .venv
# Windows PowerShell
. .\.venv\Scripts\Activate.ps1

pip install -e .[dev]

python -m jersey_planning_scraper.cli --ref "S/2025/10312"

python -m jersey_planning_scraper.cli --url "https://www.gov.je/citizen/Planning/pages/PlanningApplicationDetail.aspx?r=S%2F2025%2F10312&s=1"

python -m jersey_planning_scraper.cli --ref "S/2025/10312" --out out.json

Create refs.txt with one reference or URL per line:

S/2025/10312
S/2025/10311
https://www.gov.je/citizen/Planning/pages/PlanningApplicationDetail.aspx?r=S%2F2025%2F10310&s=1

Run:

python -m jersey_planning_scraper.cli --refs refs.txt --out out.jsonl --sleep 0.25

{
  "reference": "S/2025/10312",
  "property": "Le Clos Rondin La Rue du Rondin, Field No. MY426, Field No. MY426, St. Mary, JE3 3DA",
  "description": "Demolish 18m replica telegraph pole, 3 No. antennas, 1 No. dish and 3 base cabinets...",
  "location_map_url": null,
  "coordinates": {
    "lat": 49.237207,
    "lng": -2.176871
  },
  "applicant": "Mrs Emma Militis",
  "agent": "MS Planning Ltd",
  "type": "Major application",
  "status": "Approved",
  "officer_responsible": "Aaron Elliott",
  "constraints": "GZ - Green Zone, WPSA - Water Pollution Safeguard Area",
  "validated_date": "2025-11-11",
  "advertised_date": "2025-11-18",
  "end_publicity_date": "2025-12-02",
  "site_visited_date": null,
  "committee_meeting_date": null,
  "decision_date": "2025-12-17",
  "appeal_lodged_date": null,
  "appeal_hearing_date": null,
  "appeal_decision": "n/a",
  "appeal_decision_date": null,
  "source_url": "https://www.gov.je/citizen/Planning/pages/PlanningApplicationDetail.aspx?r=S%2F2025%2F10312&s=1",
  "fetched_at": "2026-04-06T00:00:00Z"
}

Run tests:

pytest -q

Install in editable mode:

pip install -e .[dev]

• The scraper warms up cookies by requesting https://www.gov.je/ before detail fetches.
• Label extraction is case-insensitive and whitespace-tolerant.
• URL fragments are ignored by design.
• Coordinate detection avoids unrelated global links (for example sitemap links containing map).

You are responsible for legal and ethical usage. Before scraping at scale:

• verify terms and allowed usage,
• avoid overloading servers,
• respect robots/traffic constraints and use delay in batch mode.

MIT License. See LICENSE.