Inside carBot: market intelligence starts with uncertainty, not scraping.

Collecting listings is easy to demo and hard to mistake for a finished system. The difficult part begins after download: deciding which number is mileage rather than a warranty limit, whether two URLs describe the same physical car, which records are comparable, and when the system should decline to score at all.

A listing is a claim, not a row.

A seller-written page mixes facts, formatting accidents, sales copy, and omissions. The same vehicle may reappear with a new URL, a translated description, a changed asking price, or incomplete identity data. Auction records have different lifecycle semantics from ordinary advertisements. A parser that writes one clean row and discards the source has destroyed the evidence needed to debug every later decision.

carBot therefore stores raw and normalized representations together. A parsed field carries a value, an extraction source, the original fragment, and a rule-based confidence class. That class is deliberately not presented as a calibrated probability. It is operational metadata: enough to route weak records to review, compare parser strategies, and explain why a value was accepted.

The pipeline is evidence, state, model, then action.

carBot is an asynchronous Python pipeline backed by PostgreSQL, with a typed inspection API and an analyst-facing web application. Source adapters run independently; normalization and persistence happen per listing; market recomputation, scoring, identity aggregation, and alert evaluation run as explicit post-processing stages. Expensive global stages are serialized so two source loops cannot simultaneously rebuild the same models or double the process's peak memory.

The core record is boring by design.

The database does not overwrite messy evidence with a clean fiction. It stores raw fields beside normalized values, confidence classes, parser version, content hash, and lifecycle timestamps. A unique URL fingerprint makes repeated ingestion idempotent; a separate identity link can later mark an exact cross-source duplicate. On a partial refresh, null detail fields do not erase previously known values. Price changes become append-only events instead of silently replacing history. The reduced record below uses synthetic values and omits private fields.

{
  "identity": {
    "url_fingerprint": "stable-source-key",
    "duplicate_of": null
  },
  "raw": {
    "price": "localized seller text",
    "description": "retained source evidence"
  },
  "parsed": {
    "mileage_km": {
      "value": 142000,
      "confidence": "medium",
      "source": "labeled-description-fragment"
    }
  },
  "lineage": {
    "content_hash": "change-detection-hash",
    "parser_version": "ruleset-version"
  },
  "decision": {
    "score": 82.4,
    "method": "guarded-regression",
    "sample_count": 41
  }
}

The parser ranks evidence instead of trusting the first match.

Regex is not the problem; context is. A description may contain the car's mileage, a warranty limit, and a distance to the seller, all followed by the same unit. carBot collects candidates, rejects known contexts, prefers labeled evidence, applies plausibility guards, and records why the winning value was selected. Year extraction similarly rejects numbers embedded in full dates, while make and model normalization uses ordered aliases so a specific multi-word model wins before a shorter ambiguous token.

Source precedence matters too. Titles are compact and often high-signal for engine codes and model names; descriptions are richer but noisier. The parser tries the narrow evidence first, falls back to broader text, and keeps raw fragments for review. Domain catalogs handle aliases, trims, fuel terms, gearbox language, power units, localized number formats, and currency separation.

# Simplified evidence-ranking pattern.
def choose_mileage(text: str) -> ScoredField:
    candidates = collect_mileage_candidates(text)

    usable = [
        c for c in candidates
        if not in_warranty_context(c, text)
        and not in_location_distance_context(c, text)
        and plausible_mileage(c.value)
    ]

    if not usable:
        return ScoredField(None, confidence=Confidence.NONE, source="not_found")

    winner = min(usable, key=lambda c: (c.pattern_priority, -c.confidence_rank))
    return ScoredField(winner.value, winner.confidence, winner.pattern_name)

There is an important distinction here: confidence metadata makes uncertainty visible, but it does not magically calibrate it. The current classes are rule-derived labels, not probabilities. Market cohort queries presently rely mainly on presence, plausibility, status, and deduplication rather than weighting every sample by parser confidence. Tightening that boundary, and isolating extractor failures per field instead of degrading an entire parse result after one exception, are two concrete improvements still on the roadmap.

Comparability matters more than model sophistication.

A cheap car is not necessarily a good deal; it may simply be compared with the wrong cars. carBot first partitions active, non-duplicate listings by vehicle identity and market context, excludes populations with incompatible price semantics, clips extreme asking prices with an interquartile-range rule, and computes medians. It prefers a recent exact-year cohort, falls back to longer history when the segment is thin, and widens the year window only as a last resort.

This baseline is deliberately an asking-price model. Listings are correlated, seller-selected, and not equivalent to completed transactions. The median is useful for ranking and investigation, not a claim about realizable sale value. That distinction survives every layer of the scoring design.

-- Reduced form of the cohort query.
WITH base AS (
  SELECT cohort_key, price, mileage
  FROM active_listings
  WHERE duplicate_of IS NULL
    AND price IS NOT NULL
),
quartiles AS (
  SELECT cohort_key,
         percentile_cont(0.25) WITHIN GROUP (ORDER BY price) AS q1,
         percentile_cont(0.75) WITHIN GROUP (ORDER BY price) AS q3
  FROM base
  GROUP BY cohort_key
),
clipped AS (
  SELECT b.*
  FROM base b
  JOIN quartiles q USING (cohort_key)
  WHERE b.price BETWEEN q.q1 - :iqr_multiplier * (q.q3 - q.q1)
                    AND q.q3 + :iqr_multiplier * (q.q3 - q.q1)
)
SELECT cohort_key,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY price) AS median_price,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY mileage) AS median_mileage,
       count(*) AS sample_count
FROM clipped
GROUP BY cohort_key;

The score is a guarded ranking heuristic, not an oracle.

The engine refuses obvious invalid inputs first: exact duplicates, missing vehicle identity, and records without a price remain explicitly unscored. For eligible records it tries an ordinary least-squares estimate with mileage, age, and power; falls back to a smaller mileage-and-age model; then falls back again to the comparable-cohort median. Each successful result stores the method, sample count, expected price, residual, and relevant quality metadata.

The hierarchy matters more than the regression. A fitted model is accepted only when it has enough samples, passes an in-sample fit threshold, and predicts within plausibility bounds. Otherwise the system takes the less specific but more robust cohort path. The final score can also incorporate relative mileage, seller context, and non-negated damage language. Its job is to order an analyst's queue, not certify a vehicle's value.

# Reduced scoring control flow; weights and thresholds are omitted.
def estimate_price(record, models, cohorts):
    for model in (models.ols_mileage_age_power, models.ols_mileage_age):
        prediction = model.try_predict(record)
        if prediction and prediction.fit_is_usable and prediction.is_plausible:
            return Estimate(
                value=prediction.value,
                method=model.name,
                evidence={"samples": model.n, "r_squared": model.r_squared},
            )

    cohort = cohorts.best_available_for(record)
    if cohort and cohort.sample_count >= cohort.minimum_required:
        return Estimate(
            value=cohort.median_price,
            method="cohort_median",
            evidence={"samples": cohort.sample_count, "window": cohort.window},
        )

    return None  # Unscored is a valid result.

The current OLS implementation solves normal equations directly and uses in-sample R-squared. That keeps the dependency surface small and the model explainable, but unscaled predictors, collinearity, and small year bands can make coefficients unstable. A production-grade next step is QR- or SVD-based fitting, regularization where justified, and time-split validation. Until then, regression remains one guarded estimator inside a fallback system, not the headline claim.

The runner is part of the analytical method.

An unattended pipeline can keep returning HTTP 200 while quietly becoming useless. carBot watches rolling success rate, latency percentiles, throttling responses, empty-result pages, and extraction coverage for key fields. A sudden drop in model or mileage extraction is treated as selector drift even when requests still succeed. Source loops have independent pacing and cooldown state, while a cross-process database lock prevents a second runner from starting.

Global post-processing is intentionally serialized. Cohort rebuilds, rescoring, identity aggregation, and alert evaluation touch shared data and can be memory-intensive; running two copies after simultaneous source completions would increase peak memory and create conflicting bulk updates. A lock plus a recency cooldown collapses redundant work. The trade-off is explicit: the current runner is an in-process orchestrator, not a durable distributed workflow engine, so process restarts resume from database state rather than from a persisted job graph.

The API and analyst UI expose the evidence needed to operate this system: low-confidence records, score method and details, price history, source health, extraction drift, identity events, runner state, and alert configuration. The dashboard is not decoration around the scraper. It is where model assumptions become inspectable.

Listing identity and vehicle identity are different problems.

A URL fingerprint answers "have we seen this endpoint before?" It does not prove that two different advertisements describe different cars. carBot therefore keeps source-level deduplication separate from vehicle identity. An exact validated VIN can link appearances across sources and time; image-derived plate candidates remain confidence-bearing evidence for review rather than an automatic canonical key.

Raw identifiers, plate crops, registry payloads, and source-specific retrieval methods stay outside this article. The publishable engineering point is the confidence ladder: weak evidence may suggest a link, but only stronger evidence is allowed to collapse records.

Alert matching is pure; delivery semantics are explicit.

Watch filters are evaluated as side-effect-free predicates over a listing. A result is emitted only for a meaningful event, such as a new match or a price change that crosses the configured threshold. The event key includes listing, filter, and event identity, and a database uniqueness constraint suppresses ordinary repeats while still allowing a later price change to produce a new notification.

This is testable and practical, but it is not magical exactly-once delivery. If an external notification succeeds and persistence of the dedup record fails, a retry can send twice. The single-runner lock narrows that window; a transactional outbox and provider idempotency key would close it more rigorously. Naming that semantic is part of operating the system responsibly.

Five lessons that transfer beyond automotive data.

Practically, those principles support deal triage, price-change monitoring, market research, parser quality review, auction analysis, identity timelines, and source-health operations. The same architecture also applies to property listings, procurement feeds, marketplace risk, and other vertical datasets where the inputs are claims rather than measurements.

The next milestone is calibration.

The highest-value work is no longer another source adapter. It is a measured feedback loop: turn analyst corrections into parser fixtures, estimate precision and recall by field, gate or weight cohort samples using observed extraction quality, and track score stability across time. On the modeling side, robust linear algebra, regularization where justified, time-split validation, and prediction intervals would make the regression path easier to trust and easier to reject.

Identity matching needs the same discipline: privacy-preserving candidate generation, explicit evidence grades, and human review before fuzzy links become canonical. Alert delivery can move to a transactional outbox with provider-level idempotency. These are not glamorous additions, but they are what turns a useful internal ranking system into a defensible intelligence platform.