# Unified intelligence engine — VECTOR v5

Prediction research is now part of the main engine. `/api/live` attaches its technical context and historical evidence directly to each BUY/SELL side. There is no separate Prediction Lab panel. The main terminal includes an **Engine evidence & trade history** section for deeper inspection.

Heavy work remains on a background execution path, not in the trade submission path. One `MT5_Intelligence_Observer.mq5` on a second chart handles calendar, trade-history ingestion and intelligence updates. It replaces the older prediction observer; do not run both. `MT5_Web_Bridge.mq5` stays on the first chart for quotes and orders. These are components of one deployed application, account, storage namespace and dashboard—not separate trading systems.

## Installation and upgrade

1. Keep execution disabled while upgrading. Back up the existing order ledger and research data. Deploy the entire package; new routes are `/api/intelligence`, `/api/analytics`, and `/api/model`.
2. Add settings from `.env.example`. `ANALYTICS_ENABLED=true` enables evidence capture. `AUTO_TRAIN_SHADOW=true` allows automatic experimental training; neither enables trading. `NEWS_GATE_MODE=block` remains the explicit calendar protection setting. Keep the existing role tokens and account binding.
3. Compile `MT5_Intelligence_Observer.mq5` in MetaEditor, inspect warnings, remove the old `MT5_Prediction_Observer` from its chart, and attach the new observer there. Configure matching CloudURL, BridgeToken, BridgeID and ExpectedLogin. Use PollSeconds=60, HistoryDays=90, RescanHours=24 initially. The existing HTTPS WebRequest allowlist is reused. This EA has no trade functions and needs no DLL.
4. Keep the main bridge attached. Feature cache version changes to 2.1.0; the next delta packet may request a full history resync. The existing bridge handles that response. The live ranking and execution thresholds are unchanged.
5. Verify the calendar, deals and intelligence worker on a demo account. The observer sends up to 100 deals per request and advances its local cursor only after HTTP 200. Daily rescans detect corrections in the configured history window. Broaden HistoryDays for older positions and commission adjustments; imports cannot recover history the broker does not provide.
6. Connect the web terminal. The BUY/SELL card shows current technical context, same-policy paper sample counts and experimental model output only when a compatible fresh forecast exists. The deeper evidence section separates broker trades from paper results. No probability is fabricated for a new installation.
7. Add trusted, permitted RSS/Atom URLs if desired, as described in NEWS_SETUP.md. The unified observer also updates the news pipeline. Legacy `/api/research` remains compatible but is no longer polled by the main UI or required by the new observer.

## What has been implemented

### Technical challenger features

- ATR-normalised EMA20 slope and acceleration, EMA20–50 separation and 20-bar directional efficiency.
- Trend/range/transition and low/normal/high volatility categories, with explicit heuristic thresholds.
- Confirmed swing sequences and displacement breaks through pivots that were already confirmed at the time; later closed-bar retests are distinguished from the break.
- Zone source timestamp, age, touches, depth and a transparent quality score. No source timestamp means unknown quality.
- H1/H4 directional context with M5 agreement as an entry-confirmation feature.
- Prior UTC day/week high/low observations and current session levels. Coverage is shown; incomplete history is not presented as a complete level set. UTC boundaries are research conventions, not exchange calendars.
- Per-instrument/session spread baselines from the last 200 prior M15 samples, with a 30-sample minimum. A heuristic challenger uses a prior 90th-percentile spread limit. Session thresholds remain unknown during warm-up.
- A challenger eligibility flag and score recorded beside the original technical decision. These do **not** change the existing execution gates or imply proven improvement.

### Durable records

Account-scoped, append-only records: `forecast`, `outcome`, `analysis`, `candle`, `market`, `deal`, `command`, `model`, `experiment`. They have no application TTL. Configure persistent, no-eviction storage and backups. No-TTL Redis is not an immutable offsite audit service; loss of the database still loses unbacked data.

Closed-candle histories are stored as daily full anchors plus changes, with parent links and reconstruction checksums. Corrections trigger a full anchor when deltas would not reproduce the source. Forecast input records preserve the quoted price, broker specifications, peer currency strength and calendar snapshot. M5 outcome paths are retained separately. This avoids repeating all 260 candles on every observation, but long-term evidence still consumes storage; monitor usage and export regularly.

A first-write-wins forecast ID and a separate terminal outcome prevent retries from relabelling old predictions. Missing data and ambiguous candle paths are explicitly excluded. A five-minute background-worker lease prevents ordinary concurrent capture. If a worker crashes, the next attempt resumes idempotently after expiry. This does not provide distributed transaction guarantees across all archive and state writes; backups and export are still required.

The dashboard displays the latest 200 records per stream, explicitly labelled. Complete exports use insertion cursors, not page offsets, so new records do not reorder earlier pages. Reports from a limited window can have incomplete position histories; those are excluded. Use a complete export for full-history analysis.

### Actual broker trades

Imports deal ticket, position/order IDs, volume, entry/exit classification, broker timestamp, fill price, profit, commission, swap, fee, magic, reason and comment. Corrections are retained as new versions; reports use the latest imported version per ticket. Deposits/withdrawals and standalone charges are separate account operations, not winning/losing trades.

Partial entries and exits are grouped by position ID. Missing openings, inconsistent volumes/directions and netting reversals are flagged/excluded rather than guessed. Current implementation does not allocate reversal deals into separate trading cycles. Standalone account commissions remain unallocated unless the broker associates them with trades.

R-multiples and requested-versus-filled slippage require an exact acknowledged **deal-ticket** match to one recorded opening deal with the expected volume. Prefix comments, order-ticket ambiguity, multiple additions and partial opening fills do not establish an exact risk budget. Such trades retain net cash P&L but show unknown R. Original technical/news context is recorded on new execution commands and archived by the worker; pre-upgrade trades do not have reconstructed indicator values invented for them.

Exit reasons are preserved. Stop-modification history and the intent behind manual interventions are not imported. Actual trade MAE/MFE requires a separate, time-aligned bid/ask path; a trade statement alone cannot provide it.

### Forecast outcomes and analysis

Forward paper forecasts continue to use the fixed 2.5R/12-future-M15 research policy, independently of your actual discretionary exits. Candle evaluation retains the conservative frozen-spread approximation and excludes entry-bar/both-hit ambiguity. Actual broker trades, forward paper observations and historical replay are labelled as separate evidence streams.

Reports include symbol, direction, session, regime, volatility, zone type, score bands, news status, technical-only, technical-plus-news, challenger-plus-news and omit-one-filter cohorts. Policies and streams are grouped separately. Statistics include mean gross/net R, win rate, profit factor, closed-trade drawdown, sample counts and descriptive uncertainty. Naive intervals are not dependence-adjusted; the offline report also supplies UTC-day block-bootstrap intervals when at least 20 observed days exist. Neither replaces broader robustness assessment. Closed-trade drawdown excludes intratrade floating losses.

### Automatic shadow models

The unified worker checks at most hourly once at least 200 same-policy terminal outcomes exist. It uses up to the most recent 1,000 outcomes. Logistic regression and a separate one-dimensional logistic probability calibrator are implemented in Python's standard library; no client dependencies or external model API are required.

Decision timestamps are split chronologically 60% / 20% / 20%; same-time observations stay together. Labels crossing a partition boundary are purged. Minimum sizes after purging: 100 training, 30 calibration and 30 test observations, with both outcome classes in every partition. Scaling is fitted on training data only. Timeouts count as non-target outcomes.

Test Brier score, log loss, reliability bins and a training-prevalence baseline are saved. A model must beat that baseline on both scores to be labelled a shadow candidate; otherwise probability display is withheld. Every attempted training run after sufficient data is logged, including withheld runs. These minimums and comparisons do not establish statistical significance or live profitability. Repeated hourly evaluations are exploratory and can lead to selection bias; reserve a genuinely untouched period before promotion.

Models have no authority to change live gates, weights, position size, stops or orders. New probabilities only appear for compatible forecasts made **after the end of the model's entire evaluation period**, and are suppressed when the main card's decision candle changes or the worker is stale. A rolling forecast can remain active for hours while the technical context changes; in that case current technical features remain available but probability is withheld.

Estimated net expectancy uses model target probability and training-period conditional net-R means only if costs were configured; it is a shadow estimate, not a profit forecast for discretionary exits. No model has been trained on your real data in this delivery.

## Offline tools

From the extracted package root, use Python 3.12+; no pip installation is needed.

Export the complete archive (set `VECTOR_ORIGIN` to your HTTPS application origin and `OPERATOR_TOKEN` in your private shell environment):

```bash
python tools/analytics_cli.py export --out archive.json
python tools/analytics_cli.py analyze archive.json --out history_report.json
```

Train a specific policy independently and include expanding-window evaluations:

```bash
python tools/analytics_cli.py train archive.json --policy POLICY_ID_FROM_OUTCOMES --out challenger.json
```

Add `--publish` to import the shadow model/report into your own application using the operator token. Imported reports are labelled operator-supplied, not independently certified by the API. They cannot enable live trading. The automatic background trainer uses the same chronological train/calibration/test procedure; the CLI additionally runs expanding-window evaluation blocks.

### Tick data and historical replay

`MT5_Export_Ticks.mq5` is an optional **script**, not another continuously attached EA. Compile it under MQL5/Scripts and run for one symbol and a verified historical broker UTC-offset period. It exports JSON under MQL5/Files, including `ts` (UTC, converted with the explicitly supplied historical offset), raw broker `time_msc`, bid and ask. Use the correct historical offset and split exports across DST transitions. Do not blindly use today's offset for old ticks. Limit each run to at most 31 days; smaller ranges reduce terminal memory use. Export is only as complete as the broker's available history.

Two replay input formats are supported:

1. `{ "snapshots": [...], "ticks": { "XAUUSD": [...] } }`. Each snapshot uses the documented bridge packet schema with contemporaneous account/contract metadata and 220–300 **closed** candles for each of M5/M15/H1/H4. `ts` must be increasing. Ticks contain `ts`, `bid`, `ask`. Current broker specifications must not be silently passed off as historical specifications.
2. `{ "archive": <complete archive export>, "ticks": { "XAUUSD": [...] } }`. Add `--archived`; this reconstructs stored candle chains and reruns the engine on recorded decisions using their as-of currency strength and calendar. It is replay of observed decisions, not a substitute for collecting years of historical snapshots.

```bash
python tools/analytics_cli.py replay replay_input.json --commission-per-lot 7 --slippage-ticks 2 --max-gap-seconds 60 --out replay_results.json
python tools/analytics_cli.py replay recorded_input.json --archived --commission-per-lot 7 --out recorded_replay.json
```

The numerical values above are examples, not broker cost recommendations. Commission is round-trip account currency per lot. Entry is the recorded executable quote; configured adverse slippage is applied to the exit. BUY exits use bid, SELL exits use ask. Stops fill at the observed breached price, allowing losses beyond 1R. A target receives no favourable gap credit. Tick-gap/horizon coverage failures are excluded. MAE/MFE describe sampled ticks, not unseen prices between them. Post-stop target reach is examined only when the subsequent original-horizon tick path is adequately covered. Sparse ticks cannot establish exact execution order or latency.

For actual trade excursions:

```bash
python tools/analytics_cli.py analyze archive.json --ticks broker_paths.json --out history_with_paths.json
```

Here `broker_paths.json` maps **broker symbol names** (e.g. `GOLD`, not necessarily `XAUUSD`) to ordered rows with `time_msc`, bid and ask, using the same broker time basis as deals. The exporter can be run with CanonicalSymbol set to an alphanumeric broker name, or the JSON mapping key can be changed. Excursions require a known original stop and exact entry link. Partial-exit paths are whole-position lifetime price excursions, not volume-weighted cash exposure.

## Validation and limitations

Automated tests exercise cost accounting, partial closes, orphan/reversal exclusion, account isolation, archive idempotence, future-data rejection, chronological purging, train-only scaling, model withholding, tick outcomes and HTTP/UI integration. Synthetic tests are not trading results. No live terminal, real Redis/Lua integration, broker feed, MetaEditor compilation or cloud deployment has been available for acceptance testing.

The unified observer performs synchronous HTTP requests in its own EA thread. The trade bridge remains independent. Measure actual worker duration, cloud function limits and Redis traffic before leaving it unattended. An interrupted worker may retry; durable deal IDs and forecast IDs prevent ordinary duplicate records. The new archive increases storage requirements. This release is an engineering implementation, not a certified production trading system or validated prediction model.