CHANGELOG-PROPOSAL.txt
2 days ago
QA-LOOP.md
2 days ago
STUDY-CONSOLIDATION.md
2 days ago
STUDY-MCP.md
2 days ago
STUDY-WORKSPACE.md
2 days ago
STUDY-WPAI-LEARNINGS.md
2 days ago
STUDY-CONSOLIDATION.md
97 lines
| 1 | # STUDY-CONSOLIDATION.md — AI Engine consolidation roadmap |
| 2 | |
| 3 | > **Living planning doc.** Born 2026-06-29 from a strategic review (chatbot + MCP + WP7) after the |
| 4 | > Gemini Interactions engine shipped in 3.5.6 with four silent regressions nobody caught until we |
| 5 | > drove the live API by hand. The lesson: **product breadth is outrunning verification, docs, and UX.** |
| 6 | > This phase is **consolidation, not another feature wave** — done little by little, one item per |
| 7 | > few pulses, surfaced via `/pulse`. Read `STRATEGY.md` and `STUDY-MCP.md` first for posture. |
| 8 | |
| 9 | ## Core principle |
| 10 | |
| 11 | Do **not** freeze chatbot features (unrealistic for a solo continuous shipper). Instead put a |
| 12 | **verification gate** and a **param schema** in front of new work, so features stop adding unverified |
| 13 | surface. Consolidate the main paths incrementally behind that gate. The threat is not competitors; |
| 14 | it is complexity outrunning verification. |
| 15 | |
| 16 | ## The evidence this is needed |
| 17 | |
| 18 | 3.5.6 shipped the Gemini Interactions engine as the **default** for all Google users with: dropped RAG/ |
| 19 | content-aware context, dropped chat history (restored discussions), broken PDF-in-chat, and a broken |
| 20 | connection test. All four were invisible to code review and the test suite; only live API driving caught |
| 21 | them. Fixed in commit `f93d978e`. This is the proof, not a hypothetical. |
| 22 | |
| 23 | ## Backlog (ranked — pull the highest-value item that fits the time) |
| 24 | |
| 25 | - [~] **1. Pre-release smoke gate for the chatbot/engine path.** Highest leverage. The `labs/tests` |
| 26 | harness we used would have caught every 3.5.6 regression. Make a runnable battery (text, history, |
| 27 | context/RAG, image, function-call non-stream + stream, file/PDF) across each provider env, run before |
| 28 | any release. This is the structural fix for the failure mode above. |
| 29 | **Slice one DONE 2026-07-02**: `node labs/tests/test-smoke.js` drives every configured AI env through |
| 30 | CONNECT (/ai/test_connection), TEXT, HISTORY (messages array), and STREAM (SSE live + end events), |
| 31 | which covers the 3.5.6 connection-test and dropped-history classes directly. Models are picked |
| 32 | dynamically from settings (overridable via `smoke.models` in query-api.local.conf). First live run: |
| 33 | 6 envs (OpenAI, Anthropic, Google, OpenRouter, OVH, Mistral), all green, exit code wired for CI-style |
| 34 | use. Run before any `pnpm zip`. |
| 35 | **Slice two DONE 2026-07-03**: FUNC (non-stream via /simpleChatbotQuery) + FUNCSTRM (streamed via |
| 36 | mwai-ui/v1/chats/submit with a guest nonce from /start_session) run function calling through the real |
| 37 | chatbot pipeline against fixture bots (botId starting with "smoke", mapped per env; smoke-openai + |
| 38 | smoke-anthropic created on ai.nekod.net, wired to the Code Engine "[Test] Get Magic Word" snippet). |
| 39 | Validated against the real regression: with the 84ecb22b Sonnet 5 fix reverted, FUNCSTRM fails with |
| 40 | the exact user-reported "each thinking block must contain thinking" error; with the fix, all green. |
| 41 | Checks retry once so transient provider slowness (OVH) doesn't fail the gate. |
| 42 | **Slice three, MCP round-trip DONE 2026-07-23**: `node labs/tests/test-smoke.js mcp` (also runs on |
| 43 | full runs) drives the real MCP Streamable HTTP transport (bearer self-discovered from settings): |
| 44 | MCPINIT handshake, then wp_create_post/wp_update_post_meta with nested-array + unicode meta read |
| 45 | back per-key and deep-compared (a double-serialized value is called out explicitly), block-JSON |
| 46 | `\u` escapes + shortcode + unicode content through wp_get_post, fixture post force-deleted after. |
| 47 | Covers the corruption classes Dave Hilditch's PSA documented (f3a2f4fd/fcda9415, cb806da4, |
| 48 | 600d86eb; window was 2.8.5 → 3.5.9). Validated against the real bug: with `maybe_serialize()` |
| 49 | reintroduced in wp_update_post_meta, METAUPD fails with "DOUBLE-SERIALIZED"; with the fix, all green. |
| 50 | **Slice three, RAG-through-chat DONE 2026-07-24**: RAGCHAT drives /simpleChatbotQuery on the |
| 51 | smoke-rag fixture bot (wired to the new Internal (WordPress DB) vector env, intern01 on |
| 52 | ai.nekod.net) and expects a fact that only exists in the knowledge base. Validated against a |
| 53 | simulated dropped-context regression. **Slice three remaining TODO**: image-in-chat and |
| 54 | PDF-in-chat; printed as SKIP so the coverage gap stays visible. |
| 55 | - [ ] **2. Gemini external MCP consumption.** OpenAI (`openai.php:~403`) and Anthropic (`anthropic.php`) |
| 56 | consume `$query->mcpServers`; the new default Google engine (`google-interactions.php`) does NOT — it |
| 57 | only wires Google's built-in tools. MCP is the core moat, so the default Google engine being unable to |
| 58 | use it is a real gap. Best shape: a provider-neutral MCP-to-function bridge. See |
| 59 | `project_gemini_no_external_mcp` memory. |
| 60 | - [ ] **3. Schema-driven chatbot params.** Replace the manual `MWAI_CHATBOT_FRONT_PARAMS` / |
| 61 | `MWAI_CHATBOT_SERVER_PARAMS` + 4-location sync (documented as a footgun in CLAUDE.md) with one |
| 62 | schema-driven model. Shrinks the regression surface; more ROI than a full `chat_submit()` rewrite. |
| 63 | - [ ] **4. MCP capability governance + onboarding.** MCP = remote admin automation, not "just a chatbot |
| 64 | feature." Role presets, mutating-tool audit log default-on (easy off), per-client/tool allowlists, and |
| 65 | onboarding that makes the admin power level unmistakable. `require_approval => never` (openai.php:440, |
| 66 | realtime.php:387) makes server-side allowlists + warnings more important. |
| 67 | - [ ] **5. Chatbot hygiene pass.** Remove/guard the `[ERROR]` test path (`ChatbotContext.js:985`); narrow |
| 68 | the `eval()` function-action surface (`ChatbotContext.js:589`, `:840`); audit the error-HTML |
| 69 | `dangerouslySetInnerHTML` rendering with a tight sanitize pass. |
| 70 | - [ ] **6. WP7 directory reconciliation.** `STRATEGY.md` says WP7 code lives in `/labs/wp7-integration/`, |
| 71 | but the actual files are flat `labs/wpai-*.php`. Reconcile the doc or move the files before this area |
| 72 | grows. Cheap; do it before the WP7 surface expands. |
| 73 | - [ ] **7. (Longer-term) Incremental `chat_submit()` / `ChatbotContext` decomposition.** Split the |
| 74 | monster submission pipeline behind the smoke gate — incrementally, NOT a big-bang rewrite (the rewrite |
| 75 | is itself a regression risk). Only after items 1 and 3 land. |
| 76 | |
| 77 | ## Already-decided (the review re-derived these — confirmations, not new work) |
| 78 | |
| 79 | - Deprioritize Realtime + Assistants unless users are loudly asking (`feedback_deprioritized_bugs`). |
| 80 | - MCP URL-token fallback is temporary; re-eval after 2026-12-27 (`project_mcp_url_token_revisit`). |
| 81 | - WP7 = stable connector layer for a moving framework; Abilities bridge planned (`project_wp7_mcp_eta`). |
| 82 | - Keep **image-only** behavior for image-gen Gemini models this release (no inferring mixed text+image |
| 83 | from chat text — it would risk the verified image fix). |
| 84 | |
| 85 | ## Progress Log |
| 86 | |
| 87 | - 2026-06-29: Roadmap created from the chatbot/MCP/WP7 strategic review. Baseline established; nothing |
| 88 | ticked yet. Context: Gemini Interactions hardening (commit f93d978e) just closed the regressions that |
| 89 | motivated this. |
| 90 | - 2026-07-02: Item 1 slice one shipped. `labs/tests/test-smoke.js` gates CONNECT/TEXT/HISTORY/STREAM |
| 91 | across all 6 AI envs in one command (~2 min, all green on first full run). Slice two (RAG, function |
| 92 | calling, PDF) needs site fixtures and is listed as SKIP in the gate output. |
| 93 | - 2026-07-03: Item 1 slice two shipped. FUNC + FUNCSTRM cover function calling (non-stream + streamed) |
| 94 | through the real chatbot pipeline via smoke-* fixture bots. Proven against the same-day Sonnet 5 |
| 95 | streaming bug (84ecb22b): gate fails with the fix reverted, passes with it. Full run: 27 passed, |
| 96 | 0 failed, 9 skipped (envs without fixture bots). Retry-once added for flaky providers. |
| 97 |