CHANGELOG-PROPOSAL.txt
2 days ago
QA-LOOP.md
2 days ago
STUDY-CONSOLIDATION.md
2 days ago
STUDY-MCP.md
2 days ago
STUDY-WORKSPACE.md
2 days ago
STUDY-WPAI-LEARNINGS.md
2 days ago
QA-LOOP.md
110 lines
| 1 | # QA-LOOP.md |
| 2 | |
| 3 | A periodic, self-driving QA loop for AI Engine. Each run picks **one fresh corner** of the |
| 4 | plugin, tries hard to break it, and either confirms it is solid or proposes a small fix. |
| 5 | Run it from time to time, let it tick every 30 minutes, and read the summaries. |
| 6 | |
| 7 | ## How to run |
| 8 | |
| 9 | ``` |
| 10 | /loop 30m Run one QA-LOOP round on AI Engine (see QA-LOOP.md): pick an area NOT in the |
| 11 | rounds log below, try a feature/model/environment/UI/code path, be creative, and try to |
| 12 | trick it. Prioritise real user-facing friction over security. Report the result and, if you |
| 13 | find something, propose the smallest fix. Append the round to the log at the bottom of QA-LOOP.md. |
| 14 | ``` |
| 15 | |
| 16 | Stop it anytime by asking to stop the loop (the cron is session-only and also auto-expires after 7 days). |
| 17 | |
| 18 | ## Why this loop exists |
| 19 | |
| 20 | The thing that quietly kills a plugin is not a dramatic bug. It is a user who tries something, |
| 21 | it does not work nicely, and they drop it **without ever saying a word**. This loop hunts for |
| 22 | that: the rough edge, the corrupted output, the confusing error, the "why is this off by one line." |
| 23 | |
| 24 | **Security is deliberately not the focus.** Multiple security teams already watch AI Engine. |
| 25 | Over-hardening just adds complexity and new edge cases for real users. If a round wanders into |
| 26 | security, note it and move back to friction and correctness. |
| 27 | |
| 28 | ## Principles |
| 29 | |
| 30 | - **One area per round.** Different from everything in the rounds log. Breadth over depth. |
| 31 | - **Try to trick it.** Malformed input, truncated streams, weird Unicode, huge numbers, empty |
| 32 | values, unusual-but-real model output. Think like a confused user or a cheap model, not an attacker. |
| 33 | - **Confirming robustness is a win.** Not every round needs a fix. "This corner is solid" is a |
| 34 | valuable, honest result. Do not manufacture a fix to feel productive. |
| 35 | - **Smallest possible fix.** If a fix is complex, propose the simpler alternative instead. Match |
| 36 | the surrounding code; reuse patterns already in the repo (e.g. the `min(500, …)` cap convention). |
| 37 | - **Do not churn deliberate behaviour alone.** When a behaviour is intentional and the call is a |
| 38 | product judgement, present options and ask rather than deciding unilaterally. |
| 39 | - **Leave dated breadcrumbs.** For deprecated/dead paths, compat shims, or deferred decisions, add |
| 40 | a `// TODO: Re-evaluate after <today + 6 months>` (absolute date, never a version number). |
| 41 | - **Prove it.** Reproduce logic in Node/PHP CLI, or test against the live site, before claiming a |
| 42 | result. Read code to confirm a suspicion; do not report speculation as fact. |
| 43 | |
| 44 | ## Guardrails (hard rules) |
| 45 | |
| 46 | - Never touch `Version:` / `MWAI_VERSION` / readme `Stable tag:` / the readme `== Changelog ==` |
| 47 | (all owned by Nekofy). |
| 48 | - Never partial-POST `settings/update` — it replaces the entire options object and wipes envs/keys. |
| 49 | Always read-modify-write the full object. |
| 50 | - Never commit without explicit approval. Group related changes into one coherent commit; past-tense, |
| 51 | one-line messages ending with ".". No mention of the assistant. |
| 52 | - Exclude build bundles from diffs (`app/chatbot.js`, `app/index.js`, `app/vendor.js`, |
| 53 | `premium/forms.js`, `premium/library-search.js`, etc.). Revert orphan bundle rebuilds that have no |
| 54 | matching source change. |
| 55 | - No em-dashes anywhere. |
| 56 | - Format PHP with `pcf fix <file>` (AI-Engine-only tool). Check the bundle mtime before browser- |
| 57 | verifying a JS edit; `pnpm build` once if stale. |
| 58 | |
| 59 | ## Areas to rotate through |
| 60 | |
| 61 | Pick one that is under-covered. This list is a starting point, not a limit; invent new angles. |
| 62 | |
| 63 | - Chatbot rendering (markdown, code blocks, tables, lists, links, RTL, multibyte, emoji, HTML docs) |
| 64 | - Function calling / feedback loop / MCP tools (malformed args, multi-call, depth, timeouts) |
| 65 | - Models and providers (OpenAI, Anthropic, Google, OpenRouter, Perplexity, custom OpenAI-compatible) |
| 66 | - Environments and model resolution (bare model ids, wrong env, missing pricing) |
| 67 | - Embeddings / Knowledge / vector DBs (Pinecone, Qdrant, Chroma) / Smart Search |
| 68 | - AI Forms (placeholders, multi-file upload, submission) |
| 69 | - Discussions / conversation memory / title generation / history limits |
| 70 | - Usage stats / cost calculation / guest and user limits / the limit message UX |
| 71 | - Streaming / abort / mid-stream errors / stop button |
| 72 | - Content-aware / placeholders / templates |
| 73 | - Admin UI (settings, Playground, chatbot builder, Insights) |
| 74 | - Files / uploads / transcription / image generation and editing |
| 75 | - REST endpoints / parameter validation / sanitisation boundaries |
| 76 | - i18n and translatable strings |
| 77 | |
| 78 | ## Test environment |
| 79 | |
| 80 | - Live site: `https://ai.nekod.net/` (wp-admin available; local via `/etc/hosts`). |
| 81 | - Guest nonce: `POST /wp-json/mwai/v1/start_session` → `restNonce`. |
| 82 | - Chat submit: `POST /wp-json/mwai-ui/v1/chats/submit`. |
| 83 | - Guest usage limits are enforced; to bypass for a test, drop a temporary mu-plugin using the |
| 84 | `mwai_stats_credits` filter gated on a custom request header, and remove it right after. |
| 85 | - Env ids: OpenAI `9nx9mjyd`, Anthropic `e7944erg`, Google `q6ve6g9k`, Internal Test `intern01`. |
| 86 | - Read-only MCP tools (`wp_get_post`, `wp_get_posts`, `mcp_ping`, …) are safe for probing; never use |
| 87 | write/destructive MCP tools against the live site during a probe. |
| 88 | |
| 89 | ## Report format (per round) |
| 90 | |
| 91 | 1. **Area + trick:** what was tested and how you tried to break it. |
| 92 | 2. **Result:** solid (clean bill of health) or issue found. |
| 93 | 3. **If an issue:** severity, the smallest fix or a proposal, and whether it was applied or deferred. |
| 94 | 4. **Log it:** append a one-line entry below. |
| 95 | |
| 96 | ## Rounds log |
| 97 | |
| 98 | Append newest at the bottom. Keep entries to one line so future rounds can scan and avoid repeats. |
| 99 | |
| 100 | - 2026-07-25 Gemini function-call result format + retired-model error hint — fixed both. |
| 101 | - 2026-07-25 Multibyte / CJK / emoji end-to-end pipeline — solid (mb-aware length, exact round-trip). |
| 102 | - 2026-07-25 Bare-URL underscore mangling in chatbot — fixed (URL placeholder protection). |
| 103 | - 2026-07-25 Content-aware injecting non-viewable posts — fixed (publicly-viewable / read_post guard). |
| 104 | - 2026-07-26 Markdown tables / lists rendering — fixed stray `<br/>` before nested lists; `1)` lists noted. |
| 105 | - 2026-07-26 Placeholder engine `[N/A]` catch-all — removed (corrupted content-aware pages + templates). |
| 106 | - 2026-07-26 MCP list-tool limits (`wp_get_posts` etc.) — capped at 500 to avoid big-site timeouts. |
| 107 | - 2026-07-26 Parameter validation (temperature / reasoning / maxTokens) — solid; temp clamp-to-1 noted. |
| 108 | - 2026-07-26 Usage cost for unknown / custom models — solid (records tokens, price null, no crash). |
| 109 | - 2026-07-26 Function-call streaming argument robustness — solid; legacy `function_call` path TODO'd. |
| 110 |