=========================== Migrating to ActingWeb 3.13 =========================== .. warning:: This guide shows the API **as it was** during this migration and must not be copied into new code. ActingWeb is now at a later version; see :doc:`../guides/index` for the current API. Start here ========== 3.13 bundles four largely independent pieces of work: a **property-list** storage change with pre-upgrade steps, an **MCP** behaviour change, an **MCP security fix**, and a **DynamoDB scalability** change with a required backfill. Which of them you need depends only on where you are coming from. Upgrading from 3.12.x or earlier -------------------------------- **Do all four sections, and do them in the order below**, because two of them have work that must happen *before* you deploy the new version: 1. **Property lists** — step 1 sweeps for pre-existing damage and **must run on the old version**. 3.13 turns damage that was previously invisible into a visible error, so a sweep afterwards tells you less than a sweep before. 2. **Security: MCP trust-cache bypass** — the "find trust rows that predate exact resolver matching" audit is also a before-you-upgrade step. 3. **DynamoDB scalability** — the reverse-lookup backfill is **required** for any deployment using reverse lookup, and has a rollback window worth understanding before you start. 4. **MCP structuredContent** and the remaining behaviour changes — these need code changes on your side if you are affected, but no pre-upgrade sweep. If you do not use list properties, do not expose MCP, and do not use reverse lookup, the only thing left that can affect you is "Property reads now raise on backend faults" — read that and you are done. Upgrading from a 3.13 release candidate --------------------------------------- Do the sections that were added after the rc you are running. Nothing in this table is optional for the rcs it names — the earlier rcs' pre-upgrade steps still apply if you never did them. Every row names sections by their exact title, so the table works as a checklist and not only as prose. .. list-table:: :header-rows: 1 :widths: 18 82 * - You are on - Still to do * - ``rc1`` - Everything except the reverse-lookup **backfill**, which you have already run: ``Property lists``, ``Migration refuses damaged lists``, ``Security: MCP trust-cache authorization bypass`` and ``MCP: structuredContent is now opt-in`` are all new to you. Also re-read the rest of the DynamoDB section, because three of its changes touch a code path rather than only docs: ``register_diffs()`` now resolves subscriptions before consulting suspension state, ``create_subscription()`` invalidates the actor's cached subscription list, and actor deletion writes a tombstone before the wipe begins. * - ``rc2`` - Everything except the DynamoDB section, whose backfill you have already run: ``Property lists``, ``Migration refuses damaged lists``, ``Security: MCP trust-cache authorization bypass`` and ``MCP: structuredContent is now opt-in`` are all new to you. * - ``rc3`` - ``Property lists``, ``Migration refuses damaged lists`` and ``MCP: structuredContent is now opt-in``. **Run the step-1 sweep before upgrading** — see the ``rc4`` row for why. * - ``rc4`` - ``Property lists`` and ``Migration refuses damaged lists``. **Run the step-1 sweep before upgrading**: you are on the last release that hides pre-existing list damage. 3.13.0 surfaces it as a 409 instead of silently compacting past it, so a list that has been quietly broken for months starts erroring the moment you deploy — and the sweep is easier to reason about before that than after. * - ``rc5`` - ``Migration refuses damaged lists``, plus "What changed after ``rc6``" below. * - ``rc6`` - Only "What changed after ``rc6``" below. What changed after ``rc6`` -------------------------- Between ``rc6`` and ``3.13.0`` the release gained four changes. **None requires operator action, and there is no data migration between them** — but two are worth knowing about: - **A list-metadata fix that matters if you are mid-migration.** Before ``3.13.0``, a concurrent write could put ``format: 1`` back over a completed migration's format flip by writing a whole cached metadata dictionary back; migration then deleted the v1 rows and the list read back **empty, with no error**. If you have been running ``actingweb-migrate-property-lists`` against a live deployment on ``rc5`` or ``rc6``, run ``actingweb-verify-property-lists`` once after upgrading. There is no repair step to run — the fix is preventative — but a list that was destroyed this way is already gone and you would rather know. - **MCP revocation now evicts the in-process caches.** A revoked token, a deleted trust relationship or a downgraded permission previously kept working from a warm process for up to five minutes. It no longer does, **within the process that served the revocation** — a multi-worker or multi-container deployment still serves the stale answer from every *other* process until its TTL expires. If your threat model depended on the old behaviour being instantaneous, it never was. - **A repair-tool guard, added after consumer verification of GA.** ``--repair`` now refuses a v1 list that is damaged *and* carries v2-format rows, because that combination is a reverted migration rather than a hole, and repairing it strands the data while reporting success. If you swept with ``--repair`` on ``rc5`` or ``rc6``, see the warning under "Step 2" above. - **A stale-instance write now warns.** A ``ListProperty`` retained across a migration writes its next item in the old format's row shape, where nothing reads it — and ``verify()`` calls the list healthy. That now logs a WARNING. It affects code that stashes the list object; ordinary ``actor.property_lists.`` access builds a fresh one per use. The other two are an opt-in PostgreSQL diagnostic (``ACTINGWEB_PG_DELETE_DIAGNOSTICS``, off by default) and a change in how unsupported-``MCP-Protocol-Version`` rejections are logged: a lone rejection is now INFO rather than unlogged, and a sustained run from one origin escalates to WARNING once. If you alert on WARNING from ``actingweb.handlers.mcp``, that is a new line you may see — it means a client is retrying instead of falling back, which is worth investigating rather than silencing. .. warning:: **A green test suite is not evidence this release landed correctly.** The read-path and capacity fixes are invisible to functional tests — they change how much a request costs, not what it returns. Use the operation-profile recipe in "Proving the fixes actually landed" at the end of this document to confirm them against your own hot paths. Property lists: storage format, repair, and fail-fast reads =========================================================== .. note:: Applies to every deployment that uses **list properties** (``actor.property_lists.*``, ``/properties/`` on a list), and is independent of the MCP and DynamoDB sections. If you use only scalar properties, read just "Property reads now raise on backend faults" below and skip the rest. Read this section before upgrading, not after. One of the changes turns pre-existing, previously-invisible data damage into a visible error, so the order of operations matters. Why this release exists ----------------------- ``ListProperty`` could destroy data. ``insert()`` on DynamoDB overwrote every shifted row with the last value it read, so a single call into a non-empty list corrupted it. Backend read failures were swallowed and reported as "row absent", turning a transient fault into a permanent hole. Delete and insert were multi-write shift loops over a stored length counter with no transaction, so an interruption anywhere in the middle left a hole, a duplicate, or a length that disagreed with reality — and reads silently compacted past the damage, so nothing ever surfaced it. 3.13 fixes the write paths, adds repair tooling, makes reads fail loudly instead of quietly, and introduces a storage format in which the whole class of defect is structurally impossible. Step 1: sweep for existing damage BEFORE upgrading -------------------------------------------------- Any list damaged by the old code is still damaged. Releases before 3.13 hid that; 3.13 will not. Find out what you have first:: actingweb-verify-property-lists Read-only. It reports, per list, the recorded length, how many rows are actually readable, which indices are missing, which are orphaned, and any adjacent duplicate residue. .. warning:: **Run it with the same environment your application uses.** The library defaults ``AWS_DB_PREFIX`` to ``demo_actingweb``. If your deployment uses a different prefix — and if a demo deployment exists in the account — an unset prefix silently sweeps the wrong tables and reports them clean. A clean report from the wrong table looks exactly like good news. The script now prints the backend, region and prefix it is about to use, and warns when the prefix is an unset default; check that line before trusting the result. If your items carry an identifying field, pass it:: actingweb-verify-property-lists --identity-key id Duplicate detection otherwise compares raw stored bytes, which finds the duplicate an interrupted shift leaves — but stops finding it the moment either copy is edited. That false negative hits precisely the lists that have been used since the damage. ``--identity-key`` compares on the field that identifies the item instead, and keeps working after an edit. Step 2: repair what it found ---------------------------- :: actingweb-verify-property-lists --repair ``--repair`` calls ``compact()``, which rewrites the surviving rows densely, removes orphans, corrects the recorded length, and preserves ``description``/``explanation``/``created_at``. It closes holes. **It refuses a reverted migration, by design.** A v1 list that is damaged **and** reports ``foreign_format_rows > 0`` is not an ordinary hole. It is a list whose migration to v2 was reverted: the v1 rows are gone, the metadata still claims v1 with the old length, and **the items are almost certainly alive in the v2 rows**. Compacting it would rewrite the empty v1 range, set the length to what it found, report the list healthy, and leave the only surviving copy as unreferenced residue for the next ``clear()``/``delete()``/migrate re-run to sweep. So ``--repair`` skips those lists, logs a WARNING naming the shape, and counts them as still unhealthy. Recover them first — the v2 rows are readable directly, and re-running the migration against corrected metadata restores the list. ``--repair-reverted`` overrides the refusal and is the wrong answer unless you have looked at the rows and decided to abandon them. .. warning:: Earlier releases logged those rows at INFO as "harmless to reads; clear it by re-running ``--migrate``". In this shape that advice deletes the data. If you ran a sweep on ``3.13.0rc5`` or ``rc6`` and acted on that line, check the affected lists before assuming they are empty by design. **It does not resolve duplicates, by design.** A duplicate always means an item was destroyed; collapsing one copy would bless that loss as intentional rather than surface it. Duplicates are reported and left for you to decide about. Repair does not recover a destroyed item — nothing can; the row is gone. .. warning:: **Repair is not crash-safe, and an interrupted repair leaves damage it will not itself fix.** ``compact()`` rewrites surviving rows at their new positions before deleting the tail, so an interruption between the two leaves a copy at both the old and the new position. Measured on a 4-slot list with one hole: interrupting after the first move leaves ``[a, c, c, d]``, after the second ``[a, c, d, d]`` — both with the length still reading 4, so **the list reads back with no error at all**. ``verify()`` does catch it (the duplicate is adjacent and byte-identical, which is exactly what its heuristic looks for), so a follow-up sweep reports the list unhealthy. But re-running ``--repair`` will not remove the copy: duplicates are preserved by design, so the tool declines to touch the residue its own interruption created, and the list stays one item too long until you resolve it by hand. In practice: run repair when the actor is not taking writes, and **run the sweep again afterwards** rather than assuming it worked. The same shape applies to ``compact()`` on a v2 list, where it is a rank rebalance — see the guide's warning box. **What "not taking writes" does and does not buy you.** Nothing excludes an application write while a whole-list rewrite runs: there is no lock, and adding one is deliberately not part of 3.13. A concurrent write during ``compact()``, ``--repair`` or ``--migrate`` can still be lost, so the advice above is a real operational requirement rather than a nice-to-have. What 3.13 *does* guarantee is that such a write can no longer corrupt the list's **format** *from a stale cache*: a concurrent write updates only the metadata fields it changes, merged into a fresh read. In ``v3.13.0rc6`` and earlier it wrote a whole cached dictionary back, putting ``format: 1`` over a completed migration, and the result was total silent loss — see the CHANGELOG. The window is narrowed, not closed. It was unbounded in time (a retained ``ListProperty`` could revert a migration that finished hours earlier); it is now the gap between reading the metadata row and writing it back, since there is no compare-and-set to condition that write on. Quiescing writes is what closes the rest. **Update (3.14):** this remaining gap is closed. Metadata writes now condition on the exact bytes they read via a bounded compare-and-swap retry loop, so a migration that completes inside the old window is merged onto rather than overwritten. See the 3.14 migration guide and the property-lists guide's "Concurrency during a whole-list rewrite" section for the current behavior; this page is left as written for 3.13. Step 3: expect these breaking changes after upgrading ----------------------------------------------------- **List reads now raise on corruption instead of compacting past it.** ``to_list()``, ``slice()`` and ``to_list_from_rows()`` raise ``ListCorruptionError`` (an ``IndexError`` subclass) when an item inside the list's recorded length is missing from storage. Every HTTP path that serves list content returns **409** with ``{"error": "list_corrupted", "list": ..., "detail": ..., "remedy": "compact"}``. This is the change that makes step 1 non-optional: a hole that earlier releases silently skipped becomes a raised exception or a 409. Audit your call sites — anything that reads a list and cannot tolerate an exception needs a ``ListCorruptionError`` handler, and the underlying list needs repairing:: from actingweb.property_list import ListCorruptionError **Property reads now raise on backend faults.** ``DbProperty.get()`` raises ``actingweb.db.exceptions.DbError`` when the backend itself fails (timeout, throttle, connection error). ``None`` now means "the row does not exist", and only that. Code that relied on a fault degrading to ``None`` will now see an exception. This applies to scalar properties too, not just lists. **Every list mutation now checks its writes.** ``append``, ``__setitem__``, ``__delitem__``, ``insert``, ``clear``, ``delete`` and metadata writes raise ``RuntimeError`` instead of continuing past a failed write. **``GET /properties//items`` response shape changed.** Was a bare JSON array; is now ``{"items": [{"index": i, "item": ...}, ...], "count": n}``. The ``index`` is the same one ``action=update``/``action=delete`` accept, so GET and POST are consistent. (Flask gained this route for the first time; it previously existed only on FastAPI.) **``PUT /properties/?index=N`` beyond the list length returns 404.** It previously padded the list with ``None`` up to ``N`` — both a spec violation and an unbounded-write vector. ``index == length`` still appends. The bulk item POST gained the equivalent bound. **``index()`` honours negative ``start``/``stop``.** It previously ran ``range(start, ...)`` unnormalized, so ``index(value, -1)`` could return ``-1`` as an index. Both storage formats now match ``list.index``. Step 4: decide when lists are allowed to change format ------------------------------------------------------ .. danger:: **Automatic conversion of existing lists is OFF by default** (``ACTINGWEB_LAZY_MIGRATION_MAX_LENGTH=0``). Leave it that way for this release's deployment: no list changes format, so rolling back stays a pure code rollback with no data to reconcile. Convert later, deliberately. This does not make v2 opt-in -- every list created after the upgrade is v2 with no action from you. Only the conversion of data you already have waits for your go-ahead. The reason is not latency, it is rollback. A pre-3.13 process does not error on a migrated list -- it reads it as **empty**, silently. The metadata row still exists, so the list "exists"; a v2 list stores no ``length`` field, and an older reader takes the absence as zero. A write from that process then lands in v1 storage and the list **forks**: two versions, two disjoint views, nothing reporting an error. ``--downgrade`` cannot reconcile a forked list; it overwrites v1 storage with the v2 content and destroys whatever the older process wrote. Deployment gives you at most a brief mixed-version window. Rollback gives you no window at all: deploy, let lazy migration convert lists for hours or days, then roll back for an unrelated reason, and every list that migrated reads as empty in production. Recovery is ``--downgrade`` one list at a time -- there is no bulk mode -- from a v2-capable checkout, against a database served by the code you just rolled back to. That is not a procedure to discover during an incident. Migration forward is automatic, fleet-wide and inline. Recovery back is manual and per-list. The general rule behind the setting: **no list may become v2 until every process that might serve it can read v2, including the release you would roll back to.** Once 3.13 has been live long enough that rollback is off the table, migrate as a deliberate operator action:: actingweb-migrate-property-lists # dry run actingweb-migrate-property-lists --migrate **Repair (step 2) before you migrate, and treat the dry run's exit code as the gate.** Migration refuses a list with holes or orphans, and the dry run names them and exits ``1``; a dry run that exits ``0`` is telling you the migration has nothing to trip over. This matters more than it sounds: migration renumbers survivors, so a hole that goes through it stops existing *and* stops being reportable. See "Migration refuses damaged lists" below. Step 5: understand the new storage format ----------------------------------------- Every **new** list is created in a new internal format ("v2") that stores items under sort keys rather than dense integer positions. Delete and insert become single writes: there is no shift loop, no stored length to disagree with, and therefore no way to produce the holes and duplicates this release exists to clean up. Existing lists keep working unchanged, indefinitely, on the hardened old paths. Nothing forces you to migrate. **Lazy migration.** With ``ACTINGWEB_LAZY_MIGRATION_MAX_LENGTH`` set to a positive number, a healthy v1 list of at most that many items migrates on its next mutation. Three deliberate limits: - **Damaged lists are never migrated automatically.** Migration closes holes in flight, which is correct when an operator runs the bulk script and reads its report — but doing it silently under an ordinary ``append()`` would erase the evidence: the hole closes, the destroyed item stays destroyed, duplicate residue becomes indistinguishable from real data, and ``verify()`` starts reporting the list healthy. Repair is always your decision. Damaged lists keep serving v1 and keep raising until you act. - **It is off unless you enable it** (``ACTINGWEB_LAZY_MIGRATION_MAX_LENGTH`` defaults to 0) — see step 4 for why. It is also inline and synchronous when enabled: one ``append()`` to a 40-item list performs the entire migration inside that request. The threshold is checked at the moment of the mutation, so "large lists stay v1 until I run the script" is not quite true: a ``clear()`` + ``extend()`` whole-list rewrite migrates a list of any original size, because the first ``append()`` inside ``extend()`` sees length 0. ``ACTINGWEB_LAZY_MIGRATION_MAX_LENGTH=0`` is the only setting that makes "nothing migrates without me" true. **Bulk migration** (step 4 above) is for lists too large or too idle to migrate lazily, and — with lazy migration off by default — for everything else too. It refuses damaged lists on the same grounds lazy migration does; repair them first. ``migrate_to_v2()`` is idempotent and safe to interrupt and re-run. **List names containing ``#`` are refused.** ``#`` is reserved for internal v2 storage keys. New lists cannot be created with it; existing lists that have it keep working as v1 forever and are refused by migration. Rename them if you want them migrated. They are correctly isolated from v2 lists whose name is a prefix of theirs, so this is a migration blocker, not a data risk. **Positional reads cost more under v2.** ``lst[i]`` re-reads the list's key ordering before resolving the position, because resolving against a stale ordering returns the wrong item. A ``for i in range(len(lst)): lst[i]`` loop is therefore two queries per item. Use ``to_list()``, ``to_indexed_list()`` or plain iteration — each is a single query for the whole list, in both formats. If you have such a loop, this is the one change likely to show up in your latency numbers. **Downgrade is unsupported.** ``migrate_property_lists.py --downgrade`` exists as an emergency converter only. If you use it, roll the application back to a pre-v2 release *first*. With lazy migration at its default of 0 this is "only" the ordinary rule that the reader must precede the data; if you have raised the limit, it is sharper — a downgraded list at or under it is a lazy-migration candidate again, so a still-running v2-aware application migrates it straight back on its next write. PostgreSQL: one schema migration -------------------------------- ``properties.name`` widens from ``VARCHAR(255)`` to ``TEXT`` (metadata-only, no table rewrite):: cd actingweb/db/postgresql/migrations && alembic upgrade head Apply it before running any v2 workload. The downgrade reverses it and will fail loudly if any name exceeds 255 characters at that point, which is intentional — downgrading the schema underneath live v2 keys should not silently truncate them. Repair API, if you would rather not use the scripts --------------------------------------------------- .. code-block:: python lst = actor.property_lists.notes report = lst.verify(identity_key="id") # read-only if not report["healthy"]: lst.compact() # closes holes, keeps duplicates ``verify()``/``compact()`` also work on v2 lists, for a different purpose there: detecting and rebalancing sort keys that have grown long from repeated inserts at the same position. Migration refuses damaged lists =============================== .. note:: A refinement of the property-list steps above, and it only affects you if you run ``actingweb-migrate-property-lists`` (step 4). Do the steps above first — they are the ones with the pre-upgrade work. ``actingweb-migrate-property-lists --migrate`` and ``ListProperty.migrate_to_v2()`` now **refuse a list with holes or orphans**, and the dry run reports those lists as needing repair instead of counting them as "would migrate". A dry run that finds any exits ``1``. The reason is that migrating damaged data is not just lossy, it is *unreportably* lossy. Migration renumbers the surviving rows, so the hole is gone afterwards — and so is the evidence. The migrated list verifies healthy, and nothing remains to say an item ever went missing. An earlier form of the dry run said nothing about holes at all, so a sweep over a fleet containing one could report "0 refused, 0 errors" and an operator would proceed in good faith, past the last moment at which the damage was still visible. Lazy migration has always refused damaged lists, for exactly this reason. The deliberate path now applies the same rule. What to do: **repair first** (step 2), then migrate. If you have looked at the damage and decided to migrate anyway, ``--migrate-damaged`` (or ``migrate_to_v2(allow_damaged=True)``) does it, and says clearly in the log what it is giving up. Duplicate residue is **not** affected — it never blocked migration and still does not. A duplicate stays visible after conversion, because v2's ``verify()`` reports duplicates just as v1's does, so migrating it destroys no evidence. Only holes and orphans gate. MCP: ``structuredContent`` is now opt-in ======================================== .. note:: Applies only if you expose MCP tools. If you do not, skip to the next section. Independent of the property-list section above and the security and DynamoDB sections below. What changed ------------ Previously, a tool hook returning a dict with ``content`` plus extra top-level keys had those extras **promoted** into ``structuredContent`` automatically. That promotion is removed. ``structuredContent`` is now emitted only when your hook sets that key explicitly, and only when its value is a JSON object. Why: at least one major MCP client discards **every** text content block when ``structuredContent`` is present. Under the old behaviour, adding a single scalar key to a hook's return value silently deleted that tool's entire prose payload — the model received only the promoted keys, with no error on either side. Neither reference server implementation promotes keys this way. Are you affected? ----------------- You are affected if any tool hook returns a dict with a ``content`` key **and** other top-level keys besides ``isError``, ``_meta`` and ``structuredContent``. Those extras were being serialized into ``structuredContent``; now they are dropped. You are **not** affected if your hooks already set ``structuredContent`` explicitly, return ``content`` alone, or return dicts with no ``content`` key (those take the legacy text-wrap path, which never emitted ``structuredContent``). How to migrate -------------- The migration is a no-op against the old release, so you can **migrate first and upgrade afterwards** — no coordinated deploy window. The explicit passthrough already exists in ``v3.11.0``/``v3.12.0``, so a hook updated as below produces byte-identical output on both. For a tool whose payload is genuinely structured, name the key and serialize the same object into the text block: .. code-block:: python # Before — relied on promotion return { "content": [{"type": "text", "text": f"Found {len(results)} results"}], "results": results, "count": len(results), } # After — explicit, and the text carries the same object payload = {"results": results, "count": len(results)} return { "structuredContent": payload, "content": [{"type": "text", "text": json.dumps(payload)}], } For a tool whose payload is **prose**, drop the extras instead of promoting them. That is the case the old behaviour broke, and returning prose alone is now correct. Two caveats the library cannot enforce for you ----------------------------------------------- 1. **Some clients ignore ``structuredContent`` entirely** (see ``modelcontextprotocol#1411``). Always keep the same data serialized in a text content block, per the specification's backwards-compatibility guidance. The text block is the only thing that always arrives. 2. **A request with no ``MCP-Protocol-Version`` header negotiates ``2025-03-26``**, where ``structuredContent`` is suppressed *even when set explicitly*. If you verify with ``curl``, send ``-H 'MCP-Protocol-Version: 2025-06-18'`` or you will wrongly conclude your migrated hook is broken. Two related fixes in the same release -------------------------------------- - **An explicit ``isError`` is now honoured on the legacy text-wrap path.** A hook returning ``{"isError": True, ...}`` with no ``content`` key previously reached the wire with no ``isError`` field at all, so the failure was reported to the client as a success. ``isError`` is honoured only when you set it — it is never inferred from an ``"error"`` key or any other heuristic, so apps that return ``{"error": ...}`` on their normal error path see no change. The text serialization is unchanged. - **A warning now fires when a tool declares ``output_schema`` but returns no ``structuredContent``.** Spec-conforming clients reject that result outright (``Tool X has an output schema but did not return structured content``). This was already broken before this release; it is now visible in your logs. Note that ``output_schema`` and ``structuredContent`` are independent — declaring a schema has never caused structured output to be emitted. Security: MCP trust-cache authorization bypass ============================================== .. note:: **Read this even if you skipped the MCP section above.** It applies in addition to every other section here. If you run MCP with a single client per actor, most of it does not apply to you — but the ``resources/read`` and resolver-matching items still can, so read the summary before skipping ahead. See the ``SECURITY`` section of ``CHANGELOG.rst`` for what changed and why. This section is the operational checklist: what to check *before* upgrading, and what to expect *after*. 1. Find trust rows that predate exact resolver matching -------------------------------------------------------- The MCP client-id-to-trust resolver now requires an exact ``oauth_client_id`` match, or — for legacy rows without that field — an exact reconstructed peer-id match gated on ``established_via`` being an OAuth2-family value. A row that matched only because the old resolver did substring containment will stop resolving after upgrading, and that client's next request returns an empty ``tools/list`` or a ``-32003`` on ``tools/call``/``prompts/get``/``resources/read`` instead of authenticating as before. Find at-risk rows before upgrading — any trust row where: - ``oauth_client_id`` is unset (``None``/absent), **and** - ``established_via`` is not ``"oauth2_client"`` (or the legacy peer id does not start with ``oauth2:`` or ``oauth2_client:``), **or** - the peer id is not in the three-segment ``::`` shape trust creation constructs. These are almost always trust rows created before ``v3.3`` (2025-10-04, when ``oauth_client_id`` was introduced) that have not been through an OAuth2 re-authorization since. Rows created by either live MCP path (client registration, or the MCP OAuth2 authorize callback) already carry ``oauth_client_id`` and are unaffected. 2. Audit Flask (sync) resource URIs against your trust type's patterns ------------------------------------------------------------------------ ``resources/read`` had **no authorization check at all** on the Flask (sync) transport before this release (see the ``SECURITY`` note in ``CHANGELOG.rst``) — every resource read was served regardless of trust type. It is now checked identically to the FastAPI (async) transport, which was never affected. The default ``mcp_client`` trust type only allows reading: ``public/*``, ``shared/*``, ``notes://*``, ``usage://*``, and ``actingweb://properties/all`` (see ``actingweb/trust_type_registry.py``). If your application serves custom resource URIs outside that pattern set to Flask-based MCP clients, those reads have been working *only because the check was dead* and will start returning ``-32003`` after upgrading. Audit your resource hooks' URIs against the ``resources.patterns`` of every trust type your app registers, and widen the pattern set (or register a custom trust type) before upgrading if legitimate traffic depends on a URI outside the default allowlist. 3. Understand the orphaning consequence of a resolver miss ------------------------------------------------------------- If a client's trust row stops resolving per step 1 above, that client does not stay locked out forever: the next OAuth2 authorization it completes creates a **new** trust row with a newly constructed peer id (since the old row is simply never matched, not deleted). This has one consequence worth knowing in advance: **per-peer permission overrides are silently abandoned.** Per-peer permission overrides are stored keyed by ``f"{actor_id}:{peer_id}"`` (see ``TrustPermissions.get_permission_key()`` in ``actingweb/trust_permissions.py``). Any overrides an operator applied to the old peer id do not transfer to the new row's peer id — the re-authorized client silently reverts to its trust type's *default* permissions. If you have applied custom per-peer overrides to any MCP client, check whether that client's trust row is at risk per step 1, and if so, re-apply the overrides to the new peer id after it re-authorizes. 4. Note the backfill timing for ``oauth_client_id`` ------------------------------------------------------- ``oauth_client_id`` is backfilled onto a trust row when a client completes an OAuth2 **re-authorization**, not merely on its next request (see ``actingweb/interface/trust_manager.py``). Between upgrading and a given client's next full re-authorization, that client's resolution rests entirely on the legacy peer-id fallback described in step 1 — so a row that fails the step-1 check does not "self-heal" on its own until the client goes through the OAuth2 flow again, not just makes another API call. DynamoDB scalability: reverse lookup, capacity, and table management ==================================================================== .. note:: This section and everything below it is the DynamoDB-scalability half of 3.13, and it is the one with the **required backfill**. It is independent of the property-list, MCP and security sections above; upgrading from 3.12.x means doing all of them. ActingWeb 3.13 is a **DynamoDB scalability release**. It fixes two measured superlinear cost defects (full-table scans on per-actor reads; a ``DescribeTable`` control-plane call on every accessor construction), removes hot-path read amplification, defaults auto-created tables to on-demand billing, and — the one change that affects behaviour for deployments that change nothing — **flips the default property reverse-lookup mechanism to lookup-table mode**, backed by a redesigned (v2, digest-keyed) lookup table. Nothing in this release requires code changes. What it may require is **one operational step**: running the lookup-table backfill script so reverse lookups (find-actor-by-email/oauthId) are served from the new table instead of deprecated fallbacks. How urgent that step is depends on how your app resolves the logged-in user — check rather than assume: - If it calls ``Actor.get_from_creator()`` (the ``creator-index`` GSI on the actor table), reverse lookup is **not** on your login path and deferring the backfill is low-risk. - If it calls ``get_from_property()`` / ``get_by_property()`` on ``email``, ``oauthId`` or another indexed property, that *is* your login path and it runs on deprecated fallbacks — or returns ``None`` — until the backfill completes. .. note:: **Validating this release requires a DynamoDB backend.** Every change here is DynamoDB-specific, so a green test suite run against PostgreSQL (``DATABASE_BACKEND=postgresql``) exercises none of it — an easy trap, because the suite passing reads as "validated". Run against DynamoDB Local or a real table; see `Rehearse the migration first`_. Which deployment am I? ---------------------- Find your row; do what its column says. "Implicit legacy" means you never called ``with_legacy_property_index()`` and never set ``USE_PROPERTY_LOOKUP_TABLE``. .. list-table:: :header-rows: 1 :widths: 34 33 33 * - State before 3.13 - After upgrading (no config change) - Action needed * - Implicit legacy; properties table **has** the ``property-index`` GSI (created by an older ActingWeb) - Reverse lookups keep working via the **deprecated GSI fallback** (a warning is logged per hit; an ERROR is logged at startup) - Run the backfill (below). Optionally delete the now-unused GSI to halve property write cost — only after verifying. * - Implicit legacy; properties table has **no** GSI (created before the GSI existed). Reverse lookups were **crashing** before 3.13. - Reverse lookups now work IF the lookup table is populated; until then they return ``None`` (startup ERROR tells you) - Run the backfill (below). * - Lookup mode already enabled (``with_legacy_property_index(False)``) on the **v1** lookup table - Reverse lookups keep working via the **deprecated v1 fallback** (warning per hit; startup ERROR) - Run the backfill, verify, then **drop the v1 ``_property_lookup`` table**. * - Explicit legacy (``with_legacy_property_index(True)`` or ``USE_PROPERTY_LOOKUP_TABLE=false``) - Unchanged: legacy GSI path stays active. If the table lacks the GSI you now get an actionable error instead of an opaque crash. - None immediately. Legacy mode is deprecated — plan the migration. * - Fresh deployment - Lookup-table mode, on-demand billing, no legacy GSI — the intended end state - None. Rehearse the migration first ---------------------------- The backfill is the riskiest step in this release and it is easy to treat as a one-shot production procedure. It isn't: the **whole** migration — new-table creation, the three-tier fallback, the backfill script, the startup tripwire — runs end-to-end against DynamoDB Local at zero cost and zero risk. Do it once before touching production:: docker run -d -p 8000:8000 amazon/dynamodb-local export AWS_DB_HOST=http://localhost:8000 export AWS_DEFAULT_REGION=us-west-1 export AWS_ACCESS_KEY_ID=fake AWS_SECRET_ACCESS_KEY=fake export AWS_DB_PREFIX=rehearsal Then, in order: 1. Start the app (or create a couple of actors with properties on the indexed names). Confirm the startup ERROR tripwire fires — it names the resolved lookup table and the exact script to run. 2. ``python -m actingweb.db.verify_tables`` — confirms which tables exist. 3. Run the backfill, then restart. The per-hit ``DEPRECATED: reverse lookup ...`` warnings and the startup ERROR should both stop. 4. Optionally flip ``AWS_DB_AUTO_CREATE_TABLES=false`` and confirm the app still serves. An operator who has done this once is not doing it for the first time in production. Required: run the backfill (all upgrading deployments using reverse lookup) --------------------------------------------------------------------------- The lookup table is **derived data**; the properties table is the source of truth. The script rebuilds the v2 table from it — streaming, rate-limited, resumable and idempotent:: # With the SAME environment as the app (AWS_DB_PREFIX, region, # credentials, INDEXED_PROPERTIES if customised): poetry run python scripts/backfill_property_lookup.py --dry-run poetry run python scripts/backfill_property_lookup.py --rps 50 Notes: - ``--rps`` caps items/second — a full-table scan against production should not brown-out serving traffic. ``--segments N`` parallelises. - Interrupted runs resume from ``--checkpoint-file``. - Values are copied **verbatim** (no normalisation) — runtime lookups are exact-match. - A value shared by two actors is reported as a **collision** and not overwritten (exit code 1). Resolve manually and re-run — re-runs are idempotent. Verify, then clean up: 1. Watch logs: reverse lookups should stop logging ``DEPRECATED: reverse lookup ... served from ...`` warnings, and the startup ERROR should disappear on the next restart. .. warning:: Do **not** verify the backfill with ``describe-table``'s ``ItemCount`` — it refreshes only about every six hours, so a freshly backfilled table still reports ``ItemCount: 0``. That reads as a no-op backfill when the rows are actually there. Use the real count:: aws dynamodb scan --table-name _property_lookup_v2 \ --select COUNT 2. **v1 cohort**: drop the old table — ``aws dynamodb delete-table --table-name _property_lookup``. (Never touch ``_properties`` — that is the source of truth.) 3. **GSI cohort (optional, saves ~half of property write cost)**: delete the legacy index:: aws dynamodb update-table --table-name _properties \ --global-secondary-index-updates '[{"Delete":{"IndexName":"property-index"}}]' Only after the backfill is verified: deleting the GSI removes the fallback the un-backfilled state depends on. Note that deleting the GSI makes a later return to legacy mode impossible without recreating the index. Rollback — check *before* you upgrade ------------------------------------- Setting ``USE_PROPERTY_LOOKUP_TABLE=false`` (or ``with_legacy_property_index(enable=True)``) restores the legacy path. (3.13 also fixed a bug where the env variable was silently ignored by apps using the fluent builder — the rollback now actually works.) .. warning:: **Rollback requires the legacy GSI to exist on your properties table.** Legacy mode queries ``property-index``; a properties table created before that index existed does not have it, and tables are never altered in place. Such a deployment has **no rollback path** for reverse lookup — worth knowing before upgrading, not after. Check first:: aws dynamodb describe-table --table-name _properties \ --query 'Table.GlobalSecondaryIndexes[].IndexName' ``null`` means no GSI, therefore no rollback. (The same applies after you delete the GSI in the cleanup step above.) Direct use of the lookup accessor degrades during the migration window ---------------------------------------------------------------------- The three-tier read path (v2 → deprecated v1 → legacy GSI) lives in ``DbProperty.get_actor_id_from_property()``. Code that constructs ``DbPropertyLookup()`` directly and calls ``.get()`` gets the **v2-only** path with no fallback, so between the upgrade deploy and a completed backfill it returns ``None`` while the application itself is perfectly healthy — which reads like "the migration broke actor lookup". Operational scripts (actor management, debugging, seeding, cleanup) are the usual offenders. Audit yours: resolve actors through ``get_actor_id_from_property()`` — or the interface-level ``ActorInterface.get_by_property()`` — which is the supported entry point and carries the fallbacks. Recommended: convert old auto-created tables to on-demand billing ----------------------------------------------------------------- Tables the library auto-created before 3.13 are PROVISIONED with tiny capacities — typically ``_property_lookup`` (2 RCU / 1 WCU: a hard wall on the login path) and ``_peertrustees``. New tables are on-demand; **existing tables are not changed automatically**. Convert them in place:: aws dynamodb update-table --table-name _peertrustees \ --billing-mode PAY_PER_REQUEST - AWS allows one billing-mode switch per table per 24 hours. - **Never delete and recreate a table to change billing** — the v1 lookup table (pre-backfill) and every other table hold live data. - Check for other PROVISIONED stragglers: ``aws dynamodb list-tables`` + ``describe-table``. Recommended: disable auto-creation in production ------------------------------------------------ If your tables are managed by CloudFormation/Terraform, turn off the library's table auto-creation and slim the runtime role:: AWS_DB_AUTO_CREATE_TABLES=false # env, or: app.with_dynamodb(auto_create_tables=False) With auto-creation off the library never calls ``DescribeTable`` or ``CreateTable``, so both can be dropped from the runtime IAM policy. If your IAM policy lists tables by name, add the new ``_property_lookup_v2`` table. Three things this hands you responsibility for. **1. Every required table must already exist.** The library will no longer create them, and — deliberately, so a role without ``DescribeTable`` does not pay an ``AccessDenied`` per accessor construction — it will not tell you one is missing. The authoritative list is in :ref:`required-dynamodb-tables`; verify it with operator credentials rather than by inspection:: poetry run python -m actingweb.db.verify_tables Use the same ``AWS_DB_PREFIX``, region and credentials as the app. If the app selects reverse-lookup mode with ``with_legacy_property_index(...)`` rather than ``USE_PROPERTY_LOOKUP_TABLE``, pass ``--legacy``/``--lookup-table`` explicitly — the CLI can only read the environment. Run this **before** dropping the permissions. ``_subscription_suspensions`` is the one to watch: before 3.13 its accessor had no auto-create guard, so long-lived deployments frequently never created it, and a missing suspensions table degrades silently rather than crashing (an ERROR is now logged on each failed check). **2. Do not create the new tables in the same deploy as the library bump.** Table pre-warm runs at integration time, so a new container's first cold start can create ``_property_lookup_v2`` *before* your CloudFormation/Terraform resource is created — the stack then fails with ``ResourceInUseException`` and rolls back. Pick one owner: - declare the new tables in a deploy **before** the library upgrade (still on 3.12), then bump the library, then disable auto-creation; **or** - let auto-creation own them and never declare them in IaC. Doing both in one deploy is the race. **3. The IAM change is account-wide, not library-scoped.** Dropping ``DescribeTable``/``CreateTable`` from the runtime role also breaks **your own** code that probes tables — boto3 ``table.load()`` or ``describe_table()``, pynamodb ``Model.exists()``. A caller that catches broadly will just log a warning and quietly disable a feature while re-issuing the denied call on every construction. Audit for those probes before flipping the flag; where you have one, honour the same switch rather than restoring the permission:: from actingweb.db.dynamodb import auto_create_enabled if auto_create_enabled(): ... # safe to probe / create else: ... # tables are managed externally — skip the probe Proving the fixes actually landed --------------------------------- Both headline fixes are invisible to functional tests: a ``Scan`` → ``Query`` conversion returns identical data, and removing ``DescribeTable`` calls changes no behaviour. What *is* observable is the **operation profile** — count the DynamoDB API calls a hot path issues. This works against DynamoDB Local, so it costs nothing and needs no production access. .. warning:: The obvious implementation does not work. Registering a ``before-call.dynamodb`` handler on a botocore session **silently counts nothing** under pynamodb: ``pynamodb.connection.base.Connection.session`` resolves via ``botocore.session.get_session()``, which returns a *new* ``Session`` per call, so a handler on any session you can reach never sees pynamodb's traffic. The failure mode is an empty counter — which makes ``assert scans == 0`` pass **vacuously**, worse than no test at all. Patch ``BaseClient._make_api_call`` instead; every botocore client funnels through it:: import collections import botocore.client counts = collections.Counter() original = botocore.client.BaseClient._make_api_call def _counting(self, operation_name, api_params): if self.meta.service_model.service_name == "dynamodb": counts[operation_name] += 1 return original(self, operation_name, api_params) monkeypatch.setattr(botocore.client.BaseClient, "_make_api_call", _counting) Exercise a hot path and assert the profile. Measured on identical code against 3.12.0 and 3.13.0: .. list-table:: :header-rows: 1 :widths: 40 30 30 * - Exercise - 3.12.0 - 3.13.0 * - ``DbPropertyList().fetch(actor_id)`` - ``{'DescribeTable': 1, 'Scan': 1}`` - ``Query``; **Scan: 0** * - ``DbTrustList().fetch(actor_id)`` - ``{'DescribeTable': 1, 'Scan': 1}`` - ``Query``; **Scan: 0** * - 25 × ``DbProperty()`` construction - ``{'DescribeTable': 25}`` - ``{}`` — **0 calls** ``DescribeTable: 25 → 0`` for 25 constructions is exactly the one-call-per-construction defect and its removal. Sanity-check that the counter is live (assert a non-zero count for some operation) before trusting a zero. In production, ``DescribeTable`` volume is visible in CloudTrail attributed to the runtime role. Expect a large drop on the upgrade alone (memoisation fixes the warm steady state) and **zero** only after ``AWS_DB_AUTO_CREATE_TABLES=false`` — the per-container pre-warm sweep is the residual. Make sure the measurement window contains a cold start, or the zero is vacuous. Behaviour changes to be aware of -------------------------------- - **Reverse-lookup matching is now (name, value)**, not value-only: the legacy GSI matched on value alone, so a value shared across *different* property names could resolve "cross-name". The lookup table cannot, and the migration fallback tiers (v1 table, legacy GSI, PostgreSQL scan) now also filter by property name — so the value-only cross-name match is gone on every path, not just the primary one. The fallbacks remain only to serve un-backfilled deployments and are removed in the next major release. - **Non-indexed property names return None** from ``Actor.get_from_property()`` / ``ActorInterface.get_by_property()`` (with a warning) instead of silently querying the legacy path. Add names to ``with_indexed_properties()`` to make them resolvable. - **Cross-actor lookup collisions are refused and logged** instead of silently overwritten (DynamoDB previously last-writer-wins; PostgreSQL already refused — the backends now agree, including idempotent re-creates). - **Trust/peer-trustee list ordering is deterministic** (range-key sorted) instead of arbitrary scan order. - **Subscription-suspension and peer-trustee-list accessors now auto-create their tables** like every other accessor (previously a fresh deployment crashed on first use of suspension). - The lookup table stores **SHA-256 digests, not plaintext values** — update your data map (see ``docs/reference/security.rst``). No changes needed for PostgreSQL schemas (the lookup table and its Alembic migration already exist); the default flip applies to both backends.