Migrating to ActingWeb 3.13

Warning

This guide shows the API as it was during this migration and must not be copied into new code. ActingWeb is now at a later version; see Guides for the current API.

Start here

3.13 bundles four largely independent pieces of work: a property-list storage change with pre-upgrade steps, an MCP behaviour change, an MCP security fix, and a DynamoDB scalability change with a required backfill. Which of them you need depends only on where you are coming from.

Upgrading from 3.12.x or earlier

Do all four sections, and do them in the order below, because two of them have work that must happen before you deploy the new version:

1. Property lists — step 1 sweeps for pre-existing damage and must run on the old version. 3.13 turns damage that was previously invisible into a visible error, so a sweep afterwards tells you less than a sweep before.

2. Security: MCP trust-cache bypass — the “find trust rows that predate exact resolver matching” audit is also a before-you-upgrade step.

3. DynamoDB scalability — the reverse-lookup backfill is required for any deployment using reverse lookup, and has a rollback window worth understanding before you start.

4. MCP structuredContent and the remaining behaviour changes — these need code changes on your side if you are affected, but no pre-upgrade sweep.

If you do not use list properties, do not expose MCP, and do not use reverse lookup, the only thing left that can affect you is “Property reads now raise on backend faults” — read that and you are done.

Upgrading from a 3.13 release candidate

Do the sections that were added after the rc you are running. Nothing in this table is optional for the rcs it names — the earlier rcs’ pre-upgrade steps still apply if you never did them.

Every row names sections by their exact title, so the table works as a checklist and not only as prose.

You are on	Still to do
rc1	Everything except the reverse-lookup backfill, which you have already run: Property lists, Migration refuses damaged lists, Security: MCP trust-cache authorization bypass and MCP: structuredContent is now opt-in are all new to you. Also re-read the rest of the DynamoDB section, because three of its changes touch a code path rather than only docs: register_diffs() now resolves subscriptions before consulting suspension state, create_subscription() invalidates the actor’s cached subscription list, and actor deletion writes a tombstone before the wipe begins.
rc2	Everything except the DynamoDB section, whose backfill you have already run: Property lists, Migration refuses damaged lists, Security: MCP trust-cache authorization bypass and MCP: structuredContent is now opt-in are all new to you.
rc3	Property lists, Migration refuses damaged lists and MCP: structuredContent is now opt-in. Run the step-1 sweep before upgrading — see the rc4 row for why.
rc4	Property lists and Migration refuses damaged lists. Run the step-1 sweep before upgrading: you are on the last release that hides pre-existing list damage. 3.13.0 surfaces it as a 409 instead of silently compacting past it, so a list that has been quietly broken for months starts erroring the moment you deploy — and the sweep is easier to reason about before that than after.
rc5	Migration refuses damaged lists, plus “What changed after rc6” below.
rc6	Only “What changed after rc6” below.

What changed after rc6

Between rc6 and 3.13.0 the release gained four changes. None requires operator action, and there is no data migration between them — but two are worth knowing about:

• A list-metadata fix that matters if you are mid-migration. Before 3.13.0, a concurrent write could put format: 1 back over a completed migration’s format flip by writing a whole cached metadata dictionary back; migration then deleted the v1 rows and the list read back empty, with no error. If you have been running actingweb-migrate-property-lists against a live deployment on rc5 or rc6, run actingweb-verify-property-lists once after upgrading. There is no repair step to run — the fix is preventative — but a list that was destroyed this way is already gone and you would rather know.

• MCP revocation now evicts the in-process caches. A revoked token, a deleted trust relationship or a downgraded permission previously kept working from a warm process for up to five minutes. It no longer does, within the process that served the revocation — a multi-worker or multi-container deployment still serves the stale answer from every other process until its TTL expires. If your threat model depended on the old behaviour being instantaneous, it never was.

• A repair-tool guard, added after consumer verification of GA. --repair now refuses a v1 list that is damaged and carries v2-format rows, because that combination is a reverted migration rather than a hole, and repairing it strands the data while reporting success. If you swept with --repair on rc5 or rc6, see the warning under “Step 2” above.

• A stale-instance write now warns. A ListProperty retained across a migration writes its next item in the old format’s row shape, where nothing reads it — and verify() calls the list healthy. That now logs a WARNING. It affects code that stashes the list object; ordinary actor.property_lists.<name> access builds a fresh one per use.

The other two are an opt-in PostgreSQL diagnostic (ACTINGWEB_PG_DELETE_DIAGNOSTICS, off by default) and a change in how unsupported-MCP-Protocol-Version rejections are logged: a lone rejection is now INFO rather than unlogged, and a sustained run from one origin escalates to WARNING once. If you alert on WARNING from actingweb.handlers.mcp, that is a new line you may see — it means a client is retrying instead of falling back, which is worth investigating rather than silencing.

Warning

A green test suite is not evidence this release landed correctly. The read-path and capacity fixes are invisible to functional tests — they change how much a request costs, not what it returns. Use the operation-profile recipe in “Proving the fixes actually landed” at the end of this document to confirm them against your own hot paths.

Property lists: storage format, repair, and fail-fast reads

Note

Applies to every deployment that uses list properties (actor.property_lists.*, /properties/<name> on a list), and is independent of the MCP and DynamoDB sections. If you use only scalar properties, read just “Property reads now raise on backend faults” below and skip the rest.

Read this section before upgrading, not after. One of the changes turns pre-existing, previously-invisible data damage into a visible error, so the order of operations matters.

Why this release exists

ListProperty could destroy data. insert() on DynamoDB overwrote every shifted row with the last value it read, so a single call into a non-empty list corrupted it. Backend read failures were swallowed and reported as “row absent”, turning a transient fault into a permanent hole. Delete and insert were multi-write shift loops over a stored length counter with no transaction, so an interruption anywhere in the middle left a hole, a duplicate, or a length that disagreed with reality — and reads silently compacted past the damage, so nothing ever surfaced it.

3.13 fixes the write paths, adds repair tooling, makes reads fail loudly instead of quietly, and introduces a storage format in which the whole class of defect is structurally impossible.

Step 1: sweep for existing damage BEFORE upgrading

Any list damaged by the old code is still damaged. Releases before 3.13 hid that; 3.13 will not. Find out what you have first:

actingweb-verify-property-lists

Read-only. It reports, per list, the recorded length, how many rows are actually readable, which indices are missing, which are orphaned, and any adjacent duplicate residue.

Warning

Run it with the same environment your application uses. The library defaults AWS_DB_PREFIX to demo_actingweb. If your deployment uses a different prefix — and if a demo deployment exists in the account — an unset prefix silently sweeps the wrong tables and reports them clean. A clean report from the wrong table looks exactly like good news. The script now prints the backend, region and prefix it is about to use, and warns when the prefix is an unset default; check that line before trusting the result.

If your items carry an identifying field, pass it:

actingweb-verify-property-lists --identity-key id

Duplicate detection otherwise compares raw stored bytes, which finds the duplicate an interrupted shift leaves — but stops finding it the moment either copy is edited. That false negative hits precisely the lists that have been used since the damage. --identity-key compares on the field that identifies the item instead, and keeps working after an edit.

Step 2: repair what it found

actingweb-verify-property-lists --repair

--repair calls compact(), which rewrites the surviving rows densely, removes orphans, corrects the recorded length, and preserves description/explanation/created_at. It closes holes.

It refuses a reverted migration, by design. A v1 list that is damaged and reports foreign_format_rows > 0 is not an ordinary hole. It is a list whose migration to v2 was reverted: the v1 rows are gone, the metadata still claims v1 with the old length, and the items are almost certainly alive in the v2 rows. Compacting it would rewrite the empty v1 range, set the length to what it found, report the list healthy, and leave the only surviving copy as unreferenced residue for the next clear()/delete()/migrate re-run to sweep.

So --repair skips those lists, logs a WARNING naming the shape, and counts them as still unhealthy. Recover them first — the v2 rows are readable directly, and re-running the migration against corrected metadata restores the list. --repair-reverted overrides the refusal and is the wrong answer unless you have looked at the rows and decided to abandon them.

Warning

Earlier releases logged those rows at INFO as “harmless to reads; clear it by re-running --migrate”. In this shape that advice deletes the data. If you ran a sweep on 3.13.0rc5 or rc6 and acted on that line, check the affected lists before assuming they are empty by design.

It does not resolve duplicates, by design. A duplicate always means an item was destroyed; collapsing one copy would bless that loss as intentional rather than surface it. Duplicates are reported and left for you to decide about. Repair does not recover a destroyed item — nothing can; the row is gone.

Warning

Repair is not crash-safe, and an interrupted repair leaves damage it will not itself fix. compact() rewrites surviving rows at their new positions before deleting the tail, so an interruption between the two leaves a copy at both the old and the new position. Measured on a 4-slot list with one hole: interrupting after the first move leaves [a, c, c, d], after the second [a, c, d, d] — both with the length still reading 4, so the list reads back with no error at all.

verify() does catch it (the duplicate is adjacent and byte-identical, which is exactly what its heuristic looks for), so a follow-up sweep reports the list unhealthy. But re-running --repair will not remove the copy: duplicates are preserved by design, so the tool declines to touch the residue its own interruption created, and the list stays one item too long until you resolve it by hand.

In practice: run repair when the actor is not taking writes, and run the sweep again afterwards rather than assuming it worked. The same shape applies to compact() on a v2 list, where it is a rank rebalance — see the guide’s warning box.

What “not taking writes” does and does not buy you. Nothing excludes an application write while a whole-list rewrite runs: there is no lock, and adding one is deliberately not part of 3.13. A concurrent write during compact(), --repair or --migrate can still be lost, so the advice above is a real operational requirement rather than a nice-to-have. What 3.13 does guarantee is that such a write can no longer corrupt the list’s format from a stale cache: a concurrent write updates only the metadata fields it changes, merged into a fresh read. In v3.13.0rc6 and earlier it wrote a whole cached dictionary back, putting format: 1 over a completed migration, and the result was total silent loss — see the CHANGELOG.

The window is narrowed, not closed. It was unbounded in time (a retained ListProperty could revert a migration that finished hours earlier); it is now the gap between reading the metadata row and writing it back, since there is no compare-and-set to condition that write on. Quiescing writes is what closes the rest.

Update (3.14): this remaining gap is closed. Metadata writes now condition on the exact bytes they read via a bounded compare-and-swap retry loop, so a migration that completes inside the old window is merged onto rather than overwritten. See the 3.14 migration guide and the property-lists guide’s “Concurrency during a whole-list rewrite” section for the current behavior; this page is left as written for 3.13.

Step 3: expect these breaking changes after upgrading

List reads now raise on corruption instead of compacting past it. to_list(), slice() and to_list_from_rows() raise ListCorruptionError (an IndexError subclass) when an item inside the list’s recorded length is missing from storage. Every HTTP path that serves list content returns 409 with {"error": "list_corrupted", "list": ..., "detail": ..., "remedy": "compact"}.

This is the change that makes step 1 non-optional: a hole that earlier releases silently skipped becomes a raised exception or a 409. Audit your call sites — anything that reads a list and cannot tolerate an exception needs a ListCorruptionError handler, and the underlying list needs repairing:

from actingweb.property_list import ListCorruptionError

Property reads now raise on backend faults. DbProperty.get() raises actingweb.db.exceptions.DbError when the backend itself fails (timeout, throttle, connection error). None now means “the row does not exist”, and only that. Code that relied on a fault degrading to None will now see an exception. This applies to scalar properties too, not just lists.

Every list mutation now checks its writes. append, __setitem__, __delitem__, insert, clear, delete and metadata writes raise RuntimeError instead of continuing past a failed write.

``GET /properties/<name>/items`` response shape changed. Was a bare JSON array; is now {"items": [{"index": i, "item": ...}, ...], "count": n}. The index is the same one action=update/action=delete accept, so GET and POST are consistent. (Flask gained this route for the first time; it previously existed only on FastAPI.)

``PUT /properties/<name>?index=N`` beyond the list length returns 404. It previously padded the list with None up to N — both a spec violation and an unbounded-write vector. index == length still appends. The bulk item POST gained the equivalent bound.

``index()`` honours negative ``start``/``stop``. It previously ran range(start, ...) unnormalized, so index(value, -1) could return -1 as an index. Both storage formats now match list.index.

Step 4: decide when lists are allowed to change format

Danger

Automatic conversion of existing lists is OFF by default (ACTINGWEB_LAZY_MIGRATION_MAX_LENGTH=0). Leave it that way for this release’s deployment: no list changes format, so rolling back stays a pure code rollback with no data to reconcile. Convert later, deliberately.

This does not make v2 opt-in – every list created after the upgrade is v2 with no action from you. Only the conversion of data you already have waits for your go-ahead.

The reason is not latency, it is rollback. A pre-3.13 process does not error on a migrated list – it reads it as empty, silently. The metadata row still exists, so the list “exists”; a v2 list stores no length field, and an older reader takes the absence as zero. A write from that process then lands in v1 storage and the list forks: two versions, two disjoint views, nothing reporting an error. --downgrade cannot reconcile a forked list; it overwrites v1 storage with the v2 content and destroys whatever the older process wrote.

Deployment gives you at most a brief mixed-version window. Rollback gives you no window at all: deploy, let lazy migration convert lists for hours or days, then roll back for an unrelated reason, and every list that migrated reads as empty in production. Recovery is --downgrade one list at a time – there is no bulk mode – from a v2-capable checkout, against a database served by the code you just rolled back to. That is not a procedure to discover during an incident.

Migration forward is automatic, fleet-wide and inline. Recovery back is manual and per-list. The general rule behind the setting: no list may become v2 until every process that might serve it can read v2, including the release you would roll back to.

Once 3.13 has been live long enough that rollback is off the table, migrate as a deliberate operator action:

actingweb-migrate-property-lists              # dry run
actingweb-migrate-property-lists --migrate

Repair (step 2) before you migrate, and treat the dry run’s exit code as the gate. Migration refuses a list with holes or orphans, and the dry run names them and exits 1; a dry run that exits 0 is telling you the migration has nothing to trip over. This matters more than it sounds: migration renumbers survivors, so a hole that goes through it stops existing and stops being reportable. See “Migration refuses damaged lists” below.

Step 5: understand the new storage format

Every new list is created in a new internal format (“v2”) that stores items under sort keys rather than dense integer positions. Delete and insert become single writes: there is no shift loop, no stored length to disagree with, and therefore no way to produce the holes and duplicates this release exists to clean up.

Existing lists keep working unchanged, indefinitely, on the hardened old paths. Nothing forces you to migrate.

Lazy migration. With ACTINGWEB_LAZY_MIGRATION_MAX_LENGTH set to a positive number, a healthy v1 list of at most that many items migrates on its next mutation. Three deliberate limits:

• Damaged lists are never migrated automatically. Migration closes holes in flight, which is correct when an operator runs the bulk script and reads its report — but doing it silently under an ordinary append() would erase the evidence: the hole closes, the destroyed item stays destroyed, duplicate residue becomes indistinguishable from real data, and verify() starts reporting the list healthy. Repair is always your decision. Damaged lists keep serving v1 and keep raising until you act.

• It is off unless you enable it (ACTINGWEB_LAZY_MIGRATION_MAX_LENGTH defaults to 0) — see step 4 for why. It is also inline and synchronous when enabled: one append() to a 40-item list performs the entire migration inside that request.

The threshold is checked at the moment of the mutation, so “large lists stay v1 until I run the script” is not quite true: a clear() + extend() whole-list rewrite migrates a list of any original size, because the first append() inside extend() sees length 0. ACTINGWEB_LAZY_MIGRATION_MAX_LENGTH=0 is the only setting that makes “nothing migrates without me” true.

Bulk migration (step 4 above) is for lists too large or too idle to migrate lazily, and — with lazy migration off by default — for everything else too. It refuses damaged lists on the same grounds lazy migration does; repair them first. migrate_to_v2() is idempotent and safe to interrupt and re-run.

List names containing ``#`` are refused. # is reserved for internal v2 storage keys. New lists cannot be created with it; existing lists that have it keep working as v1 forever and are refused by migration. Rename them if you want them migrated. They are correctly isolated from v2 lists whose name is a prefix of theirs, so this is a migration blocker, not a data risk.

Positional reads cost more under v2. lst[i] re-reads the list’s key ordering before resolving the position, because resolving against a stale ordering returns the wrong item. A for i in range(len(lst)): lst[i] loop is therefore two queries per item. Use to_list(), to_indexed_list() or plain iteration — each is a single query for the whole list, in both formats. If you have such a loop, this is the one change likely to show up in your latency numbers.

Downgrade is unsupported. migrate_property_lists.py --downgrade exists as an emergency converter only. If you use it, roll the application back to a pre-v2 release first. With lazy migration at its default of 0 this is “only” the ordinary rule that the reader must precede the data; if you have raised the limit, it is sharper — a downgraded list at or under it is a lazy-migration candidate again, so a still-running v2-aware application migrates it straight back on its next write.

PostgreSQL: one schema migration

properties.name widens from VARCHAR(255) to TEXT (metadata-only, no table rewrite):

cd actingweb/db/postgresql/migrations && alembic upgrade head

Apply it before running any v2 workload. The downgrade reverses it and will fail loudly if any name exceeds 255 characters at that point, which is intentional — downgrading the schema underneath live v2 keys should not silently truncate them.

Repair API, if you would rather not use the scripts

lst = actor.property_lists.notes
report = lst.verify(identity_key="id")   # read-only
if not report["healthy"]:
    lst.compact()                        # closes holes, keeps duplicates

verify()/compact() also work on v2 lists, for a different purpose there: detecting and rebalancing sort keys that have grown long from repeated inserts at the same position.

Migration refuses damaged lists

Note

A refinement of the property-list steps above, and it only affects you if you run actingweb-migrate-property-lists (step 4). Do the steps above first — they are the ones with the pre-upgrade work.

actingweb-migrate-property-lists --migrate and ListProperty.migrate_to_v2() now refuse a list with holes or orphans, and the dry run reports those lists as needing repair instead of counting them as “would migrate”. A dry run that finds any exits 1.

The reason is that migrating damaged data is not just lossy, it is unreportably lossy. Migration renumbers the surviving rows, so the hole is gone afterwards — and so is the evidence. The migrated list verifies healthy, and nothing remains to say an item ever went missing. An earlier form of the dry run said nothing about holes at all, so a sweep over a fleet containing one could report “0 refused, 0 errors” and an operator would proceed in good faith, past the last moment at which the damage was still visible.

Lazy migration has always refused damaged lists, for exactly this reason. The deliberate path now applies the same rule.

What to do: repair first (step 2), then migrate. If you have looked at the damage and decided to migrate anyway, --migrate-damaged (or migrate_to_v2(allow_damaged=True)) does it, and says clearly in the log what it is giving up.

Duplicate residue is not affected — it never blocked migration and still does not. A duplicate stays visible after conversion, because v2’s verify() reports duplicates just as v1’s does, so migrating it destroys no evidence. Only holes and orphans gate.

MCP: structuredContent is now opt-in

Note

Applies only if you expose MCP tools. If you do not, skip to the next section. Independent of the property-list section above and the security and DynamoDB sections below.

What changed

Previously, a tool hook returning a dict with content plus extra top-level keys had those extras promoted into structuredContent automatically. That promotion is removed. structuredContent is now emitted only when your hook sets that key explicitly, and only when its value is a JSON object.

Why: at least one major MCP client discards every text content block when structuredContent is present. Under the old behaviour, adding a single scalar key to a hook’s return value silently deleted that tool’s entire prose payload — the model received only the promoted keys, with no error on either side. Neither reference server implementation promotes keys this way.

Are you affected?

You are affected if any tool hook returns a dict with a content key and other top-level keys besides isError, _meta and structuredContent. Those extras were being serialized into structuredContent; now they are dropped.

You are not affected if your hooks already set structuredContent explicitly, return content alone, or return dicts with no content key (those take the legacy text-wrap path, which never emitted structuredContent).

How to migrate

The migration is a no-op against the old release, so you can migrate first and upgrade afterwards — no coordinated deploy window. The explicit passthrough already exists in v3.11.0/v3.12.0, so a hook updated as below produces byte-identical output on both.

For a tool whose payload is genuinely structured, name the key and serialize the same object into the text block:

# Before — relied on promotion
return {
    "content": [{"type": "text", "text": f"Found {len(results)} results"}],
    "results": results,
    "count": len(results),
}

# After — explicit, and the text carries the same object
payload = {"results": results, "count": len(results)}
return {
    "structuredContent": payload,
    "content": [{"type": "text", "text": json.dumps(payload)}],
}

For a tool whose payload is prose, drop the extras instead of promoting them. That is the case the old behaviour broke, and returning prose alone is now correct.

Two caveats the library cannot enforce for you

1. Some clients ignore ``structuredContent`` entirely (see modelcontextprotocol#1411). Always keep the same data serialized in a text content block, per the specification’s backwards-compatibility guidance. The text block is the only thing that always arrives.

2. A request with no ``MCP-Protocol-Version`` header negotiates ``2025-03-26``, where structuredContent is suppressed even when set explicitly. If you verify with curl, send -H 'MCP-Protocol-Version: 2025-06-18' or you will wrongly conclude your migrated hook is broken.

Two related fixes in the same release

• An explicit ``isError`` is now honoured on the legacy text-wrap path. A hook returning {"isError": True, ...} with no content key previously reached the wire with no isError field at all, so the failure was reported to the client as a success. isError is honoured only when you set it — it is never inferred from an "error" key or any other heuristic, so apps that return {"error": ...} on their normal error path see no change. The text serialization is unchanged.

• A warning now fires when a tool declares ``output_schema`` but returns no ``structuredContent``. Spec-conforming clients reject that result outright (Tool X has an output schema but did not return structured content). This was already broken before this release; it is now visible in your logs. Note that output_schema and structuredContent are independent — declaring a schema has never caused structured output to be emitted.

Security: MCP trust-cache authorization bypass

Note

Read this even if you skipped the MCP section above. It applies in addition to every other section here. If you run MCP with a single client per actor, most of it does not apply to you — but the resources/read and resolver-matching items still can, so read the summary before skipping ahead.

See the SECURITY section of CHANGELOG.rst for what changed and why. This section is the operational checklist: what to check before upgrading, and what to expect after.

1. Find trust rows that predate exact resolver matching

The MCP client-id-to-trust resolver now requires an exact oauth_client_id match, or — for legacy rows without that field — an exact reconstructed peer-id match gated on established_via being an OAuth2-family value. A row that matched only because the old resolver did substring containment will stop resolving after upgrading, and that client’s next request returns an empty tools/list or a -32003 on tools/call/prompts/get/resources/read instead of authenticating as before.

Find at-risk rows before upgrading — any trust row where:

• oauth_client_id is unset (None/absent), and

• established_via is not "oauth2_client" (or the legacy peer id does not start with oauth2: or oauth2_client:), or

• the peer id is not in the three-segment <source>:<email-or-client>:<client_id> shape trust creation constructs.

These are almost always trust rows created before v3.3 (2025-10-04, when oauth_client_id was introduced) that have not been through an OAuth2 re-authorization since. Rows created by either live MCP path (client registration, or the MCP OAuth2 authorize callback) already carry oauth_client_id and are unaffected.

2. Audit Flask (sync) resource URIs against your trust type’s patterns

resources/read had no authorization check at all on the Flask (sync) transport before this release (see the SECURITY note in CHANGELOG.rst) — every resource read was served regardless of trust type. It is now checked identically to the FastAPI (async) transport, which was never affected.

The default mcp_client trust type only allows reading: public/*, shared/*, notes://*, usage://*, and actingweb://properties/all (see actingweb/trust_type_registry.py). If your application serves custom resource URIs outside that pattern set to Flask-based MCP clients, those reads have been working only because the check was dead and will start returning -32003 after upgrading. Audit your resource hooks’ URIs against the resources.patterns of every trust type your app registers, and widen the pattern set (or register a custom trust type) before upgrading if legitimate traffic depends on a URI outside the default allowlist.

3. Understand the orphaning consequence of a resolver miss

If a client’s trust row stops resolving per step 1 above, that client does not stay locked out forever: the next OAuth2 authorization it completes creates a new trust row with a newly constructed peer id (since the old row is simply never matched, not deleted). This has one consequence worth knowing in advance: per-peer permission overrides are silently abandoned.

Per-peer permission overrides are stored keyed by f"{actor_id}:{peer_id}" (see TrustPermissions.get_permission_key() in actingweb/trust_permissions.py). Any overrides an operator applied to the old peer id do not transfer to the new row’s peer id — the re-authorized client silently reverts to its trust type’s default permissions. If you have applied custom per-peer overrides to any MCP client, check whether that client’s trust row is at risk per step 1, and if so, re-apply the overrides to the new peer id after it re-authorizes.

4. Note the backfill timing for oauth_client_id

oauth_client_id is backfilled onto a trust row when a client completes an OAuth2 re-authorization, not merely on its next request (see actingweb/interface/trust_manager.py). Between upgrading and a given client’s next full re-authorization, that client’s resolution rests entirely on the legacy peer-id fallback described in step 1 — so a row that fails the step-1 check does not “self-heal” on its own until the client goes through the OAuth2 flow again, not just makes another API call.

DynamoDB scalability: reverse lookup, capacity, and table management

Note

This section and everything below it is the DynamoDB-scalability half of 3.13, and it is the one with the required backfill. It is independent of the property-list, MCP and security sections above; upgrading from 3.12.x means doing all of them.

ActingWeb 3.13 is a DynamoDB scalability release. It fixes two measured superlinear cost defects (full-table scans on per-actor reads; a DescribeTable control-plane call on every accessor construction), removes hot-path read amplification, defaults auto-created tables to on-demand billing, and — the one change that affects behaviour for deployments that change nothing — flips the default property reverse-lookup mechanism to lookup-table mode, backed by a redesigned (v2, digest-keyed) lookup table.

Nothing in this release requires code changes. What it may require is one operational step: running the lookup-table backfill script so reverse lookups (find-actor-by-email/oauthId) are served from the new table instead of deprecated fallbacks.

How urgent that step is depends on how your app resolves the logged-in user — check rather than assume:

• If it calls Actor.get_from_creator() (the creator-index GSI on the actor table), reverse lookup is not on your login path and deferring the backfill is low-risk.

• If it calls get_from_property() / get_by_property() on email, oauthId or another indexed property, that is your login path and it runs on deprecated fallbacks — or returns None — until the backfill completes.

Note

Validating this release requires a DynamoDB backend. Every change here is DynamoDB-specific, so a green test suite run against PostgreSQL (DATABASE_BACKEND=postgresql) exercises none of it — an easy trap, because the suite passing reads as “validated”. Run against DynamoDB Local or a real table; see Rehearse the migration first.

Which deployment am I?

Find your row; do what its column says. “Implicit legacy” means you never called with_legacy_property_index() and never set USE_PROPERTY_LOOKUP_TABLE.

State before 3.13	After upgrading (no config change)	Action needed
Implicit legacy; properties table has the property-index GSI (created by an older ActingWeb)	Reverse lookups keep working via the deprecated GSI fallback (a warning is logged per hit; an ERROR is logged at startup)	Run the backfill (below). Optionally delete the now-unused GSI to halve property write cost — only after verifying.
Implicit legacy; properties table has no GSI (created before the GSI existed). Reverse lookups were crashing before 3.13.	Reverse lookups now work IF the lookup table is populated; until then they return None (startup ERROR tells you)	Run the backfill (below).
Lookup mode already enabled (with_legacy_property_index(False)) on the v1 lookup table	Reverse lookups keep working via the deprecated v1 fallback (warning per hit; startup ERROR)	Run the backfill, verify, then drop the v1 ``<prefix>_property_lookup`` table.
Explicit legacy (with_legacy_property_index(True) or USE_PROPERTY_LOOKUP_TABLE=false)	Unchanged: legacy GSI path stays active. If the table lacks the GSI you now get an actionable error instead of an opaque crash.	None immediately. Legacy mode is deprecated — plan the migration.
Fresh deployment	Lookup-table mode, on-demand billing, no legacy GSI — the intended end state	None.

Rehearse the migration first

The backfill is the riskiest step in this release and it is easy to treat as a one-shot production procedure. It isn’t: the whole migration — new-table creation, the three-tier fallback, the backfill script, the startup tripwire — runs end-to-end against DynamoDB Local at zero cost and zero risk. Do it once before touching production:

docker run -d -p 8000:8000 amazon/dynamodb-local

export AWS_DB_HOST=http://localhost:8000
export AWS_DEFAULT_REGION=us-west-1
export AWS_ACCESS_KEY_ID=fake AWS_SECRET_ACCESS_KEY=fake
export AWS_DB_PREFIX=rehearsal

Then, in order:

1. Start the app (or create a couple of actors with properties on the indexed names). Confirm the startup ERROR tripwire fires — it names the resolved lookup table and the exact script to run.

2. python -m actingweb.db.verify_tables — confirms which tables exist.

3. Run the backfill, then restart. The per-hit DEPRECATED: reverse lookup ... warnings and the startup ERROR should both stop.

4. Optionally flip AWS_DB_AUTO_CREATE_TABLES=false and confirm the app still serves.

An operator who has done this once is not doing it for the first time in production.

Required: run the backfill (all upgrading deployments using reverse lookup)

The lookup table is derived data; the properties table is the source of truth. The script rebuilds the v2 table from it — streaming, rate-limited, resumable and idempotent:

# With the SAME environment as the app (AWS_DB_PREFIX, region,
# credentials, INDEXED_PROPERTIES if customised):
poetry run python scripts/backfill_property_lookup.py --dry-run
poetry run python scripts/backfill_property_lookup.py --rps 50

Notes:

• --rps caps items/second — a full-table scan against production should not brown-out serving traffic. --segments N parallelises.

• Interrupted runs resume from --checkpoint-file.

• Values are copied verbatim (no normalisation) — runtime lookups are exact-match.

• A value shared by two actors is reported as a collision and not overwritten (exit code 1). Resolve manually and re-run — re-runs are idempotent.

Verify, then clean up:

1. Watch logs: reverse lookups should stop logging DEPRECATED: reverse lookup ... served from ... warnings, and the startup ERROR should disappear on the next restart.

   Warning

   Do not verify the backfill with describe-table’s ItemCount — it refreshes only about every six hours, so a freshly backfilled table still reports ItemCount: 0. That reads as a no-op backfill when the rows are actually there. Use the real count:

   aws dynamodb scan --table-name <prefix>_property_lookup_v2 \
     --select COUNT
   

2. v1 cohort: drop the old table — aws dynamodb delete-table --table-name <prefix>_property_lookup. (Never touch <prefix>_properties — that is the source of truth.)

3. GSI cohort (optional, saves ~half of property write cost): delete the legacy index:

   aws dynamodb update-table --table-name <prefix>_properties \
     --global-secondary-index-updates '[{"Delete":{"IndexName":"property-index"}}]'
   

   Only after the backfill is verified: deleting the GSI removes the fallback the un-backfilled state depends on. Note that deleting the GSI makes a later return to legacy mode impossible without recreating the index.

Rollback — check before you upgrade

Setting USE_PROPERTY_LOOKUP_TABLE=false (or with_legacy_property_index(enable=True)) restores the legacy path. (3.13 also fixed a bug where the env variable was silently ignored by apps using the fluent builder — the rollback now actually works.)

Warning

Rollback requires the legacy GSI to exist on your properties table. Legacy mode queries property-index; a properties table created before that index existed does not have it, and tables are never altered in place. Such a deployment has no rollback path for reverse lookup — worth knowing before upgrading, not after. Check first:

aws dynamodb describe-table --table-name <prefix>_properties \
  --query 'Table.GlobalSecondaryIndexes[].IndexName'

null means no GSI, therefore no rollback. (The same applies after you delete the GSI in the cleanup step above.)

Direct use of the lookup accessor degrades during the migration window

The three-tier read path (v2 → deprecated v1 → legacy GSI) lives in DbProperty.get_actor_id_from_property(). Code that constructs DbPropertyLookup() directly and calls .get() gets the v2-only path with no fallback, so between the upgrade deploy and a completed backfill it returns None while the application itself is perfectly healthy — which reads like “the migration broke actor lookup”.

Operational scripts (actor management, debugging, seeding, cleanup) are the usual offenders. Audit yours: resolve actors through get_actor_id_from_property() — or the interface-level ActorInterface.get_by_property() — which is the supported entry point and carries the fallbacks.

Recommended: convert old auto-created tables to on-demand billing

Tables the library auto-created before 3.13 are PROVISIONED with tiny capacities — typically <prefix>_property_lookup (2 RCU / 1 WCU: a hard wall on the login path) and <prefix>_peertrustees. New tables are on-demand; existing tables are not changed automatically. Convert them in place:

aws dynamodb update-table --table-name <prefix>_peertrustees \
  --billing-mode PAY_PER_REQUEST

• AWS allows one billing-mode switch per table per 24 hours.

• Never delete and recreate a table to change billing — the v1 lookup table (pre-backfill) and every other table hold live data.

• Check for other PROVISIONED stragglers: aws dynamodb list-tables + describe-table.

Recommended: disable auto-creation in production

If your tables are managed by CloudFormation/Terraform, turn off the library’s table auto-creation and slim the runtime role:

AWS_DB_AUTO_CREATE_TABLES=false        # env, or:
app.with_dynamodb(auto_create_tables=False)

With auto-creation off the library never calls DescribeTable or CreateTable, so both can be dropped from the runtime IAM policy. If your IAM policy lists tables by name, add the new <prefix>_property_lookup_v2 table.

Three things this hands you responsibility for.

1. Every required table must already exist. The library will no longer create them, and — deliberately, so a role without DescribeTable does not pay an AccessDenied per accessor construction — it will not tell you one is missing. The authoritative list is in Required DynamoDB tables; verify it with operator credentials rather than by inspection:

poetry run python -m actingweb.db.verify_tables

Use the same AWS_DB_PREFIX, region and credentials as the app. If the app selects reverse-lookup mode with with_legacy_property_index(...) rather than USE_PROPERTY_LOOKUP_TABLE, pass --legacy/--lookup-table explicitly — the CLI can only read the environment.

Run this before dropping the permissions. <prefix>_subscription_suspensions is the one to watch: before 3.13 its accessor had no auto-create guard, so long-lived deployments frequently never created it, and a missing suspensions table degrades silently rather than crashing (an ERROR is now logged on each failed check).

2. Do not create the new tables in the same deploy as the library bump. Table pre-warm runs at integration time, so a new container’s first cold start can create <prefix>_property_lookup_v2 before your CloudFormation/Terraform resource is created — the stack then fails with ResourceInUseException and rolls back. Pick one owner:

• declare the new tables in a deploy before the library upgrade (still on 3.12), then bump the library, then disable auto-creation; or

• let auto-creation own them and never declare them in IaC.

Doing both in one deploy is the race.

3. The IAM change is account-wide, not library-scoped. Dropping DescribeTable/CreateTable from the runtime role also breaks your own code that probes tables — boto3 table.load() or describe_table(), pynamodb Model.exists(). A caller that catches broadly will just log a warning and quietly disable a feature while re-issuing the denied call on every construction. Audit for those probes before flipping the flag; where you have one, honour the same switch rather than restoring the permission:

from actingweb.db.dynamodb import auto_create_enabled

if auto_create_enabled():
    ...  # safe to probe / create
else:
    ...  # tables are managed externally — skip the probe

Proving the fixes actually landed

Both headline fixes are invisible to functional tests: a Scan → Query conversion returns identical data, and removing DescribeTable calls changes no behaviour. What is observable is the operation profile — count the DynamoDB API calls a hot path issues. This works against DynamoDB Local, so it costs nothing and needs no production access.

Warning

The obvious implementation does not work. Registering a before-call.dynamodb handler on a botocore session silently counts nothing under pynamodb: pynamodb.connection.base.Connection.session resolves via botocore.session.get_session(), which returns a new Session per call, so a handler on any session you can reach never sees pynamodb’s traffic. The failure mode is an empty counter — which makes assert scans == 0 pass vacuously, worse than no test at all.

Patch BaseClient._make_api_call instead; every botocore client funnels through it:

import collections

import botocore.client

counts = collections.Counter()
original = botocore.client.BaseClient._make_api_call

def _counting(self, operation_name, api_params):
    if self.meta.service_model.service_name == "dynamodb":
        counts[operation_name] += 1
    return original(self, operation_name, api_params)

monkeypatch.setattr(botocore.client.BaseClient, "_make_api_call", _counting)

Exercise a hot path and assert the profile. Measured on identical code against 3.12.0 and 3.13.0:

Exercise	3.12.0	3.13.0
DbPropertyList().fetch(actor_id)	{'DescribeTable': 1, 'Scan': 1}	Query; Scan: 0
DbTrustList().fetch(actor_id)	{'DescribeTable': 1, 'Scan': 1}	Query; Scan: 0
25 × DbProperty() construction	{'DescribeTable': 25}	{} — 0 calls

DescribeTable: 25 → 0 for 25 constructions is exactly the one-call-per-construction defect and its removal. Sanity-check that the counter is live (assert a non-zero count for some operation) before trusting a zero.

In production, DescribeTable volume is visible in CloudTrail attributed to the runtime role. Expect a large drop on the upgrade alone (memoisation fixes the warm steady state) and zero only after AWS_DB_AUTO_CREATE_TABLES=false — the per-container pre-warm sweep is the residual. Make sure the measurement window contains a cold start, or the zero is vacuous.

Behaviour changes to be aware of

• Reverse-lookup matching is now (name, value), not value-only: the legacy GSI matched on value alone, so a value shared across different property names could resolve “cross-name”. The lookup table cannot, and the migration fallback tiers (v1 table, legacy GSI, PostgreSQL scan) now also filter by property name — so the value-only cross-name match is gone on every path, not just the primary one. The fallbacks remain only to serve un-backfilled deployments and are removed in the next major release.

• Non-indexed property names return None from Actor.get_from_property() / ActorInterface.get_by_property() (with a warning) instead of silently querying the legacy path. Add names to with_indexed_properties() to make them resolvable.

• Cross-actor lookup collisions are refused and logged instead of silently overwritten (DynamoDB previously last-writer-wins; PostgreSQL already refused — the backends now agree, including idempotent re-creates).

• Trust/peer-trustee list ordering is deterministic (range-key sorted) instead of arbitrary scan order.

• Subscription-suspension and peer-trustee-list accessors now auto-create their tables like every other accessor (previously a fresh deployment crashed on first use of suspension).

• The lookup table stores SHA-256 digests, not plaintext values — update your data map (see docs/reference/security.rst).

No changes needed for PostgreSQL schemas (the lookup table and its Alembic migration already exist); the default flip applies to both backends.