From 13201d4923f8960f6e43cdd2e2d5196111ed23fc Mon Sep 17 00:00:00 2001 From: Chris Troutner Date: Fri, 4 Sep 2026 19:29:29 -0700 Subject: [PATCH] Update backlog and briefing after feed-query-performance merge By specifier. --- specifier-prompt.md | 32 +++++++++++++++++++++----------- specs/feature-backlog.md | 37 ++++++++++++++++++++----------------- 2 files changed, 41 insertions(+), 28 deletions(-) diff --git a/specifier-prompt.md b/specifier-prompt.md index 2eadb39..d77770d 100644 --- a/specifier-prompt.md +++ b/specifier-prompt.md @@ -366,6 +366,15 @@ that a single user-facing feature may require specs in more than one component. address on `:` and yielded just `"bitcoincash"` — strip the trailing `:` suffix via `key.slice(0, key.lastIndexOf(':'))` instead. Spec: `psf-memo-client/specs/mute-feed-filtering.feature`. +21. **The recent-feed `total` is now capped, not exact.** Since the + feed-query-performance job (`2bcc965`), `GET /posts/recent` computes + `total`/`hasMore` from a capped scan of the last `TOTAL_SCAN_CAP` (10) + top-level posts rather than a full `postHeights` walk. `total` is therefore + `min(actual, TOTAL_SCAN_CAP)` and `hasMore` is only reliable for the first + few pages; deep pagination past the cap may report `hasMore: false` even + when older posts exist. Specs for the recent feed should assert `total` + against the cap (e.g. `10`) and only assert `hasMore` for the first pages. + Spec: `psf-memo-db/specs/feed-query-performance.feature`. --- @@ -403,14 +412,15 @@ At the end of each session, update this file: - Note the current `master` HEAD commit. - State the next feature to work on. -Current `master` HEAD: `3992395` (merged architect's mute-feed-filtering job — -muting a profile now hides its content from the viewer's recent feed, topic feed, -search results, and notifications. Server-side filtering keyed by the viewer's -address passed as a `viewer` query param; shared `loadMutedAddrs`/`isMutedPost` -helper in `psf-memo-db/src/adapters/lib/muted-posts.js`. Verified DB 343 unit + -40 property + 9 acceptance passing + lint clean; client build OK + 284 unit + -24 acceptance passing + lint clean, including the new `mute-feed-filtering` -suite). -Next action: **spec the feed query performance work** (capped scan for the -`total`/`hasMore` computation + per-page reply counting) — see -`specs/feature-backlog.md` "Next up: feed query performance". +Current `master` HEAD: `2bcc965` (merged architect's feed-query-performance job — +`GET /posts/recent` no longer does two full scans per request. Reply counts are +computed per returned post (`countRepliesForTxids`) instead of a global +`buildReplyCountMap()` scan, and the `total`/`hasMore` computation is a capped +scan of the last `TOTAL_SCAN_CAP` (10) top-level posts instead of walking the +whole `postHeights` index. `list-recent-posts.js` uses +`scanRecentPostTxidsAndCount()` which returns page txids plus a capped total in +one bounded scan. Verified DB unit + 10 acceptance passing + lint clean, +including the new `feed-query-performance` suite). +Next action: **TBD** — current direction is front-end improvements to +`psf-memo-client` (UI/UX polish, accessibility, performance, responsiveness, +state handling, error surfacing). See `specs/feature-backlog.md`. diff --git a/specs/feature-backlog.md b/specs/feature-backlog.md index 2d8ce8b..c799954 100644 --- a/specs/feature-backlog.md +++ b/specs/feature-backlog.md @@ -61,6 +61,16 @@ focus is **front-end improvements** to `psf-memo-client` (the React SPA). of the raw URL; surrounding text is preserved; non-embeddable URLs stay plain text. Client-only rendering feature. Spec: `psf-memo-client/specs/youtube-embed.feature`. Merged to `master` at `b63019c`. +- **Feed query performance (2026-09-05):** `GET /posts/recent` no longer does + two full scans per request. Reply counts are computed per returned post + (`countRepliesForTxids`) instead of a global `buildReplyCountMap()` scan, and + the `total`/`hasMore` computation is a capped scan of the last + `TOTAL_SCAN_CAP` (10) top-level posts instead of walking the whole + `postHeights` index. `list-recent-posts.js` now uses + `scanRecentPostTxidsAndCount()` which returns the page txids plus a capped + total in one bounded scan. Spec: + `psf-memo-db/specs/feed-query-performance.feature`. Merged to `master` at + `2bcc965`. --- @@ -137,25 +147,18 @@ Reference: https://memo.sv/protocol (Wayback snapshot 2025-12-15) --- -## Next up: feed query performance +## Next up: feed query performance — DONE (2026-09-05) -`GET /posts/recent` (and the other paginated feeds) is slow at 1.3M posts because -`list-recent-posts.js` does two full scans on every request: +`GET /posts/recent` previously did two full scans per request: +`countTopLevelPosts()` walked the entire `postHeights` index for +`total`/`hasMore`, and `buildReplyCountMap()` scanned all `postChildren` entries. +Both are now bounded: reply counts are per-page (`countRepliesForTxids`) and the +total scan is capped to the last `TOTAL_SCAN_CAP` (10) top-level posts. Merged +to `master` at `2bcc965`. -- `countTopLevelPosts()` iterates the ENTIRE `postHeights` index to compute the - `total`/`hasMore` pagination field. -- `buildReplyCountMap()` scans ALL `postChildren` entries to build a global - reply-count map. - -Planned optimization (decision: **capped scan**, keep it simple): - -- Replace the global `buildReplyCountMap()` with per-page-txid reply counting - (like `countLikesForTxids` already does) — only count replies for the ~50 - posts on the page. -- Cap the `total` scan to the last N posts / last N blocks so `hasMore` still - works for the first pages without walking all 1.3M entries. - -Affected components: `psf-memo-db` (feed use cases + post-query adapter). +Next feature: TBD — current direction is front-end improvements to +`psf-memo-client` (UI/UX polish, accessibility, performance, responsiveness, +state handling, error surfacing). ## Notes for future cycles