From 22928d764a174755d8c9ac5d208e70f3dd343237 Mon Sep 17 00:00:00 2001 From: Chris Troutner Date: Fri, 18 Sep 2026 09:28:25 -0700 Subject: [PATCH] Spec notifications query performance Bound GET /posts/notifications/:addr to the viewer's activity in a configurable block window instead of full-scanning likes, postChildren, and follows. Adds the followeeHeights index spec, its backfill spec, and the notifications-query-performance spec. By specifier. --- .../specs/backfill-followee-index.feature | 36 +++++++ .../notifications-query-performance.feature | 95 +++++++++++++++++++ .../specs/followee-heights-indexing.feature | 49 ++++++++++ specs/feature-backlog.md | 32 ++++++- 4 files changed, 208 insertions(+), 4 deletions(-) create mode 100644 psf-memo-db/specs/backfill-followee-index.feature create mode 100644 psf-memo-db/specs/notifications-query-performance.feature create mode 100644 psf-memo-indexer/specs/followee-heights-indexing.feature diff --git a/psf-memo-db/specs/backfill-followee-index.feature b/psf-memo-db/specs/backfill-followee-index.feature new file mode 100644 index 0000000..0ffacb0 --- /dev/null +++ b/psf-memo-db/specs/backfill-followee-index.feature @@ -0,0 +1,36 @@ +# Scenarios: Backfill Followee Index - 1, Backfill Followee Index - 2 +# +# followeeHeights is a new notification read index. Existing follows store +# records predate it, so the backfill utility writes one followeeHeights entry +# per follows record at that record's latest block height. It rebuilds from the +# follows store only, is idempotent, and leaves the follows store unchanged. +# +# Fixture "follows-by-followee": +# viewer = bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue +# follows: followerA:viewer unfollow false at 690400 +# followerB:viewer unfollow true at 690600 +# followerC:other unfollow false at 690200 +Feature: Backfill Followee Index + + Background: + Given a psf-memo-db instance with follows and followeeHeights stores + Given the fixture "follows-by-followee" is loaded into the follows store + + Scenario Outline: Backfill Followee Index - 1 backfill indexes every follow record at its height + When the followee index backfill utility is run + Then the followeeHeights store contains entry for followee from at block height marked unfollow + + Examples: + | followee | follower | height | unfollow | count | + | bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue | bitcoincash:qq3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zygrg4dtdzf | 690400 | false | 1 | + | bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue | bitcoincash:qqenxvenxvenxvenxvenxvenxvenxvenxvn254yg3p | 690600 | true | 1 | + | bitcoincash:qpzyg3zyg3zyg3zyg3zyg3zyg3zyg3zygs7fn3s6pt | bitcoincash:qp24242424242424242424242424242425wtjflljr | 690200 | false | 1 | + + Scenario Outline: Backfill Followee Index - 2 backfill is idempotent + When the followee index backfill utility is run + And the followee index backfill utility is run again + Then the followeeHeights store contains entry for followee from at block height marked unfollow + + Examples: + | followee | follower | height | unfollow | count | + | bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue | bitcoincash:qq3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zygrg4dtdzf | 690400 | false | 1 | diff --git a/psf-memo-db/specs/notifications-query-performance.feature b/psf-memo-db/specs/notifications-query-performance.feature new file mode 100644 index 0000000..fb3ef7f --- /dev/null +++ b/psf-memo-db/specs/notifications-query-performance.feature @@ -0,0 +1,95 @@ +# Scenarios: Notifications Query Performance - 1, Notifications Query Performance - 2, Notifications Query Performance - 3, Notifications Query Performance - 4 +# +# GET /posts/notifications/:addr originally scanned the ENTIRE likes, +# postChildren, and follows stores and loaded a post for every candidate, so a +# request cost O(all likes + all replies + all follows) and the /notifications +# page blocked on it. This feature bounds the work to the viewer's activity +# inside a configurable block window (NOTIFICATION_BLOCK_WINDOW, default +# 25000; cutoff = status.chainBlockHeight - window): +# +# - The viewer's own posts are read from addrPostHeights, range-limited to +# the window. +# - Likes and replies are read by prefix-scanning postLikes and postChildren +# for those posts only; the global likes and postChildren stores are never +# iterated. +# - Follows are read from the followeeHeights index, range-limited to the +# window; the global follows store is never iterated. +# - pagination.total counts only the in-window notifications. +# +# A notification is drawn from the viewer's content inside the window, so an +# interaction with a post older than the window is not returned even when the +# interaction itself is recent. +# +# Fixture "notifications-window" (chainBlockHeight 700000): +# viewer = bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue +# followerA = bitcoincash:qq3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zygrg4dtdzf +# followerB = bitcoincash:qqenxvenxvenxvenxvenxvenxvenxvenxvn254yg3p +# oldActor = bitcoincash:qpzyg3zyg3zyg3zyg3zyg3zyg3zyg3zygs7fn3s6pt +# liker = bitcoincash:qp24242424242424242424242424242425wtjflljr +# replier = bitcoincash:qpnxvenxvenxvenxvenxvenxvenxvenxvc5j32tdvn +# posts: post-recent (viewer, 690000), post-old (viewer, 600000), +# post-other (followerA, 690000) +# postLikes: post-recent:like-recent (liker, 690100) +# post-old:like-old (oldActor, 690200) +# post-other:like-other (followerA, 690300) +# postChildren: post-recent:reply-recent (replier, 690150) +# post-old:reply-old (oldActor, 690250) +# post-other:reply-other (followerA, 690300) +# followeeHeights: viewer:690400:followerA unfollow false +# viewer:690550:followerB unfollow false +# viewer:690600:followerB unfollow true +# viewer:600000:oldActor unfollow false +# filler: 100 unrelated likes, 100 unrelated postChildren entries, and 100 +# unrelated followeeHeights entries so a full-store scan is observable. +Feature: Notifications Query Performance + + Background: + Given a psf-memo-db instance with posts, postHeights, addrPostHeights, postChildren, likes, postLikes, follows, followeeHeights, and status stores + Given the fixture "notifications-window" is loaded + + Scenario Outline: Notifications Query Performance - 1 in-window notifications paginate by the configured window + Given a notification window of blocks + When the client requests /posts/notifications/ with limit and offset + Then the response contains notifications + And the response pagination shows total and hasMore + + Examples: + | window | limit | offset | count | total | hasMore | + | 25000 | 2 | 0 | 2 | 3 | true | + | 25000 | 2 | 2 | 1 | 3 | false | + | 100000 | 10 | 0 | 6 | 6 | false | + + Scenario Outline: Notifications Query Performance - 2 in-window notifications are ordered newest first + Given a notification window of 25000 blocks + When the client requests /posts/notifications/ with limit and offset + Then the response notification at index has type from + + Examples: + | limit | offset | index | type | actor | + | 5 | 0 | 0 | follow | bitcoincash:qq3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zygrg4dtdzf | + | 5 | 0 | 1 | reply | bitcoincash:qpnxvenxvenxvenxvenxvenxvenxvenxvc5j32tdvn | + | 5 | 0 | 2 | like | bitcoincash:qp24242424242424242424242424242425wtjflljr | + + Scenario Outline: Notifications Query Performance - 3 out-of-window and unrelated interactions are excluded + Given a notification window of 25000 blocks + When the client requests /posts/notifications/ with limit and offset + Then the response contains no notification from + + Examples: + | limit | offset | excluded_actor | + | 10 | 0 | bitcoincash:qpzyg3zyg3zyg3zyg3zyg3zyg3zyg3zygs7fn3s6pt | + | 10 | 0 | bitcoincash:qqenxvenxvenxvenxvenxvenxvenxvenxvn254yg3p | + + Scenario Outline: Notifications Query Performance - 4 likes and replies are read per viewer post and follows per index + Given a notification window of 25000 blocks + When the client requests /posts/notifications/ with limit and offset + Then the postLikes store was read at most entries + And the postChildren store was read at most entries + And the followeeHeights store was read at most entries + And the likes store was not iterated + And the follows store was not iterated + + Examples: + | limit | offset | max_post_likes | max_children | max_followee | + | 2 | 0 | 1 | 1 | 3 | + | 10 | 0 | 1 | 1 | 3 | diff --git a/psf-memo-indexer/specs/followee-heights-indexing.feature b/psf-memo-indexer/specs/followee-heights-indexing.feature new file mode 100644 index 0000000..db4a121 --- /dev/null +++ b/psf-memo-indexer/specs/followee-heights-indexing.feature @@ -0,0 +1,49 @@ +# Scenarios: Followee Heights Indexing - 1, Followee Heights Indexing - 2, Followee Heights Indexing - 3 +# +# GET /posts/notifications/:addr lists the follows of the viewer. The follows +# store is keyed :, so finding the viewer's +# followers requires scanning the whole follows store. This feature adds a +# followee-keyed, height-ordered index that the read side can range-scan for +# only the viewer's recent follows: +# +# followeeHeights key = :: +# followeeHeights value = { followerAddr, followeePkHash, unfollow, txid, seen, blockHeight } +# +# Every follow action (follow or unfollow) writes one entry at its block +# height. The read side takes the newest entry per follower and ignores +# unfollows, so an unfollow removes the notification without deleting the +# event history. The follows store remains the source of truth for the +# current follow graph; followeeHeights is a notification read index. +Feature: Followee Heights Indexing + + Background: + Given a psf-memo-db instance with follows and followeeHeights stores + Given a psf-memo-indexer configured to write to that database + + Scenario Outline: Followee Heights Indexing - 1 a follow records an entry for the followee + When the indexer processes a Memo follow of from at block height + Then the followeeHeights store contains entry for followee from at block height marked unfollow false + + Examples: + | follower | followee | height | count | + | bitcoincash:qqlrzp23w08434twmvr4fxw672whkjy0py26r63g3d | bitcoincash:qr95sy3j9xwd2ap32xkykttr4cvcu7as4y0qverfuy | 690100 | 1 | + | bitcoincash:qpm2qsznhks23z7629mms6s4cwef74vcwvy22gdx6a | bitcoincash:qr95sy3j9xwd2ap32xkykttr4cvcu7as4y0qverfuy | 690200 | 1 | + + Scenario Outline: Followee Heights Indexing - 2 an unfollow records a second entry marked unfollow + When the indexer processes a Memo follow of from at block height + And the indexer processes a Memo unfollow of from at block height + Then the followeeHeights store contains entry for followee from at block height marked unfollow false + And the followeeHeights store contains entry for followee from at block height marked unfollow true + + Examples: + | follower | followee | height | unfollowHeight | count | + | bitcoincash:qqlrzp23w08434twmvr4fxw672whkjy0py26r63g3d | bitcoincash:qr95sy3j9xwd2ap32xkykttr4cvcu7as4y0qverfuy | 690100 | 690900 | 1 | + + Scenario Outline: Followee Heights Indexing - 3 reprocessing a follow is idempotent + When the indexer processes a Memo follow of from at block height + And the indexer processes the same Memo follow of from again + Then the followeeHeights store contains entry for followee from at block height marked unfollow false + + Examples: + | follower | followee | height | count | + | bitcoincash:qpm2qsznhks23z7629mms6s4cwef74vcwvy22gdx6a | bitcoincash:qr95sy3j9xwd2ap32xkykttr4cvcu7as4y0qverfuy | 690300 | 1 | diff --git a/specs/feature-backlog.md b/specs/feature-backlog.md index 6af1b5a..7ac5d93 100644 --- a/specs/feature-backlog.md +++ b/specs/feature-backlog.md @@ -31,7 +31,27 @@ focus is **front-end improvements** to `psf-memo-client` (the React SPA). ## In progress -_(none)_ +- **Notifications query performance (2026-09-18):** `GET /posts/notifications/:addr` + is slow because it full-scans the `likes`, `postChildren`, and `follows` + stores and loads a post for every candidate. The fix bounds the work to the + viewer's activity inside a configurable block window + (`NOTIFICATION_BLOCK_WINDOW`, default 25000; cutoff = + `status.chainBlockHeight - window`): the viewer's posts are read from + `addrPostHeights` within the window, likes and replies are prefix-scanned + from `postLikes`/`postChildren` per post, and follows are read from a new + followee-keyed, height-ordered `followeeHeights` index maintained by the + indexer. `pagination.total` counts only in-window notifications. Affected + components: `psf-memo-indexer` (index writes) and `psf-memo-db` (read path, + config). Specs: + `psf-memo-indexer/specs/followee-heights-indexing.feature`, + `psf-memo-db/specs/backfill-followee-index.feature`, + `psf-memo-db/specs/notifications-query-performance.feature`. + **Developer documentation:** update + `psf-memo-indexer/dev-docs/psf-memo-db.md` (new `followeeHeights` store, + `NOTIFICATION_BLOCK_WINDOW` config, read-route notes) and + `psf-memo-indexer/dev-docs/design-decisions-and-tradeoffs.md` (why per-object + scans plus a block window replace the full-store scans), and let the architect + record the per-task what/why summary under `docs/reviews/`. ## Recently completed @@ -324,10 +344,14 @@ Reference: https://memo.sv/protocol (Wayback snapshot 2025-12-15) small; handoffs are blocked on a dirty tree. See `docs/process-improvements.md` for the what/why, commits, and verification evidence. -## Next up: TBD +## Next up: Notifications query performance -Current direction is front-end improvements to `psf-memo-client` (UI/UX polish, -accessibility, performance, responsiveness, state handling, error surfacing). +Specification complete and awaiting coder handoff. Make +`GET /posts/notifications/:addr` bounded to the viewer's activity in a +configurable block window instead of full-scanning `likes`, `postChildren`, and +`follows`. See **In progress** above for the specs and the developer +documentation deliverable. Current direction otherwise remains front-end +improvements to `psf-memo-client`. ## Notes for future cycles