mirror of
https://github.com/Permissionless-Software-Foundation/psf-memo.git
synced 2026-09-21 16:52:01 -07:00
Spec notifications query performance
Bound GET /posts/notifications/:addr to the viewer's activity in a configurable block window instead of full-scanning likes, postChildren, and follows. Adds the followeeHeights index spec, its backfill spec, and the notifications-query-performance spec. By specifier.
This commit is contained in:
Chris Troutner
@@ -0,0 +1,36 @@
|
||||
# Scenarios: Backfill Followee Index - 1, Backfill Followee Index - 2
|
||||
#
|
||||
# followeeHeights is a new notification read index. Existing follows store
|
||||
# records predate it, so the backfill utility writes one followeeHeights entry
|
||||
# per follows record at that record's latest block height. It rebuilds from the
|
||||
# follows store only, is idempotent, and leaves the follows store unchanged.
|
||||
#
|
||||
# Fixture "follows-by-followee":
|
||||
# viewer = bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue
|
||||
# follows: followerA:viewer unfollow false at 690400
|
||||
# followerB:viewer unfollow true at 690600
|
||||
# followerC:other unfollow false at 690200
|
||||
Feature: Backfill Followee Index
|
||||
|
||||
Background:
|
||||
Given a psf-memo-db instance with follows and followeeHeights stores
|
||||
Given the fixture "follows-by-followee" is loaded into the follows store
|
||||
|
||||
Scenario Outline: Backfill Followee Index - 1 backfill indexes every follow record at its height
|
||||
When the followee index backfill utility is run
|
||||
Then the followeeHeights store contains <count> entry for followee <followee> from <follower> at block height <height> marked unfollow <unfollow>
|
||||
|
||||
Examples:
|
||||
| followee | follower | height | unfollow | count |
|
||||
| bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue | bitcoincash:qq3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zygrg4dtdzf | 690400 | false | 1 |
|
||||
| bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue | bitcoincash:qqenxvenxvenxvenxvenxvenxvenxvenxvn254yg3p | 690600 | true | 1 |
|
||||
| bitcoincash:qpzyg3zyg3zyg3zyg3zyg3zyg3zyg3zygs7fn3s6pt | bitcoincash:qp24242424242424242424242424242425wtjflljr | 690200 | false | 1 |
|
||||
|
||||
Scenario Outline: Backfill Followee Index - 2 backfill is idempotent
|
||||
When the followee index backfill utility is run
|
||||
And the followee index backfill utility is run again
|
||||
Then the followeeHeights store contains <count> entry for followee <followee> from <follower> at block height <height> marked unfollow <unfollow>
|
||||
|
||||
Examples:
|
||||
| followee | follower | height | unfollow | count |
|
||||
| bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue | bitcoincash:qq3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zygrg4dtdzf | 690400 | false | 1 |
|
||||
@@ -0,0 +1,95 @@
|
||||
# Scenarios: Notifications Query Performance - 1, Notifications Query Performance - 2, Notifications Query Performance - 3, Notifications Query Performance - 4
|
||||
#
|
||||
# GET /posts/notifications/:addr originally scanned the ENTIRE likes,
|
||||
# postChildren, and follows stores and loaded a post for every candidate, so a
|
||||
# request cost O(all likes + all replies + all follows) and the /notifications
|
||||
# page blocked on it. This feature bounds the work to the viewer's activity
|
||||
# inside a configurable block window (NOTIFICATION_BLOCK_WINDOW, default
|
||||
# 25000; cutoff = status.chainBlockHeight - window):
|
||||
#
|
||||
# - The viewer's own posts are read from addrPostHeights, range-limited to
|
||||
# the window.
|
||||
# - Likes and replies are read by prefix-scanning postLikes and postChildren
|
||||
# for those posts only; the global likes and postChildren stores are never
|
||||
# iterated.
|
||||
# - Follows are read from the followeeHeights index, range-limited to the
|
||||
# window; the global follows store is never iterated.
|
||||
# - pagination.total counts only the in-window notifications.
|
||||
#
|
||||
# A notification is drawn from the viewer's content inside the window, so an
|
||||
# interaction with a post older than the window is not returned even when the
|
||||
# interaction itself is recent.
|
||||
#
|
||||
# Fixture "notifications-window" (chainBlockHeight 700000):
|
||||
# viewer = bitcoincash:qqg3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zye3kwllue
|
||||
# followerA = bitcoincash:qq3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zygrg4dtdzf
|
||||
# followerB = bitcoincash:qqenxvenxvenxvenxvenxvenxvenxvenxvn254yg3p
|
||||
# oldActor = bitcoincash:qpzyg3zyg3zyg3zyg3zyg3zyg3zyg3zygs7fn3s6pt
|
||||
# liker = bitcoincash:qp24242424242424242424242424242425wtjflljr
|
||||
# replier = bitcoincash:qpnxvenxvenxvenxvenxvenxvenxvenxvc5j32tdvn
|
||||
# posts: post-recent (viewer, 690000), post-old (viewer, 600000),
|
||||
# post-other (followerA, 690000)
|
||||
# postLikes: post-recent:like-recent (liker, 690100)
|
||||
# post-old:like-old (oldActor, 690200)
|
||||
# post-other:like-other (followerA, 690300)
|
||||
# postChildren: post-recent:reply-recent (replier, 690150)
|
||||
# post-old:reply-old (oldActor, 690250)
|
||||
# post-other:reply-other (followerA, 690300)
|
||||
# followeeHeights: viewer:690400:followerA unfollow false
|
||||
# viewer:690550:followerB unfollow false
|
||||
# viewer:690600:followerB unfollow true
|
||||
# viewer:600000:oldActor unfollow false
|
||||
# filler: 100 unrelated likes, 100 unrelated postChildren entries, and 100
|
||||
# unrelated followeeHeights entries so a full-store scan is observable.
|
||||
Feature: Notifications Query Performance
|
||||
|
||||
Background:
|
||||
Given a psf-memo-db instance with posts, postHeights, addrPostHeights, postChildren, likes, postLikes, follows, followeeHeights, and status stores
|
||||
Given the fixture "notifications-window" is loaded
|
||||
|
||||
Scenario Outline: Notifications Query Performance - 1 in-window notifications paginate by the configured window
|
||||
Given a notification window of <window> blocks
|
||||
When the client requests /posts/notifications/<viewer> with limit <limit> and offset <offset>
|
||||
Then the response contains <count> notifications
|
||||
And the response pagination shows total <total> and hasMore <hasMore>
|
||||
|
||||
Examples:
|
||||
| window | limit | offset | count | total | hasMore |
|
||||
| 25000 | 2 | 0 | 2 | 3 | true |
|
||||
| 25000 | 2 | 2 | 1 | 3 | false |
|
||||
| 100000 | 10 | 0 | 6 | 6 | false |
|
||||
|
||||
Scenario Outline: Notifications Query Performance - 2 in-window notifications are ordered newest first
|
||||
Given a notification window of 25000 blocks
|
||||
When the client requests /posts/notifications/<viewer> with limit <limit> and offset <offset>
|
||||
Then the response notification at index <index> has type <type> from <actor>
|
||||
|
||||
Examples:
|
||||
| limit | offset | index | type | actor |
|
||||
| 5 | 0 | 0 | follow | bitcoincash:qq3zyg3zyg3zyg3zyg3zyg3zyg3zyg3zygrg4dtdzf |
|
||||
| 5 | 0 | 1 | reply | bitcoincash:qpnxvenxvenxvenxvenxvenxvenxvenxvc5j32tdvn |
|
||||
| 5 | 0 | 2 | like | bitcoincash:qp24242424242424242424242424242425wtjflljr |
|
||||
|
||||
Scenario Outline: Notifications Query Performance - 3 out-of-window and unrelated interactions are excluded
|
||||
Given a notification window of 25000 blocks
|
||||
When the client requests /posts/notifications/<viewer> with limit <limit> and offset <offset>
|
||||
Then the response contains no notification from <excluded_actor>
|
||||
|
||||
Examples:
|
||||
| limit | offset | excluded_actor |
|
||||
| 10 | 0 | bitcoincash:qpzyg3zyg3zyg3zyg3zyg3zyg3zyg3zygs7fn3s6pt |
|
||||
| 10 | 0 | bitcoincash:qqenxvenxvenxvenxvenxvenxvenxvenxvn254yg3p |
|
||||
|
||||
Scenario Outline: Notifications Query Performance - 4 likes and replies are read per viewer post and follows per index
|
||||
Given a notification window of 25000 blocks
|
||||
When the client requests /posts/notifications/<viewer> with limit <limit> and offset <offset>
|
||||
Then the postLikes store was read at most <max_post_likes> entries
|
||||
And the postChildren store was read at most <max_children> entries
|
||||
And the followeeHeights store was read at most <max_followee> entries
|
||||
And the likes store was not iterated
|
||||
And the follows store was not iterated
|
||||
|
||||
Examples:
|
||||
| limit | offset | max_post_likes | max_children | max_followee |
|
||||
| 2 | 0 | 1 | 1 | 3 |
|
||||
| 10 | 0 | 1 | 1 | 3 |
|
||||
@@ -0,0 +1,49 @@
|
||||
# Scenarios: Followee Heights Indexing - 1, Followee Heights Indexing - 2, Followee Heights Indexing - 3
|
||||
#
|
||||
# GET /posts/notifications/:addr lists the follows of the viewer. The follows
|
||||
# store is keyed <followerAddr>:<followeePkHash>, so finding the viewer's
|
||||
# followers requires scanning the whole follows store. This feature adds a
|
||||
# followee-keyed, height-ordered index that the read side can range-scan for
|
||||
# only the viewer's recent follows:
|
||||
#
|
||||
# followeeHeights key = <followeePkHash>:<padded blockHeight>:<followerAddr>
|
||||
# followeeHeights value = { followerAddr, followeePkHash, unfollow, txid, seen, blockHeight }
|
||||
#
|
||||
# Every follow action (follow or unfollow) writes one entry at its block
|
||||
# height. The read side takes the newest entry per follower and ignores
|
||||
# unfollows, so an unfollow removes the notification without deleting the
|
||||
# event history. The follows store remains the source of truth for the
|
||||
# current follow graph; followeeHeights is a notification read index.
|
||||
Feature: Followee Heights Indexing
|
||||
|
||||
Background:
|
||||
Given a psf-memo-db instance with follows and followeeHeights stores
|
||||
Given a psf-memo-indexer configured to write to that database
|
||||
|
||||
Scenario Outline: Followee Heights Indexing - 1 a follow records an entry for the followee
|
||||
When the indexer processes a Memo follow of <followee> from <follower> at block height <height>
|
||||
Then the followeeHeights store contains <count> entry for followee <followee> from <follower> at block height <height> marked unfollow false
|
||||
|
||||
Examples:
|
||||
| follower | followee | height | count |
|
||||
| bitcoincash:qqlrzp23w08434twmvr4fxw672whkjy0py26r63g3d | bitcoincash:qr95sy3j9xwd2ap32xkykttr4cvcu7as4y0qverfuy | 690100 | 1 |
|
||||
| bitcoincash:qpm2qsznhks23z7629mms6s4cwef74vcwvy22gdx6a | bitcoincash:qr95sy3j9xwd2ap32xkykttr4cvcu7as4y0qverfuy | 690200 | 1 |
|
||||
|
||||
Scenario Outline: Followee Heights Indexing - 2 an unfollow records a second entry marked unfollow
|
||||
When the indexer processes a Memo follow of <followee> from <follower> at block height <height>
|
||||
And the indexer processes a Memo unfollow of <followee> from <follower> at block height <unfollowHeight>
|
||||
Then the followeeHeights store contains <count> entry for followee <followee> from <follower> at block height <height> marked unfollow false
|
||||
And the followeeHeights store contains <count> entry for followee <followee> from <follower> at block height <unfollowHeight> marked unfollow true
|
||||
|
||||
Examples:
|
||||
| follower | followee | height | unfollowHeight | count |
|
||||
| bitcoincash:qqlrzp23w08434twmvr4fxw672whkjy0py26r63g3d | bitcoincash:qr95sy3j9xwd2ap32xkykttr4cvcu7as4y0qverfuy | 690100 | 690900 | 1 |
|
||||
|
||||
Scenario Outline: Followee Heights Indexing - 3 reprocessing a follow is idempotent
|
||||
When the indexer processes a Memo follow of <followee> from <follower> at block height <height>
|
||||
And the indexer processes the same Memo follow of <followee> from <follower> again
|
||||
Then the followeeHeights store contains <count> entry for followee <followee> from <follower> at block height <height> marked unfollow false
|
||||
|
||||
Examples:
|
||||
| follower | followee | height | count |
|
||||
| bitcoincash:qpm2qsznhks23z7629mms6s4cwef74vcwvy22gdx6a | bitcoincash:qr95sy3j9xwd2ap32xkykttr4cvcu7as4y0qverfuy | 690300 | 1 |
|
||||
@@ -31,7 +31,27 @@ focus is **front-end improvements** to `psf-memo-client` (the React SPA).
|
||||
|
||||
## In progress
|
||||
|
||||
_(none)_
|
||||
- **Notifications query performance (2026-09-18):** `GET /posts/notifications/:addr`
|
||||
is slow because it full-scans the `likes`, `postChildren`, and `follows`
|
||||
stores and loads a post for every candidate. The fix bounds the work to the
|
||||
viewer's activity inside a configurable block window
|
||||
(`NOTIFICATION_BLOCK_WINDOW`, default 25000; cutoff =
|
||||
`status.chainBlockHeight - window`): the viewer's posts are read from
|
||||
`addrPostHeights` within the window, likes and replies are prefix-scanned
|
||||
from `postLikes`/`postChildren` per post, and follows are read from a new
|
||||
followee-keyed, height-ordered `followeeHeights` index maintained by the
|
||||
indexer. `pagination.total` counts only in-window notifications. Affected
|
||||
components: `psf-memo-indexer` (index writes) and `psf-memo-db` (read path,
|
||||
config). Specs:
|
||||
`psf-memo-indexer/specs/followee-heights-indexing.feature`,
|
||||
`psf-memo-db/specs/backfill-followee-index.feature`,
|
||||
`psf-memo-db/specs/notifications-query-performance.feature`.
|
||||
**Developer documentation:** update
|
||||
`psf-memo-indexer/dev-docs/psf-memo-db.md` (new `followeeHeights` store,
|
||||
`NOTIFICATION_BLOCK_WINDOW` config, read-route notes) and
|
||||
`psf-memo-indexer/dev-docs/design-decisions-and-tradeoffs.md` (why per-object
|
||||
scans plus a block window replace the full-store scans), and let the architect
|
||||
record the per-task what/why summary under `docs/reviews/`.
|
||||
|
||||
## Recently completed
|
||||
|
||||
@@ -324,10 +344,14 @@ Reference: https://memo.sv/protocol (Wayback snapshot 2025-12-15)
|
||||
small; handoffs are blocked on a dirty tree. See `docs/process-improvements.md`
|
||||
for the what/why, commits, and verification evidence.
|
||||
|
||||
## Next up: TBD
|
||||
## Next up: Notifications query performance
|
||||
|
||||
Current direction is front-end improvements to `psf-memo-client` (UI/UX polish,
|
||||
accessibility, performance, responsiveness, state handling, error surfacing).
|
||||
Specification complete and awaiting coder handoff. Make
|
||||
`GET /posts/notifications/:addr` bounded to the viewer's activity in a
|
||||
configurable block window instead of full-scanning `likes`, `postChildren`, and
|
||||
`follows`. See **In progress** above for the specs and the developer
|
||||
documentation deliverable. Current direction otherwise remains front-end
|
||||
improvements to `psf-memo-client`.
|
||||
|
||||
## Notes for future cycles
|
||||
|
||||
|
||||
Reference in New Issue
Block a user