From df39b47eb1faaa9cb4e9ebcf3b22f518803cfe75 Mon Sep 17 00:00:00 2001 From: Chris Troutner Date: Fri, 4 Sep 2026 17:36:26 -0700 Subject: [PATCH] Spec feed query performance: per-page reply counting and capped total scan By specifier. --- .../specs/efficient-post-pagination.feature | 12 +++--- .../specs/feed-query-performance.feature | 38 +++++++++++++++++++ 2 files changed, 44 insertions(+), 6 deletions(-) create mode 100644 psf-memo-db/specs/feed-query-performance.feature diff --git a/psf-memo-db/specs/efficient-post-pagination.feature b/psf-memo-db/specs/efficient-post-pagination.feature index 62641b7..8e816e6 100644 --- a/psf-memo-db/specs/efficient-post-pagination.feature +++ b/psf-memo-db/specs/efficient-post-pagination.feature @@ -36,13 +36,13 @@ Feature: Efficient post pagination | bitcoincash:qaddr-a | 2 | 0 | post-200-a,post-100 | 2 | false | | bitcoincash:qaddr-b | 1 | 0 | post-200-b | 1 | false | - Scenario Outline: Efficient post pagination - 3 reply counts are computed from a single postChildren scan + Scenario Outline: Efficient post pagination - 3 reply counts are computed per returned post When the client requests /posts/recent with limit and offset Then the response post with txid has replyCount - And the postChildren store was iterated exactly once + And the postChildren store was iterated at most times Examples: - | limit | offset | txid | replyCount | - | 3 | 0 | post-200-a | 1 | - | 3 | 0 | post-200-b | 0 | - | 3 | 0 | post-100 | 0 | + | limit | offset | txid | replyCount | max_iterations | + | 3 | 0 | post-200-a | 1 | 3 | + | 3 | 0 | post-200-b | 0 | 3 | + | 3 | 0 | post-100 | 0 | 3 | diff --git a/psf-memo-db/specs/feed-query-performance.feature b/psf-memo-db/specs/feed-query-performance.feature new file mode 100644 index 0000000..067ed0f --- /dev/null +++ b/psf-memo-db/specs/feed-query-performance.feature @@ -0,0 +1,38 @@ +# Scenarios: Feed query performance - 1, Feed query performance - 2 +# +# GET /posts/recent (and the other paginated feeds) is slow at 1.3M posts +# because list-recent-posts.js does two full scans on every request: +# - countTopLevelPosts() iterates the ENTIRE postHeights index to compute +# the total/hasMore pagination field. +# - buildReplyCountMap() scans ALL postChildren entries to build a global +# reply-count map. +# This feature specifies the capped-scan optimization: +# - reply counts are computed per returned post (per-page), not globally. +# - the total scan is capped to the last TOTAL_SCAN_CAP top-level posts so +# hasMore still works for the first pages without walking the whole index. +Feature: Feed query performance + + Background: + Given a psf-memo-db instance with posts, postHeights, addrPostHeights, postChildren, likes, and postLikes stores + Given the fixture "many-posts-with-replies" is loaded into the posts and likes stores + + Scenario Outline: Feed query performance - 1 reply counts are computed per returned post + When the client requests /posts/recent with limit and offset + Then the response post with txid has replyCount + And the postChildren store was read at most entries + + Examples: + | limit | offset | txid | replyCount | max_entries | + | 3 | 0 | post-020 | 2 | 3 | + | 3 | 0 | post-019 | 1 | 3 | + | 3 | 0 | post-018 | 0 | 3 | + + Scenario Outline: Feed query performance - 2 the total scan is capped + When the client requests /posts/recent with limit and offset + Then the response pagination shows total and hasMore + And the postHeights store was read at most entries + + Examples: + | limit | offset | total | hasMore | max_entries | + | 3 | 0 | 10 | true | 13 | + | 5 | 0 | 10 | true | 15 |