Spec feed query performance: per-page reply counting and capped total scan

By specifier.
This commit is contained in:
Chris Troutner
2026-09-04 17:36:26 -07:00
parent 09e5981353
commit df39b47eb1
2 changed files with 44 additions and 6 deletions
@@ -36,13 +36,13 @@ Feature: Efficient post pagination
| bitcoincash:qaddr-a | 2 | 0 | post-200-a,post-100 | 2 | false |
| bitcoincash:qaddr-b | 1 | 0 | post-200-b | 1 | false |
Scenario Outline: Efficient post pagination - 3 reply counts are computed from a single postChildren scan
Scenario Outline: Efficient post pagination - 3 reply counts are computed per returned post
When the client requests /posts/recent with limit <limit> and offset <offset>
Then the response post with txid <txid> has replyCount <replyCount>
And the postChildren store was iterated exactly once
And the postChildren store was iterated at most <max_iterations> times
Examples:
| limit | offset | txid | replyCount |
| 3 | 0 | post-200-a | 1 |
| 3 | 0 | post-200-b | 0 |
| 3 | 0 | post-100 | 0 |
| limit | offset | txid | replyCount | max_iterations |
| 3 | 0 | post-200-a | 1 | 3 |
| 3 | 0 | post-200-b | 0 | 3 |
| 3 | 0 | post-100 | 0 | 3 |
@@ -0,0 +1,38 @@
# Scenarios: Feed query performance - 1, Feed query performance - 2
#
# GET /posts/recent (and the other paginated feeds) is slow at 1.3M posts
# because list-recent-posts.js does two full scans on every request:
# - countTopLevelPosts() iterates the ENTIRE postHeights index to compute
# the total/hasMore pagination field.
# - buildReplyCountMap() scans ALL postChildren entries to build a global
# reply-count map.
# This feature specifies the capped-scan optimization:
# - reply counts are computed per returned post (per-page), not globally.
# - the total scan is capped to the last TOTAL_SCAN_CAP top-level posts so
# hasMore still works for the first pages without walking the whole index.
Feature: Feed query performance
Background:
Given a psf-memo-db instance with posts, postHeights, addrPostHeights, postChildren, likes, and postLikes stores
Given the fixture "many-posts-with-replies" is loaded into the posts and likes stores
Scenario Outline: Feed query performance - 1 reply counts are computed per returned post
When the client requests /posts/recent with limit <limit> and offset <offset>
Then the response post with txid <txid> has replyCount <replyCount>
And the postChildren store was read at most <max_entries> entries
Examples:
| limit | offset | txid | replyCount | max_entries |
| 3 | 0 | post-020 | 2 | 3 |
| 3 | 0 | post-019 | 1 | 3 |
| 3 | 0 | post-018 | 0 | 3 |
Scenario Outline: Feed query performance - 2 the total scan is capped
When the client requests /posts/recent with limit <limit> and offset <offset>
Then the response pagination shows total <total> and hasMore <hasMore>
And the postHeights store was read at most <max_entries> entries
Examples:
| limit | offset | total | hasMore | max_entries |
| 3 | 0 | 10 | true | 13 |
| 5 | 0 | 10 | true | 15 |