psf-memo/psf-memo-indexer/dev-docs/psf-memo-db.md

psf-memo-db Architecture

psf-memo-db is the persistence layer for the Memo indexer. It is structurally modeled on psf-slp-db: multiple LevelDB instances behind a Koa REST API, with no business logic beyond backup and restore.

Why a separate service

Concern	How separation helps
File locking	Only one process opens LevelDB files; indexers use HTTP
Parallelism	Multiple indexer workers could share one DB API (future)
Backups	Close all DBs, zip, reopen—without stopping indexer logic in the same process
Operational familiarity	PSF already runs psf-slp-db beside psf-slp-indexer-g2

Tradeoff: Every write is an HTTP round trip. Local indexing pays latency versus embedded LevelDB, but gains operational consistency with the SLP stack.

Server bootstrap

Mirrors psf-slp-db/bin/server.js:

index.js
  └── bin/server.js
        ├── Koa + middleware (logger, bodyparser 100mb, CORS, error handler)
        ├── controllers.initAdapters()  → level-db.openDbs()
        ├── controllers.initUseCases()  → no-op start
        ├── controllers.attachRESTControllers()
        └── app.listen(PORT)  // default 5021

Intentionally omitted from psf-slp-db: MongoDB, Passport auth, IPFS/Helia, wallet, usage tracking, /slp query routes. noMongo: true and useIpfs: false in config.

LevelDB instances

Opened in src/adapters/level-db.js:

leveldb/current/{name}  // valueEncoding: 'json'

Instance	Cache hint	Indexer writes
status	64 MB	Block height sync state
posts	512 MB	Post and reply bodies
postParents	64 MB	Reply → parent link
postChildren	64 MB	Parent → children
likes	64 MB	Like events
names	64 MB	Display names
profiles	64 MB	Profile text
profilePics	64 MB	Avatar URLs
follows	64 MB	Follow graph edges
addrPostHeights	64 MB	Posts by address ordered by height
postLikes	64 MB	Like txids grouped by liked post
followeeHeights	64 MB	Follow/unfollow events keyed by followee and height
rooms	64 MB	Topic posts and follows
processErrors	64 MB	Skipped / invalid txs
ptxs	64 MB	Processed tx markers

Posts receive a larger cache because they are the highest-volume social object.

REST API shape

All routes are under /level with a consistent CRUD pattern generated from ENTITY_CONFIG in crud-handlers.js:

Method	Path pattern	Body (create)
POST	/level/{entity}	{ <idField>, <dataField> }
GET	/level/{entity}/:key	—
PUT	/level/{entity}/:key	{ <dataField> }
DELETE	/level/{entity}/:key	—

Entity route names

Route	idField	dataField	Example key
post	txid	postData	transaction id
postparent	txid	parentData	child txid
postchild	key	childData	parentTxid:childTxid
like	txid	likeData	like txid
name	addr	nameData	cash address
profile	addr	profileData	cash address
profilepic	addr	profilePicData	cash address
follow	key	followData	follower:followeePkHash
followeeheight	key	followeeHeightData	followeePkHash:paddedHeight:followerAddr
room	key	roomData	composite
processerror	txid	errorData	txid
ptx	txid	ptxData	txid

Read routes (query use cases)

Method	Path	Query params
GET	/profile/recent	limit (default 100, max 100), offset (default 0)
GET	/posts/recent	limit (default 100, max 100), offset (default 0)
GET	/posts/notifications/:addr	limit (default 100, max 100), offset (default 0)

Returns profiles or posts sorted by block height (newest first), using the blockHeight field stored on each entity document at indexing time. Tie-breaker: seen timestamp descending.

The seen field is Unix epoch milliseconds from the block header time (block.time * 1000 at indexing time).

Response shape (/profile/recent):

{
  "profiles": [
    {
      "addr": "bitcoincash:q...",
      "text": "...",
      "txid": "...",
      "seen": 1500000000000,
      "blockHeight": 600000
    }
  ],
  "pagination": { "limit": 100, "offset": 0, "total": 42, "hasMore": false }
}

Response shape (/posts/recent):

{
  "posts": [
    {
      "txid": "...",
      "addr": "bitcoincash:q...",
      "text": "...",
      "seen": 1500000000000,
      "blockHeight": 600000
    }
  ],
  "pagination": { "limit": 100, "offset": 0, "total": 42, "hasMore": false }
}

Implementation: profile-query / post-query adapter (LevelDB scan) → list-recent-profiles / list-recent-posts use case → REST controller.

Tradeoff: Full scan of profiles on each request; suitable for moderate corpus sizes. A height-indexed store would be needed for very large archives.

Notifications

GET /posts/notifications/:addr returns likes on the viewer's posts, replies to the viewer's posts, and follows of the viewer, newest first. Work is bounded to the viewer's activity inside a configurable block window (NOTIFICATION_BLOCK_WINDOW, default 25000; cutoff = status.chainBlockHeight - window):

• the viewer's posts are read from addrPostHeights, range-limited to the window;
• likes and replies are prefix-scanned from postLikes and postChildren for those posts only (the global likes and postChildren stores are never iterated);
• follows are read from followeeHeights, range-limited to the window, keeping the newest entry per follower and ignoring unfollows (the follows store is never iterated).

pagination.total counts only in-window notifications. Because a notification is drawn from the viewer's content inside the window, an interaction with a post older than the window is not returned even when the interaction itself is recent. The indexer writes followeeHeights on every follow/unfollow; existing databases need the one-time util/follow/backfill-followee-index.js backfill.

Status (special case)

Matches SLP status semantics:

• GET /level/status/:statusKey
• POST /level/status with { statusKey, statusData }
• PUT /level/status with { statusData } — key is always status
• DELETE /level/status/:statusKey

Indexer expects statusKey = status containing:

{
  "startBlockHeight": 524999,
  "syncedBlockHeight": 800000,
  "chainBlockHeight": 800001
}

Backup and restore

• POST /level/backup — body { height, epoch } → zip leveldb/current to leveldb/zips/memo-indexer-{height}.zip
• POST /level/restore — body { height } → unzip matching archive and process.exit(0)

Implemented in src/adapters/db-backup.js.

Health

GET /health → { status: 'ok' } for load balancers and compose checks.

Indexer adapter mapping

The indexer uses src/adapters/entity-db.js:

createEntityDb('post', 'txid', 'postData')
// → POST http://localhost:5021/level/post

status-db.js uses dedicated status endpoints rather than the generic factory.

Data modeling notes

Not normalized like SQL. LevelDB stores are document keyed for fast lookup by txid or address, similar to the Go db/item/memo objects but without sharding.

Denormalized block height. Every entity written by the indexer includes a blockHeight field (the block in which the memo transaction was confirmed, or tip + 1 for unconfirmed txs). This avoids ptx lookups when serving /recent query routes. The ptxs store remains for idempotency only.

Common fields on indexed documents:

Entity	Key	Stored fields (includes)
post	txid	addr, text, seen, blockHeight
profile	addr	text, txid, seen, addr, blockHeight
name	addr	name, txid, seen, addr, blockHeight
profilePic	addr	url, txid, seen, addr, blockHeight
like	txid	addr, postTxid, seen, tip, blockHeight
follow	composite key	followerAddr, followeePkHash, unfollow, txid, seen, blockHeight
followeeHeights	followeePkHash:paddedHeight:followerAddr	followerAddr, followeePkHash, unfollow, txid, seen, blockHeight
postParent / postChild	txid / parentTxid:childTxid	parentTxid, childTxid, blockHeight
room	composite key	room, txid, seen, type, blockHeight (+ addr for follows)
processError	txid	error, ts, blockHeight

No secondary indexes in v1. Queries like “all posts by address” may require scanning or a future memo-query adapter—out of scope for the indexer write path.

JSON values keep debugging simple; binary serialization would save space but break parity with psf-slp-db tooling.

Configuration

Variable	Default	Purpose
PORT	5021	REST listen (SLP uses 5020)
SVC_ENV	development	Config profile
BACKUP_QTY	3	Retained zip backups
EXIT_ON_MISSING_BACKUP	false	Fail restore if zip missing
NOTIFICATION_BLOCK_WINDOW	25000	Blocks before the chain tip that still count as a notification

Testing

Unit tests cover:

• bin/server.js startup with stubbed controllers
• Level controller status and post CRUD with mocked LevelDB

Run: npm test in the psf-memo-db directory.

Production

Docker build context is the monorepo root; the Dockerfile copies psf-memo-db/ and lives in psf-memo-indexer/production/docker/memo-db/. See architecture.md.