psf-memo-indexer/dev-docs/theory-of-operation.md

Theory of Operation

This document walks through how the indexer behaves at runtime—from cold start through steady-state chain following—including the Memo-specific parsing rules.

Prerequisites

1. BCH full node with RPC and ZMQ enabled (rawtx, rawblock on the configured port, default 28332).
2. psf-memo-db listening (default http://localhost:5021).
3. Environment variables in .env or Docker-mounted .env files (see .env-example).

Phase 1: Initialization

Both indexer processes:

1. Construct Adapters, UseCases, and Controllers.
2. Block indexer enables keyboard listener (q stops after current block).
3. TX indexer starts Express on TX_REST_API_PORT and waits for /tx-start.

Status bootstrap (status-db.js):

• GET /level/status/status from psf-memo-db.
• If missing (fresh DB), create:

{
  "startBlockHeight": START_BLOCK_HEIGHT - 1,
  "syncedBlockHeight": START_BLOCK_HEIGHT - 1,
  "chainBlockHeight": <current RPC block count>
}

If EXIT_ON_MISSING_BACKUP=true and status is missing, the process exits instead of reinitializing—same safety valve as the SLP indexer when a backup is expected but absent.

Phase 2: Initial Block Download (IBD)

Only the block indexer runs IBD.

nextHeight = syncedBlockHeight + 1
tip        = RPC getblockcount

while nextHeight <= tip:
    processBlock(nextHeight)
    updateIndexedBlockHeight(nextHeight)
    optionally backup every 1000 blocks
    nextHeight++

Per-block pipeline (index-blocks.processBlock)

sequenceDiagram
  participant BI as Block_indexer
  participant RPC as Full_node_RPC
  participant FB as filter_block
  participant TX as transaction_adapter
  participant H as action_handlers
  participant DB as psf_memo_db

  BI->>RPC: getblockhash(height)
  BI->>RPC: getblock(hash)
  BI->>FB: filterMemoTxs(txids[])
  loop each candidate txid
    FB->>TX: isMemoTx(txid)
  end
  loop each memo txid (parallel, up to MEMO_TX_CONCURRENCY)
    BI->>BI: processMemoTx(txid)
    BI->>DB: reads/writes via handlers
  end

Step 1 — Fetch block: getblock returns the list of transaction ids in the block (verbosity 1).

Step 2 — Filter: filterMemoTxs runs up to FILTER_CONCURRENCY (default 20) concurrent isMemoTx checks (p-queue). Each check loads the transaction (with in-memory cache in transaction.js) and scans all outputs for a Memo OP_RETURN. Results are returned in block tx order (parallel checks use a set, then the original block list is filtered).

Why scan all outputs? Unlike SLP, which conventionally places token data in vout[0], Memo actions may appear in any output’s scriptPubKey. The Go reference iterates every TxOut.

Step 3 — Process: processMemoTxs runs up to MEMO_TX_CONCURRENCY (default 20) concurrent processMemoTx calls via p-queue. There is no DAG sort—Memo social actions do not form token-like dependency chains within a block. Multiple OP_RETURN outputs in a single transaction are still handled sequentially inside processMemoTx.

Phase 3: Processing a single Memo transaction

Idempotency

Before heavy work, the indexer checks GET /level/ptx/{txid}. If present, the tx is skipped. After successful handling, it POSTs a ptx record { processedAt, blockHeight }.

• seen on indexed entities (profiles, posts, likes, etc.) is the on-chain block header time in Unix epoch milliseconds (block.time * 1000 from the BCH full node).
• processedAt on ptx records is indexer wall-clock time in milliseconds (Date.now() when the tx was processed).

Nuance: If a handler partially fails mid-tx, v1 does not roll back prior writes in that tx. The Go indexer uses richer DB transactions; PSF v1 relies on idempotency only at tx boundaries. See tradeoffs doc.

Signer address

Memo actions are attributed to the first P2PKH input that yields an address from the unlocking script (getSignerAddress in memo-parser.js, backed by @psf/bch-js where possible).

If no address is found:

• Log processErrors with reason.
• Skip action handlers (no anonymous posts in v1).

This mirrors the Go SetLockHash loop over inputs.

OP_RETURN decoding

parseScriptPushDatas walks Bitcoin script bytes (push opcodes 0x01–0x4e, OP_RETURN 0x6a, etc.) and collects pushdata buffers.

A Memo action is recognized when any pushdata begins with:

0x6d <action_byte>

decodeMemoOpReturn returns { action, prefix, pushDatas } where action is a string key (post, reply, like, …) mapped in memo-codes.js.

Handler dispatch

dispatchMemoAction routes to src/use-cases/action-types/*.js. Each handler:

1. Validates pushdata count and field sizes (against MAX_POST_SIZE etc. from Go memo.go, typically 65000 bytes in modern limits).
2. On validation failure: write processErrors, return without throwing (invalid Memo txs do not halt the block).
3. On success: POST / PUT to the appropriate /level/{entity} route.

Multi-output transactions

If one transaction contains multiple Memo OP_RETURN outputs, each decoded output dispatches separately in a loop. Rare in practice but supported.

Action-specific behavior (v1)

Action	Pushdata layout	Side effects beyond primary store
setName	prefix + UTF-8 name	names by address
post	prefix + message	posts by txid; skip create if post exists
reply	prefix + 32-byte parent hash + message	postParents, postChildren, then post body
like	prefix + 32-byte post hash	likes; optional tip satoshis to post author outputs
setProfile	prefix + text	profiles
follow / unfollow	prefix + 20-byte pk hash	follows keyed by follower and followee
setProfilePic	prefix + URL	profilePics
topicMessage	prefix + topic + message	post + rooms index
topicFollow / topicUnfollow	prefix + topic name	rooms follow record

Like-tip logic: if the liked post exists and the liker is not the author, outputs paying the author’s address are summed (values converted from BCH decimal to satoshis) and stored on the like record.

Phase 4: IBD completion and TX indexer handoff

When nextBlockHeight > tip:

1. Block indexer connects ZMQ.
2. GET /tx-start on TX indexer.
3. Enters ZMQ block loop (500 ms sleep, rawblock queue).

TX indexer:

1. Already listening on Express.
2. After /tx-start, connects ZMQ and processes rawtx.
3. Maintains seenTxs Set + FIFO queue capped at seenTxMax to drop duplicate ZMQ notifications cheaply before hitting ptxDb.
4. Calls the same processMemoTx with blockHeight = chain tip + 1 for unconfirmed txs.

Phase 5: Steady state

Event source	Process	Behavior
New block (ZMQ)	Block indexer	Resolve height from block header, update status, processBlock
New mempool tx (ZMQ)	TX indexer	Dedupe → processMemoTx
Every 1000 blocks	Block indexer	POST /level/backup with { height, epoch: 1000 }

Backups

psf-memo-db closes LevelDB files, zips leveldb/current to leveldb/zips/memo-indexer-{height}.zip, reopens DBs. Old zips are pruned per BACKUP_QTY.

Restore (POST /level/restore) unzips and exits the process—expect process manager restart, same as SLP.

Memo vs SLP detection (operational difference)

	SLP indexer	Memo indexer
Detection	Lokad 534c5000 + slp-parser	Prefix 6d + action byte
Output index	Assumes vout[0] for token data	Any vout
Ordering	DAG sort within block	Sequential tx order
State model	Token UTXOs, balances	Social graph documents

Failure and retry behavior

• RPC: RetryQueue and p-retry (5 attempts) on block filter operations.
• DB HTTP: Errors bubble up; block processing logs and throws on processMemoTx failure.
• Invalid Memo payload: Logged to processErrors; block continues.
• Keyboard q: Block indexer exits after finishing current block.

Configuration reference

Variable	Default	Effect
PSF_MEMO_DB_URL	http://localhost:5021	All DB adapters
START_BLOCK_HEIGHT	525000	Genesis for indexer state
RPC_* / ZMQ_PORT	see .env-example	Full node
TX_REST_API_PORT	5455	TX control (SLP uses 5454)
seenTxMax	100000	Mempool dedupe set size
txCacheMax	100000	RPC tx cache in transaction adapter