Adding dev docs

dev-docs/README.md

@@ -0,0 +1,24 @@
+# Memo Indexer Developer Documentation
+
+This directory documents the architecture, operation, and design rationale for the Permissionless Software Foundation Memo indexing stack:
+
+| Repository | Role |
+|------------|------|
+| [psf-memo-db](../../psf-memo-db) | LevelDB persistence exposed as a REST API |
+| [psf-memo-indexer](../) | Block and mempool indexer (this repo) |
+
+Protocol reference: [memo-protocol.md](../../memo-protocol.md). Reference implementation: [memo/index](../../index) (Go).
+
+## Reading order
+
+1. **[Overview](./overview.md)** — Why these services exist and what they produce
+2. **[Architecture](./architecture.md)** — Components, processes, Clean Architecture layout, data flow
+3. **[Theory of operation](./theory-of-operation.md)** — IBD, ZMQ, parsing, indexing pipeline step-by-step
+4. **[psf-memo-db](./psf-memo-db.md)** — Database service design and REST contract
+5. **[Design decisions and tradeoffs](./design-decisions-and-tradeoffs.md)** — Comparisons to SLP and the Go indexer, scope limits, future work
+
+## Related material
+
+- Production deployment: [production/docker](../production/docker/)
+- SLP analogue: [psf-slp-indexer-g2](https://github.com/Permissionless-Software-Foundation/psf-slp-indexer-g2) and [psf-slp-db](https://github.com/Permissionless-Software-Foundation/psf-slp-db)
+- Clean Architecture primer: [Chris Troutner — Clean Architecture](https://christroutner.github.io/trouts-blog/blog/clean-architecture)

dev-docs/architecture.md

@@ -0,0 +1,177 @@
+# Architecture
+
+## System context
+
+```mermaid
+flowchart TB
+  subgraph node [BCH_full_node]
+    RPC[JSON-RPC]
+    ZMQ[ZMQ_rawtx_rawblock]
+  end
+
+  subgraph indexer [psf_memo_indexer]
+    BI[Block_indexer_process]
+    TI[TX_indexer_process]
+    BI -->|GET_tx-start| TI
+  end
+
+  subgraph db [psf_memo_db]
+    Koa[Koa_REST]
+    LDB[(LevelDB_instances)]
+    Koa --> LDB
+  end
+
+  RPC --> BI
+  RPC --> TI
+  ZMQ --> BI
+  ZMQ --> TI
+  BI -->|axios_level_CRUD| Koa
+  TI -->|axios_level_CRUD| Koa
+```
+
+## Two-process indexer design
+
+The indexer splits work the same way as [psf-slp-indexer-g2](https://github.com/Permissionless-Software-Foundation/psf-slp-indexer-g2):
+
+| Process | Responsibility | Why separate |
+|---------|----------------|--------------|
+| **Block indexer** | Historical catch-up (IBD) and confirmed blocks via ZMQ | Heavy, sequential block work; must finish before mempool indexing is trustworthy for “at tip” |
+| **TX indexer** | Unconfirmed transactions via ZMQ | High volume, duplicate ZMQ events; isolated so IBD memory/CPU does not contend with mempool flood |
+
+Coordination is deliberately minimal: after IBD, the block indexer issues `GET http://{TX_REST_API_IP}:{TX_REST_API_PORT}/tx-start`. The TX indexer polls that flag every two seconds until set, then subscribes to ZMQ.
+
+**Tradeoff:** No shared memory or message queue between processes—only HTTP and the database. Simplicity and operational parity with SLP outweigh lower latency startup.
+
+## Clean Architecture (indexer)
+
+Both repos follow [Clean Architecture](https://christroutner.github.io/trouts-blog/blog/clean-architecture): dependencies point inward; framework and I/O live at the edges.
+
+### psf-memo-indexer layers
+
+```text
+Entry points (framework)
+  psf-memo-block-indexer.js
+  psf-memo-tx-indexer.js
+        │
+        ▼
+Controllers
+  keyboard.js          — graceful stop (q key)
+  tx-rest-api.js       — Express /tx-start
+        │
+        ▼
+Use cases
+  index-blocks.js      — processBlock, processMemoTx
+  filter-block.js      — filterMemoTxs (parallel pre-check)
+  state.js             — synced block height
+  action-types/*.js    — per Memo action handlers
+  utils.js
+        │
+        ▼
+Adapters
+  rpc.js               — full node JSON-RPC
+  zmq.js               — @psf/bitcoincash-zmq-decoder
+  transaction.js       — fetch tx + memo detection cache
+  status-db.js, *-db   — axios → psf-memo-db
+  tx-indexer.js        — start signal to TX process
+  backup-db.js         — POST /level/backup
+        │
+        ▼
+Libraries
+  memo-parser.js       — OP_RETURN pushdata, signer address
+  memo-codes.js        — action prefixes and limits
+```
+
+**Rule:** Use cases contain business rules (valid pushdata counts, reply parent links, like tips). Adapters only move bytes on the network or disk API.
+
+### psf-memo-db layers
+
+```text
+index.js → bin/server.js (Koa)
+        │
+        ▼
+Controllers
+  rest-api/index.js
+  rest-api/level/*     — CRUD handlers
+  rest-api/health/
+        │
+        ▼
+Use cases
+  index.js             — minimal stub (parity with psf-slp-db)
+        │
+        ▼
+Adapters
+  level-db.js          — open/close LevelDB instances
+  db-backup.js         — zip / restore
+```
+
+Level CRUD bypasses use cases intentionally—same as `psf-slp-db`’s `/level` controller calling `adapters.level.*Db.put` directly. The DB service is a thin persistence plane, not a domain model server.
+
+## Repository layout (indexer)
+
+| Path | Purpose |
+|------|---------|
+| `config/index.js` | Environment-driven settings |
+| `src/lib/` | Pure protocol parsing (no I/O) |
+| `src/adapters/` | External systems |
+| `src/use-cases/` | Indexing orchestration and handlers |
+| `src/controllers/` | HTTP and keyboard |
+| `production/docker/` | Docker Compose, per-service Dockerfile and `.env` |
+| `test/unit/` | Mocha + c8 |
+
+## Repository layout (database)
+
+| Path | Purpose |
+|------|---------|
+| `bin/server.js` | Koa bootstrap (mirrors psf-slp-db) |
+| `config/env/` | `SVC_ENV` profiles |
+| `src/adapters/level-db.js` | Twelve LevelDB stores under `leveldb/current/` |
+| `src/controllers/rest-api/level/` | Generic CRUD + status + backup routes |
+| `leveldb/current/{name}/` | Runtime data (gitignored) |
+| `leveldb/zips/` | Epoch backups |
+
+## LevelDB schema (summary)
+
+Each store is a separate LevelDB with JSON values. Keys are txids, addresses, or composite strings depending on entity. Full detail: [psf-memo-db.md](./psf-memo-db.md).
+
+| Store | Typical key | Value role |
+|-------|-------------|------------|
+| `status` | `status` | `syncedBlockHeight`, `chainBlockHeight`, `startBlockHeight` |
+| `posts` | txid | Author address, text, timestamp |
+| `postParents` | child txid | Parent txid (reply) |
+| `postChildren` | parent txid | Reverse index for replies |
+| `likes` | like txid | Post txid, liker, optional tip |
+| `names` | address | Display name + provenance txid |
+| `profiles` | address | Profile text |
+| `profilePics` | address | Avatar URL |
+| `follows` | `follower:followeePkHash` | Follow/unfollow event |
+| `rooms` | composite | Topic posts and topic follows |
+| `processErrors` | txid | Validation / parse failures |
+| `ptxs` | txid | Idempotency marker (already processed) |
+
+## External dependencies
+
+| Package | Used by | Role |
+|---------|---------|------|
+| `@psf/bch-js` | Indexer | Script decompile for signer address; optional REST |
+| `@psf/bitcoincash-zmq-decoder` | Indexer | Decode `rawtx` / `rawblock` ZMQ messages |
+| `zeromq` | Indexer | Subscribe to full node |
+| `@chris.troutner/retry-queue` | Indexer | Retry RPC on transient failure |
+| `axios` | Indexer | psf-memo-db REST client |
+| `p-queue` / `p-retry` | Indexer | Parallel block filter with retries |
+| `express` | Indexer | TX indexer control API |
+| `level` | psf-memo-db | Embedded JSON LevelDB |
+| `koa` + `koa-router` | psf-memo-db | REST server |
+
+**Not used:** `slp-parser`, Lokad ID checks, DAG sorting, token UTXO graphs.
+
+## Deployment topology
+
+```text
+production/docker/
+├── docker-compose.yml
+├── memo-db/          → build context: ../../psf-memo-db
+├── block-indexer/    → build context: indexer repo root
+└── tx-indexer/
+```
+
+Volumes mount `.env` and start scripts per service, matching the SLP production pattern. LevelDB data can persist under `production/data/leveldb`.

dev-docs/design-decisions-and-tradeoffs.md

@@ -0,0 +1,185 @@
+# Design Decisions and Tradeoffs
+
+This document records **why** the Memo indexer stack looks the way it does, including deliberate limitations and comparisons to adjacent systems.
+
+## 1. Mirror the SLP indexer architecture
+
+**Decision:** Two processes (block + TX), separate `psf-memo-db`, Clean Architecture folders, axios DB adapters, ZMQ + RPC, epoch zip backups, Express `/tx-start` gate.
+
+**Reason:** PSF already operates `psf-slp-indexer-g2` + `psf-slp-db` in production. Reusing the pattern reduces training cost, Docker layout, and incident playbooks.
+
+**Tradeoff:** Memo does not need SLP’s DAG ordering or UTXO graph, so some SLP machinery is absent—but the **process and deployment** model stays parallel.
+
+| Aspect | SLP stack | Memo stack |
+|--------|-----------|------------|
+| DB port | 5020 | 5021 |
+| TX REST port | 5454 | 5455 |
+| Parser | `slp-parser` + Lokad ID | `memo-parser` + `0x6d` prefix |
+| DB entities | tokens, utxos, addrs | posts, likes, follows, … |
+
+## 2. Separate database service (not embedded LevelDB)
+
+**Decision:** Indexer never opens LevelDB files directly.
+
+**Pros:**
+
+- Aligns with psf-slp-db backup/restore workflow
+- Avoids multi-process file lock issues
+- Clear boundary for future read replicas or query services
+
+**Cons:**
+
+- HTTP latency on every write
+- More moving parts in development (three processes)
+
+**Alternative rejected:** Single Node process with embedded LevelDB—simpler locally but diverges from PSF production norms.
+
+## 3. Scope limited to Go indexer “core social” handlers
+
+**Decision:** v1 implements the handlers in [memo/index/node/obj/op_return/main.go](https://github.com/memocash/index/blob/master/node/obj/op_return/main.go) (`GetHandlers`), not the full [memo-protocol.md](../../memo-protocol.md) table.
+
+**Included:** name, post, reply, like, profile, profile pic, follow/unfollow, topic message, topic follow/unfollow.
+
+**Excluded (v1):**
+
+- Polls, mute, send money
+- MIP-0009 token sale actions (`0x6d30`–`0x6d32`)
+- SLP handler present in Go’s registry (`slp_tokenHandler`)
+- Planned actions (`0x6d08`, `0x6d0b`, …)
+
+**Reason:** Ship a useful social subgraph first; token and poll logic add validation and cross-tx dependencies.
+
+**Risk:** Indexer state diverges from memo.sv for excluded actions until handlers are added.
+
+## 4. No DAG sort within blocks
+
+**Decision:** Process filtered Memo txids in block order, sequentially.
+
+**SLP context:** SEND transactions may spend token outputs created earlier in the same block; DAG sort orders them correctly.
+
+**Memo context:** Social actions do not consume each other’s UTXOs in a token graph. Replies reference parent txids by hash but do not require reordering spends within the block.
+
+**Tradeoff:** If future Memo actions introduce intra-block dependencies, DAG or topological sort may be needed—unlikely for current social ops.
+
+## 5. Scan all transaction outputs for OP_RETURN
+
+**Decision:** `findMemoOutputs` checks every `vout`, not only `vout[0]`.
+
+**Reason:** Go `node/obj/saver/op_return.go` iterates all outputs; Memo has no protocol rule fixing OP_RETURN to the first output.
+
+**Cost:** Slightly more script parsing per tx during filter and process.
+
+## 6. Custom script pushdata parser (with selective bch-js)
+
+**Decision:** `parseScriptPushDatas` implements Bitcoin push opcode walking in JavaScript; `@psf/bch-js` is used for signer address extraction from inputs.
+
+**Reason:** Unit tests must run without a full node; decompile behavior must be predictable for standard `OP_RETURN` Memo txs.
+
+**Tradeoff:** Non-standard scripts may parse differently than `btcd`—edge cases go to `processErrors`.
+
+## 7. Signer = first decodable input address
+
+**Decision:** Walk `vin` until `Script.getAddressFromScriptSig` succeeds.
+
+**Matches:** Go indexer behavior when building `parse.OpReturn.Addr`.
+
+**Limitation:** Multi-input txs with multiple signers attribute the action to the first resolved address only—consistent with reference indexer, not necessarily with user intent for complex txs.
+
+## 8. Idempotency at transaction level only
+
+**Decision:** `ptxs` store marks completed txs; handlers may write multiple LevelDB keys per tx without a cross-store transaction.
+
+**Go indexer:** Uses richer `db.Save([]db.Object{...})` batches.
+
+**Failure mode:** Crash mid-handler could leave partial writes; re-run skips entire tx if `ptx` was written— or duplicates sub-objects if ptx write failed last.
+
+**Mitigation path:** Write ptx last (current code writes ptx after handlers—good) or add DB batch API.
+
+## 9. Process errors are non-fatal
+
+**Decision:** Invalid pushdata counts or oversize fields log to `processErrors` and return; they do not stop the block.
+
+**Reason:** Chain history contains malformed or experimental Memo txs; one bad tx must not halt IBD.
+
+**Ops note:** Monitor `processErrors` growth for parser bugs vs true invalid chain data.
+
+## 10. Start block 525000
+
+**Decision:** Default `START_BLOCK_HEIGHT=525000` from Go `BeginningOfMemoHeight`.
+
+**Tradeoff:** Earlier Memo experiments exist on-chain but are excluded unless the operator lowers the height and reindexes from genesis of Memo data.
+
+## 11. psf-memo-db: thin controller, no GraphQL
+
+**Decision:** Level routes call LevelDB directly; no Mongoose, no GraphQL.
+
+**Go memo/index** exposes GraphQL with attach resolvers loading from sharded DB—the read path is a separate product layer.
+
+**PSF stack** stops at indexed storage + REST CRUD for writers. Read APIs are the integrator’s responsibility.
+
+**Future:** A `memo-query.js` adapter (like `slp-query.js`) could encapsulate common reads without pulling in full GraphQL.
+
+## 12. Mempool vs confirmed indexing
+
+**Decision:** TX indexer uses `blockHeight = tip + 1` for unconfirmed txs; block indexer re-processes confirmed blocks.
+
+**Nuance:** A mempool tx may be indexed twice (mempool + block) but `ptx` idempotency prevents duplicate handler effects.
+
+**Edge case:** Tx never confirms—remains indexed with provisional height; may be acceptable for social UX or require GC policy later.
+
+## 13. Docker layout under `production/docker`
+
+**Decision:** Dockerfiles live in `psf-memo-indexer/production/docker/{memo-db,block-indexer,tx-indexer}/`, not repo roots—matching psf-slp-indexer-g2.
+
+**Reason:** Single compose file orchestrates three services; build contexts point at `psf-memo-db` sibling repo and indexer root.
+
+## 14. JavaScript vs extending the Go indexer
+
+**Decision:** New JS implementation rather than operating memocash/index directly.
+
+**Pros for PSF:**
+
+- Same language and patterns as SLP indexer maintenance
+- No CGO / Go toolchain requirement for PSF deployers
+- Independent release cycle from memocash
+
+**Cons:**
+
+- Two implementations to keep in sync with protocol changes
+- Go indexer has sharding, GraphQL, and mature handlers PSF omitted
+
+**When to prefer Go:** Full memo.sv parity, production-scale sharded cluster, SLP-in-Memo token indexing.
+
+**When to prefer PSF JS:** Unified ops with psf-slp-indexer-g2, custom downstream consumers of LevelDB REST.
+
+## 15. Testing strategy
+
+**Decision:** Unit tests with mocks for RPC, DB, and parser fixtures (Go `post_test.go` tx hex)—no mandatory e2e full node in CI.
+
+**Tradeoff:** Integration bugs (RPC format changes, ZMQ topic names) need manual or staged e2e tests.
+
+**Recommended e2e smoke:** Index block ≥ 525000 containing a known Memo post from [memo-protocol.md](../../memo-protocol.md) explorer links; verify `GET /level/post/:txid`.
+
+## Decision log (quick reference)
+
+| Topic | Choice |
+|-------|--------|
+| Architecture pattern | Clean Architecture |
+| Chain interface | RPC + ZMQ |
+| Storage | psf-memo-db REST + LevelDB |
+| Processes | 2 indexers + 1 DB |
+| Protocol breadth | Go `GetHandlers` subset |
+| Ordering | Block order, no DAG |
+| OP_RETURN location | Any output |
+| Auth on DB API | None (v1) |
+| Read API | Out of scope (v1) |
+
+## Open questions for future contributors
+
+1. **Handler parity:** Which excluded memo-protocol actions are required next (polls vs token sale)?
+2. **Query layer:** Should PSF add `memo-query` REST on psf-memo-db or a separate service?
+3. **Reorg handling:** v1 does not unwind reorgs; is depth-0 BCH acceptable for social indexing?
+4. **Address indexing:** Should posts be secondary-indexed by `addr` for feed APIs?
+5. **Alignment with Go:** Should reply/like tip logic match Go’s satoshi aggregation exactly (output script types beyond P2PKH)?
+
+Document updates belong in this `dev-docs/` folder when decisions change.

dev-docs/overview.md

@@ -0,0 +1,87 @@
+# Overview: Why the Memo Indexer Exists
+
+## The problem
+
+The [Memo protocol](https://memo.cash) encodes social actions—posts, replies, likes, follows, profiles, topics—in **Bitcoin Cash transactions** using `OP_RETURN` outputs. Wallets and explorers can read individual transactions from a full node, but useful applications need:
+
+- A **queryable history** of who posted what, when, and in reply to which parent transaction
+- **Derived indexes** (follow graph, likes per post, topic membership) without re-scanning the entire chain on every API request
+- **Consistent interpretation** of Memo action bytes (`0x6d` prefix + action code) across services
+
+Scanning the blockchain on demand for every user request does not scale. The Memo indexer exists to **materialize** protocol state into local databases so other apps (APIs, analytics, mirrors) can read indexed data quickly.
+
+## What this stack does
+
+Two cooperating Node.js services mirror the proven PSF pattern used for SLP tokens:
+
+```text
+BCH full node (RPC + ZMQ)
+        │
+        ▼
+┌───────────────────────┐     HTTP REST      ┌─────────────────┐
+│  psf-memo-indexer     │ ─────────────────► │  psf-memo-db    │
+│  (2 processes)        │   /level/*         │  (LevelDB)      │
+└───────────────────────┘                    └─────────────────┘
+        │                                              │
+        │                                              ▼
+        │                                    Query / backup APIs
+        ▼
+   Memo OP_RETURN txs
+   from blocks + mempool
+```
+
+**psf-memo-indexer** watches the chain, detects Memo transactions, validates action payloads, and writes structured records.
+
+**psf-memo-db** owns storage. It exposes CRUD over multiple LevelDB instances so the indexer (and future tools) do not open database files directly—avoiding lock contention and allowing backup/restore as a separate concern.
+
+## What v1 indexes
+
+The JavaScript indexer intentionally matches the **core social handlers** registered in the reference Go indexer’s `op_return` package—not the full Memo protocol spec:
+
+| Indexed in v1 | Not indexed in v1 |
+|---------------|-------------------|
+| Set name (`0x6d01`) | Polls (`0x6d10`–`0x6d14`) |
+| Post (`0x6d02`) | Mute / unmute |
+| Reply (`0x6d03`) | Send money (`0x6d24`) |
+| Like (`0x6d04`) | Token sale MIP-0009 actions |
+| Set profile (`0x6d05`) | SLP / Bitcom OP_RETURN in Go indexer |
+| Follow / unfollow (`0x6d06` / `0x6d07`) | Planned spec actions (`0x6d08`, `0x6d0b`, …) |
+| Set profile picture (`0x6d0a`) | |
+| Topic message / follow / unfollow (`0x6d0c`–`0x6d0e`) | |
+
+See [design-decisions-and-tradeoffs.md](./design-decisions-and-tradeoffs.md) for rationale.
+
+## Start height
+
+Indexing begins at block **525000** (`BeginningOfMemoHeight` in the Go reference). Earlier blocks are skipped because Memo activity before that height is negligible for practical deployments.
+
+## Who consumes the output
+
+This repository does **not** ship a GraphQL or public REST API for end users—that exists in the separate [memo/index](https://github.com/memocash/index) Go project. PSF’s stack is an **indexer + DB layer** intended for:
+
+- PSF infrastructure that wants Memo data beside SLP indexing patterns
+- Custom services that call `psf-memo-db` REST endpoints
+- Future query layers (e.g. a slim `memo-query` module) without reimplementing chain scanning
+
+## Operational summary
+
+| Process | Entry point | When it runs |
+|---------|-------------|--------------|
+| Block indexer | `psf-memo-block-indexer.js` | IBD from `START_BLOCK_HEIGHT` to chain tip; then ZMQ `rawblock` |
+| TX indexer | `psf-memo-tx-indexer.js` | After block indexer calls `GET /tx-start`; then ZMQ `rawtx` |
+| Database | `psf-memo-db` `index.js` | Must be running before indexers start |
+
+Typical development:
+
+```bash
+# Terminal 1
+cd psf-memo-db && npm start
+
+# Terminal 2
+cd psf-memo-indexer && npm run block-indexer
+
+# Terminal 3
+cd psf-memo-indexer && npm run tx-indexer
+```
+
+Production uses [production/docker](../production/docker/) with the same three logical services.

dev-docs/psf-memo-db.md

@@ -0,0 +1,153 @@
+# psf-memo-db Architecture
+
+`psf-memo-db` is the persistence layer for the Memo indexer. It is structurally modeled on [psf-slp-db](https://github.com/Permissionless-Software-Foundation/psf-slp-db): multiple LevelDB instances behind a Koa REST API, with no business logic beyond backup and restore.
+
+## Why a separate service
+
+| Concern | How separation helps |
+|---------|----------------------|
+| File locking | Only one process opens LevelDB files; indexers use HTTP |
+| Parallelism | Multiple indexer workers could share one DB API (future) |
+| Backups | Close all DBs, zip, reopen—without stopping indexer logic in the same process |
+| Operational familiarity | PSF already runs `psf-slp-db` beside `psf-slp-indexer-g2` |
+
+**Tradeoff:** Every write is an HTTP round trip. Local indexing pays latency versus embedded LevelDB, but gains operational consistency with the SLP stack.
+
+## Server bootstrap
+
+Mirrors `psf-slp-db/bin/server.js`:
+
+```text
+index.js
+  └── bin/server.js
+        ├── Koa + middleware (logger, bodyparser 100mb, CORS, error handler)
+        ├── controllers.initAdapters()  → level-db.openDbs()
+        ├── controllers.initUseCases()  → no-op start
+        ├── controllers.attachRESTControllers()
+        └── app.listen(PORT)  // default 5021
+```
+
+**Intentionally omitted** from psf-slp-db: MongoDB, Passport auth, IPFS/Helia, wallet, usage tracking, `/slp` query routes. `noMongo: true` and `useIpfs: false` in config.
+
+## LevelDB instances
+
+Opened in `src/adapters/level-db.js`:
+
+```javascript
+leveldb/current/{name}  // valueEncoding: 'json'
+```
+
+| Instance | Cache hint | Indexer writes |
+|----------|------------|----------------|
+| `status` | 64 MB | Block height sync state |
+| `posts` | 512 MB | Post and reply bodies |
+| `postParents` | 64 MB | Reply → parent link |
+| `postChildren` | 64 MB | Parent → children |
+| `likes` | 64 MB | Like events |
+| `names` | 64 MB | Display names |
+| `profiles` | 64 MB | Profile text |
+| `profilePics` | 64 MB | Avatar URLs |
+| `follows` | 64 MB | Follow graph edges |
+| `rooms` | 64 MB | Topic posts and follows |
+| `processErrors` | 64 MB | Skipped / invalid txs |
+| `ptxs` | 64 MB | Processed tx markers |
+
+Posts receive a larger cache because they are the highest-volume social object.
+
+## REST API shape
+
+All routes are under `/level` with a consistent CRUD pattern generated from `ENTITY_CONFIG` in `crud-handlers.js`:
+
+| Method | Path pattern | Body (create) |
+|--------|--------------|---------------|
+| `POST` | `/level/{entity}` | `{ <idField>, <dataField> }` |
+| `GET` | `/level/{entity}/:key` | — |
+| `PUT` | `/level/{entity}/:key` | `{ <dataField> }` |
+| `DELETE` | `/level/{entity}/:key` | — |
+
+### Entity route names
+
+| Route | idField | dataField | Example key |
+|-------|---------|-----------|-------------|
+| `post` | `txid` | `postData` | transaction id |
+| `postparent` | `txid` | `parentData` | child txid |
+| `postchild` | `txid` | `childData` | parent txid |
+| `like` | `txid` | `likeData` | like txid |
+| `name` | `addr` | `nameData` | cash address |
+| `profile` | `addr` | `profileData` | cash address |
+| `profilepic` | `addr` | `profilePicData` | cash address |
+| `follow` | `key` | `followData` | `follower:followeePkHash` |
+| `room` | `key` | `roomData` | composite |
+| `processerror` | `txid` | `errorData` | txid |
+| `ptx` | `txid` | `ptxData` | txid |
+
+### Status (special case)
+
+Matches SLP status semantics:
+
+- `GET /level/status/:statusKey`
+- `POST /level/status` with `{ statusKey, statusData }`
+- `PUT /level/status` with `{ statusData }` — key is always `status`
+- `DELETE /level/status/:statusKey`
+
+Indexer expects `statusKey = status` containing:
+
+```json
+{
+  "startBlockHeight": 524999,
+  "syncedBlockHeight": 800000,
+  "chainBlockHeight": 800001
+}
+```
+
+### Backup and restore
+
+- `POST /level/backup` — body `{ height, epoch }` → zip `leveldb/current` to `leveldb/zips/memo-indexer-{height}.zip`
+- `POST /level/restore` — body `{ height }` → unzip matching archive and `process.exit(0)`
+
+Implemented in `src/adapters/db-backup.js`.
+
+### Health
+
+`GET /health` → `{ status: 'ok' }` for load balancers and compose checks.
+
+## Indexer adapter mapping
+
+The indexer uses `src/adapters/entity-db.js`:
+
+```javascript
+createEntityDb('post', 'txid', 'postData')
+// → POST http://localhost:5021/level/post
+```
+
+`status-db.js` uses dedicated status endpoints rather than the generic factory.
+
+## Data modeling notes
+
+**Not normalized like SQL.** LevelDB stores are document keyed for fast lookup by txid or address, similar to the Go `db/item/memo` objects but without sharding.
+
+**No secondary indexes in v1.** Queries like “all posts by address” may require scanning or a future `memo-query` adapter—out of scope for the indexer write path.
+
+**JSON values** keep debugging simple; binary serialization would save space but break parity with psf-slp-db tooling.
+
+## Configuration
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `PORT` | `5021` | REST listen (SLP uses 5020) |
+| `SVC_ENV` | `development` | Config profile |
+| `BACKUP_QTY` | `3` | Retained zip backups |
+| `EXIT_ON_MISSING_BACKUP` | `false` | Fail restore if zip missing |
+
+## Testing
+
+Unit tests cover:
+
+- `bin/server.js` startup with stubbed controllers
+- Level controller status and post CRUD with mocked LevelDB
+
+Run: `npm test` in the `psf-memo-db` directory.
+
+## Production
+
+Docker build context is the `psf-memo-db` repo root; Dockerfile lives in [psf-memo-indexer/production/docker/memo-db/](../production/docker/memo-db/). See [architecture.md](./architecture.md#deployment-topology).

dev-docs/theory-of-operation.md

@@ -0,0 +1,193 @@
+# Theory of Operation
+
+This document walks through how the indexer behaves at runtime—from cold start through steady-state chain following—including the Memo-specific parsing rules.
+
+## Prerequisites
+
+1. **BCH full node** with RPC and ZMQ enabled (`rawtx`, `rawblock` on the configured port, default `28332`).
+2. **psf-memo-db** listening (default `http://localhost:5021`).
+3. Environment variables in `.env` or Docker-mounted `.env` files (see `.env-example`).
+
+## Phase 1: Initialization
+
+Both indexer processes:
+
+1. Construct `Adapters`, `UseCases`, and `Controllers`.
+2. Block indexer enables keyboard listener (`q` stops after current block).
+3. TX indexer starts Express on `TX_REST_API_PORT` and waits for `/tx-start`.
+
+**Status bootstrap** (`status-db.js`):
+
+- `GET /level/status/status` from psf-memo-db.
+- If missing (fresh DB), create:
+
+```json
+{
+  "startBlockHeight": START_BLOCK_HEIGHT - 1,
+  "syncedBlockHeight": START_BLOCK_HEIGHT - 1,
+  "chainBlockHeight": <current RPC block count>
+}
+```
+
+If `EXIT_ON_MISSING_BACKUP=true` and status is missing, the process exits instead of reinitializing—same safety valve as the SLP indexer when a backup is expected but absent.
+
+## Phase 2: Initial Block Download (IBD)
+
+Only the **block indexer** runs IBD.
+
+```text
+nextHeight = syncedBlockHeight + 1
+tip        = RPC getblockcount
+
+while nextHeight <= tip:
+    processBlock(nextHeight)
+    updateIndexedBlockHeight(nextHeight)
+    optionally backup every 1000 blocks
+    nextHeight++
+```
+
+### Per-block pipeline (`index-blocks.processBlock`)
+
+```mermaid
+sequenceDiagram
+  participant BI as Block_indexer
+  participant RPC as Full_node_RPC
+  participant FB as filter_block
+  participant TX as transaction_adapter
+  participant H as action_handlers
+  participant DB as psf_memo_db
+
+  BI->>RPC: getblockhash(height)
+  BI->>RPC: getblock(hash)
+  BI->>FB: filterMemoTxs(txids[])
+  loop each candidate txid
+    FB->>TX: isMemoTx(txid)
+  end
+  loop each memo txid
+    BI->>BI: processMemoTx(txid)
+    BI->>DB: reads/writes via handlers
+  end
+```
+
+**Step 1 — Fetch block:** `getblock` returns the list of transaction ids in the block (verbosity 1).
+
+**Step 2 — Filter:** `filterMemoTxs` runs up to 20 concurrent `isMemoTx` checks (`p-queue`). Each check loads the transaction (with in-memory cache in `transaction.js`) and scans **all outputs** for a Memo `OP_RETURN`.
+
+**Why scan all outputs?** Unlike SLP, which conventionally places token data in `vout[0]`, Memo actions may appear in any output’s `scriptPubKey`. The Go reference iterates every `TxOut`.
+
+**Step 3 — Process:** For each Memo txid, `processMemoTx` runs sequentially within the block (no DAG sort—Memo social actions do not form token-like dependency chains within a block).
+
+## Phase 3: Processing a single Memo transaction
+
+### Idempotency
+
+Before heavy work, the indexer checks `GET /level/ptx/{txid}`. If present, the tx is skipped. After successful handling, it `POST`s a ptx record `{ processedAt, blockHeight }`.
+
+**Nuance:** If a handler partially fails mid-tx, v1 does not roll back prior writes in that tx. The Go indexer uses richer DB transactions; PSF v1 relies on idempotency only at tx boundaries. See tradeoffs doc.
+
+### Signer address
+
+Memo actions are attributed to the **first P2PKH input** that yields an address from the unlocking script (`getSignerAddress` in `memo-parser.js`, backed by `@psf/bch-js` where possible).
+
+If no address is found:
+
+- Log `processErrors` with reason.
+- Skip action handlers (no anonymous posts in v1).
+
+This mirrors the Go `SetLockHash` loop over inputs.
+
+### OP_RETURN decoding
+
+`parseScriptPushDatas` walks Bitcoin script bytes (push opcodes `0x01`–`0x4e`, `OP_RETURN 0x6a`, etc.) and collects pushdata buffers.
+
+A Memo action is recognized when **any** pushdata begins with:
+
+```text
+0x6d <action_byte>
+```
+
+`decodeMemoOpReturn` returns `{ action, prefix, pushDatas }` where `action` is a string key (`post`, `reply`, `like`, …) mapped in `memo-codes.js`.
+
+### Handler dispatch
+
+`dispatchMemoAction` routes to `src/use-cases/action-types/*.js`. Each handler:
+
+1. Validates pushdata count and field sizes (against `MAX_POST_SIZE` etc. from Go `memo.go`, typically 65000 bytes in modern limits).
+2. On validation failure: write `processErrors`, return without throwing (invalid Memo txs do not halt the block).
+3. On success: `POST` / `PUT` to the appropriate `/level/{entity}` route.
+
+### Multi-output transactions
+
+If one transaction contains multiple Memo `OP_RETURN` outputs, each decoded output dispatches separately in a loop. Rare in practice but supported.
+
+## Action-specific behavior (v1)
+
+| Action | Pushdata layout | Side effects beyond primary store |
+|--------|-----------------|-----------------------------------|
+| `setName` | prefix + UTF-8 name | `names` by address |
+| `post` | prefix + message | `posts` by txid; skip create if post exists |
+| `reply` | prefix + 32-byte parent hash + message | `postParents`, `postChildren`, then post body |
+| `like` | prefix + 32-byte post hash | `likes`; optional `tip` satoshis to post author outputs |
+| `setProfile` | prefix + text | `profiles` |
+| `follow` / `unfollow` | prefix + 20-byte pk hash | `follows` keyed by follower and followee |
+| `setProfilePic` | prefix + URL | `profilePics` |
+| `topicMessage` | prefix + topic + message | post + `rooms` index |
+| `topicFollow` / `topicUnfollow` | prefix + topic name | `rooms` follow record |
+
+Like-tip logic: if the liked post exists and the liker is not the author, outputs paying the author’s address are summed (values converted from BCH decimal to satoshis) and stored on the like record.
+
+## Phase 4: IBD completion and TX indexer handoff
+
+When `nextBlockHeight > tip`:
+
+1. Block indexer connects ZMQ.
+2. `GET /tx-start` on TX indexer.
+3. Enters ZMQ block loop (500 ms sleep, `rawblock` queue).
+
+TX indexer:
+
+1. Already listening on Express.
+2. After `/tx-start`, connects ZMQ and processes `rawtx`.
+3. Maintains `seenTxs` Set + FIFO queue capped at `seenTxMax` to drop duplicate ZMQ notifications cheaply before hitting `ptxDb`.
+4. Calls the same `processMemoTx` with `blockHeight = chain tip + 1` for unconfirmed txs.
+
+## Phase 5: Steady state
+
+| Event source | Process | Behavior |
+|--------------|---------|----------|
+| New block (ZMQ) | Block indexer | Resolve height from block header, update `status`, `processBlock` |
+| New mempool tx (ZMQ) | TX indexer | Dedupe → `processMemoTx` |
+| Every 1000 blocks | Block indexer | `POST /level/backup` with `{ height, epoch: 1000 }` |
+
+### Backups
+
+`psf-memo-db` closes LevelDB files, zips `leveldb/current` to `leveldb/zips/memo-indexer-{height}.zip`, reopens DBs. Old zips are pruned per `BACKUP_QTY`.
+
+Restore (`POST /level/restore`) unzips and **exits the process**—expect process manager restart, same as SLP.
+
+## Memo vs SLP detection (operational difference)
+
+| | SLP indexer | Memo indexer |
+|---|-------------|--------------|
+| Detection | Lokad `534c5000` + `slp-parser` | Prefix `6d` + action byte |
+| Output index | Assumes `vout[0]` for token data | Any vout |
+| Ordering | DAG sort within block | Sequential tx order |
+| State model | Token UTXOs, balances | Social graph documents |
+
+## Failure and retry behavior
+
+- **RPC:** `RetryQueue` and `p-retry` (5 attempts) on block filter operations.
+- **DB HTTP:** Errors bubble up; block processing logs and throws on `processMemoTx` failure.
+- **Invalid Memo payload:** Logged to `processErrors`; block continues.
+- **Keyboard `q`:** Block indexer exits after finishing current block.
+
+## Configuration reference
+
+| Variable | Default | Effect |
+|----------|---------|--------|
+| `PSF_MEMO_DB_URL` | `http://localhost:5021` | All DB adapters |
+| `START_BLOCK_HEIGHT` | `525000` | Genesis for indexer state |
+| `RPC_*` / `ZMQ_PORT` | see `.env-example` | Full node |
+| `TX_REST_API_PORT` | `5455` | TX control (SLP uses 5454) |
+| `seenTxMax` | `100000` | Mempool dedupe set size |
+| `txCacheMax` | `100000` | RPC tx cache in transaction adapter |