Paralell processing of TXs within a block

dev-docs/architecture.md

@@ -60,8 +60,8 @@ Controllers
         │
         ▼
 Use cases
-  index-blocks.js      — processBlock, processMemoTx
-  filter-block.js      — filterMemoTxs (parallel pre-check)
+  index-blocks.js      — processBlock, processMemoTx (parallel per block)
+  filter-block.js      — filterMemoTxs (parallel pre-check, block order preserved)
   state.js             — synced block height
   action-types/*.js    — per Memo action handlers
   utils.js
@@ -157,7 +157,7 @@ Each store is a separate LevelDB with JSON values. Keys are txids, addresses, or
 | `zeromq` | Indexer | Subscribe to full node |
 | `@chris.troutner/retry-queue` | Indexer | Retry RPC on transient failure |
 | `axios` | Indexer | psf-memo-db REST client |
-| `p-queue` / `p-retry` | Indexer | Parallel block filter with retries |
+| `p-queue` / `p-retry` | Indexer | Parallel block filter and parallel per-tx processing within a block |
 | `express` | Indexer | TX indexer control API |
 | `level` | psf-memo-db | Embedded JSON LevelDB |
 | `koa` + `koa-router` | psf-memo-db | REST server |

dev-docs/design-decisions-and-tradeoffs.md

@@ -51,15 +51,21 @@ This document records **why** the Memo indexer stack looks the way it does, incl
 
 **Risk:** Indexer state diverges from memo.sv for excluded actions until handlers are added.
 
-## 4. No DAG sort within blocks
+## 4. Parallel processing within blocks (no DAG sort)
 
-**Decision:** Process filtered Memo txids in block order, sequentially.
+**Decision:** Filter and process Memo txs concurrently within a block using `p-queue`. Default concurrency is 20 for both phases (`FILTER_CONCURRENCY`, `MEMO_TX_CONCURRENCY`). Filter results are reordered to match block tx order before processing starts.
 
-**SLP context:** SEND transactions may spend token outputs created earlier in the same block; DAG sort orders them correctly.
+**SLP context:** SEND transactions may spend token outputs created earlier in the same block; DAG sort orders them correctly and writes must stay serial when UTXO state overlaps.
 
-**Memo context:** Social actions do not consume each other’s UTXOs in a token graph. Replies reference parent txids by hash but do not require reordering spends within the block.
+**Memo context:** Social actions do not consume each other’s UTXOs in a token graph. Most handler writes are keyed by `txid` and are independent across transactions. Parallel `processMemoTx` is safe for typical blocks.
 
-**Tradeoff:** If future Memo actions introduce intra-block dependencies, DAG or topological sort may be needed—unlikely for current social ops.
+**Soft ordering:** Likes that reference a post in the same block may compute `tip = 0` if the post handler has not finished yet. Replies still persist parent/child links and post bodies regardless of completion order. Profile/follow handlers keyed by address can race if the same signer emits multiple updates in one block (rare).
+
+**Within a single tx:** Multiple Memo `OP_RETURN` outputs in one transaction are still dispatched sequentially inside `processMemoTx`.
+
+**Tradeoff:** Higher throughput vs. HTTP contention on `psf-memo-db`. Lower `MEMO_TX_CONCURRENCY` if the DB service becomes the bottleneck.
+
+**Alternative rejected:** Full serial processing—simple but leaves RPC/DB latency on the table during IBD when blocks contain many unrelated Memo txs.
 
 ## 5. Scan all transaction outputs for OP_RETURN
 

dev-docs/theory-of-operation.md

@@ -63,7 +63,7 @@ sequenceDiagram
   loop each candidate txid
     FB->>TX: isMemoTx(txid)
   end
-  loop each memo txid
+  loop each memo txid (parallel, up to MEMO_TX_CONCURRENCY)
     BI->>BI: processMemoTx(txid)
     BI->>DB: reads/writes via handlers
   end
@@ -71,11 +71,11 @@ sequenceDiagram
 
 **Step 1 — Fetch block:** `getblock` returns the list of transaction ids in the block (verbosity 1).
 
-**Step 2 — Filter:** `filterMemoTxs` runs up to 20 concurrent `isMemoTx` checks (`p-queue`). Each check loads the transaction (with in-memory cache in `transaction.js`) and scans **all outputs** for a Memo `OP_RETURN`.
+**Step 2 — Filter:** `filterMemoTxs` runs up to `FILTER_CONCURRENCY` (default 20) concurrent `isMemoTx` checks (`p-queue`). Each check loads the transaction (with in-memory cache in `transaction.js`) and scans **all outputs** for a Memo `OP_RETURN`. Results are returned in **block tx order** (parallel checks use a set, then the original block list is filtered).
 
 **Why scan all outputs?** Unlike SLP, which conventionally places token data in `vout[0]`, Memo actions may appear in any output’s `scriptPubKey`. The Go reference iterates every `TxOut`.
 
-**Step 3 — Process:** For each Memo txid, `processMemoTx` runs sequentially within the block (no DAG sort—Memo social actions do not form token-like dependency chains within a block).
+**Step 3 — Process:** `processMemoTxs` runs up to `MEMO_TX_CONCURRENCY` (default 20) concurrent `processMemoTx` calls via `p-queue`. There is no DAG sort—Memo social actions do not form token-like dependency chains within a block. Multiple OP_RETURN outputs in a **single** transaction are still handled sequentially inside `processMemoTx`.
 
 ## Phase 3: Processing a single Memo transaction