Forked from ipfs-torlist-service

2026-09-23 01:22:03 -07:00 · 2021-09-09 18:23:26 -07:00
commit 2f2ebdc603
134 changed files with 58872 additions and 0 deletions
@@ -0,0 +1,73 @@
+# Developer Documentation
+
+This is living documentation that will be updated, edited, and changed over time, using the same version control as the rest of the code. The purpose of this documentation is to capture and explain how this tor-list-api and [tor-list-frontend](https://github.com/Permissionless-Software-Foundation/tor-list-frontend) [SPA](https://en.wikipedia.org/wiki/Single-page_application) interact with the [P2WDB](https://github.com/Permissionless-Software-Foundation/ipfs-p2wdb-service) to create a permissionless, censorship-resistant database for storing and sharing deep-web websites at [TorList.cash](https://torlist.cash).
+
+- [Specification](./specification.md)
+
+# Overview
+
+There are three major pieces of software behind the tor-list concept. They work together to form a censorship-resistant application for curating information about websites.
+
+![tor-list major subcomponents](./diagrams/software-interaction.png)
+
+- [tor-list-frontend](https://github.com/Permissionless-Software-Foundation/tor-list-frontend) is the web based user interface (UI) which can be seen at [TorList.cash](https://torlist.cash/).
+- [tor-list-api](https://github.com/Permissionless-Software-Foundation/tor-list-api) is the back end REST API that maintains a local database of information that tor-list-frontend reads from.
+- [P2WDB](https://github.com/Permissionless-Software-Foundation/ipfs-p2wdb-service) is the pay-to-write global database with a REST API for interfacing with the other two pieces of software.
+
+The arrows in the image represent the information flow between the three pieces of software:
+
+- tor-list-frontend displays information about websites. It _reads_ this information from tor-list-api.
+- tor-list-frontend is also a web wallet. It can generate the needed transactions to _write_ information to the P2WDB.
+- tor-list-api imports data from the global database into its local database using a [webhook](https://en.wikipedia.org/wiki/Webhook).
+
+This architecture keeps the global database highly censorship resistant, while allowing local installations to maintain tight control over the user experience. The goal is to have many curation sites similar to [TorList.cash](https://torlist.cash). Each site would cater to their niche by customizing the display of the data, but they would all tap into the same global database of information.
+
+**Example:** [TorList.cash](https://torlist.cash) will prevent the display of sites that are associated with drugs, guns, pornography, or gambling. That data can still exist in the global database, it just won't be displayed on Torlist.cash. Other websites, using a fork of the same software, can display those categories. They could also impose stricter rules to filter the data even more aggressively. (Imagine a site that focuses just on cryptocurrency trading, and filters out all sites that are not strictly about cryptocurrency trading.) Each site competes with the others for users, but work together to maintain the global database.
+
+# Back End
+
+This section provides additional information on the tor-list-api and P2WDB back end software.
+
+## P2WDB
+
+The heart of the censorship resistance is the pay-to-write database ([P2WDB](https://github.com/Permissionless-Software-Foundation/ipfs-p2wdb-service)). This is an [OrbitDB](https://orbitdb.org/) peer-to-peer (p2p) database. The write-access rules have been customized to allow anyone to write to the database, so long they prove that a sufficient quantity [PSF tokens](https://psfoundation.cash) have been burned, to pay for the write.
+
+Because OrbitDB is a p2p database, no one party holds the 'official' copy of the database. Instead, like a blockchain, the database is replicated among several peers, and they coordinate updates to the database using consensus rules. Peers are free to leave or enter the network. Each peer independently verifies the database entries have sufficient proof-of-burn.
+
+## tor-list-api
+
+The [tor-list-api](https://github.com/Permissionless-Software-Foundation/tor-list-api) replicates a copy of the global P2WDB, but has the ability to apply localized filters to the data before passing it on to tor-list-frontend to be displayed. The most important filter is the `blacklist` model, which allows administrators of the site to mark database entries as 'blacklisted', to prevent them from being displayed.
+
+tor-list-api is based on this [koa-api-boilerplate](https://github.com/christroutner/koa-api-boilerplate), which is a production ready REST API that includes many features for building a web app. This includes user management and authentication, REST API scaffolding, API documentation, Docker container generation, and extensive test coverage. It's intended to be customized for the needs of the website administrator.
+
+- [Specification](./specification.md)
+
+# Workflows
+
+This section describes the protocols for the database interactions between the three main software components.
+
+These are just a brief, high-level overview. Review the [Specifications](./specification.md) for more details.
+
+## Writing to the Global Database
+
+Adding data to the global P2WDB is a result of the interaction between tor-list-frontend and the P2WDB. tor-list-api is not involved.
+
+Writing data follows these steps:
+
+- To add an entry to the P2WDB, tor-list-frontend collects the data the user wants to add to the database, but it also collects several pieces of required information 'behind the scenes':
+  - The users BCH address.
+  - A cleartext message (The website URL)
+  - A signature generated from the cleartext message and the BCH address.
+  - It burns the required amount of PSF tokens, and generates a transaction ID (TXID) as proof of this burn.
+- tor-list-frontend then communicates with P2WDB via its REST API to submit all the data.
+- The P2WDB REST API will then evaluate the data and attempt to update the p2p database using the TXID.
+- Each peer on the network will independently validate the new database entry.
+- tor-list-api will receive a webhook call to its POST endpoint. This event will trigger the import of the new data into the apps local Mongo database.
+
+## Reading from the Local Database
+
+tor-list-frontend reads data from the local database stored by tor-list-api, and does not read the global database directly. This gives tor-list-api the opportunity to filter and modify the data locally for a more controlled user experience.
+
+The most important filter that tor-list-api runs against the data is the blacklist filter. Entries in the OrbitDB are identified by a [CID](https://docs.ipfs.io/concepts/content-addressing/) or 'hash'. The blacklist filter is a collection of CIDs that have been 'blacklisted' for removal before being displayed on the front end. This allows the administrators of the website to prevent spam and abuse.
+
+Entries on tor-list-frontend are organized relative to their [merit](https://github.com/Permissionless-Software-Foundation/bch-message-lib/blob/master/lib/merit.js). This merit value will change with time. Every 24 hours, the entries displayed at TorList.cash have their merit recalculated, and the merit value in the local database is updated.
@@ -0,0 +1,72 @@
+# Tor-List-API Specification
+
+This document contains a high-level, human-readable specification for the four major architectural areas of the tor-list-api:
+
+- Entities
+- Use Cases
+- Controllers (inputs)
+- Adapters (outputs)
+
+This reflects the [Clean Architecture](https://github.com/Permissionless-Software-Foundation/ipfs-p2wdb-service/blob/master/dev-docs/clean-architecture.md) design pattern.
+
+## Entities
+
+Entities make up the core business concepts. If these entities change, they fundamentally change the entire app.
+
+### Website
+
+The tor-list concept is all about curating deep-web websites that may have unusual or ephemeral address and may exist on different parts of the internet, like IPFS, Tor, or lightly supported TLDs like `.eth`. These Websites Entities have the following properties:
+
+- **url** - not restricted to common TLDs. This could have an address to an IPFS gateway, a .onion, .eth, or other type of address.
+- **address** - A blockchain address associated with the website. This address is used to evaluate the merit value of the site. In the future, it can be used by the site owner to communicate additional information using the [PS001 specification](https://github.com/Permissionless-Software-Foundation/specifications/blob/master/ps001-media-sharing.md).
+- **blockchain** - The blockchain the address above belongs to. Default value is `BCH`
+- **merit** - A merit score for the website. This is used to rank sites within their category. The value is calculated and updated every 24 hours.
+- **merit timestamp** - A timestamp of when the merit of this entity was last calculated.
+- **mainCategory** - A string categorizing the website. (e.g. Tor websites)
+- **subCategory** - A string categorizing the website within the main category. (e.g. 'Email' services, within Tor websites)
+- **description** - A long-form string describing the website.
+- **isBlacklisted** - A Boolean. Set to `false` by default.
+
+### Blacklist
+
+The blacklist entity points at a Website entity. It's purpose is to identify P2WDB entries that should not be displayed on the front-end. It has the following properties:
+
+- **hash** - The OrbitDB hash of the Website entity, used to identify it in the P2WDB.
+- **txid** - The TXID used as a key to identify the Website entity in the P2WDB.
+- **reason** - A string indicating the reason why an admin chose to blacklist this website.
+
+### Category
+
+The Category Entity holds data about entries that fit within a single category. Having an entity reduces the computational load, because data can quickly be looked up in the Category Entity, and the values of that Category Entity can be recalculated asynchrously. e.g. The time information is retrieved and the time information is updated are different.
+
+- **websitesAll** - An array of Mongo IDs of Website Entities. Every Website Entity assigned to the category should appear in this list.
+- **websitesTop** - An array of Mongo IDs of Website Entities. This contains a sorted list of the top 100 websites, based on merit.
+
+## Use Cases
+
+Use cases are verbs or actions that is done _to_ an Entity or _between_ Entities.
+
+- **Delete** - This is actually _not_ a use case. It's mentioned here to point out that deleting websites is not necessary. Users can instead move their PSF tokens to a different address, and the merit will go to zero. This will cause the website entry to drop to the bottom of the rankings for its category. Entries with zero merit will be ignored, or possibly garbage-collected.
+- **Add to Blacklist** - This is an action that can only be performed by a logged-in user with admin privledges. This allows the admin to prevent entries that violate the ToS from being displayed.
+- **Get Merit** - Read the merit value of the Website Entity.
+- **Update Merit** - Recalculate and update the merit value of the Website Entity.
+
+## Controllers
+
+Controllers are inputs to the system. When a controller is activated, it causes the system to react in some way.
+
+- **POST Webhook** - This POST REST endpoint will be triggered by a webhook generated by the P2WDB. This will notify the tor-list-api that a new entry has been added to the P2WDB that matches the `appId` of `torlist`. It's a new entry that should be evaluated for inclusion in the tor-list-api local database.
+- **GET by Category** - Return the 100 top-ranked Website Entities within a specific category, and with a merit above zero.
+- **POST Blacklist** - Creates a new Blacklist Entity. This endpoint is only accessible by an Admin user.
+- **24 hour Timer** - A timer which triggers every 24 hours, sets off an update of the database. When it triggers, the following actions should take place:
+  - All the website entities should be added to a retry-queue. Highest-merit websites are processed first, lowest merit last.
+  - The server will recalculate the merit for each Website Entity in the queue.
+    - TODO: Figure out a good garbage collection scheme to remove websites that consistently have no merit.
+  - The server will update the 'top 100' in each Category Entity.
+
+## Adapters
+
+Adapters are output libraries so that the business logic doesn't need to know any specific information about the I/O. They are essentially the output of the application.
+
+- **localdb** - An adapter for the local database (MongoDB).
+- **ipfs** - An adapter for IPFS and the ipfs-coord library. It allows the app to use the JSON-RPC over IPFS.