hentag-scraper

Scraper and sync tools for hentag page data.

Setup

Install dependencies:

bun install

Set environment variables (for example in .env):

DATA_DIR=./data
MAX_PAGE=2000000
MAX_CONCURRENT_FETCHES=4
SLEEP_TIME=0
START_PAGE=1
END_PAGE=2000000
RETRY_MAX_ATTEMPTS=6
RETRY_BASE_DELAY_MS=750
RETRY_MAX_DELAY_MS=30000
GLOBAL_COOLDOWN_MS=5000
MIN_REQUEST_INTERVAL_MS=200
DIRECTION=asc

Scripts

1) Fetch missing page JSON files

index.ts fetches page JSON files into DATA_DIR for a selected range. Existing files are skipped.

bun run index.ts

Uses:

• START_PAGE, END_PAGE, DIRECTION for range/order
• MAX_CONCURRENT_FETCHES for worker count
• retry/rate-limit settings from env.ts

2) Import local JSON files into SQLite

save-data.ts reads all .json files from DATA_DIR and upserts them into data.db.

bun run save-data.ts

3) Incremental API sync directly into SQLite

main.ts fetches pages from the API and writes directly to data.db.

It stops when it encounters the first work already current in the database:

• gallery.metadata_id == work.id
• gallery.metadata_updated_at == work.lastModified

This is intended for incremental updates from newest pages onward.

bun run main.ts

Uses:

• START_PAGE to choose where to begin
• MAX_PAGE as the upper bound
• retry/rate-limit settings from env.ts

Database

SQLite file: data.db

Tables:

• gallery
• tags
• gallery_tags
• gallery_location

Schema/upsert logic is centralized in gallery-repository.ts.

Notes

• Runtime: Bun
• Type check:

bunx tsc --noEmit