Skip to main content

Overview

The Knowledge API lets you manage knowledge bases of documents that are processed asynchronously. Upload files or URLs, organize them into folders, digitize and rewrite content, and read embedding status for search/retrieval readiness.

Typical Flow​

  1. Create a knowledge base for your partner account
  2. Upload documents (files or URLs) into the knowledge base
  3. Documents are automatically digitized (OCR/parsing)
  4. Optionally rewrite documents using LLM processing
  5. Final rewritten content is embedded automatically
  6. Poll documents or knowledge-base embedding status until ready
  7. Retrieve processed content or download rewritten bundles

Features​

  • File and URL upload: Upload documents directly or provide URLs for automatic fetching
  • Batch URL import: Submit up to 1,000 URLs in a single request
  • Automatic digitization: Supported uploads are extracted, OCRed, or transcribed into text
  • Optional knowledge-base character limit: Callers can enforce a total digitized-character limit atomically as content is committed; this is a KB-wide limit, not a per-document cap
  • LLM rewriting: Documents are rewritten using configurable content-type templates (article, youtube, book)
  • Automatic embedding: Rewritten documents are queued for embedding automatically; clients do not need to start KB embedding manually
  • Folder organization: Organize documents into folders within knowledge bases
  • Processing spend visibility: See the stored cost of each document's latest rewrite attempt—including LLM-generated output that was rejected—and successful Apify digitization, with knowledge-base and folder totals
  • Embedding status: Track per-document and knowledge-base embedding progress for search and retrieval
  • Bulk operations: Rewrite all documents in a folder or knowledge base at once

Spend in the Web UI​

The Knowledge web UI displays the spend stored with each document's current processing state. It combines:

  • The estimated list-price cost of the latest rewrite attempt. New attempts persist one combined cost calculated from the usage and price of every model involved; this keeps mixed economy/primary flows accurate. If the LLM generated output before the application rejected or truncated it, that consumed usage remains part of spend even though no final rewrite is published.
  • The actual Apify usageTotalUsd share stored with the latest successful Apify digitization. Because Apify runs are batched, the run total is divided evenly across its unique inputs before it is saved on each accepted document.

Previous rewrites, failed digitization fallbacks, retries, and other historical attempts are not added. Only the latest rewrite attempt is retained; an LLM-generated rejected attempt replaces the previous rewrite component because its tokens were actually consumed. A source rejected before any LLM call has no rewrite spend.

Rewrite generation uses Hidoba Messages API v3 as its primary transport. Knowledge Processor uses its direct Google Vertex backup only when it cannot establish a connection to Messages API before dispatch. Authentication, access, and quota rejections never bypass Messages API. Content-filter and truncated results are not sent through the backup. Timed-out, malformed, and HTTP error results stop the current job without an automatic provider or queue retry because the original request may already have incurred spend.

  • The knowledge-base dropdown always shows the estimated total for every knowledge base.
  • An open document always shows its own Spend in the status row.
  • Per-folder and per-file amounts are always visible in the sidebar.
  • The sidebar can sort folders and files by Name (ascending), Price (highest first), or Size (largest first). Size uses the final rewrite character count when available, falling back to the digitized character count. The sort preference is saved in the current browser.

Knowledge Processor 3.1.5 and later stores the multi-model estimate in rewrite_cost_usd. Older completed rewrites fall back to rewrite_input_tokens_total, rewrite_output_tokens, and rewrite_model. The currently configured USD prices per 1 million tokens are:

ModelInputOutput
Gemini 3 Flash Preview$0.50$3.00
Gemini 3.5 Flash-Lite$0.30$2.50

Rewrite amounts are list-price estimates, while the Apify component is the actual run cost allocated to that input. Neither is an invoice record. A ≥ prefix means one or more rewrite or digitization components lack complete stored cost data, so the displayed total is a lower bound. n/a means processing exists but none of its spend components can be calculated. — means no priced rewrite or Apify digitization is stored in that scope.

Apify spend is persisted starting with Knowledge Processor 3.1.4. Earlier Apify digitizations cannot be reconstructed safely from document metadata and show no Apify component until the document is digitized through Apify again.

Rejected rewrite-attempt usage and mixed-model rewrite cost are persisted starting with Knowledge Processor 3.1.5. Rejected attempts recorded by earlier releases did not save their token usage on the document, so they cannot be priced from document metadata alone.

:::important Important Considerations

  • Supported uploads: PDF, DOC/DOCX, PPT/PPTX, XLS/XLSX, EPUB, MD/MARKDOWN, HTML/HTM, TXT/PY/JS/CSS, CSV/JSON/XML/YAML/YML, MP4/MOV/M4V/AVI, MP3/WAV/M4A, and JPG/JPEG/PNG/TIF/TIFF/GIF/WebP. Archives such as ZIP, TAR, GZ/TGZ, RAR, 7Z, BZ2, and XZ are blocked.
  • File size limit: 100MB per upload by default
  • Character limits: character_limit is optional and limits total digitized characters across one knowledge base. It is enforced after OCR/parsing, so it is independent from the upload file-size limit. Omit it for unrestricted processing.
  • Async processing: Digitization, rewriting, and embedding run in the background — poll document and KB status
  • Embedding source: Only final rewritten content is embedded. Updating rewritten content automatically refreshes embeddings.
  • Audio/video access: Audio and video transcription can be disabled per partner. If disabled, audio/video documents fail digitization with processing_error.
  • Rate limits: Requests are rate-limited :::