Overview

The Knowledge API lets you manage knowledge bases of documents that are processed asynchronously. Upload files or URLs, organize them into folders, digitize and rewrite content, and read embedding status for search/retrieval readiness.

Typical Flow

1. Create a knowledge base for your partner account, optionally with durable character, token, and USD budgets
2. Upload documents and URLs, or create a YouTube channel folder
3. Documents are automatically digitized (OCR/parsing/transcript extraction)
4. Optionally rewrite documents using LLM processing
5. Final rewritten content is embedded automatically
6. Poll document, budget, channel, or knowledge-base embedding status until ready
7. Retrieve processed content or download rewritten bundles

Features

• File and URL upload: Upload documents directly or provide URLs for automatic fetching
• Batch URL import: Submit up to 1,000 URLs in a single request
• Automatic digitization: Supported uploads are extracted, OCRed, or transcribed into text
• Knowledge-base budgets: Optionally cap stored digitized characters, lifetime model tokens, and lifetime customer charges across every ingestion route
• LLM rewriting: Documents are rewritten using configurable content-type templates (article, youtube, book)
• Automatic embedding: Rewritten documents are queued for embedding automatically; clients do not need to start KB embedding manually
• Folder organization: Organize documents into regular folders or YouTube channel folders that discover and import videos daily
• Processing spend visibility: See the stored cost of each document's latest rewrite attempt—including LLM-generated output that was rejected—and successful Apify digitization, with knowledge-base and folder totals
• Embedding status: Track per-document and knowledge-base embedding progress for search and retrieval
• Bulk operations: Rewrite all documents in a folder or knowledge base at once

Knowledge-base budgets

Each knowledge base can have three optional, independent limits:

Limit	Counts	Restored by deleting content?
Stored characters	Accepted digitized characters for non-deleted documents	Yes
Model tokens	Lifetime recorded input and output tokens from rewriting and embedding	No
Customer charges	Lifetime recorded customer charges, including bounded Apify discovery/digitization and model work	No

null means unlimited. A configured value of 0 blocks further consumption. Limits may be set when the KB is created or changed later through the limits endpoint. Lowering a limit below existing usage is valid: it pauses new paid work immediately, while already-dispatched work keeps its reservation and settles normally.

Budget admission applies to file uploads, individual and batch URL imports, YouTube videos, channel discovery, manual processing, bulk rewrites, retries, fallbacks, and embedding. Before a paid request, the processor checks current usage and outstanding reservations under the KB database lock. It reserves a bounded amount before dispatch, records actual usage after completion, and releases only unused verified capacity. Unknown historical usage or an ambiguous provider outcome pauses capped work rather than treating it as free.

When a budget blocks processing, the document stays visible with a BUDGET_BLOCKED: processing error and durable pending work. It automatically resumes after a relevant limit increases or character capacity is freed. Cached extracted text is retained when it is too large for the character budget, so resumption does not repeat extraction. Deleting content restores only stored-character capacity; it never refunds lifetime tokens or charges.

Request-level character_limit values on upload, batch, manual-processing, and content-update endpoints remain supported. When both a request-level limit and the stored KB character limit are present, the smaller limit applies.

YouTube channel folders

A folder can be a persistent YouTube channel source. Create it with type=youtube_channel, a channel URL or @handle, and a model tier. The service discovers regular videos, Shorts, and completed livestreams through the configured Apify channel actor. Active and upcoming livestreams remain deferred until a later scan confirms completion.

The first scan is scheduled immediately, subject to the KB budget. Subsequent complete metadata scans run every 24 hours. Newly discovered videos are imported as ordinary URL documents and processed newest first through the normal transcript, rewrite, and indexing pipeline. Existing documents elsewhere in the KB are linked as duplicates and are never reprocessed or moved. Explicitly deleted channel videos remain suppressed.

After a handle or alias resolves, the source uses its canonical channel identity. A later discovery result for a different channel pauses that source before it imports any videos from the mismatched page. Channel discovery itself is a customer charge and pauses with the rest of the KB when a budget is exhausted.

Spend in the Web UI

The Knowledge web UI displays the spend stored with each document's current processing state. This display is separate from the KB's lifetime budget ledger: it is useful for understanding the current document, while the budget includes all settled attempts, retries, embeddings, and channel discovery. The document display combines:

• The estimated list-price cost of the latest rewrite attempt. New attempts persist one combined cost calculated from the usage and price of every model involved; this keeps mixed economy/primary flows accurate. If the LLM generated output before the application rejected or truncated it, that consumed usage remains part of spend even though no final rewrite is published.
• The actual Apify usageTotalUsd share stored with the latest successful Apify digitization. Because Apify runs are batched, the run total is divided evenly across its unique inputs before it is saved on each accepted document.

Previous rewrites, failed digitization fallbacks, retries, and other historical attempts are not added. Only the latest rewrite attempt is retained; an LLM-generated rejected attempt replaces the previous rewrite component because its tokens were actually consumed. A source rejected before any LLM call has no rewrite spend.

Rewrite generation uses Hidoba Messages API v3 as its primary transport. Knowledge Processor uses its direct Google Vertex backup only when it cannot establish a connection to Messages API before dispatch. Authentication, access, and quota rejections never bypass Messages API. Content-filter and truncated results are not sent through the backup. Timed-out, malformed, and HTTP error results stop the current job without an automatic provider or queue retry because the original request may already have incurred spend.

• The knowledge-base dropdown always shows the estimated total for every knowledge base.
• An open document always shows its own Spend in the status row.
• Per-folder and per-file amounts are always visible in the sidebar.
• The sidebar can sort folders and files by Name (ascending), Price (highest first), or Size (largest first). Size uses the final rewrite character count when available, falling back to the digitized character count. The sort preference is saved in the current browser.

Knowledge Processor 3.1.5 and later stores the multi-model estimate in rewrite_cost_usd. Older completed rewrites fall back to rewrite_input_tokens_total, rewrite_output_tokens, and rewrite_model. The currently configured USD prices per 1 million tokens are:

Model	Input	Output
Gemini 3 Flash Preview	$0.50	$3.00
Gemini 3.5 Flash-Lite	$0.30	$2.50

Rewrite amounts are list-price estimates, while the Apify component is the actual run cost allocated to that input. Neither is an invoice record. A ≥ prefix means one or more rewrite or digitization components lack complete stored cost data, so the displayed total is a lower bound. n/a means processing exists but none of its spend components can be calculated. — means no priced rewrite or Apify digitization is stored in that scope.

Apify spend is persisted starting with Knowledge Processor 3.1.4. Earlier Apify digitizations cannot be reconstructed safely from document metadata and show no Apify component until the document is digitized through Apify again.

Rejected rewrite-attempt usage and mixed-model rewrite cost are persisted starting with Knowledge Processor 3.1.5. Rejected attempts recorded by earlier releases did not save their token usage on the document, so they cannot be priced from document metadata alone.

:::important Important Considerations

• Supported uploads: PDF, DOC/DOCX, PPT/PPTX, XLS/XLSX, EPUB, MD/MARKDOWN, HTML/HTM, TXT/PY/JS/CSS, CSV/JSON/XML/YAML/YML, MP4/MOV/M4V/AVI, MP3/WAV/M4A, and JPG/JPEG/PNG/TIF/TIFF/GIF/WebP. Archives such as ZIP, TAR, GZ/TGZ, RAR, 7Z, BZ2, and XZ are blocked.
• File size limit: 100MB per upload by default
• Character limits: A request-level character_limit is optional and limits total digitized characters across one knowledge base. A persisted KB character limit uses the same accounting. Both are enforced after OCR/parsing and are independent from the upload file-size limit.
• Token and USD limits: Persisted KB limits can stop new paid rewriting, embedding, digitization, and channel discovery. See the limits endpoint for usage, reservations, remaining capacity, and blocking reasons.
• Async processing: Digitization, rewriting, and embedding run in the background — poll document and KB status
• Embedding source: Only final rewritten content is embedded. Updating rewritten content automatically refreshes embeddings.
• Audio/video access: Audio and video transcription can be disabled per partner. If disabled, audio/video documents fail digitization with processing_error.
• Rate limits: Requests are rate-limited :::