fluree.doc ========== .. py:module:: fluree.doc .. autoapi-nested-parse:: Read documents with an extraction model reached over HTTP. :func:`read` parses a file (PDF, Word, PowerPoint, HTML, Markdown, text, spreadsheets, transcripts, email) the way ``fluree doc parse`` does, sends its text and structure with your ontology and catalog to a reader, and returns its reading: a Fluree DB transaction holding the run (the document read, the release that read it), the facts in your ontology's terms, the claims behind them and the mentions, each anchored to character offsets in the parser's text (what ``fluree doc parse --format text`` prints):: import fluree reading = fluree.doc.read("invoice.pdf", ontology="invoices.ttl", catalog="suppliers.ttl", model="reader-small") for m in reading.mentions(): print(m["text"], m["entity"]) with fluree.connect("./data") as conn: conn.create("invoices").insert(reading.to_jsonld()) ``include=`` says what comes with the facts, which always come: ``"claims"``, ``"mentions"``, both (the default) or ``"none"``. The read is the ``fluree`` CLI's own, so a document is cut into calls, sent, resent when the reader has no capacity yet or rate-limits it, and kept the same way. Waits (a model still starting can take minutes) are logged as warnings on the ``fluree.engine`` logger. Python does no vision escalation: scanned pages and images are read with the parser's deterministic tier only, and :attr:`Reading.notes` says what it couldn't read. **Signing in.** 1. ``api_key`` with ``auth_header``: that key, sent as is in that header, to ``url``. 2. ``api_key`` alone: a Fluree.ai API key (``flk_…``) is traded for an access token (reused for the hour it lasts), or, when the service won't trade it (a developer key, which calls the reader itself), sent as it is; any other key is sent as ``Authorization: Bearer``, as the CLI's ``[doc.reader] api_key`` is. 3. No ``api_key``: this machine's ``fluree login``, else the ``FLUREE_AI_API_KEY`` environment variable (as in 2). Each is sent only to its own service; a ``url`` elsewhere is called with no credential. With no ``url``, a Fluree.ai sign-in reads through the account. **Rows and terms.** The reading is kept as the reader wrote it, in the extraction vocabulary (``xt:``, ``https://ns.flur.ee/extract#``): :attr:`Reading.transactions` holds one transaction per call (a document longer than one call's limit takes several, each its own run, its offsets moved into the whole document's text). :meth:`Reading.mentions` and :meth:`Reading.claims` are rows read out of it. Module Contents --------------- .. py:class:: Usage What a read was billed, as the reader reported it. .. py:attribute:: input_tokens :type: int .. py:attribute:: windows :type: int .. py:attribute:: cache_read_tokens :type: int .. py:attribute:: cache_write_tokens :type: int .. py:attribute:: fuel :type: float | None .. py:attribute:: note :type: str The same, as one line: ``1,032 tokens, cache 900 read / 0 written``. .. py:class:: Reading One document's reading, as the reader wrote it. .. py:attribute:: transactions :type: list[dict[str, Any]] One Fluree DB transaction (JSON-LD) per call: the run in its ``txn-meta``, the facts and the claims annotating them in its ``@graph``, the mentions in a named graph of their own. .. py:attribute:: fact_count :type: int Statements in the facts, the claims aside. .. py:attribute:: new_entities :type: list[str] Entities the runs minted for things the catalog doesn't hold. .. py:attribute:: linked :type: list[str] Catalog records the mentions link to. .. py:attribute:: usage :type: Usage .. py:attribute:: calls :type: int Calls the document took; :attr:`cached` of them were answered from the cache. .. py:attribute:: cached :type: int .. py:attribute:: release :type: str | None .. py:attribute:: runs :type: list[dict[str, Any]] :value: [] One per call: ``run`` (the run's IRI), ``release``, ``request_id`` (the endpoint's ID for the call) and ``cached``. .. py:attribute:: ontology_notes :type: list[str] :value: [] What the reader set aside or adapted in the ontology. .. py:attribute:: notes :type: list[str] :value: [] What the parse left out: pages it couldn't read (no vision escalation from Python), an email's attachments, and the like. .. py:property:: mention_count :type: int .. py:property:: claim_count :type: int .. py:method:: to_jsonld() -> dict[str, Any] The reading as a transaction, ready for ``Ledger.insert``: the run becomes the commit's metadata. A document read in several calls has a transaction per call; insert each of :attr:`transactions`. .. py:method:: to_trig() -> str The reading as TriG: the run in ``<#txn-meta>``, each claim behind a fact reifying it. .. py:method:: to_turtle() -> str The reading's statements as Turtle, without the run. .. py:method:: mentions() -> list[dict[str, Any]] One row per mention (``xt:Occurrence``), in text order: ``id``, ``text`` (``nif:anchorOf``), ``begin`` and ``end`` (offsets in the parser's text), ``class`` (``xt:occurrenceClass``), ``entity`` (the catalog record it links to, ``xt:entity``, or the entity the run minted, ``xt:localEntity``, and then ``new_entity``), ``confidence`` (``xt:linkConfidence`` for a link, else ``xt:score``) and ``element`` (the structure element it sits in). .. py:method:: claims() -> list[dict[str, Any]] One row per claim (``xt:Claim``): ``id``, ``subject``, ``predicate``, ``object`` (an IRI, or a literal's text), ``status`` (``xt:assertionStatus``), ``accepted``, ``score``, and ``asserted`` (the claim annotates its fact, which is in the facts). .. py:function:: read(path: str | os.PathLike[str] | None = None, *, text: str | None = None, ontology: str | os.PathLike[str], catalog: str | os.PathLike[str] | Sequence[str | os.PathLike[str] | tuple[str | os.PathLike[str], str]] | None = None, model: str, cache: str | None = None, include: str | Sequence[str] = _INCLUDE, url: str | None = None, api_key: str | None = None, auth_header: str | None = None, doc_iri: str | None = None, base_iri: str = 'urn:fluree:doc:', cache_dir: str | os.PathLike[str] | None = None) -> Reading Read one document with ``model`` and return its :class:`Reading`. Give a file as ``path``, or plain text as ``text``. A text is read under ``doc_iri``, by default ``text/``, so two different texts never share a document's nodes. ``ontology`` is a ``.ttl``, ``.nt``, ``.jsonld`` or ``.json`` file. ``catalog`` is one such file, or a list of them, each optionally a ``(file, class IRI)`` pair to take only that class's records; a bare 2-tuple is read as such a pair unless its second item is itself an RDF file. ``cache`` (``"ontology"``, ``"catalog"`` or ``"off"``) lets the reader keep that much for the next request sharing it. ``include`` says what comes with the facts: ``"claims"``, ``"mentions"``, both (the default, as ``("claims", "mentions")`` or ``"claims,mentions"``), or ``"none"``. With ``cache_dir``, parses and answers are kept there, so reading the same request again costs nothing. Raises :class:`fluree.errors.InvalidRequestError` for arguments that can't make a read (a bad ``cache``, ``auth_header`` without ``api_key``, no way to sign in), :class:`fluree.errors.PermissionDeniedError` when a credential is refused, and :class:`fluree.errors.FlureeError` for a reader's other refusals, with its message.