On-Prem Legal AI
A private AI workspace for legal teams that runs on hardware in their own office, so documents and questions never leave the building.
The 93 second recording walks through model management, a document collection, a cited answer, a workflow, the audit log and the system console. The names and documents in it are synthetic.
The situation
Law firms and clinics cannot paste privileged documents into a public chatbot. So they either skip AI and lose the productivity, or use it anyway and accept the risk. Both options are bad, and this product existed to close the gap with a preconfigured computer, roughly the size of a thick book, that ships with local models and a browser interface reachable from any device in the office.
The idea we kept coming back to was network accessed intelligence. Where a NAS made files a shared network resource, this makes inference a shared, governed, auditable resource the firm owns. My job was the software side of that promise. Deployment, hardening, the interface, and the retrieval layer for legal documents.
How it works
The diagram is the pipeline. The frames below it are taken from the recording.
- StoreLocal modelsOllama serves open models such as Llama, Mistral, Qwen and Gemma. Admins pull new ones from the settings page.
- StoreDocument collectionsPDF, Word, Outlook messages and spreadsheets are parsed, chunked and embedded into a local vector store.
- RetrievalA question pulls the relevant passages from the collections the user is allowed to see.
- Cited answerThe model answers with citations that open the exact passage, its page and a relevance score.
- Tools and workflowsMCP servers and n8n workflows extend the assistant without sending data off the machine.
- GateAudit logEvery prompt, retrieval, upload and workflow run is logged with user and timestamp.
- GateAir gap switchA kill switch cuts WAN traffic while models, knowledge and workflows keep running on the LAN.
Manage models from the browser
Admins pull models by name from the settings page and watch the download. The box runs as many as it has room for.
Build a document collection
A data room takes any mix of formats. Each file is parsed, chunked and embedded locally, and access stays scoped to the group you grant.
Check the answer against the source
Opening a citation shows the exact passage, its page and a relevance score, so a lawyer can verify the claim without reading logs.
Automate without leaving the box
Workflows are built visually in the bundled n8n instance and call the local model, so an intake or conflict check runs on the same hardware.
Keep an audit trail
The admin log records every prompt, retrieval, upload and workflow run with the user who triggered it.
Watch the machine
A separate console shows GPU load, inference throughput, storage and network, with a kill switch that cuts WAN traffic while everything keeps running on the LAN.
Decisions that mattered
Build on Open WebUI, then own the fork
Starting from a proven interface bought the chat, document and admin surfaces. The fork added audit logging, a monitoring dashboard, offline bundles and hardened MCP tool execution.
Lawyer readable first, telemetry second
Show which documents an answer used before showing anything about tokens or retrieval scores. The detailed logs stay available but one click away.
Harden like a public server
Fail2Ban, AppArmor, HSTS, a strict content security policy and nosniff headers, even though the box lives on an office network.
A retry loop for models without native tool calling
Smaller local models do not all support function calling, so tool use runs through a ReAct style loop that retries and shows each call in the interface.
Where it landed
Local model management, document collections with cited answers, workflows and an audit trail on hardware the customer owns. The point of the product held. Models, documents and answers stayed on the box.
- Local models, no cloud accounts
- Answers open the passage they came from
- Audit log and air gap switch for compliance conversations
- Legal retrieval quality was still open work when the engagement ended. Evaluating legal specific embeddings and OCR quality for scanned filings was on the list, so this page makes no clause level accuracy claims.
- Multi user load and multi GPU concurrency were not validated.
- The client stays unnamed here.
Notes from the build
Written in my own words, after the fact. Opinions, not claims.
This ran alongside the BusinessFrame work for the same client, and it pulled me toward the operations side of AI. Getting a model to answer is the easy part. Getting a box to a law firm that keeps working without anyone from us on site, that a paralegal can administer, and that a compliance officer will sign off on is the actual product.
The note I kept coming back to in my own list was make it lawyer readable. A retrieval score means nothing to the person reading the answer. They want to know which document it came from and whether they can trust the passage. That reordered the whole interface.
I would want an evaluation set of real legal questions with known answers before I called the retrieval done. We never got there, and I would rather say that than claim it.
Nawap Bastola · 2025 to 2026