Document Processing

Document Processing

The core of AutoDoc: detecting placeholders in an uploaded .docx template, collecting values for them, and generating a merged document without changing the template's page count where possible. All of this runs on src/autodoc/services/document_processing.py and src/autodoc/documents/routes.py.

Placeholder syntax

Both styles are detected simultaneously: - <placeholder> — the style used by the real conveyance-deed template this project was built around - {placeholder} — the more common convention elsewhere (avoids colliding with HTML, which <> can, and did — an early version of the UI rendered <plotno> as an actual unknown HTML tag before this was fixed with proper escaping)

PLACEHOLDER_PATTERN in document_processing.py matches both in one regex. Detection scans every paragraph in the document body and every table cell (extract_placeholders()).

Values format

Members never type brackets. A pasted or individually-entered field is always a bare name (plotno, not <plotno> or {plotno}) — canonical_key(token) strips whichever bracket style wraps a detected placeholder and lowercases it, so <plotNo>, {plotno}, and a bare plotno all resolve to the same key.

Two UI input modes, kept in sync: - Paste all at once — one line per field: fieldname: value (colon, for hand-typing) or fieldname<TAB>value (tab, for pasting two columns straight out of Excel). Auto-detected per line. Plain CSV (comma-delimited) was deliberately not used — real values here (Indian-formatted amounts like 24,57,856.00, addresses with commas) are comma-heavy, which makes naive comma-splitting unreliable. - Individual fields — a table, one labeled input box per detected placeholder.

Switching between the two modes converts whatever's currently entered, so nothing is lost (onFillModeChange() in index.html).

Computed aliases

apply_computed_aliases() fills in a couple of fields the member never types directly, mirroring the original template's business rules: - allotteeresidence — copied from allotteeaddress if not separately provided - convertvalueinwords — the value field converted to Indian-style words (crore/lakh/thousand grouping) via number_to_words_indian()

These are template-specific; add more here as new templates need them.

Real page counting (why LibreOffice)

A .docx file has no stored page count — pagination depends on the renderer (fonts, margins, page size). Word computes it live via COM automation; this server has no Word. Instead, AutoDoc uses a portable, no-root LibreOffice install (see cPanel deployment for how it's set up) to render to PDF and count pages with pypdf:

• convert_to_pdf(docx_path, output_dir, soffice_path) — runs soffice --headless --convert-to pdf, each call with its own throwaway profile directory so concurrent conversions never collide.
• count_pdf_pages(docx_path, soffice_path) — wraps the above and counts the resulting PDF's pages.

Both return None (never raise) if soffice_path isn't configured, times out, or fails — callers must treat a missing count as "unknown," not an error. This is what lets local development (no LibreOffice on the Windows dev machine) still work: page counting and page-fit shrinking are simply skipped, falling back to a plain merge.

Merging: run-level replacement

Naively rebuilding a whole paragraph's text loses per-run formatting (bold/italic/font-size boundaries). Instead, _replace_placeholder_in_place() isolates each replaced occurrence into its own run, cloning the original run's formatting (rPr) via copy.deepcopy, so surrounding text keeps its formatting and the inserted value inherits the placeholder's original size. Verified to correctly preserve bold prefixes, italic suffixes, and explicit font sizes around a replaced value.

If an occurrence spans multiple runs (rare — usually only from a mid-placeholder formatting change in the original document), it's left for a fallback whole-paragraph substitution (_fallback_replace_token()) so the content is still correct, at the cost of losing that one paragraph's per-run formatting.

Page-fit generation

generate_with_page_fit() is the heart of the "don't change the page count" requirement:

1. Merge all values into the document at their normal (template) size.
2. Check the real page count once, via LibreOffice.
3. If it's still within the template's original page count, done.
4. If it grew, collect every place a font size is explicitly set in the whole document — every run with its own size, plus every paragraph/character style with a defined size (_collect_whole_document_font_targets()) — and shrink all of them together, uniformly, in 0.2pt steps down to a fixed 8pt floor, re-checking the page count after each step, until it fits or bottoms out.

Why whole-page, not per-value

An earlier version shrunk only the inserted values' runs. This looked visually inconsistent — some fields (with lots of surrounding whitespace) shrunk noticeably while others didn't need to, and it didn't help when the overflow came from a value that wrapped onto an extra line by itself. Shrinking the whole page uniformly (matching what a human would do by hand — reduce the whole document's font slightly) looks consistent and fixes overflow regardless of which field caused it.

Why one page-check, not one per placeholder

The original Word/win32com script this feature reproduces checks the page count after every single placeholder replacement, using Word's instant in-process repagination. Reproducing that exactly on Linux means launching a LibreOffice process per check — with a real template's 16 placeholders, that reliably exceeded the server's request timeout (a LibreOffice cold start is a few seconds, not instant). The current approach merges everything first and checks/shrinks as a group, typically needing only a handful of LibreOffice calls total regardless of placeholder count.

Minimum font size

Fixed internally at 8pt (min_font_size_pt default in generate_with_page_fit). This was originally an admin-configurable setting; removed on request since a hardcoded sane floor was judged sufficient and one less thing to misconfigure.

Two modes: fit vs. simple

Both GET /documents/{id}/generate and GET /documents/{id}/preview accept ?fit=true|false (default true): - fit=true — the full page-fit algorithm above - fit=false — fill_placeholders_simple(), a plain one-shot substitution with no page-count guarantee, instant since it never invokes LibreOffice

The UI exposes this as a "Page count" dropdown per document: "Keep original page count" vs. "Just replace placeholders".

Everything server-side

An earlier design considered a "download a package (template + values.txt) to run locally with the original Word script" option. This was explicitly rejected — AutoDoc is a commercial product, and no step should require a member's own PC. Preview and Download are both fully server-side; the only artifact of the "run it elsewhere" idea that remains is fill_placeholders_simple() itself, repurposed as the fast "simple" mode rather than a handoff format.