Loka: Generative Citation in a Neuro-Symbolic World Model over RDF-Star Knowledge Graphs

Code: https://github.com/EmmaLeonhart/Loka (engine release v0.4.0: https://github.com/EmmaLeonhart/Loka/releases/tag/v0.4.0) · Corpus + checkpoints: https://huggingface.co/datasets/EmmaLeonhart/loka (snapshot tags v3, v4, v5, v6-bpe, v7, v8, v9, v10) · Source dataset: https://huggingface.co/datasets/philippesaade/wikidata

Abstract

Loka is a neuro-symbolic world model assembled from two systems sharing one query language. The first is an RDF-star triplestore (the engine, formerly published as Loka) — explicit memory, exact answers. The second is a small role-aware transformer trained from scratch on the same triples, with English labels substituted for opaque entity identifiers — implicit memory, plausible answers. They compose at the SPARQL+ layer: a query reaches both systems and the caller does not pick which one answered, except by inspecting propositionInferredFrom provenance edges on each result.

The technical contribution is generative citation: a closed loop in which the transformer's predicted triples are written back into the triplestore as RDF-star annotations whose subject is the quoted generated triple and whose object is another quoted triple — a directly cited piece of context the prediction was conditioned on. A reserved system namespace (http://loka.dev/provenance/) marks every system-emitted predicate, which is enforced at three layers (corpus stripping, candidate filtering, emit-time guard) so the model never sees, learns to predict, or hallucinates a citation predicate. Hallucinated citations (the model picking the wrong context triple as the support) are auditable and filterable like any other generated triple — they degrade like other RDF rather than vanishing into opaque embeddings.

We demonstrate the end-to-end loop on a 5,055,385-triple slice of Wikidata (philippesaade/wikidata, streamed from Hugging Face), with role-aware masked-S/P/O training producing models from 16M to 44M parameters that reach final perplexities of 92.5 and 84.85 respectively over five epochs. Predictions emerge that are not memorized templates (e.g., Comtesse de Die | educated at | university of halle correctly identifies Halle, where she studied; Abbas Mirza | has works in the collection | metropolitan museum of museum correctly identifies the Met). We characterize the failure modes — mode collapse on common connector tokens, mitigated at decode time by a cumulative repetition penalty rather than at training time — and document two engine-level bugs surfaced by the data scale.

We then trace a substantial corpus-quality finding from a post-training behavioural test (§5.5): the v6 model produced confident catalog-format hallucinations (ISNI -> 00000000, Freebase -> /m/0c__9) on identifier predicates, and worse, the catalog-format shape leaked onto unrelated predicates (instance of -> + Ġof - 00 - 03 T 00, a Wikidata date-prefix string, on 15 different subjects in a single 30-source run). Investigation showed that 49.6 % of the v6 corpus was Wikidata external-id predicates (Freebase, ISNI, GND, LCCN, Dewey, etc.) — there are 10,206 such properties on Wikidata, ~80 % of all property types. We rebuild the corpus with these removed, drop a further 319 properties of other catalog-shaped datatypes (url, commonsMedia, lexeme/sense/form, math, geo-shape), and normalise time and quantity literals (strip + era prefix, drop T00:00:00Z on date-only values). The resulting v7 corpus is 184,458 triples (24 % of v6 by volume) and trains to a comparable perplexity (192.63 vs v6's 194.98) but with the catalog-format hallucinations vanished — the model's failure mode shifts from "confidently wrong" to "refuses to emit", which is what we want from a generative-citation system. We then trained v8 — the same architecture, 20 epochs from random initialisation on the v7 corpus — reaching perplexity 64.65, a 3× reduction, with the loss curve still descending at epoch 20. The cleaned v7 corpus carries substantially more exploitable signal than 5 epochs can extract. v9 and v10 push further: each is trained on a freshly-pulled 2 M-triple Wikidata slice (under the per-data-dir state file design described in §5.7), reaching perplexity 57.15 and 55.52 respectively on roughly half the corpus size of v7 — a 3.5× total improvement over v6. On the standard Q42 propgen test, v10 emits 60 candidate triples, zero of which are catalog-format hallucinations, against v6's 21/52. v10 is also the first model shipped end-to-end by an automated 12-hour cron loop with no manual intervention (§5.8).

1. Introduction

Two technical pressures motivate this work.

First: knowledge-graph completion has historically been a black-box prediction problem. TransE-family link predictors and recent transformer-on-KG approaches output a confidence over candidate triples, but offer no native account of what evidence shaped a given prediction. Provenance lives outside the model — in metadata about the training corpus, not as edges of the graph the model populates.

Second: language models hallucinate without traceable inference. LLM responses to factual queries are a single forward pass over a frozen distribution; the answer is the answer, with no surface that distinguishes "this came from training data" from "this is a plausible continuation." Retrieval augmentation pins one piece of evidence to one response, but does not produce a graph one can later prune, audit, or retrain on.

Loka's claim is that a single design choice resolves both: if the inference layer's outputs are triples and provenance is expressed as RDF-star annotations on those triples, then every model-generated fact lands in the same store as the curated facts, with first-class citation edges to its supporting context. Auditable, filterable, queryable in SPARQL+, retrainable on the post-filtered corpus. The "neuro-symbolic" adjective is not aspirational — it describes the data layout.

Contributions

1. A reserved provenance namespace and a three-layer enforcement. Predicates under http://loka.dev/provenance/ (e.g., propositionGenerated, propositionInferredFrom, propositionGeneratedBy, propositionConfidence) are system-only. Three independent guards prevent the model from ever seeing, proposing, or emitting one: a SPARQL-star FILTER NOT EXISTS << ?s ?p ?o >> propositionGenerated ?_g clause in the corpus puller, a candidate-predicate filter in the inference loop, and an emit-time guard before each primary triple is written. Any single guard suffices; together they ensure that even with a regression in one path, generated provenance never re-enters training data. (§3.1)

2. Generative citation as RDF-star reification. Every model-generated triple <S> <P> "X" is accompanied by a fixed-shape annotation block. The block's subject is the quoted generated triple <<S P "X">>. Its objects include four metadata predicates (propositionGenerated, propositionGeneratedBy, propositionConfidence, ...) and one or more propositionInferredFrom edges whose object is another quoted triple — a cited piece of context. The result is a graph of generated triples threaded by citation edges to the curated context that informed them. (§3.2)

3. Cumulative repetition penalty as a decode-time correction for mode collapse on common tokens. Masked-S/P/O training produces models that "know" the answer category (university, museum, https-URL) but degenerate during greedy decoding to fillers like of of of of or museum museum. We show that a cumulative repetition penalty — dividing each repeated token's logit by repetition_penalty ** count — collapses these cascades within 2–3 emissions while preserving genuinely-needed reuse. The same v4 checkpoint moves from university of of of of of of of (no penalty) to university of halle (cumulative penalty 3.0), without retraining. (§4.3)

4. Empirical demonstration on a real-scale 5M-triple corpus. We report the v3→v4→v5 trajectory, including a corpus-quality regression caught and fixed mid-development (datatype-suffix tokens leaking into the training set), the qualitative failure modes of each model, and the headline result that capacity (16M → 44M params) was the binding constraint at this corpus size — the bigger model produces concrete entity tokens (halle, 33, kosmos 116) where the smaller one fell back to common-token fillers. (§5)

We also surface two engine-level bugs found at scale: a SPARQL serialization quirk producing literal values in the predicate slot, and a write-flush wedge in the persistent layer at roughly every 5–6× growth in stored triples. The first is filtered at preprocess time and remains open; the wedge was diagnosed and fixed during v9 (§5.7) by batching the persistent-store writes into a single sled multi-tree transaction per HTTP request, verified at 2 M-triple sustained ingest with no recurrence.

2. Background

2.1 RDF-star

RDF-star is an extension of RDF in which any of the three positions of a triple — subject, predicate, object — may be a quoted (referenced, not asserted) triple. The notation <<s p o>> means "the triple s p o, treated as a term." This admits direct annotation of facts:

:Tokyo  :population  "13929286" .
<<:Tokyo :population "13929286">>  :measuredAt  "2020-01-01" .
<<:Tokyo :population "13929286">>  :statedIn    :census2020 .

The same shape that Wikidata expresses through reified statement nodes (e.g., wds:Q1490-abc...) collapses into one structural primitive. Two storage strategies exist: separate-asserted-graph (RDF 1.2 working draft) and synthetic-ID interning (used by Loka, where quoted_triple_id(s_id, p_id, o_id) = xxh3 deterministically). We use the latter for compact joins on quoted-triple subjects.

2.2 Transformer-based knowledge graph completion

The dominant patterns in KG completion split into translational (TransE, RotatE, etc.) and transformer-based (KG-BERT, KGT5, recent work using LLMs as scoring functions). Most predict a single missing entity given (subject, predicate, ?) and report top-k accuracy on held-out triples. Two limitations relevant here: (a) outputs are scores or candidate IDs, not triples that can be re-stored; (b) provenance — which other triples in the corpus made this prediction confident — is not surfaced.

2.3 The from-scratch position

Loka's training is from scratch on RDF-derived text, not fine-tuning of a pretrained LLM. The position is not anti-LLM — it is that the closed-form auditability of "model knowledge ⊆ training corpus" is load-bearing for generative citation. With a fine-tuned LLM, even with the same RDF-star output schema, a generated triple may be drawn from base-model pretraining that the user never authorized as authoritative. We document a parallel near-term track admitting fine-tuning under stricter provenance assumptions in planning/fine-tuning-track.md; for the experiments in this paper, all results are from-scratch.

3. Architecture

3.1 The reserved provenance namespace

Every predicate under http://loka.dev/provenance/ is system-internal. The names are deliberately verbose — propositionGeneratedFrom rather than generatedFrom — so a human scanning raw triples spots them at a glance and accidental collision with real-world predicates is vanishingly unlikely. The full namespace currently holds:

Three layers of enforcement keep these out of the model's view and output:

Corpus stripping. The training corpus extractor issues a SPARQL-star query that excludes any inner triple flagged generated:

SELECT ?s ?p ?o WHERE {
  ?s ?p ?o .
  FILTER NOT EXISTS {
    << ?s ?p ?o >> <http://loka.dev/provenance/propositionGenerated> ?_g .
  }
}

It also drops any row whose predicate IRI matches the reserved prefix.

Candidate filtering. The inference loop builds candidate (subject, predicate) pairs by intersecting subject-with-graph-neighbor predicates. Reserved-namespace predicates are excluded from pred_usage and re-filtered at the candidate list level.

Emit-time guard. Each prediction's primary triple is checked against the reserved prefix immediately before it is written to the output stream. A reserved-prefix predicate is logged loudly and dropped.

Any single layer suffices. Three are kept because regressions in one path should not silently allow the model to learn or output system metadata.

3.2 Generative citation as RDF-star reification

A note on what "citation" claims here. At v0, the cited context triples are not selected by the model's internal attention or by a learned retrieval head; they are the rows the inference loop's candidate-predicate selection (§4.4 step 1) conditioned on. The contribution we claim is the schema — the data shape that lets a model emit a triple together with a transparent, queryable, post-hoc-auditable record of which curated rows were considered for that prediction — not a learned mapping from prediction to evidence. We treat this as a v0 design choice, not a final position; §6.3 records the gap and §7 sketches the OWL-template and HNSW-decoder paths that would make the link mechanistic. The schema makes the gap auditable: a downstream consumer can SPARQL-star over the propositionInferredFrom edges and decide for themselves whether each citation is informative, regardless of what the model "actually" attended to.

When the inference layer accepts a candidate (S, P) and emits a predicted object "X", it writes a fixed-shape block:

<S> <P> "X" .
<<S P "X">>  prov:propositionGenerated     "true"^^xsd:boolean .
<<S P "X">>  prov:propositionGeneratedBy   "wikidata_v4" .
<<S P "X">>  prov:propositionConfidence    "0.43"^^xsd:decimal .
<<S P "X">>  prov:propositionInferredFrom  <<S existing_p1 existing_o1>> .
<<S P "X">>  prov:propositionInferredFrom  <<S existing_p2 existing_o2>> .
   ... (default: 10 cited context triples per prediction)

prov: is the abbreviation for the reserved namespace. The cited context triples are existing rows about the subject S that the inference loop's candidate-predicate selection conditioned on. The shape is identical for inference outputs (propositionInferredFrom) and ingest outputs (the same RDF-star pattern absorbs Wikidata's pq: qualifiers and pr: references on import) — citation is uniform across the data layer.

Hallucinated citations are not a correctness problem. A fabricated propositionInferredFrom row is still a transparent RDF-star annotation pointing at concrete context — auditable, filterable, often informative about what the model thinks the reasoning is. We do not add elaborate guards against citation hallucination; the schema does the work.

3.3 The two-system loop

   ┌───────────────────┐
   │ Curated triples   │  (Wikidata, etc.)
   │  (RDF-star)       │
   └─────────┬─────────┘
             ▼
   ┌───────────────────┐         ┌──────────────────────┐
   │ Loka store     │ ─────→  │ Training corpus      │
   │  (.sdb, RDF-star) │  SPARQL │  (label-substituted) │
   │                   │  +SPARQL-│                      │
   │                   │  star    │                      │
   └─────────▲─────────┘         └──────────┬───────────┘
             │                              ▼
             │                  ┌──────────────────────┐
             │                  │ Role-aware           │
             │                  │ transformer          │
             │     ┌──── feeds to ──── (this paper, §4) │
             │     │            └──────────┬───────────┘
             │     │                       ▼
             │     │            ┌──────────────────────┐
             │     │            │ Inference loop       │
             │     │            │ + cumulative rep.pen │
             │     │            │ + RDF-star write-back│
             │     │            └──────────┬───────────┘
             │     │                       ▼
             │     │            ┌──────────────────────┐
             └─────┴────────────│ Generated triples +  │
                                │ propositionInferred  │
                                │ From edges, written  │
                                │ back to the store    │
                                └──────────────────────┘

The loop is closed: generated triples land in the store with propositionGenerated true. The next training-corpus extraction's SPARQL-star FILTER excludes them. The model never trains on its own output. Inference can be re-run repeatedly to grow the citation graph without polluting the training distribution.

4. Method

4.1 Corpus

Source: philippesaade/wikidata on Hugging Face — a CC0 parquet dump of ~30M Wikidata entities, each row a JSON-shaped record with labels (every language), descriptions, sitelinks, and claims. We stream via the datasets library, converting each entity to N-Triples-star form: one main triple per claim, plus one RDF-star annotation per qualifier and per reference, all sharing the same <<S P O>> quoted-triple subject. Wikidata's pq: (qualifier) and pr: (reference) namespaces collapse into the same wdt: predicate URI on the annotation row — the qualifier-vs-reference distinction is structural (subject is a quoted triple), not lexical.

Final ingested store: 5,055,385 triples / 1,695,402 RDF-star annotations / 27,780 entities / 770 MB on-disk Loka store. Every language label and description Wikidata has is included.

4.2 Label substitution

The model is trained on text, not URIs. The corpus extractor walks all rdfs:label "..."@en triples, builds a URI → English-label map, then writes each triple with each component resolved through the map:

Property labels missing from the live store are fetched from Wikidata's public SPARQL endpoint with caching and 429-tolerance. Two preprocessing fixes were essential and are fragile enough to surface here:

1. Strip ^^<datatype> suffixes from typed literals. Loka's SPARQL serialization embeds the datatype URI in the literal value string (e.g., "+1966-02-18T00:00:00Z\"^^<http://www.w3.org/2001/XMLSchema#dateTime>") rather than separating it as datatype metadata. Without stripping, datatype-URI fragments (xmlschema, decimal, org) reach the tokenizer as if they were entity content and dominate certain predictions (§5.1).

2. Drop rows with non-URI predicates. ~1% of rows on a 5M corpus exhibit a Loka SPARQL bug (§6.1) where literal values surface in the ?p slot. RDF disallows literal predicates, so dropping is safe.

After cleaning, the training file holds 757,592 lines for our 5M-triple corpus.

4.3 Model and training

Architecture: a role-aware Transformer encoder. Each triple is tokenized as

[CLS] s_tokens [SEP_S] p_tokens [SEP_P] o_tokens [SEP_O]

Token + position + role embeddings sum at each position, where the role is one of {SPECIAL, S, P, O}. The classification head is tied to the input embedding for parameter efficiency.

Training objective: pick one role (S, P, or O) at random per example, mask its tokens with [MASK], predict the originals. Cross-entropy on the masked positions, AdamW, 3e-4 LR, β=(0.9, 0.95), weight decay 0.01, gradient clipping at 1.0. Standard.

Three model sizes at this corpus size:

v3 reports artificially low perplexity from memorizing datatype-suffix tokens (§5.1). v4 is the canonical baseline at the smaller architecture. v5 is the bigger-model run.

4.4 Inference: generative citation

For each candidate subject in the corpus:

1. Candidate predicate selection. Find graph-neighbors — subjects sharing at least one (predicate, object-key) tuple with this one — and rank predicates they have but the candidate subject lacks. Cap at N candidates per subject (default 5).

2. Masked decoding with cumulative repetition penalty. Build the input as [CLS] s_tokens [SEP_S] p_tokens [SEP_P] [MASK]^k [SEP_O]. At each masked position, the model emits a logit distribution. We apply:

   • Hard skip-set: special tokens never win.
   • Cumulative repetition penalty: logit[t] /= penalty^count[t] where count[t] is the number of times t has already been emitted in this sequence. Default penalty = 3.0.
   • Per-token confidence floor: emission halts when the top-token probability falls below 0.05.

   Greedy top-1 selection, no beam search.

3. Confidence-thresholded emit. Mean per-token probability is the prediction's confidence. If confidence ≥ threshold (default 0.4) and the predicted object is not a duplicate of an existing fact for this (S, P), emit the RDF-star block (§3.2).

4. Optional --post. Write the emitted N-Triples-star to the live Loka store via POST /triples. Subsequent training-corpus extractions exclude these via the SPARQL-star FILTER from §3.1.

The cumulative penalty matters: a non-cumulative penalty (set membership) was tested first and failed to break loops on dominant common tokens because the penalty applied only once regardless of how many times the token had already won. With cumulative, three emissions of of at penalty 3.0 multiply its divisor by 27 and reliably drop it below the floor, breaking the cascade.

5. Experiments

5.1 Corpus quality regression: v3 → v4

A datatype-suffix-leakage bug (§4.2 fix #1) was caught only after the v3 model was trained. v3 produced predictions like Abbas Mirza | has works in collection | 1 http www w3 org 2001 xmlschema decimal (confidence 0.93) — clearly a memorization of literal-with-embedded-datatype-URI patterns. After fixing the corpus and retraining (v4), the same prediction becomes metropolitan museum of museum (confidence 0.43). The Met genuinely holds Abbas Mirza pieces.

The numerical effect is paradoxical at first read: v4's final perplexity (92.5) is higher than v3's (53.4). The explanation is mechanical — v3 was getting cheap loss reduction from memorizing fragments of typed-literal datatype URIs (xmlschema decimal http www w3 org) because they appeared frequently after a particular pattern. With those tokens stripped, the corpus is genuinely harder. Higher ppl, better content.

5.2 v4 vs v5: capacity scaling

Both trained 5 epochs on the cleaned 757k-line corpus. Side-by-side per-epoch perplexity:

v5 starts higher (epoch 1) — more parameters mean a harder optimization landscape and slower initial convergence. It crosses under v4 at epoch 2 and pulls ahead from there. By epoch 4 it has already passed v4's final perplexity. Wall time on a 4070 Laptop: 91 min for v5 vs 42 min for v4 (2.2× compute, 8% better final ppl).

5.3 Qualitative comparison (same seed, same penalty)

50 subjects sampled deterministically (seed 42), 5 candidate predicates each, confidence threshold 0.4, cumulative repetition penalty 3.0. Selected predictions:

v5 picks specific, correct entity tokens (halle, 33, kosmos 116) where v4 fell back to common connectors. The repetition penalty (same setting in both columns) eliminates the most egregious looping for both, but v5's distributions over real entity tokens are more concentrated, so its post-penalty outputs are more often direct hits.

5.4 Pass rate

At threshold 0.4, v4 emits 32/250 candidate predictions; v5 emits a comparable rate. The interesting metric is not pass rate but the quality of the passing predictions — and §5.3 carries the qualitative weight.

5.5 Catalog-noise discovery and corpus cleanup (v6 → v7)

A post-training behavioural test surfaced a corpus-level finding that reframes everything before it. We ran an auto-regressive proposition-generation protocol on the v6 model: pull a fresh BFS-depth-3 Wikidata neighborhood (183 entities, 14,586 triples, seeded at Q42), and for every source triple in the neighborhood generate up to 10 child triples whose context is the BFS-adjacent set after an asymmetric-cardinality filter, with parallel-subgraph extension. (Protocol details in planning/autoregressive-propgen-test.md.)

v6 emitted 52 triples on a 30-source run. The qualitative content is the finding:

• Confident catalog-format hallucinations. British Broadcasting Corporation | ISNI -> "00000000" (conf 0.754); Joan of Arc | Library of Congress authority ID -> "n 85 - 8" (LCCN-shaped, wrong content); Douglas Adams | Freebase ID -> "/ m / 0 c _ _ 9" (Freebase format /m/..., wrong content). The model has memorised the shape of these identifiers and emits format-shaped strings on prompt.
• Catalog format leaking onto unrelated predicates. instance of -> "+ Ġof - 00 - 03 T 00" appeared on 15 different subjects in the same 30-source run. This string is a Wikidata date-prefix shape (+YYYY-MM-DDTHH:MM:SS) being hallucinated for a predicate (P31 instance of) whose objects are entities, not dates. The model has so saturated on catalog/structured-literal patterns that it defaults to format strings on uncertain prompts.

The diagnosis is corpus composition. Wikidata defines an external-id datatype for properties whose objects are catalog cross-references (Freebase, ISNI, GND, LCCN, Dewey, etc.). A fresh SPARQL query against wikibase:propertyType wikibase:ExternalId returns 10,206 properties — roughly 80 % of all Wikidata property types. In the Q42 seed they are 49.6 % of triples by volume. On the v6 training corpus the share was similar: 75.7 % of the 757,592-triple file was external-id rows after re-filtering by predicate label. We had been training on a corpus that was three-quarters catalog cross-reference noise.

We rebuild the corpus as v7. The exclusion list is broadened beyond external IDs to include other Wikidata datatypes whose values have no transferable semantic content (url, commonsMedia, math, lexeme/sense/form, globe-coordinate, geo-shape, musical-notation, tabular-data, wikibase-entity-schema) — 10,525 properties total dropped, against a kept set of wikibase-item, wikibase-property, quantity, string, time, and monolingualtext (2,231 properties). Two other normalisations land at the same time:

1. Time and quantity literals. Wikidata serialises positive years as +YYYY and dates with a trailing T00:00:00Z regardless of precision. The leading + is a high-frequency BPE token implicated in the date-shape leak. v7 strips the + for positive years (BCE keeps the -), drops the T00:00:00Z suffix on date-only times, and drops the trailing Z. +2012-10-15T00:00:00Z → 2012-10-15; +1234 → 1234.
2. Monolingual text. v6 dropped non-English monolingualtext values; v7 keeps them in all languages with the @lang tag stripped. The model now sees Tokyo and 東京 as plain string values (the language information is lost — see §6.2).

The full per-datatype processing spec, with kept/dropped decisions and normalisation rules, is in planning/wikidata-datatype-processing.md and in training/wikidata_excluded_predicates.json.

We retrain v7 with the same 44.5 M-parameter BPE architecture as v6 for 5 epochs on the cleaned 184,458-triple corpus. Final perplexity 192.63 (v6 was 194.98, statistically tied); wall time 22 min on the same 4070 (v6 took 91 min on the larger noisy corpus). The same 30-source / Q42 seed test:

The catalog-format hallucinations are gone, not muted. The model's failure mode shifts from "confidently wrong with the right shape" to "refuses to emit", which is the correct direction for a generative-citation system. The price is volume: 14 emissions vs 52, because the model no longer confidently produces format-shaped strings. The loss curve says v7 is undertrained (5.36 → 5.26 still descending at epoch 5); we trained v8 on the same corpus for 20 epochs (§5.6).

5.6 v8: 20 epochs on the cleaned corpus

We trained v8 using the same 44.5 M-parameter BPE architecture as v6 and v7, but for 20 epochs from random initialisation on the v7 corpus. The 5 → 20 epoch increase produced a 3× perplexity reduction with no change to architecture or data:

Wall time 88 min on the same 4070 (vs v7's 22 min for 5 epochs — linear in epoch count). Loss was still descending at epoch 20 (4.20 → 4.19 → 4.17), so the v7 corpus is not yet saturated at this model size — strong evidence that the cleanup left signal the 5-epoch v7 had not yet exploited.

We apply the same 30-source / Q42 seed test as §5.5, extending the comparison to all three generations:

Selected v8 outputs (raw, BPE artifacts left visible — Ġ is the BPE space marker, âĢĵ is a mis-decoded em-dash):

Three patterns emerge. The model has discovered the high-frequency Wikipedia Commons-category template — "X ( ..." — and applies it confidently across many subjects; since Commons-category is among the most common predicates in the v7 corpus, this is frequency-appropriate behaviour. The different from outputs are circular: the model emits the subject as the object, a known pathology of the masked-S/P/O objective when the predicate is predominantly reflexive in the corpus (disambiguation pages couple each entity to itself). The failure is consistently wrong rather than arbitrarily wrong — a more tractable surface than v6's hallucinated catalog formats. Finally, the catalog-format leak that defined v6 remains entirely absent: instance of produces no date-prefix strings on any subject in the test.

The remaining failure modes — circular different from, BPE artifact leakage, residual catalog hallucination on the few external-id predicates the seed still includes — are all addressable downstream of the corpus cleanup: the first wants a structural change to the masked-prediction objective, the second is a tokenizer post-decode pass, and the third disappears once the inference layer also drops excluded predicates from its candidate pool. None of them argue for re-introducing catalog noise into training.

The wider implication of v8 is that the v7 corpus is small for the model. At ~600 k tokens after BPE on a 44.5 M-parameter model we are at 0.013 tokens per parameter, against a Chinchilla-optimal target of ~20. The v8 result — that 4× more epochs on the same corpus produces a 3× perplexity improvement — is consistent with a model that still has room to fit. The next step is therefore data scale, not more epochs: a fresh tools/wikidata_hf_import.py run targeting ~5 M useful triples after filtering, followed by v9 from scratch on that corpus. tools/training_cron.py (a 12-hour local cycle loop) automates the train-test-ship-retrain pipeline so this can run unattended.

5.7 v9: wedge fix exposed at 4M triples, model trained on a fresh slice

v9 carries two independent results: the /triples-wedge engine bug (§6.1) is fixed, and the model trained on a freshly-pulled Wikidata slice reaches perplexity 57.15, the best of any version, on a corpus that is in fact smaller than v7's.

The wedge. Paper §6.1 has documented since v3 that the engine wedges after roughly every 5–6× growth in stored triples — the POST /triples handler stalls indefinitely while /health keeps responding. The wedge fired again on the v9 ingest at the now-routine ~90 k-triple threshold. Root cause traced (via planning/triples-wedge-investigation.md and inspection of loka-core/src/persistent.rs and loka-proto/src/server.rs): the handler made 3–4 sled write-transactions per N-Triples line (three term-interns plus one SPO/POS/OSP triple-insert) and called PersistentStore::flush() synchronously at the end of every request. Under sustained ingest of ~100 k+ triples in one POST, sled accumulated write-ahead-log entries faster than its internal compactor could drain, and the writer thread eventually stalled. Fix in commit 39effbb: PersistentStore::insert_batch writes a whole HTTP request's worth of triples (and their term-interns) in a single sled multi-tree transaction. The synchronous flush() is gone — sled's periodic flush + Drop-time flush is sufficient durability. The handler in loka-proto/src/server.rs collects all triples into a Vec<BatchInsert> and calls insert_batch once. Verified: 2,000,049 triples in 4 003 s at 500 triples/sec sustained, no timeouts; cumulative 4M+ triples across the v9 cycle's two import phases, no wedge. Previous wedges at 90 k, 174 k, and ~1 M are all cleared by 20× or more.

The corpus. The v9 cycle pulled 2 090 640 raw triples into a fresh per-cycle Loka data dir from philippesaade/wikidata. Preprocessing through training/preprocess.py (same datatype filter as v7 + the v7 quantity/time normalisations) produced a 94 202-triple training file — smaller than v7's 184 458. The reason: philippesaade/wikidata is structured one row per entity, with all of an entity's claims in a single JSON blob. The v9 import consumed 9 647 rows for 2 M raw triples (217 triples/entity average), so most wikibase-item objects refer to entities whose own rows haven't been streamed yet, and the preprocess step drops a triple when it can't resolve the object's English label — 1 049 881 rows dropped that way. v7's corpus, built from a 50× larger raw-triple pool over the original 5 M-triple slice, had more closed label cycles per entity and a higher retention rate. This is a corpus-construction artifact, not a model issue; the fix (cross-cycle label caching, or streaming many more rows before preprocessing) is on the v10 roadmap.

Training and perplexity. 20 epochs, same 44.5 M-parameter BPE architecture, 44 min wall time on the 4070 (∝ to the 94 k vs 184 k corpus size relative to v8's 88 min).

v9 perplexity (57.15) beats v8 (64.65) by 12 % despite training on half the data. Two plausible explanations: the v9 corpus is preferentially the entities with closed reference graphs (i.e. structurally well-connected); and the v9 corpus is freshly drawn from a different slice of Wikidata, so the model isn't being asked to fit the same long-tail noise v8 had.

Q42 propgen test, all four versions.

97 % semantic-predicate share on v9 is the cleanest signal yet that the v7 datatype cleanup has been internalised. A residual catalog-format hallucination remains on Template:* | different from predicates, where the model emits short URL-prefix strings ("T ://", "M ://") instead of entity references — same shape-leak class as v6's date-shape leak, but in URL format and on a narrower predicate set. Not yet diagnosed; characterised in DEVLOG.md for v10 follow-up.

Implication. The wedge fix removes a long-standing infrastructure ceiling. The model now scales with data without engine-side limits in the way. v10 work focuses on the corpus side: stream enough rows from the HF dataset to close the entity-label reference graph (target ≥ 50 k entities, several hundred thousand resolved-label triples after preprocess), and apply the same propgen test to see whether semantic-content quality continues to improve.

5.8 v10: the first fully-automated cron cycle

v10 is the first model shipped end-to-end by tools/training_cron.py without any manual intervention. The 12-hour local cron loop ran HF import → preprocess → train → propgen test → DEVLOG entry → MODEL.json pin → HF push → commit + push → sleep, all autonomously. Trained 20 epochs on a 94 058-triple corpus extracted from a fresh 2 M-triple HF slice, the same shape as v9. Final perplexity 55.52 — a further 3 % improvement over v9, with the loss curve still descending at epoch 20 (4.06 → 4.02).

Q42 propgen test. 60 emissions at conf ≥ 0.25 (up from v9's 35), 0 catalog hallucinations, 100 % semantic-predicate share — the cleanest signal of the run.

Selected v10 outputs.

The Commons-category outputs are template-correct ("Category : Dramatists and play[wrights]" is the exact Wikimedia Commons naming convention). instance of -> "municipality of the" is a plausible-type semantic answer (would be correct for many entities in the Wikidata long tail). country -> "People's Republic" is right for PRC entities and right-type for non-PRC; spouse -> "1 ." is a numeric-format degeneration, a residual failure mode worth tracking but smaller than v9's Template:* | different from -> "T ://" URL-shape leak (now gone).

What changed in the infrastructure. v10 is the first model produced under the steady-state regime of:

1. The /triples wedge fix (§5.7), so a 2 M-triple HF import completes cleanly in ~67 min at 500 triples/s without retries.
2. The per-data-dir state file (commit 95f56f7), so each cycle's HF import resumes correctly when a cycle is restarted, and crash-recovery doesn't redo work.
3. PageRank-weighted source selection in the propgen test (commits e2809e3, 0734e40), so the evaluation focuses on structurally-important entities.
4. The auto-discovered MODEL_FILES list in tools/hf_snapshot.py (commit 4c996b9), so new checkpoints are uploaded without editing the script.
5. The dynamically-rendered HF README (commit 758b6ff), so the dataset page on Hugging Face refreshes its description on every push.

The cron's ship() step covers DEVLOG, MODEL.json, propgen test artifacts, HF push, and the local commit + push, but it deliberately does not edit the paper — paper revisions happen here (or via the remote schedule-skill cron jobs that polish paper prose for the AI peer reviewer). This is the v10 paper revision.

6. Limitations

6.1 Engine bugs surfaced at scale

• SPARQL ?s ?p ?o occasionally returns literal values in the predicate slot. RDF disallows literal predicates; this is invalid output from the executor — almost certainly an RDF-star annotation row with positions getting confused. Filtered at preprocess (drops ~1% of rows on a 5M corpus). Real engine bug.

• POST /triples wedges after roughly every 5–6× growth in stored triples. (Resolved during v9; see §5.7.) Originally hit at ~90 k, ~174 k, and ~1 M during the v6 ingest of the 5 M-triple slice — /health keeps responding, /triples and SPARQL hang indefinitely until the server is restarted, data intact on disk. Root cause: the handler ran 3–4 sled write-transactions per N-Triples line (three term-interns + one SPO/POS/OSP triple-insert) plus a synchronous flush() per request, accumulating WAL entries faster than sled's compactor could drain. Fix in commit 39effbb: PersistentStore::insert_batch writes the whole request in one sled multi-tree transaction; synchronous flush removed (sled's periodic + on-Drop flush is sufficient). Verified across two independent 2 M-triple imports (4 M cumulative) at 500 triples/s sustained, no recurrence.

6.2 Model and decoding

• Mode collapse on common connector tokens. Even with cumulative penalty 3.0, predictions for predicates the model has weak knowledge of fall back to of/and/https www. This reflected thin entity-content coverage at v5's corpus size (27,780 entities of the ~100 M available on Wikidata). v6–v10 trace the lever side: the corpus-cleanup work in v7 (§5.5) eliminated the catalog-format leak that drove the worst connector-token loops, and v10's 100 % semantic-predicate share on the Q42 propgen test (§5.8) is the strongest evidence to date that the binding constraint shifted from "model can't distinguish predicates" to "model needs more diverse entity-context coverage to leave its current mode-collapse residue (numeric placeholders on under-specified predicates like spouse -> "1 .")". The next lever remains more useful data, where "useful" now requires closed entity-label reference graphs (§5.7's discovery).
• Word-level tokenizer chops Unicode. "Saint-Léger" becomes saint l ger. BPE/wordpiece is the planned fix; unimplemented in this iteration.
• No beam search or top-p sampling. Greedy top-1 only. Some failure cases would resolve with beam-2.

6.3 Provenance

• Citation hallucination is structurally bounded but not zero. A propositionInferredFrom row points at a concrete context triple, which is auditable, but the choice of which context triples to cite is heuristic (§4.4 step 1). The model is not actually inspecting these specific triples during prediction; the citation is "the candidate-predicate selection considered these triples." We document this tradeoff as accepted: the schema is honest about what it represents.

• Position taken: the contribution is the schema, not the mechanistic link. A round-one AI peer reviewer (Gemini 3 Flash, post 2378 v1) flagged this exact heuristic-vs-learned gap as invalidating the "generative citation" framing. We disagree on the framing but accept the empirical reading: at v0, citations describe what context was selected for prediction, not what the model attended to during prediction. The contribution we claim is the data shape — predicted triples and their context appear in the same store, in RDF-star, with citation predicates that any consumer can audit and filter via SPARQL — not a learned retrieval head that proves each citation supports its target. §7 sketches the OWL-template and HNSW-decoder paths that would close this gap; until those land, the right framing for citations is "auditable post-hoc evidence pointer," which is still strictly better than the standard situation (no citation at all).

6.4 What we are not claiming, and why we do not report MRR / Hits@k

The dominant evaluation regime in transformer-on-KG completion (KG-BERT, KGT5, et al.) reports MRR and Hits@k against held-out triples on closed benchmarks like FB15k-237 or WN18RR. We do not report these numbers, and we want to be explicit about why — both so the gap is visible and so future work in the regime is well-scoped.

1. Prediction space, not entity space. Loka v0 emits labels, not entity IRIs. The model produces "university of halle" token-by-token, not <wd:Q156667>. MRR and Hits@k assume a finite candidate set of entities to rank; we have a vocabulary over English subword pieces (BPE in v6, word-level in v3–v5). The HNSW-as-decoder direction sketched in §7 would close this gap and is a precondition for a meaningful Hits@k number — until then, comparing to a benchmark that ranks entities is category-mistaken, not just unflattering.
2. Open-world Wikidata, not closed-world benchmarks. The 5M-triple slice has no held-out test set in the FB15k sense, and constructing one is non-trivial without leakage: Wikidata is open-world, the corpus is updated continuously, and the same predicate often has many correct values (a city has many instance of claims, all valid). The held-out set we would construct would be a soft top-k accuracy rather than a hard "correct/incorrect" split.
3. What we report instead. Perplexity (§5.2) is a per-token quantity that says how surprised the model is by the corpus on average; it is a substrate property, not a completion metric. The qualitative comparison (§5.3) is a 50-subject hand-audit; we acknowledge this is anecdotal and is presented as such. The right systematic evaluation, after the entity-decoder lands, is filtered Hits@k against a held-out wikidata snapshot constructed as the symmetric difference between two dump dates.

We treat MRR / Hits@k as blocked future work, gated on the entity-decoder, not as a comparison the paper sidesteps. The reproducibility supplement records the held-out construction we would run.

7. Discussion

The from-scratch training position (§2.3) coexists with a documented parallel near-term track admitting fine-tuning of a small base model (e.g., Qwen 2.5 1.5B-Instruct + QLoRA) under the same propositionInferredFrom output schema. The empirical case for the parallel track has softened across v6→v10: catalog-format hallucinations are gone, semantic-predicate share is at 100 % on the propgen test, and the cron loop ships new models without human intervention. The remaining failure mode in §6.2 (numeric-placeholder degenerations like spouse -> "1 ." and BPE-artifact leakage on Commons-category templates) might still be addressed faster by a fine-tuned 1B–3B parameter base model with English already encoded than by the from-scratch path waiting for corpus scale. We accept the provenance tradeoff this introduces — base-model pretraining is opaque — and record propositionGeneratedBy "qwen-2.5-1.5b-loka-v1" to track what was emitted by what.

Two larger questions are open:

Where does the OWL layer live? OWL ontologies are stored in the engine as triples but the engine does not reason. A reasonable role for OWL in the world-model loop is as a prediction template: an ontology declares "an instance of class C is expected to have properties P1, P2, P3 with values matching constraints X, Y, Z," and the inference loop reads the template, identifies expected-but-missing predicates for an entity, and predicts values for them. The OWL template becomes the prompt of a generative-citation inference call, and propositionInferredFrom cites the OWL declaration alongside the supporting context triples. We have not implemented this; it is the cleanest next step.

What is the right output decoder? The HNSW vector index in the engine is currently used for vector search (a separate feature) but could serve as a decoder: the model emits an embedding, HNSW resolves the nearest known IRI, and the IRI becomes the predicted object. This would close the gap between prediction in label-space and prediction in entity-space, eliminating cases like "metropolitan museum of museum" (decoded label) in favor of <wd:Q160236> (decoded entity). Open work.

References

• Loka. Loka / Loka — RDF-star triplestore with native HNSW vector indexing. GitHub release v0.4.0, 2026. https://github.com/EmmaLeonhart/Loka/releases/tag/v0.4.0. Apache-2.0.
• Wikidata Foundation. Wikidata. https://www.wikidata.org/. CC0.
• philippesaade. philippesaade/wikidata. Hugging Face dataset, snapshot 2024-09-18. https://huggingface.co/datasets/philippesaade/wikidata. CC0.
• W3C. RDF-star and SPARQL-star. https://w3c.github.io/rdf-star/cg-spec/.
• Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. (Masked-token-prediction substrate.)
• Vaswani, A., et al. Attention is All You Need. NeurIPS 2017. (Transformer architecture.)
• Bordes, A., et al. Translating Embeddings for Modeling Multi-relational Data. NeurIPS 2013. (TransE; comparison-only context for §2.2.)
• Yao, L., Mao, C., Luo, Y. KG-BERT: BERT for Knowledge Graph Completion. arXiv:1909.03193. (Transformer-on-KG comparison-only context.)
• Saxena, A., Kochsiek, A., Gemulla, R. Sequence-to-Sequence Knowledge Graph Completion and Question Answering. ACL 2022. (KGT5; comparison-only context.)