Dreaming

Dreaming is Dense-Mem's isolated process for proposing possible Relationships between existing semantic nodes. Its durable output is a Hypothesis.

A Hypothesis is not evidence, an Entity, a Value, a Relationship, a tier, or a SemanticEdge. Dreaming cannot create semantic nodes or predicate definitions.

Boundary

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart LR
  Active["Active validated claims<br/>and facts"] --> Cycle["Dream cycle"]
  Candidate["Eligible pending candidates"] --> Cycle
  Nodes["Existing Entity / Value IDs"] --> Cycle
  Cycle --> Hypothesis["Hypothesis only"]
  Hypothesis --> Review{"Client review"}
  Review -- ignore / reject --> Isolated["Keep or close in<br/>Hypothesis lifecycle"]
  Review -- new evidence --> Remember["Normal remember"]
  Remember --> Verify["Resolve and verify"]
  Verify --> Relationship["Relationship lifecycle"]

Likelihood, confidence, repetition, feedback, or source tier cannot promote a Hypothesis. Only new evidence entering remember can create or update a Relationship.

Eligible Inputs

Dreaming may read:

• compact active validated_claim and fact Relationships
• a candidate/pending_evidence Relationship only when it has a final schema-valid insufficient VerificationEvent
• existing resolved Entity and typed-Value endpoints referenced by those Relationships
• registered predicate definitions

Equivalent cross-profile Relationships are grouped so copied positions do not multiply dream weight. Dreaming may use only the preferred position of a resolved Conflict case, with all source Relationship IDs. An open or overdue cross-profile case is excluded rather than used as a guessed premise.

A pending candidate is eligible only when:

tier = candidate
status = pending_evidence
subject Entity already exists
object Entity or Value already exists
predicate is registered and valid for both endpoint kinds
at least one schema-valid insufficient VerificationEvent exists
no identity, predicate, scope, or temporal ambiguity is open

Context retained after an unavailable verifier or invalid provider response is not yet an eligible candidate. Dense-Mem keeps that context in the placement ledger, but dreaming cannot use it until normal verification produces a valid material insufficient outcome. This prevents provider failure text from becoming a speculation seed.

Dreaming must exclude quarantined, disputed, rejected, retracted, superseded, ambiguous, and review-held records. It must also exclude records whose only support comes from a non-current source revision, even while the replacement revision is still quarantined or awaiting placement.

Allowed Output

A dream may propose only a possible registered predicate between server-supplied existing endpoint IDs:

{
  "hypothesis_id": "hyp_01J...",
  "team_id": "team-1",
  "source_owner_profile_ids": ["profile-a", "profile-b"],
  "subject_entity_id": "ent_dense_mem",
  "predicate_key": "benefits_from",
  "object_entity_id": "ent_search_freshness",
  "object_value_id": null,
  "statement": "Dense-Mem may benefit from explicit search freshness.",
  "rationale": "Active recall and embedding-job Relationships suggest a useful connection.",
  "likelihood": 0.68,
  "confidence": 0.61,
  "source_relationship_ids": ["r_recall", "r_embedding_job"],
  "source_candidate_relationship_ids": ["r_candidate_freshness"],
  "source_versions": {
    "r_recall": 4,
    "r_embedding_job": 2,
    "r_candidate_freshness": 1
  },
  "generator_kind": "provider",
  "generator_version": "dream-v2",
  "status": "proposed",
  "created_at": "2026-07-11T00:00:00Z"
}

Exactly one object form is set. All endpoint, predicate, and source IDs come from immutable server allowlists and must belong to the same team.

The generator cannot output:

• a new Entity, Value, name, alias, or ID
• an unknown predicate or registry change
• a Relationship, support link, verification verdict, tier, or status change
• evidence text claiming that the Hypothesis proves itself
• a source ID outside the supplied input set

If the idea requires a concept that has no existing semantic node, the dream item is rejected. A later user may introduce that concept through real evidence in remember.

Hypothesis Lifecycle

submitted does not mean accepted or factual. The linked ingest_id and any later relationship_outcomes[] relationship ID expose the actual placement outcome.

Generation Flow

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart TD
  Start["Start team dream cycle"] --> Read["Read bounded eligible inputs<br/>in one PostgreSQL snapshot"]
  Read --> Groups["Build diverse connected groups"]
  Groups --> Generate["Generate structured edge-shaped ideas"]
  Generate --> Schema{"Closed schema valid?"}
  Schema -- No --> Reject["Reject generated item"]
  Schema -- Yes --> Allow{"All IDs allowed and<br/>nodes/predicate still exist?"}
  Allow -- No --> Reject
  Allow -- Yes --> Existing{"Exact active Relationship<br/>already exists?"}
  Existing -- Yes --> Reject
  Existing -- No --> Recheck["Recheck source versions"]
  Recheck --> Dedup["Deduplicate normalized identity<br/>and source set"]
  Dedup --> Persist["Persist Hypothesis and cycle result"]

The generator receives compact current rows, not full evidence or verifier rationales. Source Relationship IDs explain origin but are not confirming evidence. A source candidate remains visibly identified as a candidate.

An exact pending candidate may be the target of the Hypothesis. Dreaming still does not change that candidate; it only records why the possibility may be worth seeking new evidence for.

Scheduling And Concurrency

The scheduler:

1. finds due teams from database-backed effective configuration
2. acquires a unique team/time-window PostgreSQL lease
3. records a cycle run before generation
4. enforces input, output, token, duration, and concurrency budgets
5. records success, partial rejection, failure, or cancellation
6. never uses Redis as the only run record

%%{init: {"sequence": {"rightAngles": true}}}%%
sequenceDiagram
  participant S as Scheduler
  participant P as PostgreSQL
  participant D as Dream service
  participant A as Optional AI provider

  S->>P: Claim due team and window
  P-->>S: Durable cycle_run_id and eligible IDs
  S->>D: Build bounded structured context
  opt Provider-backed generation
    D->>A: Existing endpoints, predicates, and source IDs
    A-->>D: Structured Hypothesis proposals
  end
  D->>P: Validate IDs/versions and persist accepted Hypotheses

Redis may coordinate distributed rate limits or wake workers. PostgreSQL owns leases, cycle results, and Hypotheses.

Configuration names and defaults belong in Configuration.

Client Workflow

The feature uses:

• list_dreams
• get_dream
• resolve_dream_feedback

Exact public schemas belong in Technical Reference. Feedback such as reject or stale changes only Hypothesis lifecycle.

Submit new evidence

Confirming a Hypothesis requires evidence independent of the generated text:

{
  "hypothesis_id": "hyp_01J...",
  "decision": "confirm_true",
  "evidence": [
    {
      "content": "The user confirmed that Dense-Mem benefits from explicit search freshness.",
      "source_type": "manual",
      "authority": "authoritative",
      "source_group": "user-dream-review:hyp_01J..."
    }
  ],
  "proposal": {
    "entities": [
      {
        "ref": "dense_mem",
        "name": "Dense-Mem",
        "known_entity_id": "ent_dense_mem"
      },
      {
        "ref": "search_freshness",
        "name": "search freshness",
        "known_entity_id": "ent_search_freshness"
      }
    ],
    "relationships": [
      {
        "proposal_id": "dense_mem_benefits_from_search_freshness",
        "subject_ref": "dense_mem",
        "predicate": "benefits_from",
        "object_ref": "search_freshness",
        "polarity": "+",
        "evidence": [
          {
            "evidence_index": 0,
            "start": 0,
            "end": 73
          }
        ]
      }
    ]
  }
}

The server validates ID hints against the team and still runs the normal caller-owned remember verifier and lifecycle. The Hypothesis text is not copied into the evidence. A negative decision likewise needs correcting or explicitly negative evidence if it is meant to alter knowledge.

Recall And Staleness

recall_memory may return matching Hypotheses in related_hypotheses after primary Relationship ranking when dreaming is enabled for the team. This is a server-side policy decision, not a request flag. Hypotheses never enter the primary result list.

A Hypothesis becomes stale when:

• a source Relationship is retracted, superseded, disputed, or hidden
• a source candidate becomes rejected or its ambiguity reopens
• a source revision changes and regeneration does not reinforce the idea
• a retention rule expires it
• a client marks it stale

Staleness preserves the ledger record but excludes it from normal related Hypothesis retrieval.

Risks And Mitigations

Acceptance Invariants

1. dreaming creates no Entity, Value, predicate, Relationship, support, tier, status change, or SemanticEdge
2. every endpoint and source ID already exists in the same team
3. every proposed predicate is registered for the endpoint kinds
4. pending candidates are eligible only after schema-valid insufficient verification
5. primary recall never contains Hypotheses
6. confirmation without new evidence fails
7. submitted evidence enters the normal strict remember pipeline
8. source changes can stale but never delete Hypothesis history
9. cycle retries do not duplicate run or Hypothesis records
10. equivalent authored Relationships do not multiply dream weight
11. unresolved cross-profile conflict sets are not dream premises