# Reference Encode Service

`ReferenceEncodeService` owns reusable mechanics for ad-hoc TTS reference
encoding:

- cache-keyed lookup;
- byte-bounded LRU storage;
- same-key single-flight while an encode is in progress;
- failure propagation to waiters without caching failures;
- artifact store/load conversion and caller-owned return values;
- basic cache statistics.

The service is for ad-hoc request references. Registered or uploaded voices
continue to use `SpeakerArtifactCache`, because they have a different lifetime,
key space, and invalidation path.

## API Shape

The implementation lives in `sglang_omni/scheduling/reference_encoder.py`.

```python
@dataclass(frozen=True)
class ReferenceEncodeKey:
    model_id: str
    model_revision: str
    encoder_id: str
    encoder_config_hash: str
    artifact_kind: str
    input_key: str
    options_key: str = ""


class ReferenceEncodeHook(Generic[InputT, ArtifactT, StoredT]):
    def normalize_input(self, raw_input: Any) -> InputT: ...
    def cache_key(self, item: InputT) -> ReferenceEncodeKey | None: ...
    def encode_one(self, item: InputT) -> ArtifactT: ...
    def store_artifact(self, artifact: ArtifactT) -> StoredT: ...
    def load_artifact(self, stored: StoredT) -> ArtifactT: ...
    def revalidate(self, item: InputT, key: ReferenceEncodeKey) -> bool: ...


class ReferenceEncodeService(Generic[InputT, ArtifactT, StoredT]):
    def get_or_encode(self, raw_input: Any, *, desc: str | None = None) -> ArtifactT: ...
    def stats(self) -> dict[str, int]: ...
```

`ReferenceEncodeService` is synchronous and thread-first. Existing TTS
preprocessing and encoder stages already run synchronous model code inside
`SimpleScheduler` or `ThreadedSimpleScheduler`, so adding an async surface would
force nested event-loop management without changing the underlying work.

## Responsibility Split

The service owns mechanics:

- `_inflight` single-flight map;
- `StageOutputCache` access under a service-owned lock;
- cache insertion, byte budget, and LRU eviction;
- follower waits, timeout handling, and exception fanout;
- no-poison-on-failure behavior;
- stats for hits, misses, merges, failures, uncacheable inputs, entries, bytes,
  and evictions.

The hook owns model semantics:

- request-specific input normalization;
- cacheability;
- model/checkpoint/config key parts;
- `encode_one`;
- artifact device and dtype policy;
- store/load conversion;
- revalidation for mutable local files.

## Cache-Key Contract

`ReferenceEncodeKey` must include every input that can change the encoded
artifact identity:

- model family or checkpoint identity;
- model or encoder revision;
- encoder implementation and config hash;
- artifact kind;
- normalized reference content identity;
- encode options that affect the artifact.

Local reference files should use
`reference_path_cache_key(path, trust_stat=False)` and revalidate before cache
insert. Bytes and data-URI payloads should key by the bytes or original payload
actually consumed by the model hook. Remote URLs should not be cached by URL
string alone unless an upstream fetch layer has already materialized immutable
content identity.

## Artifact Policy

Hooks should store cache-owned artifacts, usually detached CPU tensors or a
small CPU dictionary of detached tensors. `load_artifact` must return a
caller-owned object, commonly by cloning and moving to the expected dtype or
device. The service enforces the byte budget on the stored representation.

If a stored artifact is larger than `max_bytes`, the leader request and all
same-key followers still receive a result, but the artifact is not inserted into
the LRU.

## Failure And Waiters

For a cacheable key:

1. Cache hits return `hook.load_artifact(stored)`.
2. If another request is already encoding the same key, followers wait on the
   leader future.
3. The leader encodes once, stores the artifact representation, optionally
   inserts it into the LRU, resolves waiters, and removes the in-flight entry.

Leader failures are propagated to waiters and are not cached. The next request
can retry as a new leader. A follower timeout does not remove the leader's
in-flight entry.

## M4a And M4b Boundary

This document covers **M4a only**: the ad-hoc reference cache and same-key
single-flight that ships today.

**M4b (different-key batch coalescing) is not implemented and is a non-goal
here**; it is described only to mark the scope boundary. Do not add M4b runtime
code until profiling proves it is worth the extra scheduling surface.

Before building M4b, run cold-cache workloads with different reference audio per
request at concurrency 8 and 16 for FishAudio S2-Pro, Qwen3-TTS, and
MOSS-TTS Local. Track preprocessing/reference-encode p50/p95, end-to-end TTFA
and latency, throughput, cache hit/miss/merge counts, and GPU/CPU utilization.
Build M4b for a model only if different-key reference encode remains a top
bottleneck and batching gives at least 15% p95 latency reduction or 20%
throughput improvement versus M4a.

If M4b is built later, it should be an opt-in extension of
`ReferenceEncodeService`, not the default path:

- add explicit hook capability, for example `can_encode_batch()` defaulting to
  `False`, and call `encode_batch(items)` only for hooks that opt in;
- add service knobs such as `max_batch_size=1` and `max_batch_wait_ms=0`;
- preserve M4a ordering: normalize input, compute cache key, check cache,
  merge same-key inflight work, and only then enqueue distinct cache-miss
  leaders for different-key batching;
- use an internal queue that drains up to `max_batch_size` or
  `max_batch_wait_ms`, calls one hook batch encode, and then stores,
  revalidates, cache-inserts, and resolves each item independently;
- on a batch failure, retry per item so one bad reference does not fail the
  whole batch.

Model rollout should stay evidence-driven. MOSS-TTS Local already has its own
batched reference encoder and should not be migrated just to fit this generic
surface. FishAudio S2-Pro is the first plausible candidate only if profiling
shows real benefit; its batched path would decode/resample each reference, pad
waveforms, call the codec once, and split outputs while preserving parity with
`encode_one`. Qwen3-TTS should remain M4a-only unless the upstream wrapper
exposes a safe batch primitive for `create_voice_clone_prompt`.