Class: Kotoshu::Cache::ModelCache

Inherits:
BaseCache
  • Object
show all
Defined in:
lib/kotoshu/cache/model_cache.rb,
sig/kotoshu.rbs

Overview

Manages embedding model downloads from FastText CDN and GitHub.

Extends BaseCache to support FastText .vec files and ONNX models. Downloads FastText models from Facebook's public CDN.

Examples:

Downloading a FastText model

cache = ModelCache.new
vec_file = cache.get_fasttext_model('en')
model = FastTextModel.from_file(vec_file)

Downloading an ONNX model

onnx_file = cache.get_onnx_model('en')

Constant Summary collapse

AVAILABLE_MODELS =

Available models in FastText CDN and models-fasttext-onnx repository

{
  # FastText crawl vectors (300D) from Facebook Research
  # https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/
  # Selected high-resource languages
  fasttext: {
    de: { file: "cc.de.300.vec.gz", size: 1_000_000, source: "FastText Common Crawl" },
    en: { file: "cc.en.300.vec.gz", size: 2_000_000, source: "FastText Common Crawl" },
    es: { file: "cc.es.300.vec.gz", size: 1_000_000, source: "FastText Common Crawl" },
    fr: { file: "cc.fr.300.vec.gz", size: 1_000_000, source: "FastText Common Crawl" },
    pt: { file: "cc.pt.300.vec.gz", size: 1_000_000, source: "FastText Common Crawl" },
    ru: { file: "cc.ru.300.vec.gz", size: 1_000_000, source: "FastText Common Crawl" }
  },
  # ONNX models (active set) from models-fasttext-onnx repository.
  # Sizes synced with manifest.json in kotoshu/models-fasttext-onnx.
  # The repo holds .onnx for 158 languages but only the 9 below are
  # tracked and exposed — to promote a language, see
  # models-fasttext-onnx/.gitignore and re-sync this constant.
  # https://github.com/kotoshu/models-fasttext-onnx
  onnx: {
    de: { file: "fasttext.de.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
    en: { file: "fasttext.en.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
    es: { file: "fasttext.es.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
    fr: { file: "fasttext.fr.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
    pt: { file: "fasttext.pt.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
    ru: { file: "fasttext.ru.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
    zh: { file: "fasttext.zh.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
    ja: { file: "fasttext.ja.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
    ko: { file: "fasttext.ko.onnx", size: 120_000_415, source: "models-fasttext-onnx" },
  }
}.freeze
TIERS =

Model tiers published by kotoshu/models-fasttext-onnx (registry ids kotoshu://models/{lang}/{tier}). full is today's layout and behavior; fluency and mini are the tiered tiers resolved through the registry (TODO.impl/67 M1).

%i[full fluency mini].freeze
LFS_POINTER_PREFIX =

First line of a Git LFS pointer stub — what raw.githubusercontent.com serves for LFS-tracked files (*.onnx) instead of the object content. Used to detect pointer-stub downloads/caches by content: the legacy download path records the checksum of whatever it downloaded, so a pointer stub passes checksum verification forever.

"version https://git-lfs.github.com/spec/v1"
TIER_PREFERENCE =

Deterministic tier preference order, smallest footprint first. This is NOT a fallback chain — tiers are never silently substituted. It only fixes the ordering in which cached tiers are reported (e.g. ambiguity errors from tier: :any), so output is stable across machines and runs.

%i[mini fluency full].freeze
LID_LANGUAGE =

The language-identification artifact pair (plan 106): registry resource kotoshu://models/lid/lid-176 — the ONNX-converted lid.176 model (int8-per-row, dim 16) plus its vocab.json sidecar. Lives in the same registry as the embedding tiers but is NOT a language tier: it has no language of its own and the tier machinery (resource ids, TIERS) does not apply. It is stored under a top-level lid/ directory (not {lang}/models/onnx/{tier}) so #cached_resources keeps reporting per-language ids only.

"lid"
LID_TIER = 
"lid-176"
LID_REGISTRY_ID = 
ModelRegistry::Resource.id_for(LID_LANGUAGE, LID_TIER)
LID_DIRECTORY =

Directory (under the cache root) holding the LID artifact pair.

"lid"
TYPO_LANGUAGE =

The hybrid typo bi-encoder (plan 131): a language-less registry resource (kotoshu://models/typo/typo-biencoder) with a REQUIRED vocab sidecar, resolved exactly like the LID pair.

"typo"
TYPO_TIER = 
"typo-biencoder"
TYPO_REGISTRY_ID = 
ModelRegistry::Resource.id_for(TYPO_LANGUAGE, TYPO_TIER)
TYPO_DIRECTORY =

Cache directory (under the models root, beside the languages).

File.join("models", "typo")

Instance Attribute Summary

Attributes inherited from BaseCache

#cache_path, #cache_ttl, #github_url, #max_cache_size, #source_registry, #url_base

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from BaseCache

#available?, #cache_kind, #cached_data?, #clean, #clear, #clear_all, #download, #evict, #get, #initialize, #read_metadata, #reset_stats, #stats

Constructor Details

This class inherits a constructor from Kotoshu::Cache::BaseCache

Class Method Details

.normalize_tier(tier) ⇒ Symbol

Normalize and validate a tier name.

Parameters:

  • tier (String, Symbol) —

    "full", "fluency", or "mini"

Returns:

  • (Symbol) —

    the normalized tier symbol

Raises:

  • (ArgumentError) —

    for any other value (covers config model_tier / KOTOSHU_MODEL_TIER and caller typos)



107
108
109
110
111
112
113
114
115
 
# File 'lib/kotoshu/cache/model_cache.rb', line 107

def self.normalize_tier(tier)
  symbol = tier.to_s.downcase.strip.to_sym
  unless TIERS.include?(symbol)
    raise ArgumentError,
          "unknown model tier #{tier.inspect} (expected one of: #{TIERS.join(', ')})"
  end

  symbol
end

Instance Method Details

#all_available_models ⇒ Hash

List all available models across all languages.

Returns:

  • (Hash) —

    Mapping of language to available model types



623
624
625
 
# File 'lib/kotoshu/cache/model_cache.rb', line 623

def all_available_models
  AVAILABLE_MODELS
end

#available_models_for(language_code) ⇒ Array<Symbol>

Get available model types for a language.

Parameters:

  • language_code (String) —

    ISO 639-1 language code

Returns:

  • (Array<Symbol>) —

    Available model types (:fasttext, :onnx)



603
604
605
606
607
608
609
 
# File 'lib/kotoshu/cache/model_cache.rb', line 603

def available_models_for(language_code)
  lang = language_code.to_sym
  types = []
  types << :fasttext if AVAILABLE_MODELS[:fasttext][lang]
  types << :onnx if AVAILABLE_MODELS[:onnx][lang]
  types
end

#cached_resources ⇒ Array<String>

List all cached resources.

Tiered entries appear as "lang:onnx:tier"; the registry storage directory is not a model resource and is skipped.

Returns:

  • (Array<String>) —

    List of cached resource identifiers



647
648
649
650
651
652
653
654
655
656
657
658
 
# File 'lib/kotoshu/cache/model_cache.rb', line 647

def cached_resources
  Dir.glob(File.join(@cache_path, "**", "metadata.json")).filter_map do |path|
    relative = Pathname.new(path).relative_path_from(Pathname.new(@cache_path))
    parts = relative.to_s.split("/")
    next if parts.first == "registry" # registry storage, not a model

    case parts.length
    when 4 then "#{parts[0]}:#{parts[2]}" # lang/models/type/metadata.json (full)
    when 5 then "#{parts[0]}:#{parts[2]}:#{parts[3]}" # lang/models/type/tier/metadata.json
    end
  end.uniq
end

#cached_tiers(language_code) ⇒ Array<Symbol>

Model tiers currently cached for a language (disk-only lookup; never touches the network, so it is resolve-safe).

The legacy layout ({lang}/models/onnx/) counts as :full.

Parameters:

  • language_code (String) —

    ISO 639-1 language code

Returns:



192
193
194
195
196
197
198
199
 
# File 'lib/kotoshu/cache/model_cache.rb', line 192

def cached_tiers(language_code)
  lang = language_code.to_s
  onnx_dir = File.join(@cache_path, lang, "models", "onnx")
  TIER_PREFERENCE.select do |tier|
    dir = tier == :full ? onnx_dir : File.join(onnx_dir, tier.to_s)
    File.exist?(File.join(dir, "metadata.json"))
  end
end

#download_lid(force: false) ⇒ Hash

Download the LID artifact pair from the models registry (kotoshu://models/lid/lid-176), verifying the ONNX bytes against the registry sha256 before accepting them (the setup half of the two-stage model; same contract as #download_tiered_model).

Parameters:

  • force (Boolean) (defaults to: false) —

    re-fetch the registry first

Returns:

  • (Hash) —

    { onnx_path:, vocab_path:, metadata }

Raises:

  • (Kotoshu::Error) —

    no registry entry, or offline without a cached registry (the registry is fetched once per cache instance; see #registry)

  • (Kotoshu::IntegrityError) —

    sha256 mismatch (corrupt bytes are removed before raising)



356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
 
# File 'lib/kotoshu/cache/model_cache.rb', line 356

def download_lid(force: false)
  entry = registry(force: force)&.find(LID_LANGUAGE, LID_TIER)
  unless entry
    raise Kotoshu::Error, "no registry entry for #{LID_REGISTRY_ID}"
  end

  dir = File.join(@cache_path, LID_DIRECTORY)
  FileUtils.mkdir_p(dir)

  onnx_file = File.join(dir, filename_from_url(entry.urls.primary))
  used_url = download_primary_or_mirror!(entry, onnx_file)
  verify_registry_sha256!(entry, onnx_file, LID_REGISTRY_ID, used_url,
                          remediation: "Run Kotoshu.setup_lid(force: true) to re-download.")

  # The vocab sidecar is required, not optional: the reader
  # cannot score without its labels and Huffman counts.
  vocab_file = File.join(dir, filename_from_url(entry.vocab_url))
  begin
    download_file(entry.vocab_url, vocab_file)
  rescue StandardError => e
    File.delete(onnx_file) if File.exist?(onnx_file)
    raise Kotoshu::Error, "LID vocab unavailable at #{entry.vocab_url}: #{e.message}"
  end

  metadata = {
    version: entry.version,
    url: used_url,
    language: LID_LANGUAGE,
    type: "lid",
    tier: LID_TIER,
    file: File.basename(onnx_file),
    vocab_file: File.basename(vocab_file),
    checksum: entry.sha256,
    registry_id: LID_REGISTRY_ID,
    size_bytes: entry.size_bytes,
    cached_at: Time.now.utc.iso8601,
    source: "registry"
  }
  write_metadata(File.join(dir, "metadata.json"), metadata)

  { onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
end

#download_tiered_model(language_code, tier:, force_download: false) ⇒ Hash

Download and verify a tiered model from the registry.

Downloads urls.primary, falling back to urls.mirror, plus the vocab sibling; verifies the model's SHA-256 against the registry entry before accepting it into the cache. Unlike BaseCache#get, download failures and checksum mismatches RAISE rather than degrading to nil — setup callers rely on that to report :unavailable.

Parameters:

  • language_code (String) —

    ISO 639-1 language code

  • tier (String, Symbol) —

    "mini" or "fluency" (":full" is delegated to the legacy download path)

  • force_download (Boolean) (defaults to: false) —

    re-fetch the registry first

Returns:

  • (Hash) —

    { model_path:, metadata:, vocab_path? }

Raises:



257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
 
# File 'lib/kotoshu/cache/model_cache.rb', line 257

def download_tiered_model(language_code, tier:, force_download: false)
  tier = self.class.normalize_tier(tier)
  lang = language_code.to_s

  # The full tier rides the registry when the cached registry
  # knows it (primary URL, vocab sibling, sha256 — the path the
  # typo layer's rescore requires, since the legacy download
  # never fetched the vocab). Offline without a cached registry
  # the entry is nil and the legacy CDN path serves, preserving
  # the pre-registry behavior exactly.
  entry = registry_entry_for(lang, tier, force: force_download)
  if tier == :full && entry.nil?
    return get("#{lang}:onnx", force_download: force_download)
  end
  unless entry
    raise Kotoshu::Error,
          "no registry entry for #{ModelRegistry::Resource.id_for(lang, tier)}"
  end

  resource_id = tier_resource_id(lang, tier)
  dest_path = resource_dir_for(resource_id)
  FileUtils.mkdir_p(dest_path)

  filename = filename_from_url(entry.urls.primary)
  model_file = File.join(dest_path, filename)
  used_url = download_primary_or_mirror!(entry, model_file)

  verify_registry_sha256!(entry, model_file, resource_id, used_url)

  vocab_path = download_tiered_vocab(entry, dest_path)

  metadata = {
    version: entry.version,
    url: used_url,
    language: lang,
    type: "onnx",
    tier: tier.to_s,
    file: filename,
    checksum: entry.sha256,
    registry_id: ModelRegistry::Resource.id_for(lang, tier),
    size_bytes: entry.size_bytes,
    cached_at: Time.now.utc.iso8601,
    source: "registry"
  }
  metadata[:vocab_file] = File.basename(vocab_path) if vocab_path
  write_metadata(File.join(dest_path, "metadata.json"), metadata)

  result = { model_path: model_file, metadata: metadata }
  result[:vocab_path] = vocab_path if vocab_path
  result
end

#download_typo_biencoder(force: false) ⇒ Hash

Download the typo bi-encoder pair from the models registry, verifying the ONNX bytes against the registry sha256 (the setup half of the two-stage model). The vocab sidecar is required, not optional — the encoder cannot tokenize without its char vocabulary — so a registry entry without vocab_url (the pre-cut state) raises the honest error instead of caching a half-usable artifact.

Parameters:

  • force (Boolean) (defaults to: false) —

    re-fetch the registry first

Returns:

  • (Hash) —

    { onnx_path:, vocab_path:, metadata }

Raises:



453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
 
# File 'lib/kotoshu/cache/model_cache.rb', line 453

def download_typo_biencoder(force: false)
  entry = registry(force: force)&.find(TYPO_LANGUAGE, TYPO_TIER)
  unless entry
    raise Kotoshu::Error, "no registry entry for #{TYPO_REGISTRY_ID}"
  end
  if entry.vocab_url.nil? || entry.vocab_url.to_s.strip.empty?
    raise Kotoshu::Error,
          "#{TYPO_REGISTRY_ID} carries no vocab_url yet — the typo pair ships with a models registry release"
  end

  dir = File.join(@cache_path, TYPO_DIRECTORY)
  FileUtils.mkdir_p(dir)

  onnx_file = File.join(dir, filename_from_url(entry.urls.primary))
  used_url = download_primary_or_mirror!(entry, onnx_file)
  verify_registry_sha256!(entry, onnx_file, TYPO_REGISTRY_ID, used_url,
                          remediation: "Run Kotoshu::Typo.setup(force: true) to re-download.")

  vocab_file = File.join(dir, filename_from_url(entry.vocab_url))
  begin
    download_file(entry.vocab_url, vocab_file)
  rescue StandardError => e
    File.delete(onnx_file) if File.exist?(onnx_file)
    raise Kotoshu::Error, "typo vocab unavailable at #{entry.vocab_url}: #{e.message}"
  end

  metadata = {
    version: entry.version,
    url: used_url,
    language: TYPO_LANGUAGE,
    type: "typo-biencoder",
    tier: TYPO_TIER,
    file: File.basename(onnx_file),
    vocab_file: File.basename(vocab_file),
    checksum: entry.sha256,
    registry_id: TYPO_REGISTRY_ID,
    size_bytes: entry.size_bytes,
    cached_at: Time.now.utc.iso8601,
    source: "registry"
  }
  write_metadata(File.join(dir, "metadata.json"), metadata)

  { onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
end

#download_typo_matrix(language_code, force: false) ⇒ String

Download the matrix (the setup half). Raises like the other registry downloads when the registry cannot serve it yet.

Parameters:

  • language_code (String)
  • force (Boolean) (defaults to: false)
  • force: (bool force) (defaults to: false)

Returns:

  • (String) —

    path to the cached .ktm1



567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
 
# File 'lib/kotoshu/cache/model_cache.rb', line 567

def download_typo_matrix(language_code, force: false)
  lang = language_code.to_s
  entry = registry(force: force)&.find(lang, "typo-matrix")
  unless entry
    raise Kotoshu::Error,
          "no registry entry for kotoshu://models/#{lang}/typo-matrix (the matrix ships per language; derive-at-load remains the fallback)"
  end

  dir = File.join(@cache_path, lang, "models", "typo-matrix")
  FileUtils.mkdir_p(dir)
  file = File.join(dir, filename_from_url(entry.urls.mirror))
  used_url = download_primary_or_mirror!(entry, file)
  verify_registry_sha256!(entry, file, "kotoshu://models/#{lang}/typo-matrix", used_url,
                          remediation: "Run kotoshu setup #{lang} --typo --force to re-download.")

  write_metadata(File.join(dir, "metadata.json"),
                 "version" => entry.version,
                 "url" => used_url,
                 "language" => lang,
                 "type" => "typo-matrix",
                 "file" => File.basename(file),
                 "checksum" => entry.sha256,
                 "registry_id" => "kotoshu://models/#{lang}/typo-matrix",
                 "size_bytes" => entry.size_bytes,
                 # Plan 14/142: the pairing rides the cache so a
                 # later tier rebuild cannot arm a stale matrix.
                 "paired_tier_sha256" => entry.paired_vocab_sha256,
                 "cached_at" => Time.now.utc.iso8601,
                 "source" => "registry")
  file
end

#find_after_registry_refresh(language_code, tier) ⇒ Object

Plan 135: a cached registry within its TTL shadows the live one - during the v1.7.0 train, a v1.4.0 cache made a newly published resource invisible until manual deletion. When the wanted entry is absent from the cached copy, refresh the registry ONCE and look again. An offline refresh failure keeps the original miss: the legacy fallbacks (CDN for tiers, the honest no-entry error for the typo pair) apply exactly as before.



229
230
231
232
233
234
235
236
237
238
239
 
# File 'lib/kotoshu/cache/model_cache.rb', line 229

def find_after_registry_refresh(language_code, tier)
  refreshed = begin
    registry(force: true)
  rescue StandardError
    # Offline or transport failure: keep the original miss, the
    # existing fallbacks apply (Windows surfaces raw Errno
    # classes the wrapper does not always raise as Kotoshu::Error).
    nil
  end
  refreshed&.find(language_code.to_s, tier.to_s)
end

#get_fasttext_model(language_code, force_download: false) ⇒ String?

Get or download FastText model for a language.

Parameters:

  • language_code (String) —

    ISO 639-1 language code

  • force_download (Boolean) (defaults to: false) —

    Force re-download

Returns:

  • (String, nil) —

    Path to downloaded .vec file



122
123
124
125
126
127
 
# File 'lib/kotoshu/cache/model_cache.rb', line 122

def get_fasttext_model(language_code, force_download: false)
  resource_id = "#{language_code}:fasttext"
  result = get(resource_id, force_download: force_download)

  result&.dig(:model_path)
end

#get_onnx_model(language_code, force_download: false) ⇒ String?

Get or download ONNX model for a language.

Parameters:

  • language_code (String) —

    ISO 639-1 language code

  • force_download (Boolean) (defaults to: false) —

    Force re-download

Returns:

  • (String, nil) —

    Path to downloaded .onnx file



134
135
136
137
138
139
 
# File 'lib/kotoshu/cache/model_cache.rb', line 134

def get_onnx_model(language_code, force_download: false)
  resource_id = "#{language_code}:onnx"
  result = get(resource_id, force_download: force_download)

  result&.dig(:model_path)
end

#lid_cached? ⇒ Boolean

Whether the LID artifact pair is cached and complete.

Returns:

  • (Boolean) 


314
315
316
 
# File 'lib/kotoshu/cache/model_cache.rb', line 314

def lid_cached?
  !load_cached_lid.nil?
end

#load_cached(resource_id) ⇒ Hash?

Load cached resource data (implements abstract method).

Public — this is the cache-only reader BaseCache declares publicly; resolve paths use it to read without any download.

Verifies the cached model file's SHA-256 against the checksum recorded at download time. A mismatch — truncated file, disk corruption, manual edit — raises IntegrityError with a remediation hint pointing at the cache subcommand, matching the Phase C contract of TODO.impl/38-onnx-semantic-gating.md. Caches written before checksums were recorded are accepted as "unverified" (graceful degradation).

Parameters:

  • resource_id (String) —

    The resource identifier

Returns:

  • (Hash, nil) —

    Loaded model info

Raises:



677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
 
# File 'lib/kotoshu/cache/model_cache.rb', line 677

def load_cached(resource_id)
  return load_tiered_cached(resource_id) if tiered_resource_id?(resource_id)

  language = extract_language(resource_id)
  type = extract_type(resource_id)
  return nil unless language && type

  model_info = AVAILABLE_MODELS[type.to_sym][language.to_sym]
  return nil unless model_info

  metadata_path = metadata_path_for(resource_id)
  return nil unless File.exist?(metadata_path)

  metadata = read_metadata(metadata_path)
  return nil unless metadata

  # For .gz files, the decompressed version is stored without .gz extension
  filename = model_info[:file].sub('.gz', '')
  model_file = File.join(resource_dir_for(resource_id), filename)

  return nil unless File.exist?(model_file)

  verify_cached_integrity!(resource_id, metadata, model_file)

  { model_path: model_file, metadata: metadata }
end

#load_cached_lid ⇒ Hash?

Resolve the LID artifact pair from the cache only — never touches the network (the resolve half of the two-stage model).

Returns:

  • (Hash, nil) —

    { onnx_path:, vocab_path:, metadata } when cached, nil when not set up

Raises:



325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
 
# File 'lib/kotoshu/cache/model_cache.rb', line 325

def load_cached_lid
  dir = File.join(@cache_path, LID_DIRECTORY)
  metadata_path = File.join(dir, "metadata.json")
  return nil unless File.exist?(metadata_path)

  metadata = read_metadata(metadata_path)
  return nil unless metadata

  onnx_file = File.join(dir, metadata["file"].to_s)
  vocab_file = File.join(dir, metadata["vocab_file"].to_s)
  return nil unless File.exist?(onnx_file) && File.size(onnx_file).positive?
  return nil unless File.exist?(vocab_file) && File.size(vocab_file).positive?

  verify_model_pair_integrity!(metadata, onnx_file, vocab_file)

  { onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
end

#load_cached_tier(language_code, tier) ⇒ Hash?

Resolve a tiered model from the cache only — the cache half of the two-stage model for tiers, symmetric with #load_cached_lid / #load_cached_typo_biencoder. Returns the vocab sibling when the registry-served download landed it.

Parameters:

  • language_code (String)
  • tier (String, Symbol)

Returns:

  • (Hash, nil) —

    { model_path:, vocab_path?, metadata } or nil



506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
 
# File 'lib/kotoshu/cache/model_cache.rb', line 506

def load_cached_tier(language_code, tier)
  tier = self.class.normalize_tier(tier)
  resource_id = tier_resource_id(language_code.to_s, tier)
  dir = resource_dir_for(resource_id)
  metadata_path = File.join(dir, "metadata.json")
  return nil unless File.exist?(metadata_path)

  metadata = read_metadata(metadata_path)
  return nil unless metadata

  model_file = File.join(dir, metadata["file"].to_s)
  return nil unless File.exist?(model_file) && File.size(model_file).positive?

  result = { model_path: model_file, metadata: metadata }
  vocab = metadata["vocab_file"].to_s
  vocab_path = File.join(dir, vocab)
  result[:vocab_path] = vocab_path if !vocab.empty? && File.exist?(vocab_path)
  result
end

#load_cached_typo_biencoder ⇒ Hash?

Resolve the typo bi-encoder pair from the cache only — never touches the network (the suggest path resolves; setup downloads, the same two-stage split as the LID pair).

Returns:

  • (Hash, nil) —

    { onnx_path:, vocab_path:, metadata } when cached, nil when not set up

Raises:



417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
 
# File 'lib/kotoshu/cache/model_cache.rb', line 417

def load_cached_typo_biencoder
  dir = File.join(@cache_path, TYPO_DIRECTORY)
  metadata_path = File.join(dir, "metadata.json")
  return nil unless File.exist?(metadata_path)

  metadata = read_metadata(metadata_path)
  return nil unless metadata

  onnx_file = File.join(dir, metadata["file"].to_s)
  vocab_file = File.join(dir, metadata["vocab_file"].to_s)
  return nil unless File.exist?(onnx_file) && File.size(onnx_file).positive?
  return nil unless File.exist?(vocab_file) && File.size(vocab_file).positive?

  verify_model_pair_integrity!(
    metadata, onnx_file, vocab_file,
    directory: dir,
    registry_id: TYPO_REGISTRY_ID,
    remediation: "Run Kotoshu.setup_typo(language, force: true) to re-download."
  )

  { onnx_path: onnx_file, vocab_path: vocab_file, metadata: metadata }
end

#load_cached_typo_matrix(language_code, paired_tier_sha256: nil) ⇒ String?

The prebuilt KTM1 matrix (plans 136/142): cache-only and sha-verified; paired_tier_sha256 rejects a matrix cached against a different full tier (nil = no record, pre-plan-14).

Parameters:

  • language_code (String)
  • paired_tier_sha256: (String? paired_tier_sha256) (defaults to: nil)

Returns:

  • (String, nil) 


533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
 
# File 'lib/kotoshu/cache/model_cache.rb', line 533

def load_cached_typo_matrix(language_code, paired_tier_sha256: nil)
  entry = begin
    registry&.find(language_code.to_s, "typo-matrix")
  rescue StandardError
    nil
  end
  return nil unless entry

  dir = File.join(@cache_path, language_code.to_s, "models", "typo-matrix")
  file = File.join(dir, filename_from_url(entry.urls.mirror))
  return nil unless File.exist?(file) && File.size(file).positive?
  return nil unless Digest::SHA256.file(file).hexdigest == entry.sha256

  # Plan 142: the rows are index-parallel to exactly the tier
  # vocab they were derived over. When the caller states the tier
  # it will pair with, a cached matrix recorded against a
  # different tier is stale - answer nil so the caller derives
  # instead of arming wrong slates. Matrices cached before the
  # pairing was recorded carry no field and stay usable.
  if paired_tier_sha256
    metadata = read_metadata(File.join(dir, "metadata.json"))
    recorded = metadata && metadata["paired_tier_sha256"]
    return nil if recorded && recorded != paired_tier_sha256
  end

  file
end

#model_info(language_code, model_type) ⇒ Hash?

Get model info for a language and type.

Parameters:

  • language_code (String) —

    ISO 639-1 language code

  • model_type (Symbol) —

    Model type (:fasttext, :onnx)

Returns:

  • (Hash, nil) —

    Model info or nil if not available



616
617
618
 
# File 'lib/kotoshu/cache/model_cache.rb', line 616

def model_info(language_code, model_type)
  AVAILABLE_MODELS.dig(model_type, language_code.to_sym)
end

#registry_entry_for(language_code, tier, force: false) ⇒ ModelRegistry::Resource?

Look up the registry entry for a (language, tier) model.

Reads the cached registry.json; downloads it once (per cache instance) when absent or stale — unless offline mode is active, in which case a stale cached copy is reused and a missing one raises instead of ever hitting the network.

Parameters:

  • language_code (String) —

    ISO 639-1 language code

  • tier (String, Symbol) —

    one of TIERS

  • force (Boolean) (defaults to: false) —

    re-fetch the registry first

Returns:

Raises:



214
215
216
217
218
219
 
# File 'lib/kotoshu/cache/model_cache.rb', line 214

def registry_entry_for(language_code, tier, force: false)
  return registry(force: true)&.find(language_code.to_s, tier.to_s) if force

  registry&.find(language_code.to_s, tier.to_s) ||
    find_after_registry_refresh(language_code, tier)
end

#supports_resource?(resource_id) ⇒ Boolean

Check if a resource type is supported.

Parameters:

  • resource_id (String) —

    The resource identifier (e.g., "en:fasttext")

Returns:

  • (Boolean) —

    True if supported



631
632
633
634
635
636
637
638
639
 
# File 'lib/kotoshu/cache/model_cache.rb', line 631

def supports_resource?(resource_id)
  return true if tiered_resource_id?(resource_id)

  parts = resource_id.split(":")
  return false unless parts.size == 2

  language, type = parts
  AVAILABLE_MODELS[type.to_sym]&.key?(language.to_sym)
end

#tier_from_resource_id(resource_id) ⇒ Symbol?

Tier of a tiered resource id ("en:onnx:mini" -> :mini). Returns nil for legacy two-part ids.

Parameters:

  • resource_id (String) —

    resource identifier

Returns:

  • (Symbol, nil) 


179
180
181
182
183
 
# File 'lib/kotoshu/cache/model_cache.rb', line 179

def tier_from_resource_id(resource_id)
  return nil unless tiered_resource_id?(resource_id)

  resource_id.split(":")[2].to_sym
end

#tier_resource_id(language_code, tier) ⇒ String

Resource identifier for a (language, tier) model.

The full tier maps to today's two-part id ("lang:onnx") and therefore to today's on-disk layout — backwards compatible. fluency/mini get a third path segment so tiers coexist:

models/{lang}/models/onnx/                # full (unchanged)
models/{lang}/models/onnx/mini/           # mini
models/{lang}/models/onnx/fluency/        # fluency

Parameters:

  • language_code (String) —

    ISO 639-1 language code

  • tier (String, Symbol) —

    one of TIERS

Returns:

  • (String) —

    resource id ("en:onnx" or "en:onnx:mini")



156
157
158
159
160
 
# File 'lib/kotoshu/cache/model_cache.rb', line 156

def tier_resource_id(language_code, tier)
  tier = self.class.normalize_tier(tier)
  lang = language_code.to_s
  tier == :full ? "#{lang}:onnx" : "#{lang}:onnx:#{tier}"
end

#tiered_resource_id?(resource_id) ⇒ Boolean

Whether a resource id carries a tier segment ("lang:onnx:tier").

Parameters:

  • resource_id (String) —

    resource identifier

Returns:

  • (Boolean) —

    true for well-formed tiered ids



166
167
168
169
170
171
172
 
# File 'lib/kotoshu/cache/model_cache.rb', line 166

def tiered_resource_id?(resource_id)
  parts = resource_id.to_s.split(":")
  return false unless parts.size == 3

  _lang, type, tier = parts
  type == "onnx" && TIERS.include?(tier.to_sym)
end

#typo_biencoder_cached? ⇒ Boolean

Whether the typo bi-encoder pair is cached and complete (both files, checksum-verified).

Returns:

  • (Boolean) 


405
406
407
 
# File 'lib/kotoshu/cache/model_cache.rb', line 405

def typo_biencoder_cached?
  !load_cached_typo_biencoder.nil?
end