On-device text-to-image and "More like this" search (#939, #66): working implementation on a fork

[jasontitus]

I know the README says code PRs aren't accepted, so I'm posting this here instead of opening one. It's a working implementation of the text-to-image and similar-image search requested in #939 and #66, available as a branch on my fork in case any of it is useful:

• Branch: jasontitus/aves@feature/smart-search, 2 commits on top of current develop, merges cleanly
• Diff: compare develop...jasontitus:feature/smart-search

In #66 you said shipping a model as part of Aves is out. Here, no model is in the APK: models are downloaded on demand, only after the user enables the feature. The ONNX Runtime library, though, is bundled (+23–33 MB), which is why open question 1 below suggests restricting it to some flavors or a separate APK.

Tested on real devices, 64-bit and 32-bit:

• Amazon Fire Max 11 (4 GB, Fire OS 8): the 64-bit build with both models, and the 32-bit (armeabi-v7a) build with the Standard model. On 32-bit, download, full indexing, text search and "More like this" all work, about 4–5× slower than on 64-bit.
• Samsung Galaxy A35 5G (6 GB, Android 16): both models, dark theme.
• 2 GB RAM emulator (Android 11): Standard only, memory-constrained configuration.

I'd be glad to reshape any of this (split it, move it behind a flavor, drop parts) if you're interested, and equally fine if it doesn't fit your plans.

---

Adds optional on-device semantic search to Aves:

• Search by description. Type "dog on a beach" or "birthday cake" to get a ranked page of matching photos and videos.
• More like this. A new viewer action lists items that look similar to the current one.

Everything runs on the device. The only network traffic is a one-time model download, which the user explicitly starts. Photos, videos and queries are never sent anywhere.

The feature is off by default. It is designed for low- and mid-range devices with 2–4 GB of RAM; Amazon Fire tablets were the reference target.

How it works

Models	CLIP-style dual encoders: an image encoder and a text encoder that map into the same embedding space
Runtime	ONNX Runtime Android 1.30.0 (MIT, Maven Central), running int8-quantized models in ORT format. Weights are memory-mapped; the arena and memory patterns are off; weight prepacking is off on low-RAM devices
Indexing	Separate WorkManager work, not part of the existing analysis work. When 20 items or fewer are pending (new photos, for example), they are indexed inline with 1 thread. It resumes after interruption, pauses on low battery, high temperature or when not charging (optional), and can be scheduled to run while charging
Storage	Append-only int8 vector store (~532 bytes per item for the Standard model) with CRC-checked records, tombstones and compaction. Records are keyed by entry ID plus a content fingerprint, so an edited, rotated or replaced file never returns a stale vector
Search	Brute-force dot product over candidates, restricted on the Dart side to entries that are visible (hidden albums, vaults and the bin are never searched). Results are kept when their score stands out from the candidate distribution (z-score); short queries use prompt ensembling
UI	A suggestion tile on the search page. Enter runs a smart search, except for title syntax (/regex/, "exact", -neg, YEAR=…) or a file-name-like query that matches item titles (IMG_2041); filtering by title stays available from the query suggestion chip. Also: recent smart searches, a ranked results page (fixed order, no date headers), the viewer "More like this" action (also available as a quick action), and a settings section (enable, model, Wi-Fi only, charging only, index status, delete)

Models, licenses and download

Tier	Model	License	Download	Embedding	Availability
Standard (default)	OpenCLIP ViT-B/32 laion2b_s34b_b79k	MIT	158 MB (image 96 MB, text 68 MB, BPE vocab 1.4 MB)	512-d	All devices
Best quality	Meta Perception Encoder PE-Core-B16-224	Apache-2.0	448 MB (image 99 MB, text 370 MB)	1024-d	64-bit, ≥ 3.5 GB RAM (hidden otherwise)

• Hosting. The files are hosted at huggingface.co/sliderforthewin/aves-smart-search. The app pins a specific commit (ebc8231d…) and embeds each file's SHA-256 and size. Files are verified while being copied from the DownloadManager destination into private no_backup storage, and are only used once verified. The model card lists the sources, the modifications and the licenses.
• Reproducibility. scripts/smart_search/convert.py reproduces the files: OpenCLIP → ONNX (opset 18) → dynamic int8 per-channel → ORT. With the listed versions, the ViT-B/32 conversion is byte-identical across runs.
• Network conditions. Downloads go through DownloadManager, over Wi-Fi only by default (configurable), and continue in the background. Everything is removed by "Delete model and index".
• Attribution. The licenses page lists ONNX Runtime, OpenCLIP, Perception Encoder and the OpenAI CLIP BPE vocabulary. The privacy policy (terms.md/terms.txt) and the README permissions table mention the model download.

Quality

Flickr30k 1k test split, text → image retrieval over 5,000 captions. Images are first downscaled to thumbnails, as the app does.

Model	fp32 R@1	shipped int8 R@1	R@5	on-device R@1 (Fire Max 11)
ViT-B/32 (Standard)	0.662	0.662	0.879	0.653
PE-Core-B16-224 (Best quality)	0.803	0.797	0.951	0.797

Naive int8 quantization of the ViT-B/32 text encoder gives R@1 0.41. One projection layer with activation outliers is kept in fp32.

APK size (libre flavor, release, split per ABI)

ABI	develop	this branch	Δ
arm64-v8a	53.4 MB	86.8 MB	+33.4 MB
armeabi-v7a	51.6 MB	75.2 MB	+23.6 MB

Almost all of the growth is the ONNX Runtime native library. Models are never bundled. See open question 1.

Performance

Amazon Fire Max 11 (MediaTek MT8188, 4 GB), release or profile build:

	Standard	Best quality
Download + verify + install (Wi-Fi)	72 s	–
First index, 1,318 photos	~2¼ min	~8½ min
Image embedding (bench, A55 cores only, 4 threads / 2 threads / 2 threads with prepacking off)	139 / 241 / 354 ms	–
Image embedding, 32-bit app (armeabi-v7a, A55, 4 / 2 threads)	426 / 609 ms	n/a
Text query encode (bench, 1 phrasing, 4 threads)	106 ms	–
In-app text search, first query (encode + search, 1,315 items)	733 + 36 ms	2,323 + 31 ms
In-app text search, later queries	350–371 + 5–47 ms	322–577 + 4–31 ms
Switch Standard → Best quality (448 MB download, install, activate)	–	76 s
App PSS: idle → indexing → after indexing	405 → ~610 → 545 MB	–

The PSS "after" figure includes GPU texture memory for thumbnail grids, which grows the same way when scrolling the collection without smart search. The encoders themselves are released after idle; no model file stays mapped.

Amazon Fire Max 11, 32-bit (armeabi-v7a) app, profile build, Standard model (Best quality needs a 64-bit device). The 32-bit ONNX Runtime is much slower than on 64-bit, but it works end to end:

	Standard, 32-bit	same device, 64-bit
Download + verify + install	33 s	72 s (earlier run, different network conditions)
First index, 1,318 photos	~11 min (~2 items/s), PSS peak 544 MB	~2¼ min
Text search, all queries (encode + search)	1,238–1,582 + 4–15 ms	350–733 + 5–47 ms
Cold app start	1.2 s	1.4 s
"More like this"	Works	Works
Fresh start → after browsing 7 results pages and a 12 MP photo	406 → 977 MB, of which 680 MB is GPU textures for thumbnails; models released (no mapped model file)	similar pattern

Samsung Galaxy A35 5G (Exynos 1380, 6 GB, Android 16), profile build, 108 items (71 videos, including 14 empty files):

	Standard	Best quality
Download + verify + install	11 s	18 s (including the switch)
First index	~30 s	~35 s, PSS peak 649 MB
Text search, first query	1,299 + 12 ms	1,965 + 16 ms
Text search, later queries	228–345 + 2–7 ms	187–248 + 2–3 ms

2 GB RAM emulator (arm64, Android 11, profile build):

	Standard
Model tiers offered	Standard only; Best quality hidden
First index, 1,301 items	~65 s (host CPU, not representative of device speed)
App PSS: indexing peak → idle after encoder release	439 MB → 268 MB
Text search: first query (cold)	842 ms encode + 37 ms search
Text search: warm queries (4 phrasings)	344–580 ms encode + 2–75 ms search

Encoders load lazily. They are released after 20 s (image) or 60 s (text) of idle time, and on TRIM_MEMORY_RUNNING_LOW and above. Indexing runs with 1–2 threads, spinning is disabled, and the scheduler checks battery and thermal state between batches.

Testing

Automated

• Dart: 16 new tests in test/model/smart_search_test.dart, all 90 passing. They cover:
  • fingerprint semantics and most-recent-first inline indexing;
  • charging-only scheduling, and the fallback to app resume when scheduling isn't possible;
  • coalescing of overlapping analysis completions;
  • auto-indexing after a download completes, without the settings open;
  • visibility and hidden-album filtering, and results dropping items hidden after the search;
  • "More like this" ordering, garbage collection only after a full analysis, and the disabled state;
  • calendar layout on results, search scoping (album vs. results vs. everything);
  • the Enter-key rules (words like "sunset" search even when a title contains them; file names like IMG_2041 filter), including file names inside calendar day stacks, and no search from the bin.
• Kotlin: ClipTokenizerTest, 260 golden strings against the reference Python tokenizer, including ftfy and HTML-entity cases; VectorStoreTest, 16 tests covering CRC corruption, torn writes, tombstones, compaction, churn, generation guards and statistics.
  • The Android unit tests now run on the JUnit Platform (separate first commit). CI doesn't run gradle tests today; see open question 5.
• flutter analyze is clean.

On device and emulator. All of the following passed:

Scenario	Result
Fresh install → promo → enable → download → leave settings immediately	Model installed and indexing started automatically (12 s)
Offline when starting the download	Shows "Waiting for network"; resumes when back online
Disable mid-index	Stops at the next batch
Charging only, unplugged	Pauses; charging work scheduled (API < 31) or resumed on app resume
New photo	Indexed inline (1 item, 1 thread)
Deleted photos (1,080 at once)	Vectors tombstoned and compacted; index size matches live items exactly
SIGKILL mid-index (40 of 600 done)	Relaunch indexed exactly the remaining 560; index is byte-exact (no duplicates or torn records)
Video "More like this"	Works (uses the video thumbnail): the video, then the photo it was made from, then other musicians
2 GB RAM device	Best quality hidden; Standard indexes with prepacking off; memory as above
"More like this" from a view intent (other app)	Correctly not offered
"More like this" on a burst frame (3-shot stack, frame 2)	Works; results show the frames unstacked, then the source photo, then similar scenes
Model switch Standard → Best quality on the Fire, settings closed	Standard keeps answering searches during the 448 MB download; Best quality installed and activated after 76 s, indexing started 3 s later
Enter on "sunset" in a real library where one title contains "sunset"	Runs a smart search (before this change it silently filtered to that 1 item)
Android 16 (Samsung), dark theme	Fresh install through the search page tip, download, index, Best quality switch; dialogs and settings legible
Empty (0-byte) photos and videos	Recorded as unreadable, not retried; settings show "94 items indexed · 14 items could not be read"
Video "More like this" on the Samsung	Similar indoor clips from the same place
32-bit (armeabi-v7a) app on the Fire	Download, install, full index of 1,318 photos, text search and "More like this" all work; about 4–5× slower than 64-bit (numbers above)

Issues found by the device testing, all fixed in this branch:

• a query word contained in a title (such as "sunset") filtered by title instead of searching;
• on Fire OS 8, re-downloading after a reinstall crashed the system downloads provider, because of stale media store entries at the same paths. Each download attempt now uses its own directory, and installed files are located through DownloadManager (which may rename them);
• empty files were retried on every analysis and kept the index "in progress". They are now recorded as unreadable and shown as such.

Open questions

1. ORT in which flavors / F-Droid. ORT Android is an MIT-licensed prebuilt AAR from Maven Central. F-Droid builds from source and may not accept the prebuilt native library. Options:

   • ship smart search only in the play and izzy flavors (the code is already isolated behind the channel, so a no-op handler is easy);
   • build ORT from source for F-Droid;
   • accept the +23–33 MB everywhere.

   Happy to implement whichever you prefer.

2. Model hosting ownership. The files are on a Hugging Face repo that I control. They're pinned by commit and hash, so they can't change under the app, but you may prefer a repo you own: an HF org or GitHub Releases on deckerst/aves_extra, for example. The conversion script is in this branch, so moving them is a copy plus a URL/commit change in ModelCatalog.kt.

3. Promo on the search page. A dismissible "Find photos by describing them" tile is shown until dismissed or enabled. Keep it, or rely on settings only?

4. Translations. Only app_en.arb has the new strings (the Weblate workflow is left to you).

5. CI. Should the gradle unit tests (:app:testLibreDebugUnitTest) be added to the workflow? They need testOptions.unitTests.isReturnDefaultValues = true for android.util.Log.

6. Test data size. The tokenizer test adds the 1.4 MB CLIP BPE vocabulary (MIT, license file alongside) under android/app/src/test/resources. It could be fetched at test time instead, if you'd rather not have the binary in the repo.

🤖 Generated with Claude Code

[deckerst]

Thanks for taking the time to work on this.

As you know, this repo does not accept PRs, and that means external work in general. Especially code from AI.

Out of curiosity, how's internationalisation? In other words, is searching for images using non-English text supported by that kind of models?

[jasontitus]

Thanks for asking about internationalisation. I checked this separately rather than treating Unicode tokenization as proof of multilingual search. The branch's Standard model was trained on the English subset of LAION-2B; the PE-Core pack is not a documented multilingual retrieval checkpoint. A more promising candidate, not included in the branch/app, is the paired image + text SigLIP 2 Base/32 256 OpenCLIP conversion (revision cd1efb47643f2794413dd79ef24397c175032780). I exported both towers and kept the sensitive MLP output projections fp32 while dynamically quantizing the other eligible weights to int8.

For a small 24-image XM3600 redistribution gallery, each language's first human caption was a query; numbers are text-to-image Recall@1 (target ranked among those 24 photos):

Pack / measured runtime	English	German	Japanese	Hindi
Existing Standard int8, desktop ORT	0.83	0.58	0.21	0.04
Existing Best-quality int8, desktop ORT	0.79	0.54	0.04	0.04
SigLIP 2 selective-int8, Fire ORT harness	0.88	0.92	0.92	0.71

These used the same selected gallery, but the SigLIP 2 row ran in a separate Android harness with its own tokenizer/preprocessing; this is directional, not a controlled in-app or population-quality comparison. On a separate 100-image sample, the quantized SigLIP 2 pair scored 0.84 / 0.94 / 0.76 / 0.51 (EN/DE/JA/HI), versus its own fp32 pair's 0.81 / 0.94 / 0.72 / 0.50. Larger galleries, hard negatives and more languages are needed.

Size/download: Its ARM .ort image + text files total 564 MB, and the matching tokenizer adds 34 MB: 598 MB installed (decimal), versus ~166 MB Standard and ~470 MB Best quality for the current packs. Gzip packaging measured ~463 MB transfer for the candidate, but it must be unpacked for memory-mapped inference; I have not measured a real model download or installation in Aves.

Fire performance/memory: On a 64-bit Fire Max 11, ONNX Runtime Android 1.30 with two CPU threads and weight prepacking off, the standalone ARM-ORT harness completed 24 images and 96 pretokenized queries: warm medians 162 ms/image, 140 ms/query; cold session creation 63 + 108 ms; process PSS 485 MB after inference. That is not a measured peak, and it is not directly comparable to the existing app's indexing PSS or first-query timings. A smaller ~463 MB installed mixed int4-image/int8-text option ran at 450 ms/image and shifted many rankings, so I wouldn't call it quality-preserving.

So: non-English image search looks feasible with a different matched model pair, especially for Japanese and Hindi in this sample. The current branch does not support this candidate yet; integration needs its exact tokenizer, a new model/index ID and reindexing, pinned downloads, and app-level memory/indexing tests before it could be offered as a tier. No photos or queries need leave the device for inference.

[jasontitus]

Yes—there is now an optional multilingual SigLIP2 image/text model. I tested searches in Japanese (森の川), Hindi (जंगल में नदी), German (ein wald mit einem bach), and English (purple flowers) on a Kindle Fire; the corresponding forest/river and flower photos appeared among the best matches. This is an on-device model downloaded only when selected (570 MiB); language quality can vary.

For anyone who wants to try the fork rather than build it, I published the exact signed, non-debuggable arm64 APK tested on that Fire: https://github.com/jasontitus/aves/releases/tag/smart-search-preview-1.15.4-1 . It installs separately as deckers.thibault.aves.preview and does not replace upstream Aves. The Fire scan completed with 1,435 items indexed and 3 unreadable items. A 64-bit device with at least 3.5 GB RAM is required. APK SHA-256: 82af82a22ea0ee40d755597b498267ab399c77f5b70514b4f06d40c97cd88090.

This is an independent preview on my fork, not an upstream release.