core, ui: resize images in the core for sharper previews and images (#7941)

* WIP: Haskell impl of resizeImageToStrSize * WIP: add FFI export wrapper * WIP: switch ios resizeImageToStrSizeSync to backend * WIP: fix resizeImageToStrSizeSync * WIP: add JNI wrappers * Add non-base64 resize and cWriteImage * WIP: add chat_write_image * fix resizeToStr * fixes * fix syntax, raise jpeg quality back * add chatWriteImage * Fix cfArgs in tests * Use jpeg-turbo encoder * add nix pins * bump jpeg-turbo * add CI deps * add encoding funs to android lib * bump jpeg-turbo for 20.04 fix * jpeg-turbo fix for 8.10 * android fixes * Bump jpeg-turbo * set static flag for jpeg-turbo * Fix nix for aarch64/android * Fix android C api * Fix resize in android * Bump hs-libjpeg-turbo * target correct override * override android lib from android pkgs * add macos build tweaks * bump hs-jpeg-turbo for 8.10 fixes * more build fixes * make ios compile * Always use pkgconfig for libjpeg-turbo * WIP: debug libjpeg in ubuntu pkg-config CI * Add pkg-config to dockerfiles * Add libturbojpeg to dockerfiles * Use -dev version of libturbojpeg in dockerfiles * Install pkg-config in windows CI * Disable flag for custom libturbojpeg linking options Workarounds relocation error on x86_64-linux with gcc: /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libturbojpeg.a(turbojpeg.c.o): relocation R_X86_64_TPOFF32 against `errStr' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: failed to set dynamic section sizes: bad value * WIP: debug libjpeg in windows pkg-config CI * WIP: debug libjpeg package in windows CI * core: image resize module and buffer FFI * ci: link libjpeg-turbo statically on all platforms * ui: resize images in the core * core: rename image resize functions to chat_resize_image and chat_resize_image_uri * ci: restore whitespace in build.yml * ci: build libjpeg-turbo 3.1.4.1 from a submodule * core: hs-jpeg-turbo from github.com/simplex-chat * nodejs: resize images * python: resize images * core: hs-jpeg-turbo from gitlab.com, the GitHub import lacks LFS objects * core: generate /img previews * ci: link libjpeg-turbo on Windows without pkg-config * ci: build libjpeg-turbo on Windows with the GHC toolchain * docs: plan for image resizing in the core * docs: update image resizing plan * ios: prevent crash when wallpaper image cannot be resized * android, desktop: raise an error when an image cannot be resized * docs: update spec references for image resizing * docker: build the CLI image from a clone without submodules * docs: apply review fixes to image resizing plan * docker: build the CLI from a git worktree with the submodule present * docs: apply second review to image resizing plan * docs: point image resizing plan at the rewritten commit * core: choose image format by transparency in chat_resize_image * nodejs: resize images to the app sizes * python: resize images to the app sizes * nodejs, python: describe the core modules as used for image resizing * docs: add license notices of the image libraries * ci: describe the avatar method without attributing it to the apps * ci, ios, nix: remove the libjpeg-turbo build outside cabal * python: name resize_image in the core module text * nodejs, python: describe resizing to the app image sizes * core: refuse images larger than the supported size * core: resize to JPEG images with a side above 65500 pixels * docs: record image resizing decisions for the bundled libjpeg-turbo * core, docs: build libjpeg-turbo and stb from the fork submodules * docs: record the fork pins in the image resizing plan * core: resize images to at most 4320 pixels per side, refuse above 16384 * docs: record the image side limits in the image resizing plan * core: pin the rebased hs-jpeg-turbo fork * core: pin the hs-jpeg-turbo fork without package.yaml * docs: plan scaled JPEG decoding and EXIF orientation of images * docs: copy the JuicyPixels-stbir licence of the pinned fork * core: pin the hs-jpeg-turbo fork with scaled JPEG decoding * core: decode JPEG at the display size, up to 65500 pixels per side * core: rotate JPEG images by their EXIF orientation * docs: refer the image plan to the JPEG decoding code * docs: record the approved image processing decisions * core: limit JPEG decoding memory to 1024 MB outside iOS and Android * docs: name JPEG decoding in the libjpeg-turbo notice * docs: plan image resizing from files and the decisions of 26 Oct * core: scale images in one Catmull-Rom pass with sharpening * docs: plan image sizes at a fixed quality * core: resize images at a fixed quality to fill the size limit * core: resize images from a file path or a file descriptor * ios, android, desktop: resize picked images from their files * core: encode resized images with mozjpeg * tests: check descriptor closing by the file it refers to * core: raise JPEG quality only for a full-size image below 97% of the limit * ci: name the mozjpeg submodule in the Windows cleanup comments * docs: refer the image plan to the current code * docs: refer the image plan to the code merged with master * core: export every image resizing function from each library * core: close the file descriptor when its handle cannot be created * docs: count the C sources of mozjpeg with the arithmetic decoder * core: use the reviewed hs-jpeg-turbo and JuicyPixels-stbir forks * ci: build the armv7a Android core library on request * core: round image heights half up in the size search * ios: describe the file image resizing functions in SimpleX.h * ci: pick, preview and send images in the Claude app tests * core: decode PNG images with libspng and refuse other formats * core: encode PNG with libspng, keeping the smaller filter choice * ci: pick, preview and send images in the scripted app smoke tests * flake: build the PNG decoder package in nix builds * ios: remove the copy of a picked image after sending it * core: refuse image files larger than the decode memory limit * core: scale a JPEG image to its largest size once per resize * core: bisect JPEG quality without a step function * core: reduce the memory used to sharpen and to decode PNG images * android, desktop: pass the image format to the core as an integer * android, desktop: fuse the preview conditions of picked media * android, desktop, ci: check that the core resized the smoke test images * python: remove the docstrings of the explicit image resizing functions * docs: add the second review fixes to the image plan * ci: tap the iOS attachment button by its accessibility label * core: keep the colour profiles of resized images * docs: add the colour profiles to the image plan * ci: wait up to 30 s for the new group screens in the iOS smoke test * ci: pick the Android smoke test images from their own photo folders * ci: tap the iOS photo picker by position, as its tree is not shown * ci: pick the test card and the logo by their iOS picker positions * core: update JuicyPixels-stbir with the stb_image_resize licence terms * core: write a fixed Display P3 profile for Display P3 images only * core: do not keep the input image in memory for its colour profile * core: read the JPEG segments once for the orientation and the profile * core: read an image file into one buffer of its size * docs: add the third review fixes to the image plan * core: resize images with Display P3 or no profile, not profile bytes * ci: check the pixel sizes of the images sent in the iOS smoke test * core: update hs-jpeg-turbo with PNG chunk limits and encoder fixes * docs: list Compact ICC Profiles, libspng and miniz in the dependencies * android, desktop: reword the picked media preview comment * docs: refer the image plan to the code merged with master * ci: render the iOS snapshots before the images case * ci: dismiss the snapshot tests' camera alert before the iOS images case * core: update hs-jpeg-turbo with tests and the JuicyPixels-spng synopsis * core: pin hs-jpeg-turbo and JuicyPixels-stbir in simplex-chat repositories * update ios core library * core, ui: accept only JPEG and PNG image data * core: update JuicyPixels-stbir * docs: plan for the iOS copy of picked images --------- Co-authored-by: IC Rainbow <aenor.realm@gmail.com> Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com> Co-authored-by: Alexandre Esteves <alexfmpe@proton.me>
2026-11-04 06:32:59 +00:00 · 2026-10-30 12:14:51 +00:00
1 parent 1491d0cff8
commit fbca2cae81
@@ -277,17 +277,30 @@ def lib() -> ctypes.CDLL:
    return _lib


-QUEUE_SIZE_UNSUPPORTED = (
-    "loaded libsimplex does not export chat_migrate_init_queue; queue size needs a newer libsimplex"
-)
+def _optional_export(name: str, feature: str, argtypes: list[Any]) -> Any:
+    try:
+        fn = getattr(lib(), name)
+    except AttributeError as e:
+        raise RuntimeError(
+            f"loaded libsimplex does not export {name}; {feature} needs a newer libsimplex"
+        ) from e
+    fn.argtypes = argtypes
+    fn.restype = c_void_p
+    return fn


 def migrate_init_queue() -> Any:
    """`chat_migrate_init_queue`, which older libsimplex releases do not export."""
-    try:
-        fn = lib().chat_migrate_init_queue
-    except AttributeError as e:
-        raise RuntimeError(QUEUE_SIZE_UNSUPPORTED) from e
-    fn.argtypes = [c_char_p, c_char_p, c_char_p, c_int, POINTER(c_void_p)]
-    fn.restype = c_void_p
-    return fn
+    return _optional_export(
+        "chat_migrate_init_queue",
+        "queue size",
+        [c_char_p, c_char_p, c_char_p, c_int, POINTER(c_void_p)],
+    )
+
+
+def resize_image() -> Any:
+    return _optional_export("chat_resize_image", "image resizing", [c_char_p, c_int, c_int, c_int])
+
+
+def resize_image_uri() -> Any:
+    return _optional_export("chat_resize_image_uri", "image resizing", [c_char_p, c_int, c_int])
@@ -1,17 +1,19 @@
-"""Internal typed async wrapper around libsimplex's 8 C ABI functions.
+"""Typed async wrapper around libsimplex's C ABI functions.

 Users interact with `Bot` / `ChatApi`. This module is exposed as
-`simplex_chat.core` for tests and the api.ChatApi class only.
+`simplex_chat.core` for tests, the api.ChatApi class and image resizing
+(`resize_image`, `chat_resize_image`, `chat_resize_image_uri`).
 """

 from __future__ import annotations

 import asyncio
+import base64
 import ctypes
 import json
 from concurrent.futures import Executor
 from enum import StrEnum
-from typing import Any, TypedDict
+from typing import Any, Literal, TypedDict, overload

 from . import _native
 from .types import CR, CEvt, T
@@ -70,6 +72,12 @@ class MigrationConfirmation(StrEnum):
    ERROR = "error"


+class ImageSize(StrEnum):
+    IMAGE = "image"
+    PROFILE_PREVIEW = "profilePreview"
+    MESSAGE_PREVIEW = "messagePreview"
+
+
 class CryptoArgs(TypedDict):  # wire-format JSON; camelCase fields
    fileKey: str
    fileNonce: str
@@ -88,6 +96,12 @@ def _read_and_free(ptr: int | None) -> str:
        _native.libc().free(ctypes.c_void_p(ptr))


+def _check_c_int(name: str, value: int) -> None:
+    # ctypes silently wraps ints that do not fit C int.
+    if ctypes.c_int(value).value != value:
+        raise ValueError(f"{name} {value} does not fit C int")
+
+
 async def chat_send_cmd(ctrl: int, cmd: str) -> CR.ChatResponse:
    def _call() -> str:
        ptr = _native.lib().chat_send_cmd(ctrl, cmd.encode("utf-8"))
@@ -138,9 +152,8 @@ async def chat_migrate_init(

    `queue_size` is the size of internal queues; the core default is used when None.
    """
-    # ctypes silently wraps ints that do not fit C int.
-    if queue_size is not None and ctypes.c_int(queue_size).value != queue_size:
-        raise ValueError(f"queue_size {queue_size} does not fit C int")
+    if queue_size is not None:
+        _check_c_int("queue_size", queue_size)

    init_queue = _native.migrate_init_queue() if queue_size is not None else None

@@ -246,3 +259,62 @@ def _crypto_args_result(raw: str) -> CryptoArgs:
    if parsed.get("type") == "error":
        raise RuntimeError(parsed.get("writeError", "unknown write error"))
    raise RuntimeError(f"unexpected result: {raw[:200]}")
+
+
+async def chat_resize_image_uri(data: bytes, max_size: int) -> str:
+    _check_c_int("len(data)", len(data))
+    _check_c_int("max_size", max_size)
+    resize = _native.resize_image_uri()
+
+    def _call() -> str:
+        return _read_and_free(resize(data, len(data), max_size))
+
+    return _resized_image(await asyncio.to_thread(_call))
+
+
+async def chat_resize_image(
+    data: bytes, max_size: int, image_format: Literal["jpeg", "png"]
+) -> bytes:
+    return await _resize_image_data(data, max_size, 1 if image_format == "png" else 0)
+
+
+@overload
+async def resize_image(data: bytes, size: Literal[ImageSize.IMAGE]) -> bytes: ...
+
+
+@overload
+async def resize_image(
+    data: bytes, size: Literal[ImageSize.PROFILE_PREVIEW, ImageSize.MESSAGE_PREVIEW]
+) -> str: ...
+
+
+@overload
+async def resize_image(data: bytes, size: ImageSize) -> bytes | str: ...
+
+
+async def resize_image(data: bytes, size: ImageSize) -> bytes | str:
+    """Resize image to the app size limit: JPEG or PNG for an image, a data URI for a preview."""
+    match size:
+        case ImageSize.IMAGE:
+            return await _resize_image_data(data, 261120, -1)
+        case ImageSize.PROFILE_PREVIEW:
+            return await chat_resize_image_uri(data, 12500)
+        case ImageSize.MESSAGE_PREVIEW:
+            return await chat_resize_image_uri(data, 14000)
+
+
+async def _resize_image_data(data: bytes, max_size: int, png: int) -> bytes:
+    _check_c_int("len(data)", len(data))
+    _check_c_int("max_size", max_size)
+    resize = _native.resize_image()
+
+    def _call() -> str:
+        return _read_and_free(resize(data, len(data), max_size, png))
+
+    return base64.b64decode(_resized_image(await asyncio.to_thread(_call)))
+
+
+def _resized_image(res: str) -> str:
+    if not res:
+        raise RuntimeError("image cannot be decoded or does not fit max_size")
+    return res
@@ -0,0 +1,121 @@
+from __future__ import annotations
+
+import asyncio
+import base64
+from typing import Any
+
+import pytest
+
+from simplex_chat import core
+
+JPEG = b"\xff\xd8\xff\xe0"
+
+
+class FakeLib:
+    def __init__(self, result: str) -> None:
+        self.result = result
+        self.calls: list[tuple[str, tuple[Any, ...]]] = []
+
+    def _export(self, name: str) -> Any:
+        lib = self
+
+        class Fn:
+            argtypes: Any = None
+            restype: Any = None
+
+            def __call__(self, *args: Any) -> str:
+                lib.calls.append((name, args))
+                return lib.result
+
+        return Fn()
+
+    @property
+    def chat_resize_image(self) -> Any:
+        return self._export("chat_resize_image")
+
+    @property
+    def chat_resize_image_uri(self) -> Any:
+        return self._export("chat_resize_image_uri")
+
+
+@pytest.fixture
+def fake_lib(monkeypatch: pytest.MonkeyPatch):
+    def install(result: str) -> FakeLib:
+        lib = FakeLib(result)
+        monkeypatch.setattr(core._native, "lib", lambda: lib)
+        monkeypatch.setattr(core, "_read_and_free", lambda ptr: ptr)
+        return lib
+
+    return install
+
+
+def test_resize_image_uri_returns_the_data_uri(fake_lib):
+    lib = fake_lib("data:image/jpg;base64,AAAA")
+    assert asyncio.run(core.chat_resize_image_uri(b"image", 14000)) == "data:image/jpg;base64,AAAA"
+    assert lib.calls == [("chat_resize_image_uri", (b"image", 5, 14000))]
+
+
+@pytest.mark.parametrize(("image_format", "png"), [("jpeg", 0), ("png", 1)])
+def test_resize_image_passes_the_format_and_decodes_base64(fake_lib, image_format, png):
+    lib = fake_lib(base64.b64encode(JPEG).decode())
+    assert asyncio.run(core.chat_resize_image(b"image", 261120, image_format)) == JPEG
+    assert lib.calls == [("chat_resize_image", (b"image", 5, 261120, png))]
+
+
+def test_resize_image_leaves_the_format_to_the_core(fake_lib):
+    lib = fake_lib(base64.b64encode(JPEG).decode())
+    assert asyncio.run(core.resize_image(b"image", core.ImageSize.IMAGE)) == JPEG
+    assert lib.calls == [("chat_resize_image", (b"image", 5, 261120, -1))]
+
+
+@pytest.mark.parametrize(
+    ("size", "max_size"),
+    [(core.ImageSize.PROFILE_PREVIEW, 12500), (core.ImageSize.MESSAGE_PREVIEW, 14000)],
+)
+def test_resize_image_previews_return_the_data_uri(fake_lib, size, max_size):
+    lib = fake_lib("data:image/png;base64,AAAA")
+    assert asyncio.run(core.resize_image(b"image", size)) == "data:image/png;base64,AAAA"
+    assert lib.calls == [("chat_resize_image_uri", (b"image", 5, max_size))]
+
+
+@pytest.mark.parametrize(
+    "resize",
+    [
+        lambda: core.chat_resize_image_uri(b"image", 14000),
+        lambda: core.chat_resize_image(b"image", 261120, "jpeg"),
+        lambda: core.resize_image(b"image", core.ImageSize.IMAGE),
+    ],
+)
+def test_empty_result_raises(fake_lib, resize):
+    fake_lib("")
+    with pytest.raises(RuntimeError, match="image cannot be decoded or does not fit max_size"):
+        asyncio.run(resize())
+
+
+@pytest.mark.parametrize(
+    ("name", "resize"),
+    [
+        ("chat_resize_image_uri", lambda: core.chat_resize_image_uri(b"image", 14000)),
+        ("chat_resize_image", lambda: core.chat_resize_image(b"image", 261120, "png")),
+        ("chat_resize_image", lambda: core.resize_image(b"image", core.ImageSize.IMAGE)),
+        (
+            "chat_resize_image_uri",
+            lambda: core.resize_image(b"image", core.ImageSize.PROFILE_PREVIEW),
+        ),
+    ],
+)
+def test_old_lib_raises_clear_error(monkeypatch, name, resize):
+    monkeypatch.setattr(core._native, "lib", lambda: object())
+    message = f"loaded libsimplex does not export {name}; image resizing needs a newer libsimplex"
+    with pytest.raises(RuntimeError, match=message):
+        asyncio.run(resize())
+
+
+@pytest.mark.parametrize("max_size", [2**31, -(2**31) - 1])
+def test_max_size_outside_c_int_is_rejected_before_ffi(fake_lib, max_size):
+    lib = fake_lib("data:image/jpg;base64,AAAA")
+    with pytest.raises(ValueError, match="does not fit C int"):
+        asyncio.run(core.chat_resize_image_uri(b"image", max_size))
+    with pytest.raises(ValueError, match="does not fit C int"):
+        asyncio.run(core.chat_resize_image(b"image", max_size, "jpeg"))
+    assert lib.calls == []