core, ui: resize images in the core for sharper previews and images (#7941)

* WIP: Haskell impl of resizeImageToStrSize

* WIP: add FFI export wrapper

* WIP: switch ios resizeImageToStrSizeSync to backend

* WIP: fix resizeImageToStrSizeSync

* WIP: add JNI wrappers

* Add non-base64 resize and cWriteImage

* WIP: add chat_write_image

* fix resizeToStr

* fixes

* fix syntax, raise jpeg quality back

* add chatWriteImage

* Fix cfArgs in tests

* Use jpeg-turbo encoder

* add nix pins

* bump jpeg-turbo

* add CI deps

* add encoding funs to android lib

* bump jpeg-turbo for 20.04 fix

* jpeg-turbo fix for 8.10

* android fixes

* Bump jpeg-turbo

* set static flag for jpeg-turbo

* Fix nix for aarch64/android

* Fix android C api

* Fix resize in android

* Bump hs-libjpeg-turbo

* target correct override

* override android lib from android pkgs

* add macos build tweaks

* bump hs-jpeg-turbo for 8.10 fixes

* more build fixes

* make ios compile

* Always use pkgconfig for libjpeg-turbo

* WIP: debug libjpeg in ubuntu pkg-config CI

* Add pkg-config to dockerfiles

* Add libturbojpeg to dockerfiles

* Use -dev version of libturbojpeg in dockerfiles

* Install pkg-config in windows CI

* Disable flag for custom libturbojpeg linking options

Workarounds relocation error on x86_64-linux with gcc:
  /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libturbojpeg.a(turbojpeg.c.o): relocation R_X86_64_TPOFF32 against `errStr' can not be used when making a shared object; recompile with -fPIC
  /usr/bin/ld: failed to set dynamic section sizes: bad value

* WIP: debug libjpeg in windows pkg-config CI

* WIP: debug libjpeg package in windows CI

* core: image resize module and buffer FFI

* ci: link libjpeg-turbo statically on all platforms

* ui: resize images in the core

* core: rename image resize functions to chat_resize_image and chat_resize_image_uri

* ci: restore whitespace in build.yml

* ci: build libjpeg-turbo 3.1.4.1 from a submodule

* core: hs-jpeg-turbo from github.com/simplex-chat

* nodejs: resize images

* python: resize images

* core: hs-jpeg-turbo from gitlab.com, the GitHub import lacks LFS objects

* core: generate /img previews

* ci: link libjpeg-turbo on Windows without pkg-config

* ci: build libjpeg-turbo on Windows with the GHC toolchain

* docs: plan for image resizing in the core

* docs: update image resizing plan

* ios: prevent crash when wallpaper image cannot be resized

* android, desktop: raise an error when an image cannot be resized

* docs: update spec references for image resizing

* docker: build the CLI image from a clone without submodules

* docs: apply review fixes to image resizing plan

* docker: build the CLI from a git worktree with the submodule present

* docs: apply second review to image resizing plan

* docs: point image resizing plan at the rewritten commit

* core: choose image format by transparency in chat_resize_image

* nodejs: resize images to the app sizes

* python: resize images to the app sizes

* nodejs, python: describe the core modules as used for image resizing

* docs: add license notices of the image libraries

* ci: describe the avatar method without attributing it to the apps

* ci, ios, nix: remove the libjpeg-turbo build outside cabal

* python: name resize_image in the core module text

* nodejs, python: describe resizing to the app image sizes

* core: refuse images larger than the supported size

* core: resize to JPEG images with a side above 65500 pixels

* docs: record image resizing decisions for the bundled libjpeg-turbo

* core, docs: build libjpeg-turbo and stb from the fork submodules

* docs: record the fork pins in the image resizing plan

* core: resize images to at most 4320 pixels per side, refuse above 16384

* docs: record the image side limits in the image resizing plan

* core: pin the rebased hs-jpeg-turbo fork

* core: pin the hs-jpeg-turbo fork without package.yaml

* docs: plan scaled JPEG decoding and EXIF orientation of images

* docs: copy the JuicyPixels-stbir licence of the pinned fork

* core: pin the hs-jpeg-turbo fork with scaled JPEG decoding

* core: decode JPEG at the display size, up to 65500 pixels per side

* core: rotate JPEG images by their EXIF orientation

* docs: refer the image plan to the JPEG decoding code

* docs: record the approved image processing decisions

* core: limit JPEG decoding memory to 1024 MB outside iOS and Android

* docs: name JPEG decoding in the libjpeg-turbo notice

* docs: plan image resizing from files and the decisions of 26 Oct

* core: scale images in one Catmull-Rom pass with sharpening

* docs: plan image sizes at a fixed quality

* core: resize images at a fixed quality to fill the size limit

* core: resize images from a file path or a file descriptor

* ios, android, desktop: resize picked images from their files

* core: encode resized images with mozjpeg

* tests: check descriptor closing by the file it refers to

* core: raise JPEG quality only for a full-size image below 97% of the limit

* ci: name the mozjpeg submodule in the Windows cleanup comments

* docs: refer the image plan to the current code

* docs: refer the image plan to the code merged with master

* core: export every image resizing function from each library

* core: close the file descriptor when its handle cannot be created

* docs: count the C sources of mozjpeg with the arithmetic decoder

* core: use the reviewed hs-jpeg-turbo and JuicyPixels-stbir forks

* ci: build the armv7a Android core library on request

* core: round image heights half up in the size search

* ios: describe the file image resizing functions in SimpleX.h

* ci: pick, preview and send images in the Claude app tests

* core: decode PNG images with libspng and refuse other formats

* core: encode PNG with libspng, keeping the smaller filter choice

* ci: pick, preview and send images in the scripted app smoke tests

* flake: build the PNG decoder package in nix builds

* ios: remove the copy of a picked image after sending it

* core: refuse image files larger than the decode memory limit

* core: scale a JPEG image to its largest size once per resize

* core: bisect JPEG quality without a step function

* core: reduce the memory used to sharpen and to decode PNG images

* android, desktop: pass the image format to the core as an integer

* android, desktop: fuse the preview conditions of picked media

* android, desktop, ci: check that the core resized the smoke test images

* python: remove the docstrings of the explicit image resizing functions

* docs: add the second review fixes to the image plan

* ci: tap the iOS attachment button by its accessibility label

* core: keep the colour profiles of resized images

* docs: add the colour profiles to the image plan

* ci: wait up to 30 s for the new group screens in the iOS smoke test

* ci: pick the Android smoke test images from their own photo folders

* ci: tap the iOS photo picker by position, as its tree is not shown

* ci: pick the test card and the logo by their iOS picker positions

* core: update JuicyPixels-stbir with the stb_image_resize licence terms

* core: write a fixed Display P3 profile for Display P3 images only

* core: do not keep the input image in memory for its colour profile

* core: read the JPEG segments once for the orientation and the profile

* core: read an image file into one buffer of its size

* docs: add the third review fixes to the image plan

* core: resize images with Display P3 or no profile, not profile bytes

* ci: check the pixel sizes of the images sent in the iOS smoke test

* core: update hs-jpeg-turbo with PNG chunk limits and encoder fixes

* docs: list Compact ICC Profiles, libspng and miniz in the dependencies

* android, desktop: reword the picked media preview comment

* docs: refer the image plan to the code merged with master

* ci: render the iOS snapshots before the images case

* ci: dismiss the snapshot tests' camera alert before the iOS images case

* core: update hs-jpeg-turbo with tests and the JuicyPixels-spng synopsis

* core: pin hs-jpeg-turbo and JuicyPixels-stbir in simplex-chat repositories

* update ios core library

* core, ui: accept only JPEG and PNG image data

* core: update JuicyPixels-stbir

* docs: plan for the iOS copy of picked images

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Co-authored-by: Alexandre Esteves <alexfmpe@proton.me>
This commit is contained in:
authored and GitHub committed 2026-10-30 12:14:51 +00:00
1 parent 1491d0cff8
commit fbca2cae81
118 files changed
+4227 -348

No files matched your search

@@ -13,6 +13,7 @@ import androidx.core.graphics.scale
import boofcv.android.ConvertBitmap
import boofcv.struct.image.GrayU8
import chat.simplex.common.R
import chat.simplex.common.helpers.toUri
import chat.simplex.common.views.helpers.decodeSampledBitmapFromByteArray
import chat.simplex.common.views.helpers.errorBitmap
import chat.simplex.common.views.helpers.getFileName
@@ -20,7 +21,6 @@ import java.io.ByteArrayOutputStream
import java.io.InputStream
import java.net.URI
import kotlin.math.min
import kotlin.math.sqrt
private const val MAX_IMAGE_DIMENSION = 4320
@@ -50,20 +50,6 @@ actual fun decodeBase64Image(base64: String, width: Int): ImageBitmap? =
null
}
actual fun resizeImageToStrSize(image: ImageBitmap, maxDataSize: Long): String {
var img = image
var str = compressImageStr(img)
while (str.length > maxDataSize) {
val ratio = sqrt(str.length.toDouble() / maxDataSize.toDouble())
val clippedRatio = min(ratio, 2.0)
val width = (img.width.toDouble() / clippedRatio).toInt()
val height = img.height * width / img.width
img = Bitmap.createScaledBitmap(img.asAndroidBitmap(), width, height, true).asImageBitmap()
str = compressImageStr(img)
}
return str
}
// Inspired by https://github.com/MakeItEasyDev/Jetpack-Compose-Capture-Image-Or-Choose-from-Gallery
actual fun cropToSquare(image: ImageBitmap): ImageBitmap {
var xOffset = 0
@@ -87,32 +73,12 @@ fun Bitmap.clipToCircle(): Bitmap {
return circle
}
actual fun compressImageStr(bitmap: ImageBitmap): String {
val usePng = bitmap.hasAlpha()
val ext = if (usePng) "png" else "jpg"
return "data:image/$ext;base64," + Base64.encodeToString(compressImageData(bitmap, usePng).toByteArray(), Base64.NO_WRAP)
}
actual fun compressImageData(bitmap: ImageBitmap, usePng: Boolean): ByteArrayOutputStream {
val stream = ByteArrayOutputStream()
bitmap.asAndroidBitmap().compress(if (!usePng) Bitmap.CompressFormat.JPEG else Bitmap.CompressFormat.PNG, 85, stream)
return stream
}
actual fun resizeImageToDataSize(image: ImageBitmap, usePng: Boolean, maxDataSize: Long): ByteArrayOutputStream {
var img = image
var stream = compressImageData(img, usePng)
while (stream.size() > maxDataSize) {
val ratio = sqrt(stream.size().toDouble() / maxDataSize.toDouble())
val clippedRatio = min(ratio, 2.0)
val width = (img.width.toDouble() / clippedRatio).toInt()
val height = img.height * width / img.width
img = Bitmap.createScaledBitmap(img.asAndroidBitmap(), width, height, true).asImageBitmap()
stream = compressImageData(img, usePng)
}
return stream
}
actual fun GrayU8.toImageBitmap(): ImageBitmap = ConvertBitmap.grayToBitmap(this, Bitmap.Config.RGB_565).asImageBitmap()
actual fun ImageBitmap.hasAlpha(): Boolean = hasAlpha
@@ -142,3 +108,17 @@ actual fun isAnimImage(uri: URI, drawable: Any?): Boolean {
actual fun loadImageBitmap(inputStream: InputStream): ImageBitmap =
BitmapFactory.decodeStream(inputStream).asImageBitmap()
actual fun resizeImageFileUri(uri: URI, maxSize: Int): String =
imageFileDescriptor(uri)?.let { chatResizeImageFdUri(it, maxSize) } ?: ""
actual fun resizeImageFileData(uri: URI, maxSize: Int): String =
imageFileDescriptor(uri)?.let { chatResizeImageFd(it, maxSize, -1) } ?: ""
private fun imageFileDescriptor(uri: URI): Int? =
try {
androidAppContext.contentResolver.openFileDescriptor(uri.toUri(), "r")?.detachFd()
} catch (e: Exception) {
Log.e(TAG, "imageFileDescriptor error: ${e.stackTraceToString()}")
null
}
@@ -6,7 +6,6 @@ import android.content.pm.PackageManager
import android.net.Uri
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.compose.runtime.*
import androidx.compose.ui.graphics.ImageBitmap
import androidx.core.content.ContextCompat
import chat.simplex.common.helpers.toURI
import chat.simplex.common.platform.*
@@ -23,9 +22,9 @@ actual fun AttachmentSelection(
) {
val cameraLauncher = rememberCameraLauncher { uri: Uri? ->
if (uri != null) {
val bitmap: ImageBitmap? = getBitmapFromUri(uri.toURI())
if (bitmap != null) {
val imagePreview = resizeImageToStrSize(bitmap, maxDataSize = 14000)
val imagePreview = resizeImageFileToStrSize(uri.toURI(), maxDataSize = 14000)
?: getBitmapFromUri(uri.toURI())?.let { resizeImageToStrSize(it, maxDataSize = 14000) }
if (imagePreview != null) {
composeState.value = composeState.value.copy(preview = ComposePreview.MediaPreview(listOf(imagePreview), listOf(UploadContent.SimpleImage(uri.toURI()))))
}
}
@@ -83,6 +83,10 @@ extern char *chat_write_file(chat_ctrl ctrl, const char *path, char *ptr, int le
extern char *chat_read_file(const char *path, const char *key, const char *nonce);
extern char *chat_encrypt_file(chat_ctrl ctrl, const char *from_path, const char *to_path);
extern char *chat_decrypt_file(const char *from_path, const char *key, const char *nonce, const char *to_path);
extern char *chat_resize_image_uri(char *ptr, int length, int max_size);
extern char *chat_resize_image(char *ptr, int length, int max_size, int png);
extern char *chat_resize_image_fd_uri(int fd, int max_size);
extern char *chat_resize_image_fd(int fd, int max_size, int png);
JNIEXPORT jobjectArray JNICALL
Java_chat_simplex_common_platform_CoreKt_chatMigrateInit(JNIEnv *env, __unused jclass clazz, jstring dbPath, jstring dbKey, jstring confirm) {
@@ -273,3 +277,27 @@ Java_chat_simplex_common_platform_CoreKt_chatDecryptFile(JNIEnv *env, jclass cla
(*env)->ReleaseStringUTFChars(env, to_path, _to_path);
return res;
}
JNIEXPORT jstring JNICALL
Java_chat_simplex_common_platform_CoreKt_chatResizeImageUri(JNIEnv *env, jclass clazz, jobject buffer, jint maxSize) {
jbyte *buff = (jbyte *) (*env)->GetDirectBufferAddress(env, buffer);
jlong capacity = (*env)->GetDirectBufferCapacity(env, buffer);
return decode_and_free(env, chat_resize_image_uri(buff, capacity, maxSize));
}
JNIEXPORT jstring JNICALL
Java_chat_simplex_common_platform_CoreKt_chatResizeImage(JNIEnv *env, jclass clazz, jobject buffer, jint maxSize, jint png) {
jbyte *buff = (jbyte *) (*env)->GetDirectBufferAddress(env, buffer);
jlong capacity = (*env)->GetDirectBufferCapacity(env, buffer);
return decode_and_free(env, chat_resize_image(buff, capacity, maxSize, png));
}
JNIEXPORT jstring JNICALL
Java_chat_simplex_common_platform_CoreKt_chatResizeImageFdUri(JNIEnv *env, jclass clazz, jint fd, jint maxSize) {
return decode_and_free(env, chat_resize_image_fd_uri(fd, maxSize));
}
JNIEXPORT jstring JNICALL
Java_chat_simplex_common_platform_CoreKt_chatResizeImageFd(JNIEnv *env, jclass clazz, jint fd, jint maxSize, jint png) {
return decode_and_free(env, chat_resize_image_fd(fd, maxSize, png));
}
@@ -46,6 +46,10 @@ extern char *chat_write_file(chat_ctrl ctrl, const char *path, char *ptr, int le
extern char *chat_read_file(const char *path, const char *key, const char *nonce);
extern char *chat_encrypt_file(chat_ctrl ctrl, const char *from_path, const char *to_path);
extern char *chat_decrypt_file(const char *from_path, const char *key, const char *nonce, const char *to_path);
extern char *chat_resize_image_uri(char *ptr, int length, int max_size);
extern char *chat_resize_image(char *ptr, int length, int max_size, int png);
extern char *chat_resize_image_file_uri(const char *path, int max_size);
extern char *chat_resize_image_file(const char *path, int max_size, int png);
// As a reference: https://stackoverflow.com/a/60002045
jstring decode_to_utf8_string(JNIEnv *env, char *string) {
@@ -274,3 +278,33 @@ Java_chat_simplex_common_platform_CoreKt_chatDecryptFile(JNIEnv *env, jclass cla
(*env)->ReleaseStringUTFChars(env, to_path, _to_path);
return res;
}
JNIEXPORT jstring JNICALL
Java_chat_simplex_common_platform_CoreKt_chatResizeImageUri(JNIEnv *env, jclass clazz, jobject buffer, jint maxSize) {
jbyte *buff = (jbyte *) (*env)->GetDirectBufferAddress(env, buffer);
jlong capacity = (*env)->GetDirectBufferCapacity(env, buffer);
return decode_to_utf8_string(env, chat_resize_image_uri(buff, capacity, maxSize));
}
JNIEXPORT jstring JNICALL
Java_chat_simplex_common_platform_CoreKt_chatResizeImage(JNIEnv *env, jclass clazz, jobject buffer, jint maxSize, jint png) {
jbyte *buff = (jbyte *) (*env)->GetDirectBufferAddress(env, buffer);
jlong capacity = (*env)->GetDirectBufferCapacity(env, buffer);
return decode_to_utf8_string(env, chat_resize_image(buff, capacity, maxSize, png));
}
JNIEXPORT jstring JNICALL
Java_chat_simplex_common_platform_CoreKt_chatResizeImageFileUri(JNIEnv *env, jclass clazz, jstring path, jint maxSize) {
const char *_path = encode_to_utf8_chars(env, path);
jstring res = decode_to_utf8_string(env, chat_resize_image_file_uri(_path, maxSize));
(*env)->ReleaseStringUTFChars(env, path, _path);
return res;
}
JNIEXPORT jstring JNICALL
Java_chat_simplex_common_platform_CoreKt_chatResizeImageFile(JNIEnv *env, jclass clazz, jstring path, jint maxSize, jint png) {
const char *_path = encode_to_utf8_chars(env, path);
jstring res = decode_to_utf8_string(env, chat_resize_image_file(_path, maxSize, png));
(*env)->ReleaseStringUTFChars(env, path, _path);
return res;
}
@@ -2,7 +2,6 @@ package chat.simplex.common.model
import chat.simplex.common.platform.*
import kotlinx.serialization.*
import java.nio.ByteBuffer
@Serializable
sealed class WriteFileResult {
@@ -23,10 +22,7 @@ sealed class WriteFileResult {
// Spec: spec/services/files.md#writeCryptoFile
fun writeCryptoFile(path: String, data: ByteArray): CryptoFileArgs {
val ctrl = ChatController.getChatCtrl() ?: throw Exception("Controller is not initialized")
val buffer = ByteBuffer.allocateDirect(data.size)
buffer.put(data)
buffer.rewind()
val str = chatWriteFile(ctrl, path, buffer)
val str = chatWriteFile(ctrl, path, data.toDirectBuffer())
return when (val d = json.decodeFromString(WriteFileResult.serializer(), str)) {
is WriteFileResult.Result -> d.cryptoArgs
is WriteFileResult.Error -> throw Exception(d.writeError)
@@ -39,6 +39,12 @@ external fun chatWriteFile(ctrl: ChatCtrl, path: String, buffer: ByteBuffer): St
external fun chatReadFile(path: String, key: String, nonce: String): Array<Any>
external fun chatEncryptFile(ctrl: ChatCtrl, fromPath: String, toPath: String): String
external fun chatDecryptFile(fromPath: String, key: String, nonce: String, toPath: String): String
external fun chatResizeImageUri(buffer: ByteBuffer, maxSize: Int): String
external fun chatResizeImage(buffer: ByteBuffer, maxSize: Int, png: Int): String
external fun chatResizeImageFileUri(path: String, maxSize: Int): String
external fun chatResizeImageFile(path: String, maxSize: Int, png: Int): String
external fun chatResizeImageFdUri(fd: Int, maxSize: Int): String
external fun chatResizeImageFd(fd: Int, maxSize: Int, png: Int): String
// the code in core's canonical form (prefix and 20 characters, no separators), or null when core rejects it.
// The check character is verified by core via FFI, not re-implemented here, so the two cannot disagree.
@@ -47,6 +53,13 @@ fun parseBadgeCode(s: String): String? {
return if (canonical.isEmpty()) null else canonical
}
fun ByteArray.toDirectBuffer(): ByteBuffer {
val buffer = ByteBuffer.allocateDirect(size)
buffer.put(this)
buffer.rewind()
return buffer
}
val chatModel: ChatModel
get() = chatController.chatModel
@@ -13,13 +13,11 @@ import kotlinx.coroutines.withContext
import java.io.ByteArrayOutputStream
import java.io.InputStream
import java.net.URI
import java.util.Base64
expect fun base64ToBitmap(base64ImageString: String): ImageBitmap
expect fun decodeBase64Image(base64: String, width: Int): ImageBitmap?
expect fun resizeImageToStrSize(image: ImageBitmap, maxDataSize: Long): String
expect fun resizeImageToDataSize(image: ImageBitmap, usePng: Boolean, maxDataSize: Long): ByteArrayOutputStream
expect fun cropToSquare(image: ImageBitmap): ImageBitmap
expect fun compressImageStr(bitmap: ImageBitmap): String
expect fun compressImageData(bitmap: ImageBitmap, usePng: Boolean): ByteArrayOutputStream
expect fun GrayU8.toImageBitmap(): ImageBitmap
@@ -33,6 +31,35 @@ expect fun isAnimImage(uri: URI, drawable: Any?): Boolean
expect fun loadImageBitmap(inputStream: InputStream): ImageBitmap
expect fun resizeImageFileUri(uri: URI, maxSize: Int): String
expect fun resizeImageFileData(uri: URI, maxSize: Int): String
fun resizeImageFileToStrSize(uri: URI, maxDataSize: Long): String? =
resizeImageFileUri(uri, maxDataSize.toInt()).ifEmpty {
Log.d(TAG, "resizeImageFileToStrSize: core returned no image")
null
}
fun resizeImageFileToDataSize(uri: URI, maxDataSize: Long): ByteArray? =
resizeImageFileData(uri, maxDataSize.toInt()).ifEmpty {
Log.d(TAG, "resizeImageFileToDataSize: core returned no image")
null
}?.let { Base64.getDecoder().decode(it) }
fun resizeImageToStrSize(image: ImageBitmap, maxDataSize: Long): String {
val str = chatResizeImageUri(compressImageData(image, image.hasAlpha()).toByteArray().toDirectBuffer(), maxDataSize.toInt())
if (str.isEmpty()) throw Exception("Image resize failed")
return str
}
fun resizeImageToDataSize(image: ImageBitmap, usePng: Boolean, maxDataSize: Long): ByteArrayOutputStream {
val str = chatResizeImage(compressImageData(image, usePng).toByteArray().toDirectBuffer(), maxDataSize.toInt(), if (usePng) 1 else 0)
if (str.isEmpty()) throw Exception("Image resize failed")
val stream = ByteArrayOutputStream()
stream.write(Base64.getDecoder().decode(str))
return stream
}
@Composable
fun Base64AsyncImage(
base64ImageString: String,
@@ -15,7 +15,6 @@ import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.clip
import androidx.compose.ui.focus.FocusRequester
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.ImageBitmap
import androidx.compose.ui.graphics.painter.Painter
import androidx.compose.ui.text.TextRange
import androidx.compose.ui.text.TextStyle
@@ -362,43 +361,42 @@ suspend fun MutableState<ComposeState>.processPickedMedia(uris: List<URI>, text:
val content = ArrayList<UploadContent>()
val imagesPreview = ArrayList<String>()
uris.forEach { uri ->
var bitmap: ImageBitmap?
var preview: String? = null
val uploadContent: UploadContent? = when {
isImage(uri) -> {
// Image
val drawable = getDrawableFromUri(uri)
// Do not show alert in case it's already shown from the function above
bitmap = getBitmapFromUri(uri, withAlertOnException = !AlertManager.shared.hasAlertsShown())
fun bitmapPreview() = getBitmapFromUri(uri, withAlertOnException = !AlertManager.shared.hasAlertsShown())?.let { resizeImageToStrSize(it, maxDataSize = 14000) }
if (isAnimImage(uri, drawable)) {
// It's a gif or webp
val fileSize = getFileSize(uri)
if (fileSize != null && fileSize <= maxFileSize) {
preview = bitmapPreview()
UploadContent.AnimatedImage(uri)
} else {
bitmap = null
showLargeFileAlert(fileSize ?: 0, value.sendIncognito, chatModel.currentUser.value?.profile)
null
}
} else if (bitmap != null) {
UploadContent.SimpleImage(uri)
} else {
null
preview = resizeImageFileToStrSize(uri, maxDataSize = 14000) ?: bitmapPreview()
UploadContent.SimpleImage(uri)
}
}
else -> {
// Video
val res = getBitmapFromVideo(uri, withAlertOnException = true)
bitmap = res.preview
preview = res.preview?.let { resizeImageToStrSize(it, maxDataSize = 14000) }
val durationMs = res.duration
UploadContent.Video(uri, durationMs?.div(1000)?.toInt() ?: 0)
}
}
// content and imagesPreview must stay index-aligned and equal-length: both consumers
// (ComposeImageView and sendMessageAsync) cross-index one list by the other's index.
// Only pair them when a preview bitmap exists; otherwise skip the media entirely.
if (bitmap != null && uploadContent != null) {
// Only pair them when a preview exists; otherwise skip the media entirely.
if (preview != null && uploadContent != null) {
content.add(uploadContent)
imagesPreview.add(resizeImageToStrSize(bitmap, maxDataSize = 14000))
imagesPreview.add(preview)
} else if (uploadContent is UploadContent.Video && !AlertManager.shared.hasAlertsShown()) {
// A corrupted/undecodable video can yield a null preview frame without throwing, so
// getBitmapFromVideo shows no alert. Skip it (other picked media still send) and tell
@@ -51,8 +51,13 @@ import java.io.File
import java.net.URI
import java.nio.ByteBuffer
import java.nio.charset.CodingErrorAction
import java.util.Base64
private const val DATA_IMAGE = "data:image/"
private val IMAGE_SIGNATURES = listOf(
listOf(0xFF, 0xD8, 0xFF),
listOf(0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A)
)
private const val MAX_MARKDOWN_VIEW_SIZE: Long = 1_048_576
private const val MAX_MARKDOWN_LINE_LENGTH = 15_000
private const val MAX_MARKDOWN_DEPTH = 32
@@ -281,7 +286,15 @@ private class DataImageTransformer : ImageTransformer {
private fun imageWithinBudget(link: String, width: Int): ImageBitmap? =
link.takeIf { it.startsWith(DATA_IMAGE) }
?.let { decodeBase64Image(it.substringAfter("base64,"), width) }
?.substringAfter("base64,")
?.takeIf(::hasImageSignature)
?.let { decodeBase64Image(it, width) }
?.takeIf { it.width.toLong() * it.height <= pixels }
?.also { pixels -= it.width.toLong() * it.height }
}
private fun hasImageSignature(base64: String): Boolean {
val header = runCatching { Base64.getDecoder().decode(base64.take(12)) }.getOrNull() ?: return false
val bytes = header.map { it.toInt() and 0xFF }
return IMAGE_SIGNATURES.any { bytes.take(it.size) == it }
}
@@ -172,20 +172,14 @@ fun getThemeFromUri(uri: URI, withAlertOnException: Boolean = true): ThemeOverri
}
fun saveImage(uri: URI): CryptoFile? {
val bitmap = getBitmapFromUri(uri) ?: return null
return saveImage(bitmap)
}
fun saveImage(image: ImageBitmap): CryptoFile? {
return try {
val encrypted = chatController.appPrefs.privacyEncryptLocalFiles.get()
val ext = if (image.hasAlpha()) "png" else "jpg"
val dataResized = resizeImageToDataSize(image, ext == "png", maxDataSize = MAX_IMAGE_SIZE)
val (dataResized, ext) = resizeImageFromUri(uri) ?: return null
val destFileName = generateNewFileName("IMG", ext, File(getAppFilePath("")))
val destFile = File(getAppFilePath(destFileName))
if (encrypted) {
try {
val args = writeCryptoFile(destFile.absolutePath, dataResized.toByteArray())
val args = writeCryptoFile(destFile.absolutePath, dataResized)
CryptoFile(destFileName, args)
} catch (e: Exception) {
Log.e(TAG, "Unable to write crypto file: " + e.stackTraceToString())
@@ -194,7 +188,7 @@ fun saveImage(image: ImageBitmap): CryptoFile? {
}
} else {
val output = FileOutputStream(destFile)
dataResized.writeTo(output)
output.write(dataResized)
output.flush()
output.close()
CryptoFile.plain(destFileName)
@@ -206,14 +200,12 @@ fun saveImage(image: ImageBitmap): CryptoFile? {
}
fun desktopSaveImageInTmp(uri: URI): CryptoFile? {
val image = getBitmapFromUri(uri) ?: return null
return try {
val ext = if (image.hasAlpha()) "png" else "jpg"
val dataResized = resizeImageToDataSize(image, ext == "png", maxDataSize = MAX_IMAGE_SIZE)
val (dataResized, ext) = resizeImageFromUri(uri) ?: return null
val destFileName = generateNewFileName("IMG", ext, tmpDir)
val destFile = File(tmpDir, destFileName)
val output = FileOutputStream(destFile)
dataResized.writeTo(output)
output.write(dataResized)
output.flush()
output.close()
CryptoFile.plain(destFile.absolutePath)
@@ -223,6 +215,17 @@ fun desktopSaveImageInTmp(uri: URI): CryptoFile? {
}
}
private fun resizeImageFromUri(uri: URI): Pair<ByteArray, String>? {
val data = resizeImageFileToDataSize(uri, MAX_IMAGE_SIZE)
if (data != null) {
val ext = if (data.firstOrNull() == 0x89.toByte()) "png" else "jpg"
return data to ext
}
val image = getBitmapFromUri(uri) ?: return null
val ext = if (image.hasAlpha()) "png" else "jpg"
return resizeImageToDataSize(image, ext == "png", maxDataSize = MAX_IMAGE_SIZE).toByteArray() to ext
}
fun saveAnimImage(uri: URI): CryptoFile? {
return try {
val encrypted = chatController.appPrefs.privacyEncryptLocalFiles.get()
@@ -17,7 +17,6 @@ import java.net.URI
import java.util.*
import javax.imageio.*
import javax.imageio.stream.MemoryCacheImageOutputStream
import kotlin.math.sqrt
private fun errorBitmap(): ImageBitmap =
ImageIO.read(ByteArrayInputStream(Base64.getMimeDecoder().decode("iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAAAXNSR0IArs4c6QAAAKVJREFUeF7t1kENACEUQ0FQhnVQ9lfGO+xggITQdvbMzArPey+8fa3tAfwAEdABZQspQStgBssEcgAIkSAJkiAJljtEgiRIgmUCSZAESZAESZAEyx0iQRIkwTKBJEiCv5fgvTd1wDmn7QAP4AeIgA4oW0gJWgEzWCZwbQ7gAA7ggLKFOIADOKBMIAeAEAmSIAmSYLlDJEiCJFgmkARJkARJ8N8S/ADTZUewBvnTOQAAAABJRU5ErkJggg=="))).toComposeImageBitmap()
@@ -63,33 +62,6 @@ actual fun decodeBase64Image(base64: String, width: Int): ImageBitmap? =
null
}
actual fun resizeImageToStrSize(image: ImageBitmap, maxDataSize: Long): String {
var img = image
var str = compressImageStr(img)
while (str.length > maxDataSize) {
val ratio = sqrt(str.length.toDouble() / maxDataSize.toDouble())
val clippedRatio = kotlin.math.min(ratio, 2.0)
val width = (img.width.toDouble() / clippedRatio).toInt()
val height = img.height * width / img.width
img = img.scale(width, height)
str = compressImageStr(img)
}
return str
}
actual fun resizeImageToDataSize(image: ImageBitmap, usePng: Boolean, maxDataSize: Long): ByteArrayOutputStream {
var img = image
var stream = compressImageData(img, usePng)
while (stream.size() > maxDataSize) {
val ratio = sqrt(stream.size().toDouble() / maxDataSize.toDouble())
val clippedRatio = kotlin.math.min(ratio, 2.0)
val width = (img.width.toDouble() / clippedRatio).toInt()
val height = img.height * width / img.width
img = img.scale(width, height)
stream = compressImageData(img, usePng)
}
return stream
}
actual fun cropToSquare(image: ImageBitmap): ImageBitmap {
var xOffset = 0
var yOffset = 0
@@ -111,18 +83,6 @@ actual fun cropToSquare(image: ImageBitmap): ImageBitmap {
return croppedImage
}
actual fun compressImageStr(bitmap: ImageBitmap): String {
val usePng = bitmap.hasAlpha()
val ext = if (usePng) "png" else "jpg"
return try {
val encoded = Base64.getEncoder().encodeToString(compressImageData(bitmap, usePng).toByteArray())
"data:image/$ext;base64,$encoded"
} catch (e: Exception) {
Log.e(TAG, "resizeImageToStrSize error: $e")
throw e
}
}
actual fun compressImageData(bitmap: ImageBitmap, usePng: Boolean): ByteArrayOutputStream {
val writer = ImageIO.getImageWritersByFormatName(if (usePng) "png" else "jpg").next()
val writeParam = writer.defaultWriteParam
@@ -213,6 +173,10 @@ actual fun isAnimImage(uri: URI, drawable: Any?): Boolean {
actual fun loadImageBitmap(inputStream: InputStream): ImageBitmap =
Image.makeFromEncoded(inputStream.readBytes()).toComposeImageBitmap()
actual fun resizeImageFileUri(uri: URI, maxSize: Int): String = chatResizeImageFileUri(uri.toFile().absolutePath, maxSize)
actual fun resizeImageFileData(uri: URI, maxSize: Int): String = chatResizeImageFile(uri.toFile().absolutePath, maxSize, -1)
// https://stackoverflow.com/a/68926993
fun BufferedImage.rotate(angle: Double): BufferedImage {
val sin = Math.abs(Math.sin(Math.toRadians(angle)))
+14 -7
View File
@@ -139,13 +139,20 @@ All JNI declarations reside in [`Core.kt`](../common/src/commonMain/kotlin/chat/
| 11 | [`chatParseUri()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L33) | `external fun chatParseUri(str: String, safe: Int): String` | 33 | Parse SimpleX connection URI |
| 12 | [`chatPasswordHash()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L34) | `external fun chatPasswordHash(pwd: String, salt: String): String` | 34 | Hash password with salt |
| 13 | [`chatValidName()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L35) | `external fun chatValidName(name: String): String` | 35 | Validate/sanitize display name |
| 14 | [`chatJsonLength()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L36) | `external fun chatJsonLength(str: String): Int` | 36 | Get JSON-encoded string length |
| 15 | [`chatWriteFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L37) | `external fun chatWriteFile(ctrl: ChatCtrl, path: String, buffer: ByteBuffer): String` | 37 | Write encrypted file via core |
| 16 | [`chatReadFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L38) | `external fun chatReadFile(path: String, key: String, nonce: String): Array<Any>` | 38 | Read and decrypt file |
| 17 | [`chatEncryptFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L39) | `external fun chatEncryptFile(ctrl: ChatCtrl, fromPath: String, toPath: String): String` | 39 | Encrypt file on disk |
| 18 | [`chatDecryptFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L40) | `external fun chatDecryptFile(fromPath: String, key: String, nonce: String, toPath: String): String` | 40 | Decrypt file on disk |
| 14 | [`chatParseBadgeCode()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L36) | `external fun chatParseBadgeCode(code: String): String` | 36 | Canonical form of a badge code, or an empty string |
| 15 | [`chatJsonLength()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L37) | `external fun chatJsonLength(str: String): Int` | 37 | Get JSON-encoded string length |
| 16 | [`chatWriteFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L38) | `external fun chatWriteFile(ctrl: ChatCtrl, path: String, buffer: ByteBuffer): String` | 38 | Write encrypted file via core |
| 17 | [`chatReadFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L39) | `external fun chatReadFile(path: String, key: String, nonce: String): Array<Any>` | 39 | Read and decrypt file |
| 18 | [`chatEncryptFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L40) | `external fun chatEncryptFile(ctrl: ChatCtrl, fromPath: String, toPath: String): String` | 40 | Encrypt file on disk |
| 19 | [`chatDecryptFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L41) | `external fun chatDecryptFile(fromPath: String, key: String, nonce: String, toPath: String): String` | 41 | Decrypt file on disk |
| 20 | [`chatResizeImageUri()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L42) | `external fun chatResizeImageUri(buffer: ByteBuffer, maxSize: Int): String` | 42 | Resize encoded image to a data URI of at most `maxSize` characters |
| 21 | [`chatResizeImage()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L43) | `external fun chatResizeImage(buffer: ByteBuffer, maxSize: Int, png: Int): String` | 43 | Resize encoded image to base64 of PNG or JPEG data of at most `maxSize` bytes |
| 22 | [`chatResizeImageFileUri()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L44) | `external fun chatResizeImageFileUri(path: String, maxSize: Int): String` | 44 | Desktop: as `chatResizeImageUri`, from the file at `path` |
| 23 | [`chatResizeImageFile()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L45) | `external fun chatResizeImageFile(path: String, maxSize: Int, png: Int): String` | 45 | Desktop: as `chatResizeImage`, from the file at `path`; `png` −1 selects the format by transparency |
| 24 | [`chatResizeImageFdUri()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L46) | `external fun chatResizeImageFdUri(fd: Int, maxSize: Int): String` | 46 | Android: as `chatResizeImageUri`, from the file descriptor `fd`, which core closes |
| 25 | [`chatResizeImageFd()`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L47) | `external fun chatResizeImageFd(fd: Int, maxSize: Int, png: Int): String` | 47 | Android: as `chatResizeImage`, from the file descriptor `fd`, which core closes |
**Total: 18 external native functions** (the `ChatCtrl` type alias at [line 23](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L23) is `Long`, representing the Haskell-side controller pointer).
**Total: 25 external native functions** (the `ChatCtrl` type alias at [line 23](../common/src/commonMain/kotlin/chat/simplex/common/platform/Core.kt#L23) is `Long`, representing the Haskell-side controller pointer).
<a id="initChatControllerOnStart"></a>
<a id="chatInitTemporaryDatabase"></a>
@@ -338,7 +345,7 @@ Examples from platform files:
| `expect class VideoPlayer` | [`VideoPlayer.kt`](../common/src/commonMain/kotlin/chat/simplex/common/platform/VideoPlayer.kt#L25) | 25 |
| `expect class RecorderNative` | [`RecAndPlay.kt`](../common/src/commonMain/kotlin/chat/simplex/common/platform/RecAndPlay.kt#L17) | 17 |
| `expect val cryptor: CryptorInterface` | [`Cryptor.kt`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Cryptor.kt#L9) | 9 |
| `expect fun base64ToBitmap(base64ImageString: String): ImageBitmap` | [`Images.kt`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Images.kt#L17) | 17 |
| `expect fun base64ToBitmap(base64ImageString: String): ImageBitmap` | [`Images.kt`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Images.kt#L18) | 18 |
| `expect fun getWakeLock(timeout: Long): (() -> Unit)` | [`SimplexService.kt`](../common/src/commonMain/kotlin/chat/simplex/common/platform/SimplexService.kt#L3) | 3 |
| `expect class GlobalExceptionsHandler` | [`UI.kt`](../common/src/commonMain/kotlin/chat/simplex/common/platform/UI.kt#L24) | 24 |
| `expect fun UriHandler.sendEmail(subject: String, body: CharSequence)` | [`Share.kt`](../common/src/commonMain/kotlin/chat/simplex/common/platform/Share.kt#L7) | 7 |
+1 -1
View File
@@ -315,7 +315,7 @@ Bottom sheet (`ModalBottomSheetLayout`) presenting attachment type options:
**`processPickedMedia`** (line ~300):
1. For each URI, determines type (image, animated image, video).
2. Images: Gets bitmap, creates `SimpleImage` or `AnimatedImage` upload content.
2. Images: the preview of a non-animated image is made by core from the file (`resizeImageFileToStrSize`); otherwise, or when core returns no preview, from the bitmap; creates `SimpleImage` or `AnimatedImage` upload content.
3. Videos: Extracts thumbnail and duration, creates `Video` upload content.
4. Generates base64 preview thumbnails (max 14KB).
5. Sets `ComposePreview.MediaPreview` with thumbnails and content list.
+6 -6
View File
@@ -56,7 +56,7 @@ FileProtocol.LOCAL -> MAX_FILE_SIZE_LOCAL
## 3. CryptoFile
[`CryptoFile.kt`](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt) (62 lines)
[`CryptoFile.kt`](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt) (58 lines)
Provides encrypted file I/O backed by the chat core's native cryptography (via JNI/JNA calls to `chatWriteFile`, `chatReadFile`, `chatEncryptFile`, `chatDecryptFile`).
@@ -78,10 +78,10 @@ sealed class WriteFileResult {
| Function | Line | Signature | Description |
|---|---|---|---|
| `writeCryptoFile` | [L24](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt#L24) | `(path: String, data: ByteArray): CryptoFileArgs` | Writes data to an encrypted file via a direct `ByteBuffer`. Returns the generated key and nonce. Requires initialized `ChatController`. |
| `readCryptoFile` | [L36](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt#L36) | `(path: String, cryptoArgs: CryptoFileArgs): ByteArray` | Reads and decrypts a file given its key and nonce. Returns the plaintext bytes. Throws on error (status != 0). |
| `encryptCryptoFile` | [L47](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt#L47) | `(fromPath: String, toPath: String): CryptoFileArgs` | Encrypts an existing plaintext file to a new encrypted file. Returns the generated key and nonce. |
| `decryptCryptoFile` | [L57](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt#L57) | `(fromPath: String, cryptoArgs: CryptoFileArgs, toPath: String)` | Decrypts an encrypted file to a plaintext output file. Throws on non-empty error string. |
| `writeCryptoFile` | [L23](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt#L23) | `(path: String, data: ByteArray): CryptoFileArgs` | Writes data to an encrypted file via a direct `ByteBuffer`. Returns the generated key and nonce. Requires initialized `ChatController`. |
| `readCryptoFile` | [L32](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt#L32) | `(path: String, cryptoArgs: CryptoFileArgs): ByteArray` | Reads and decrypts a file given its key and nonce. Returns the plaintext bytes. Throws on error (status != 0). |
| `encryptCryptoFile` | [L43](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt#L43) | `(fromPath: String, toPath: String): CryptoFileArgs` | Encrypts an existing plaintext file to a new encrypted file. Returns the generated key and nonce. |
| `decryptCryptoFile` | [L53](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt#L53) | `(fromPath: String, cryptoArgs: CryptoFileArgs, toPath: String)` | Decrypts an encrypted file to a plaintext output file. Throws on non-empty error string. |
All functions delegate to native C library functions through the chat core JNI bridge.
@@ -205,7 +205,7 @@ When `auto = true`, relay approval alerts are suppressed (the file is silently r
| File | Path | Lines | Description |
|---|---|---|---|
| `CryptoFile.kt` | [`common/src/commonMain/.../model/CryptoFile.kt`](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt) | 62 | Encrypted file read/write via native crypto |
| `CryptoFile.kt` | [`common/src/commonMain/.../model/CryptoFile.kt`](../../common/src/commonMain/kotlin/chat/simplex/common/model/CryptoFile.kt) | 58 | Encrypted file read/write via native crypto |
| `Files.kt` | [`common/src/commonMain/.../platform/Files.kt`](../../common/src/commonMain/kotlin/chat/simplex/common/platform/Files.kt) | 191 | Common file path declarations, theme I/O, file helpers |
| `Files.android.kt` | [`common/src/androidMain/.../platform/Files.android.kt`](../../common/src/androidMain/kotlin/chat/simplex/common/platform/Files.android.kt) | 79 | Android file path implementations |
| `Files.desktop.kt` | [`common/src/desktopMain/.../platform/Files.desktop.kt`](../../common/src/desktopMain/kotlin/chat/simplex/common/platform/Files.desktop.kt) | 116 | Desktop file path implementations |