Files
simplex-chat/apps/ios/Shared/Views/Chat/ChatItem/MarkdownFileView.swift
T
fbca2cae81 core, ui: resize images in the core for sharper previews and images (#7941)
* WIP: Haskell impl of resizeImageToStrSize

* WIP: add FFI export wrapper

* WIP: switch ios resizeImageToStrSizeSync to backend

* WIP: fix resizeImageToStrSizeSync

* WIP: add JNI wrappers

* Add non-base64 resize and cWriteImage

* WIP: add chat_write_image

* fix resizeToStr

* fixes

* fix syntax, raise jpeg quality back

* add chatWriteImage

* Fix cfArgs in tests

* Use jpeg-turbo encoder

* add nix pins

* bump jpeg-turbo

* add CI deps

* add encoding funs to android lib

* bump jpeg-turbo for 20.04 fix

* jpeg-turbo fix for 8.10

* android fixes

* Bump jpeg-turbo

* set static flag for jpeg-turbo

* Fix nix for aarch64/android

* Fix android C api

* Fix resize in android

* Bump hs-libjpeg-turbo

* target correct override

* override android lib from android pkgs

* add macos build tweaks

* bump hs-jpeg-turbo for 8.10 fixes

* more build fixes

* make ios compile

* Always use pkgconfig for libjpeg-turbo

* WIP: debug libjpeg in ubuntu pkg-config CI

* Add pkg-config to dockerfiles

* Add libturbojpeg to dockerfiles

* Use -dev version of libturbojpeg in dockerfiles

* Install pkg-config in windows CI

* Disable flag for custom libturbojpeg linking options

Workarounds relocation error on x86_64-linux with gcc:
  /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libturbojpeg.a(turbojpeg.c.o): relocation R_X86_64_TPOFF32 against `errStr' can not be used when making a shared object; recompile with -fPIC
  /usr/bin/ld: failed to set dynamic section sizes: bad value

* WIP: debug libjpeg in windows pkg-config CI

* WIP: debug libjpeg package in windows CI

* core: image resize module and buffer FFI

* ci: link libjpeg-turbo statically on all platforms

* ui: resize images in the core

* core: rename image resize functions to chat_resize_image and chat_resize_image_uri

* ci: restore whitespace in build.yml

* ci: build libjpeg-turbo 3.1.4.1 from a submodule

* core: hs-jpeg-turbo from github.com/simplex-chat

* nodejs: resize images

* python: resize images

* core: hs-jpeg-turbo from gitlab.com, the GitHub import lacks LFS objects

* core: generate /img previews

* ci: link libjpeg-turbo on Windows without pkg-config

* ci: build libjpeg-turbo on Windows with the GHC toolchain

* docs: plan for image resizing in the core

* docs: update image resizing plan

* ios: prevent crash when wallpaper image cannot be resized

* android, desktop: raise an error when an image cannot be resized

* docs: update spec references for image resizing

* docker: build the CLI image from a clone without submodules

* docs: apply review fixes to image resizing plan

* docker: build the CLI from a git worktree with the submodule present

* docs: apply second review to image resizing plan

* docs: point image resizing plan at the rewritten commit

* core: choose image format by transparency in chat_resize_image

* nodejs: resize images to the app sizes

* python: resize images to the app sizes

* nodejs, python: describe the core modules as used for image resizing

* docs: add license notices of the image libraries

* ci: describe the avatar method without attributing it to the apps

* ci, ios, nix: remove the libjpeg-turbo build outside cabal

* python: name resize_image in the core module text

* nodejs, python: describe resizing to the app image sizes

* core: refuse images larger than the supported size

* core: resize to JPEG images with a side above 65500 pixels

* docs: record image resizing decisions for the bundled libjpeg-turbo

* core, docs: build libjpeg-turbo and stb from the fork submodules

* docs: record the fork pins in the image resizing plan

* core: resize images to at most 4320 pixels per side, refuse above 16384

* docs: record the image side limits in the image resizing plan

* core: pin the rebased hs-jpeg-turbo fork

* core: pin the hs-jpeg-turbo fork without package.yaml

* docs: plan scaled JPEG decoding and EXIF orientation of images

* docs: copy the JuicyPixels-stbir licence of the pinned fork

* core: pin the hs-jpeg-turbo fork with scaled JPEG decoding

* core: decode JPEG at the display size, up to 65500 pixels per side

* core: rotate JPEG images by their EXIF orientation

* docs: refer the image plan to the JPEG decoding code

* docs: record the approved image processing decisions

* core: limit JPEG decoding memory to 1024 MB outside iOS and Android

* docs: name JPEG decoding in the libjpeg-turbo notice

* docs: plan image resizing from files and the decisions of 26 Oct

* core: scale images in one Catmull-Rom pass with sharpening

* docs: plan image sizes at a fixed quality

* core: resize images at a fixed quality to fill the size limit

* core: resize images from a file path or a file descriptor

* ios, android, desktop: resize picked images from their files

* core: encode resized images with mozjpeg

* tests: check descriptor closing by the file it refers to

* core: raise JPEG quality only for a full-size image below 97% of the limit

* ci: name the mozjpeg submodule in the Windows cleanup comments

* docs: refer the image plan to the current code

* docs: refer the image plan to the code merged with master

* core: export every image resizing function from each library

* core: close the file descriptor when its handle cannot be created

* docs: count the C sources of mozjpeg with the arithmetic decoder

* core: use the reviewed hs-jpeg-turbo and JuicyPixels-stbir forks

* ci: build the armv7a Android core library on request

* core: round image heights half up in the size search

* ios: describe the file image resizing functions in SimpleX.h

* ci: pick, preview and send images in the Claude app tests

* core: decode PNG images with libspng and refuse other formats

* core: encode PNG with libspng, keeping the smaller filter choice

* ci: pick, preview and send images in the scripted app smoke tests

* flake: build the PNG decoder package in nix builds

* ios: remove the copy of a picked image after sending it

* core: refuse image files larger than the decode memory limit

* core: scale a JPEG image to its largest size once per resize

* core: bisect JPEG quality without a step function

* core: reduce the memory used to sharpen and to decode PNG images

* android, desktop: pass the image format to the core as an integer

* android, desktop: fuse the preview conditions of picked media

* android, desktop, ci: check that the core resized the smoke test images

* python: remove the docstrings of the explicit image resizing functions

* docs: add the second review fixes to the image plan

* ci: tap the iOS attachment button by its accessibility label

* core: keep the colour profiles of resized images

* docs: add the colour profiles to the image plan

* ci: wait up to 30 s for the new group screens in the iOS smoke test

* ci: pick the Android smoke test images from their own photo folders

* ci: tap the iOS photo picker by position, as its tree is not shown

* ci: pick the test card and the logo by their iOS picker positions

* core: update JuicyPixels-stbir with the stb_image_resize licence terms

* core: write a fixed Display P3 profile for Display P3 images only

* core: do not keep the input image in memory for its colour profile

* core: read the JPEG segments once for the orientation and the profile

* core: read an image file into one buffer of its size

* docs: add the third review fixes to the image plan

* core: resize images with Display P3 or no profile, not profile bytes

* ci: check the pixel sizes of the images sent in the iOS smoke test

* core: update hs-jpeg-turbo with PNG chunk limits and encoder fixes

* docs: list Compact ICC Profiles, libspng and miniz in the dependencies

* android, desktop: reword the picked media preview comment

* docs: refer the image plan to the code merged with master

* ci: render the iOS snapshots before the images case

* ci: dismiss the snapshot tests' camera alert before the iOS images case

* core: update hs-jpeg-turbo with tests and the JuicyPixels-spng synopsis

* core: pin hs-jpeg-turbo and JuicyPixels-stbir in simplex-chat repositories

* update ios core library

* core, ui: accept only JPEG and PNG image data

* core: update JuicyPixels-stbir

* docs: plan for the iOS copy of picked images

---------

Co-authored-by: IC Rainbow <aenor.realm@gmail.com>
Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>
Co-authored-by: Alexandre Esteves <alexfmpe@proton.me>
2026-10-30 12:14:51 +00:00

217 lines
8.3 KiB
Swift

import SwiftUI
import SimpleXChat
import Ink
private let MAX_MARKDOWN_VIEW_SIZE: Int64 = 1_048_576
private let MAX_MARKDOWN_LINE_LENGTH = 15_000
private let MAX_MARKDOWN_NESTING = 16
private let MAX_MARKDOWN_LINK_NESTING = 4
private let MAX_MARKDOWN_TABLE_ROWS = 200
private let MAX_MARKDOWN_IMAGE_DIMENSION: CGFloat = 4320
private let MAX_MARKDOWN_IMAGE_PIXELS: CGFloat = 2160 * 4320
func isMarkdownFile(_ file: CIFile) -> Bool {
let ext = (file.fileName as NSString).pathExtension.lowercased()
return file.fileSize <= MAX_MARKDOWN_VIEW_SIZE && (ext == "md" || ext == "markdown")
}
func showMarkdownFile(_ file: CIFile) {
guard let fileSource = getLoadedFileSource(file) else { return }
Task {
do {
let data = try getFileData(getAppFilePath(fileSource.filePath), fileSource.cryptoArgs)
let text = decodedText(data)
let lines = text.components(separatedBy: "\n")
let html = withinMarkdownLimits(lines)
? MarkdownParser(modifiers: [Modifier(target: .html) { escapedHTML($0.markdown) }, imagesWithinBudget()]).html(from: text)
: nil
let shownLines = lines.flatMap(chunkedLine)
let hasBidiControls = text.unicodeScalars.contains(where: isBidiControl)
await MainActor.run {
showAppSheet {
MarkdownFileView(file: file, fileSource: fileSource, lines: shownLines, html: html, hasBidiControls: hasBidiControls)
}
}
} catch {
await MainActor.run {
showAlert(NSLocalizedString("Error reading file", comment: "alert title"), message: error.localizedDescription)
}
}
}
}
private func decodedText(_ data: Data) -> String {
var text = String.UnicodeScalarView()
var parser = Unicode.UTF8.ForwardParser()
var bytes = data.makeIterator()
while true {
switch parser.parseScalar(from: &bytes) {
case let .valid(encoded): text.append(replacingControlChar(Unicode.UTF8.decode(encoded)))
case let .error(length): text.append(contentsOf: repeatElement("?" as Unicode.Scalar, count: length))
case .emptyInput: return String(text).replacingOccurrences(of: "\r\n", with: "\n").replacingOccurrences(of: "\r", with: "\n")
}
}
}
private func replacingControlChar(_ s: Unicode.Scalar) -> Unicode.Scalar {
s.value < 0x20 && s != "\t" && s != "\n" && s != "\r" ? "?" : s
}
private func withinMarkdownLimits(_ lines: [String]) -> Bool {
var tableRows = 0
var linkNesting = 0
for line in lines {
tableRows = line.contains("|") ? tableRows + 1 : 0
if line.count > MAX_MARKDOWN_LINE_LENGTH || nestingLevel(line) > MAX_MARKDOWN_NESTING || tableRows > MAX_MARKDOWN_TABLE_ROWS || !linkNestingWithinLimit(line, &linkNesting) {
return false
}
}
return true
}
private func linkNestingWithinLimit(_ line: String, _ nesting: inout Int) -> Bool {
if line.isEmpty { nesting = 0 }
for c in line {
if c == "[" {
nesting += 1
if nesting > MAX_MARKDOWN_LINK_NESTING { return false }
} else if c == "]" && nesting > 0 {
nesting -= 1
}
}
return true
}
private func nestingLevel(_ line: String) -> Int {
let content = line.drop(while: { $0 == " " || $0 == "\t" || $0 == ">" })
let prefix = line[..<content.startIndex]
let quotes = prefix.filter { $0 == ">" }.count
let listItem = content.first.map { "-*+".contains($0) || $0.isNumber } ?? false
return quotes + (listItem ? (prefix.count - quotes) / 2 : 0)
}
private func chunkedLine(_ line: String) -> [String] {
let chars = Array(line)
return stride(from: 0, to: max(chars.count, 1), by: MAX_MARKDOWN_LINE_LENGTH).map {
String(chars[$0 ..< min($0 + MAX_MARKDOWN_LINE_LENGTH, chars.count)])
}
}
private func escapedHTML(_ s: Substring) -> String {
s.replacingOccurrences(of: "&", with: "&amp;")
.replacingOccurrences(of: "<", with: "&lt;")
.replacingOccurrences(of: ">", with: "&gt;")
}
private func imagesWithinBudget() -> Modifier {
var pixels = MAX_MARKDOWN_IMAGE_PIXELS
return Modifier(target: .images) { image in
let src = image.html.dropFirst("<img src=\"".count).prefix(while: { $0 != "\"" })
guard src.hasPrefix("data:image/") else { return escapedHTML(image.markdown) }
let parts = src.split(separator: ",", maxSplits: 1)
guard parts.count == 2,
parts[0].hasSuffix(";base64"),
let data = Data(base64Encoded: String(parts[1])),
data.starts(with: [0xFF, 0xD8, 0xFF]) || data.starts(with: [0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]),
let size = UIImage(data: data)?.size,
size.width <= MAX_MARKDOWN_IMAGE_DIMENSION,
size.height <= MAX_MARKDOWN_IMAGE_DIMENSION,
size.width * size.height <= pixels
else { return "" }
pixels -= size.width * size.height
return "<img src=\"\(src)\"/>"
}
}
private func isBidiControl(_ s: Unicode.Scalar) -> Bool {
[0x061C, 0x200E, 0x200F].contains(s.value) || (0x202A...0x202E).contains(s.value) || (0x2066...0x2069).contains(s.value)
}
private func withInvisibleCharMarkers(_ line: String) -> Text {
var text = Text(verbatim: "")
var plain = ""
for s in line.unicodeScalars {
if s.properties.generalCategory == .format {
text = text + Text(verbatim: plain) + Text(verbatim: unicodeMarker(s)).foregroundColor(.orange)
plain = ""
} else {
plain.unicodeScalars.append(s)
}
}
return text + Text(verbatim: plain)
}
private func openMarkdownFileLink(_ url: URL) {
let uri = url.absoluteString
let parsed = parseSimpleXMarkdown(uri)
let format = parsed?.count == 1 ? parsed?.first?.format : nil
switch format {
case let .simplexLink(_, _, simplexUri, _)?:
if let simplexUrl = URL(string: simplexUri) {
ChatModel.shared.appOpenUrl = simplexUrl
} else {
showInvalidLinkAlert(uri)
}
case .uri? where uri.hasPrefix("https://") || uri.hasPrefix("http://"):
openBrowserAlert(uri: uri)
default:
showInvalidLinkAlert(uri)
}
}
struct MarkdownFileView: View {
@AppStorage(DEFAULT_MARKDOWN_FILE_TEXT_VIEW) private var textView = false
let file: CIFile
let fileSource: CryptoFile
let lines: [String]
let html: String?
let hasBidiControls: Bool
var body: some View {
NavigationView {
VStack(spacing: 0) {
if hasBidiControls {
Text("This file has text direction characters. The text can be shown in a different order than it is written.")
.foregroundColor(.orange)
.padding()
}
if let html, !textView {
ConditionsWebView(html: html, openLink: openMarkdownFileLink)
.padding(.horizontal)
} else {
ScrollView {
LazyVStack(alignment: .leading, spacing: 0) {
ForEach(lines.indices, id: \.self) { i in
withInvisibleCharMarkers(lines[i])
.font(.system(.body, design: .monospaced))
.textSelection(.enabled)
}
}
.padding()
}
}
}
.modifier(ThemedBackground())
.navigationTitle(file.fileName)
.navigationBarTitleDisplayMode(.inline)
.toolbar {
ToolbarItemGroup(placement: .navigationBarTrailing) {
if html != nil {
Button {
textView.toggle()
} label: {
Image(systemName: textView ? "doc.richtext" : "chevron.left.forwardslash.chevron.right")
}
}
Button {
saveCryptoFile(fileSource)
} label: {
Image(systemName: "square.and.arrow.up")
}
}
}
}
.navigationViewStyle(.stack)
}
}