Merge ScarletNX upstream #5
No reviewers
Labels
No labels
accessibility
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
GRID0-net/Inkypaint!5
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "doodlylizz/upstream-sync-2026-11"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
1046 upstream commits, build fixes
The CPU part of 83d85a5d on feat/ounce ("Bring up an Ounce game on the NVN loader"), without its NVN, shader, service and memory changes. - a64.profile: the switch_extended target, the Switch's features with FEAT_LSE, FEAT_LRCPC and FEAT_PAuth. - ir_int.c: the PAuth instructions with SCTLR's EnIA, EnIB, EnDA and EnDB clear. PAC* and AUT* keep the pointer. BRA*, BLRA* and RETA* branch to the register. XPAC* strip the code for a 39-bit address space. - ir_codegen.c: ir_int.c's ops are legal under switch_extended. - The cpu_extensions setting selects switch_extended for the recompile (module_llvm.c, the NCE analysis in recompile.c) and for the runtime JIT (guest_jit.c). snx_nx_codegen_analyze takes the features. - cpu.h: stx's local "this" is "requested", a name C++ can include. - codegen_test: cpu_extensions_test runs the PAuth, LSE and RCpc words in the step JIT and the AOT path, and checks that the A57 still refuses them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>The CPU part of bce9db47 on feat/ounce ("Make the NPDM the one Ounce flag, for the CPU, memory and GPU"), without its memory and GPU changes. loader/ounce.h has the one test, snx_npdm_ounce: the ACID magic at 0x300, past the Ounce's 0x180-byte signature and key. The loader puts it in SnxNpdm.ounce. The recompile (module_llvm.c, the NCE analysis in recompile.c) takes switch_extended from it. The runtime's npdm_layout sets g_ounce from the same test, and the runtime JIT (guest_jit.c) takes the same features from it. An Ounce game so needs no cpu_extensions=1, and the code made ahead of time and the code the JIT makes agree. Tests: loader_test parses an Ounce NPDM, a Switch one and one whose magic lies past the ACID's end. runtime_variants defines g_ounce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>tools/decodetree reads QEMU's .decode files, as QEMU's scripts/decodetree.py does, in strict C99, so the build needs no Python. It has two outputs: - QEMU's: arg_ structs, static trans_NAME(DisasContext *ctx, arg_NAME *a) declarations, the format extractors and the decode function, a switch tree. Its options are decodetree.py's: --decode, --static-decode, --translate, -w, -o. Not supported: --varinsnwidth. For a32.decode, a32-uncond.decode, t32.decode, t16.decode and a64.decode of QEMU commit a19f59a, its output is decodetree.py's, line for line, but for the order of a run of field assignments, which decodetree.py takes from a set and changes from run to run. - --a64-profile, --a64-header, --a64-decoder: the A64 tables that tools/a64dec made, from the same parse. a64.c is a64dec's backend, which load_rows feeds from decodetree.c's patterns. A pattern's fields keep the order of its line, with its format's where @format is. Two rows can overlap only when the innermost group they share is a { } group, as a64dec's one level of groups allowed. Its output is a64dec's, byte for byte, but for the tool's name in the notice, which changes SNX_A64_DECODE_HASH and so the JIT and unit cache keys, one time. tools/a64dec and cmake/a64dec.cmake are gone. cmake/decodetree.cmake has snx_decodetree_a64(target) in place of snx_a64dec(target), and snx_decodetree(target OUTPUT ... DECODE ... OPTIONS ...) for the QEMU output. Tests: a64_decode, native_codegen, llvm_module, docs_check and runtime_architecture pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>Dalvik lets any code call clone() through Object on an array or on an object of another class, but the JVM's verifier refuses a protected Object.clone call there ("Bad access to protected data"), so the whole class failed to load. The translator now calls the array type's own clone for an array receiver, and for an object it cannot prove to be a subclass of the caller it calls snx.dalvik.Runtime.cloneObject, which clones arrays directly and calls a public clone() by reflection. Verified with an emulator app whose classes clone arrays this way: they now load and run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>getFd returned the placeholder 3 and detachFd gave it to the app, so an app's native code that read a document through its descriptor read descriptor 3, which is the APK's zip, and then closed it: the app read the APK's bytes as its file, and the runtime's later reads of the APK failed ("bad local header"), as did files that took the number afterwards (a SharedPreferences file read back garbled). A java.io.FileDescriptor's host descriptor is read and made through JNI now (runtime/fds.c, snx.android.os.Fds): - getFd gives the descriptor; detachFd gives a duplicate the caller owns and closes this object, so that no Java object or cleaner closes what the caller owns; - fromFd duplicates a descriptor, adoptFd takes one over, dup duplicates; - getStatSize of one made of a descriptor reads its size from the host. Verified with an emulator app that scans a games folder through the documents provider and reads each file in native code: it lists the game with its icon and title, and the APK reads no longer fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>The USSE translator matched words against a table of letter strings and handed its handlers the fields by position (f[12], f[23 - i]), and the PICA interpreter and decompiler each switched on their own copy of the opcodes. Both instruction sets are now .decode files, as A32 and A64 are, and tools/decodetree makes their decoders: - src/vita/gpu/usse.decode (-w 64): Vita3K's 35 patterns with their fields named, the overlapping ones in { } groups in Vita3K's order. usse.c defines a trans_ function for each, its handlers read arg->field, and usse_survey runs through the same decoder. - src/3ds/gpu/pica.decode: nihstro's opcodes and formats. shader.c has a trans_ function for each instruction; vsgen.c's trans_ functions give the step that its flow analysis reads and compile the instruction when asked. - The bit operations the decoders call are in common/utilities/decodetree.h, which the ARM decoders include too. A differential harness ran 600k random USSE and PICA programs through the old and the new code: the same survey names, SPIR-V bytes and errors, and the same interpreter state but for the sign of a NaN that both addends of an add were (which one x86 gives back is the register allocation's). vsgen still makes the unused temporary that CMP's and MOVA's destination bits named, to keep the module the same. ctest: 815/818, the failures main has (3ds_memory_adopt, android_bionic, android_arm32). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>PMutex and the pm_* calls lived in common/filesystem/host.c, and the condition variables, the waits on a word, plat_yield and plat_tid in the NX platform layer, where no other driver could link them. They are now common/platform/thread.{h,c}, in a new target scarletnx-platform that links only the system's threads. scarletnx-filesystem links it, and filesystem/host.h and plat.h include its header, so the callers are unchanged. pm_tid and plat_tid were two ids for one thread, each with its own TLS cache. plat_tid is now the only one: gettid on Linux, pthread_threadid_np on Apple systems and pthread_getthreadid_np on FreeBSD (pthread_self before), kept in the TLS slot of the lock bookkeeping. The Windows timer-period request is now plat_timer_1ms, as the condition variables need it. Verified: the full build on FreeBSD and ctest (815 of 818, the same three failures as main: 3ds_memory_adopt, android_bionic, android_arm32). The Windows half is moved text, but no Windows build was run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>Each driver parsed --update, --data, --seconds and the rest by hand, with different forms (only --opt=VALUE here, only --opt VALUE there), different messages and different help text. common/utilities/cli.{h,c} holds one table of the common options: long name, short letter, older name (--recomp, --repo, --patch), value, and help line. snx_cli_match takes both value forms, says an older name once through snx_option_renamed, and refuses a common option a driver does not take in one wording. SnxCli collects a run's options (--update repeats), snx_cli_help prints the lines in one column, and snx_cli_joined joins --opt VALUE into --opt=VALUE for a reader of the = form. Verified with the new cli ctest (forms, letters, older names said once, refusals, repeated --update, joined arguments, help lines the same for each mask). No Windows toolchain here, so the Windows build is not checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>The file calls were in four places: plat_pread, the read-only maps and plat_replace_file in plat_posix.c, plat_win32.c and plat_sdl.c, the shared files and plat_mkdir in common/filesystem/host.c, the Android bridge's host_pread and common.c's snx_replace. Now file.c has plat_pread_fd (the descriptor form), plat_pread on a FILE*, plat_is_dir, plat_mkdir and snx_host_wpath, file_map.c the PFileMap maps, file_share.c the shared files and file_replace.c plat_replace_file. Each group is an object file of its own, as the tests stub them by group. common/filesystem/host.{c,h} are gone, and their includers include file.h, or thread.h for PMutex alone. plat_pread_fd keeps host_pread's errno (EBADF and EIO on Windows, pread's elsewhere), and sets EINVAL for a range past INT64_MAX, where plat_pread returned -1 with no reason. plat_is_dir and plat_replace_file no longer go through SDL: stat or GetFileAttributesW, and rename or MoveFileExW. snx_replace and plat_replace_file differed in two ways. snx_replace asked MoveFileExW for MOVEFILE_WRITE_THROUGH, and plat_replace_file did not: the merged one keeps it, as each caller replaces a file or folder that must be whole on the disk before the call returns (saves, settings, caches, compiled modules). plat_replace_file deleted its source on failure, and snx_replace kept it: the merged one keeps it, because the 3DS driver moves whole folders of saves with it, which a failed move must not delete. The NX callers that relied on the deletion (the pipeline cache and the save state writer) now remove their temporary file themselves, as the settings, the hints and the shader files already did. Verified on FreeBSD 15: the RelWithDebInfo build and ctest, where only the known 3ds_memory_adopt, android_bionic and android_arm32 fail. The Windows code was moved, not built (no cross toolchain here), and Linux and Apple were not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>A 3DS emulator app's library has R_AARCH64_TLSDESC relocations, which the linker refused ("unknown relocation type 1031"). It also had already put the library in the global group, so the app's later lookups found the symbols of a library whose code the JIT did not have and whose constructors never ran: the first call of one jumped to "no code". - Each library's PT_TLS is a module of dynamic TLS, as bionic gives a dlopen'd library's. Each thread gets the module's block (its .tdata, then zeros) the first time it reaches it. - TLSDESC descriptors point at an HLE resolver that gives the variable's offset from the thread pointer in x0 alone, as the ABI asks. DTPMOD64/DTPREL64 and __tls_get_addr give the general dynamic model. - A library whose relocations fail leaves the global group again. Verified: the app's library loads and its constructors run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>An app's theme inherits from the framework's themes on a device, which the tables lack, so a framework attribute that the app's own styles never set had no value: Material's BottomNavigationView, whose styles leave android:textAppearance to the theme, refused to inflate ("This component requires that you specify a valid TextAppearance attribute"). When the app's theme lacks textAppearance or textAppearanceSmall, Medium or Large, the theme gives the framework's TextAppearance style IDs, as Theme.DeviceDefault's do. Verified with an emulator app: its main screen with a bottom navigation bar inflates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>- DocumentsContract's createDocument, renameDocument, deleteDocument, removeDocument, copyDocument and moveDocument call the document's provider through the content resolver, and DocumentsProvider answers them, as Android's do: an app's DocumentFile.createDirectory made nothing before. - The shared storage's documents ("primary:<path>") are the files under Environment.getExternalStorageDirectory(), as Android's: an app that turns a tree into a path (the primary volume's folder, then the path) finds the same files. A folder --pick gives that is elsewhere on the host is linked into the shared storage by its name, as if the user had picked it there. - The external storage provider copies, moves and removes documents, and deletes a folder with what it holds. Verified with an emulator app that keeps its user folder in a picked tree: it makes its folders there, reads its keys from it and lists the game in it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>A drawable turned by its level, from fromDegrees at level 0 to toDegrees at 10000, about its pivot (a fraction of the bounds or a dimension), as Android's. An emulator app's navigation menu icon is a <rotate>, which failed to inflate ("invalid drawable tag rotate"). Verified with the app: its menu inflates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>One game's vertex program was refused with "global address whose base no constant buffer holds". It loads a 64-bit pointer from a table that c[0][0x110] holds, one for each vertex, and stores {0, 1, 0, 0, 0} through it: the 20 bytes of an indexed indirect draw, which it resets. No constant buffer word holds the pointer, so track_globals gives the access no buffer. The Maxwell translator takes such an access as Eden's SPIR-V backend does (shader_global_dynamic): it compares the 64-bit address with each buffer that the program binds, and accesses the first that holds it. A load from none reads 0, and a store there is dropped. global_dynamic does the same for an SM86 LDG or STG whose address register an LDG, LDS or LDL wrote. Each buffer is the 16 MiB that the renderer binds from its address, and the comparison takes the whole 64-bit address. A compute program's store logs its bytes for the buffer that takes it, as the tracked path does. An address that the flow loses on other paths is still refused. Verified: snx-sm86 translates the game's program, and spirv-val passes it. The new test_global_pointer in shader_sm86 runs three pointers from a buffer on the CPU evaluator: one inside it, one below it and one 4 GiB past it with the same low word of the difference. The first takes the store and the load, the others neither. A comparison of only the low words, and a load from none that keeps the register, each make it fail. Full ctest -j1 passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 45062622095f9d0b5f2eab071cf8d715d3be587a)A mesh pipeline must not declare more outputs than the device's maxMeshOutputMemorySize. The Vulkan spec ("Mesh Shader Output") counts four scalars for each output location, per vertex in granules of meshOutputPerVertexGranularity and per primitive in granules of meshOutputPerPrimitiveGranularity, and not the triangle indices. sm86_mesh_output_bytes counts the locations that the translation declares. The renderer logs the count of each mesh program, and a program past the limit fails with the reason, so its draws are skipped with a log line. Verified: in one game's menu, no mesh program goes past the limit of an RTX 2080 (32768 bytes). The largest, of 256 vertices and 192 primitives, needs 24576 bytes, the others 1536 to 5120. shader_sm86 compares the count with the Output variables of a translated program, and vulkan_headless_mesh checks that a program of 256 vertices with all 32 generics fails. A count of 3 scalars for each location makes the first test fail. The full ctest passes but for the two UMA tests that fail on feat/ounce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1d81338e5f4ffda35495fdeec0451209480f104f)An IPA at the centroid of a fragment input that no earlier stage writes (in_unfed_mask) emitted OpDecorate %0 Centroid: such an input is a constant and has no variable, so the decoration named id 0. spirv-val and the validation layer refuse the module ("Id is 0"), and the driver got it anyway. One menu run under the layer made 1343 modules, 10 of them with that decoration, and the layer logged 9 such errors. The decoration now goes only to an input that has a variable. Verified: shader_sm86 translates an unfed input read at the centroid and at the default place, finds no decoration of id 0, and passes spirv-val. A 110 s menu run under the validation layer logged no "Id is 0" error. The full ctest passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2b9fbe81bf2bf9d57a2b1acb8e0326f4423e22fe) Rebased-by: AO55 <AO55@reden-emu.dev>At each submit the mirror copied every page its programs had written, bytes and mask, into host memory, and once the submit was done landed the logged bytes into guest memory and the shadow. With a submit after each such dispatch, harvesting and landing took most of the GPU thread's time per dispatch in one game's menus, for pages the CPU mostly never read again. Where the host traps guest memory: - The harvest lists the written pages with their masks only, a ninth of an entry. Once the submit is done they are queued, still pending. - At the next point where nothing is being recorded (vk_mirror_publish: before the releases those submits gate are performed, at the end of a GPU job, before a CPU read of them) each page's changes by the CPU since it was brought up to date, the bytes its programs didn't log, are copied into the mirror (a byte both wrote keeps the program's), that copy is submitted, and the runs of pages are published over guest memory with the mirror as their owner, a borrowed resource. - The CPU's first touch of a page lands it whole, which is exact as none of its bytes can change in between; the landing lets go of the page's shadow (vk_mirror_landed). A CPU write after publishing now comes after the program's. - Binding skips the pages the mirror owns, and copies other owners' in sixteen at a time, however many there are. - gmem_ptr lands pending results before it takes the bytes from their owners: landing could publish them again, and the CPU then wrote a GPU-owned page. Without traps the pass copies the bytes and they land as before. The first version (f26cb868c, reverted) took the pages in and published them where results land. That runs inside a frame's recycling, which begins a command buffer: the copies began it again there, the recycling then reset its pool and descriptor pools and began it once more, and draws crashed in the driver binding descriptor sets ("vkBeginCommandBuffer while still recording" under validation). Nothing is recorded or submitted inside a landing, a recycling or a submission now (vk_rec_unsafe). Publishing only at the end of GPU jobs left the game reading stale untrapped pages after a release, and stalled it 15 seconds in two of three menu runs. Verified: vulkan_mirror checks that after the submit a written page is pending and not yet owned (publishing inside the landing fails it), that once published it is owned by the mirror and not landed, that binding it again keeps the program's bytes, and that the CPU's touch lands them with the byte the CPU wrote beside them meanwhile (skipping that copy fails it). vulkan_release checks that no release is performed between a landing and the publishing (removing it fails it). Under the validation layer the game's menus report no new message over 225 seconds; three menu runs had no crash and no stall. All tests pass but the two forced-UMA ones that fail on main too. (cherry picked from commit 6072680150841b1901b8e1636fe160fc95f1d956) Rebased-by: AO55 <AO55@reden-emu.dev>e24ae8958 turned on GPU-assisted validation each time the validation layer was on. The layer then logs the settings that it changes ("validation is adjusting settings"), and vulkan_headless_vic_validation fails on each "vulkan validation:" line. That test also took 2.86 s in place of 0.33 s. GPU-assisted validation is now the value 2 of vk_validation, and 1 is the layer alone, as on main. Verified: the macOS build and its ctest pass. The Linux validation test runs only in CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 10e8ea676e609273dc86d987d855ee2dad7a11f9) Rebased-by: AO55 <AO55@reden-emu.dev>DrawMeshTasksIndirect's draws were skipped. One game draws about 30 a frame that way in its first 3D scene. Its record is taken as {count, first}, as NV_mesh_shader lays out its own: the GPU writes it, so the CPU can't check it. The renderer copies the record on the GPU into a staging {count, 1, 1} for vkCmdDrawMeshTasksIndirectEXT, and its first into a uniform buffer bound at SNX_VKB_RS. A task program, or a mesh program without one, reads its first workgroup from there when the push flag SH_PUSH_GPU_MESH_FIRST is set. The copy ends the render pass of each indirect draw. docs/platforms/nx/ounce.md records that on a Radeon 890M the mesh draws run without a fault, and that RADV_DEBUG=hang, which waits for each draw, stalls the task programs' submissions and times out a ring by itself. Run on a Radeon 890M host before the port to main, with mesh_draws=1, from a save of that game: about 12 runs without a fault, the scene's indirect draws drawn, and a validation run with no message from mesh draws. test_mesh and test_task check a first workgroup read from the uniform buffer. On main the 77 shader, Vulkan, mesh, settings and docs tests pass with this change and the ones before it, the Vulkan ones on RADV. With the whole series on main the full suite passes but nvn_game_benchmark, which fails on main alone too: its script needs Python 3.12, and this host has 3.11. The 38 Vulkan tests pass on RADV and on lavapipe, vulkan_headless on lavapipe at a rerun: its first run there failed the check that a descriptor heap transition stays in one command buffer, which a periodic submit can split, and five runs after it passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit acef4c060bc3b39512070b9fc402559350f0a6cd)maxwell_opcodes.def becomes sm53.decode: the same 16-bit patterns, read by tools/decodetree (-w 16 on the word's top 16 bits). The rows that overlap go in { } groups, the narrower first, and decodetree refuses any other overlap, as shaderdec did. CCTL's second encoding is the CCTLT pattern. The opcode list (enum order, display names) is SM53_OPS in translator.h, and decode.c makes a trans_ function of each entry. Every word's top 16 bits, with 40 sets of low bits each, decode to the same opcode and name as before. shader_decode now checks that every opcode has an encoding and that the groups pick the narrower pattern. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>A kind is 32 bits: a punctuator's characters, packed from the first in the low byte (TK_LPAREN is '(', TK_ADD_ASSIGN '+' | '=' << 8, made from the bytes one at a time, so the same number on a big-endian host); keywords from 0x80, the built-in types from 0x100; a name the preprocessor let through is GLSLC_TK_NAME | its index, so two tokens name the same thing when their kinds are equal, and the token carries no name of its own. The lexer scans a copy of the source with zero bytes after it, which lets it read ahead without a bound check, and classifies bytes with a table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>59b33a57) into inkypaint 6a3bdc62b2