diff --git a/docs/articles/vfs.md b/docs/articles/vfs.md index ed18e908e..5524b727a 100644 --- a/docs/articles/vfs.md +++ b/docs/articles/vfs.md @@ -2,12 +2,13 @@ ## Introduction -`eepp-vfs` is an optional static module that gives applications one asynchronous, URI-based API +`eepp-vfs` is an optional module that gives applications one asynchronous, URI-based API for local files, packed resources and remote servers: | Scheme | Provider | Notes | | --- | --- | --- | | `file://` | `LocalProvider` | Native filesystem (POSIX and Win32). | +| `zip://` | `ZipProvider` | Read-only browsing and streaming reads of individual local ZIP archives. | | `res://` | `ResourceProvider` | Read-only view of every open `Pack` (`System::PackFileIndex`). | | `sftp://` | `SFTPProvider` | Built by default (`--without-vfs-sftp` disables it), libssh2 based, honors `~/.ssh/config`. | | `mem://` | `MemoryProvider` | In-memory filesystem, mainly for tests. | @@ -26,6 +27,12 @@ are useful on their own: Link the module with `eepp_module_vfs_add()` in Premake and include ``. The UI model lives in `` and is not part of the aggregate header. +The default is `eepp-vfs-static`; generate with `--with-shared-vfs` to build and link `eepp-vfs` +as a shared library. Public types and non-inline functions use `EE_VFS_API`, controlled by +`EE_VFS_EXPORTS` for the DLL and `EE_VFS_STATIC` for static builds, independently of core eepp +exports. The Premake helper sets the consumer flags and dependencies. The MinGW helper accepts +`--with-shared-vfs` as well. Combining it with `--with-static-eepp` is rejected: a shared VFS +must use the same eepp DLL as the application, including its resource and UI singletons. ## Tasks, results and threads @@ -256,6 +263,61 @@ on the server's `PATH`, or set `SFTPProviderConfig::watchAgentCommand`. The eepp same connection. - The efsw-agent is not deployed automatically. +## ZIP archives (`zip://`) + +`ZipProvider` is registered by default. Build archive locations with `makeURI()` rather than +concatenating paths: the backing file URI and the path inside the archive are separate identities. +Locations use `zip:///path/archive.zip!/member`. The host, when present, identifies a local UNC +archive. Literal `!` characters in the backing path are doubled, so a directory named `packages!` +uses `packages!!/` before the archive boundary. Member names use ordinary URI escaping. +`withAppendedSegment()` appends a member; eefm handles crossing the archive root with Up. + +```cpp +URI archive = URI::fromFSPath( "/home/me/downloads/package.zip" ); +URI root = ZipProvider::makeURI( archive ); +URI folder = root.withAppendedSegment( "docs" ); + +vfs.listAll( folder ).then( ui, []( VFSResult>&& entries ) { ... } ); +vfs.readAll( folder.withAppendedSegment( "README.txt" ) ); +vfs.copy( folder, URI::fromFSPath( "/home/me/extracted-docs" ) ); +``` + +Directories exist both for explicit directory entries (including empty directories) and implicitly +for the parents of files. Listings include uncompressed file sizes and entry timestamps. Permission bits are unspecified +(the archive API does not expose them), so copies receive the destination filesystem's defaults. Browsing +never registers an archive in `PackRegistry` or `PackFileIndex`, so archive contents do not become +application resources. Opening a missing archive reports `NotFound` and never creates a file. + +The provider is read-only: creating, replacing, renaming or deleting entries returns `ReadOnly`. +Files stream through independent readers without extracting the entire entry into memory. +Sequential reads retain the decompression position; seeking backwards replays the compressed +prefix with bounded scratch memory. `Seek` is advertised; `RandomAccess` is not. CRC failures are +reported as `IO`, including when the final read requests exactly the remaining file size. + +Changes to the backing file invalidate its cached index for new operations. Open handles keep +their original reader and metadata; modifying an archive in place can cause their reads to fail. +The normal VFS polling backend detects archive changes. Archives on network mounts use the local +I/O scheduler's mount isolation and timeouts. + +Only `file://` backing archives are supported initially. Password-protected entries and unsupported +compression methods report `Unsupported`. ZIP symlink attributes are not followed: their stored +data is exposed as a regular file. Unsafe names (absolute paths, traversal components, backslashes +or drive/stream colons) are excluded. Indexes are limited to 262,144 entries including implicit +directories, with at most eight cached indexes; existing handles can retain older indexes. + +In eefm, Enter or double-click on a local `.zip` file enters it. Files inside open through the usual +application flow, and copying entries out uses VFS transfers. Up at the archive root returns to its +containing folder and selects the archive. History and saved tabs retain archive locations. The +location bar displays a readable address such as +`zip:///home/me/downloads/package.zip!/docs`, which can also be typed or pasted there. + +For lower-level archive access, construct `Zip archive(false)` to opt out of the resource registry +and call `archive.open(path, Zip::OpenMode::ReadOnly)`. `entryCount()`, `entry()` and `openStream()` +provide metadata and streaming access. Streams borrow the archive and must be destroyed before it; +callers sharing an archive across threads must serialize access. `open()` never creates a missing +file; writing starts explicitly with `create()`. The default constructor retains resource-pack +registration for the engine and other existing pack consumers. + ## Resources (`res://`) `System::PackFileIndex` (formerly `System::VirtualFileSystem`) indexes the files of every open `Pack`. `res:///fonts/a.ttf` maps to the pack path diff --git a/include/eepp/system/iostreamzip.hpp b/include/eepp/system/iostreamzip.hpp index 6b2fffd04..f92bca916 100644 --- a/include/eepp/system/iostreamzip.hpp +++ b/include/eepp/system/iostreamzip.hpp @@ -16,12 +16,14 @@ class EE_API IOStreamZip : public IOStream { public: static IOStreamZip* New( Zip* pack, const std::string& path ); - /** @brief Open a file from a zip file + /** @brief Open a file from a zip file. The pack must outlive the stream. ** @param pack Pack to open from path ** @param path Path of the file in the pack file **/ IOStreamZip( Zip* pack, const std::string& path ); + IOStreamZip( Zip* pack, Uint64 index ); + virtual ~IOStreamZip(); ios_size read( char* data, ios_size size ); @@ -37,9 +39,10 @@ class EE_API IOStreamZip : public IOStream { bool isOpen(); protected: - std::string mPath; struct zip* mZip; struct zip_file* mFile; + Uint64 mIndex; + ios_size mSize; ios_size mPos; }; diff --git a/include/eepp/system/pack.hpp b/include/eepp/system/pack.hpp index 261157e7a..96c5969d2 100644 --- a/include/eepp/system/pack.hpp +++ b/include/eepp/system/pack.hpp @@ -14,6 +14,9 @@ class EE_API Pack : protected Mutex { public: Pack(); + /** Standalone archives can opt out of PackRegistry and PackFileIndex. */ + explicit Pack( bool registerResource ); + virtual ~Pack(); /** Creates a new pack file */ @@ -81,6 +84,7 @@ class EE_API Pack : protected Mutex { protected: bool mIsOpen; + bool mRegisterResource; void onPackOpened(); diff --git a/include/eepp/system/zip.hpp b/include/eepp/system/zip.hpp index a0b04b6df..e0c8f43f8 100644 --- a/include/eepp/system/zip.hpp +++ b/include/eepp/system/zip.hpp @@ -2,26 +2,66 @@ #define EE_SYSTEMCZIP_HPP #include +#include struct zip; namespace EE { namespace System { -/** @brief Zip files package manager. */ +/** ZIP archive reader/writer and optional resource pack. Use Zip(false) and OpenMode::ReadOnly + * for standalone browsing. Entry streams borrow the archive and must be destroyed before it. + * Serialize access to an archive and its streams when sharing them between threads. */ class EE_API Zip : public Pack { public: static Zip* New(); + struct Entry { + std::string name; + Uint64 size{ 0 }; + Int64 modified{ 0 }; + Uint32 crc{ 0 }; + }; + + enum class OpenMode : Uint8 { ReadOnly, ReadWrite }; + + enum class Error : Uint8 { + None, + NotFound, + PermissionDenied, + InvalidArchive, + Unsupported, + ReadOnly, + IO + }; + Zip(); + explicit Zip( bool registerResource ); + ~Zip(); /** Creates a new pack file */ bool create( const std::string& path ); - /** Open a pack file */ + /** Opens an existing archive for reading and writing. Missing files are never created; + * use create() explicitly. */ bool open( const std::string& path ); + /** ReadOnly supports Unicode native paths and cannot modify or create an archive. */ + bool open( const std::string& path, OpenMode mode ); + + Uint64 entryCount() const; + + bool entry( Uint64 index, Entry& result ); + + /** Opens an entry without extracting it into memory. Backward seeks replay the prefix; + * scratch space is bounded. Reads/seeks return -1 on error, including CRC failures. */ + std::unique_ptr openStream( Uint64 index ); + + Error error() const { return mError; } + + const std::string& errorMessage() const { return mErrorMessage; } + /** Close the pack file */ bool close(); @@ -80,8 +120,20 @@ class EE_API Zip : public Pack { struct zip* mZip; std::string mZipPath; + std::string mErrorMessage; + OpenMode mOpenMode{ OpenMode::ReadWrite }; + Error mError{ Error::None }; struct zip* getZip(); + + private: + bool openReadOnly( const std::string& path ); + + void setError( int code, int systemCode = 0 ); + + bool writable(); + + bool extractStream( IOStream& stream, Uint8* data, size_t size ); }; }} // namespace EE::System diff --git a/premake4.lua b/premake4.lua index 0eb85408a..08dd5e008 100644 --- a/premake4.lua +++ b/premake4.lua @@ -193,6 +193,10 @@ newoption { description = "Set the shared data directory (default: /usr/share/ecode)", } newoption { trigger = "with-static-cpp", description = "Builds statically libstdc++" } +newoption { trigger = "with-shared-vfs", description = "Builds and links the VFS module as a shared library instead of a static library." } +if _OPTIONS["with-static-eepp"] and _OPTIONS["with-shared-vfs"] then + error("--with-shared-vfs requires eepp-shared; it cannot be combined with --with-static-eepp") +end newoption { trigger = "without-vfs-sftp", description = "Disables sftp:// support in the eepp-vfs module (otherwise the vendored libssh2 is built)." } newoption { trigger = "with-lto", description = "Enables Link-Time Optimization for release builds." } @@ -1085,8 +1089,8 @@ function vfs_sftp_enabled() return _OPTIONS["without-vfs-sftp"] == nil and not os.is_real("emscripten") end -function eepp_module_vfs_add() - links { "eepp-vfs-static", "efsw-static" } +function eepp_module_vfs_dependencies() + links { "efsw-static" } includedirs { "src/modules/vfs/include/", "src/thirdparty/efsw/include" } -- The ssh_config evaluator (always built) resolves host names through Winsock. if os.is_real("windows") or os.is_real("mingw32") or os.is_real("mingw64") then @@ -1111,6 +1115,58 @@ function eepp_module_vfs_add() end end +function eepp_module_vfs_add() + includedirs { "src/modules/vfs/include/", "src/thirdparty/efsw/include" } + if vfs_sftp_enabled() then + defines { "EE_VFS_SFTP" } + end + if _OPTIONS["with-shared-vfs"] then + links { "eepp-vfs" } + else + links { "eepp-vfs-static" } + defines { "EE_VFS_STATIC" } + eepp_module_vfs_dependencies() + end +end + +function build_vfs_module( shared ) + local name = shared and "eepp-vfs" or "eepp-vfs-static" + project(name) + kind(shared and "SharedLib" or "StaticLib") + language "C++" + set_targetdir("libs/" .. os.get_real() .. "/") + includedirs { "include", "src/modules/vfs/include/", "src/modules/vfs/src/", "src/thirdparty/efsw/include" } + files { "src/modules/vfs/src/**.cpp" } + defines { shared and "EE_VFS_EXPORTS" or "EE_VFS_STATIC" } + if shared then + eepp_module_vfs_dependencies() + links { "eepp-shared" } + end + if vfs_sftp_enabled() then + defines { "EE_VFS_SFTP" } + includedirs { "src/thirdparty/libssh2/include" } + else + excludes { "src/modules/vfs/src/eepp/vfs/sftp/**.cpp" } + end + if _OPTIONS["with-static-eepp"] then + defines { "EE_STATIC" } + end + if not is_vs() then + buildoptions{ "-std=c++20" } + else + buildoptions{ "/std:c++20" } + end + build_base_cpp_configuration( name ) + + if shared then + postsymlinklib("../libs/" .. os.get_real() .. "/", "../../bin/", "eepp-vfs" ) + postsymlinklib("../libs/" .. os.get_real() .. "/", "../../bin/unit_tests/", "eepp-vfs" ) + if os.is_real("mingw32") or os.is_real("mingw64") or (os.is("windows") and not is_vs()) then + links { "winpthread" } + end + end +end + function eepp_module_backward_add( ecode_mode ) files { "src/modules/backward/src/backward.cpp" } includedirs { "src/thirdparty" } @@ -1761,27 +1817,10 @@ solution "eepp" end build_base_cpp_configuration( "efsw-agent" ) - project "eepp-vfs-static" - kind "StaticLib" - language "C++" - set_targetdir("libs/" .. os.get_real() .. "/") - includedirs { "include", "src/modules/vfs/include/", "src/modules/vfs/src/", "src/thirdparty/efsw/include" } - files { "src/modules/vfs/src/**.cpp" } - if vfs_sftp_enabled() then - defines { "EE_VFS_SFTP" } - includedirs { "src/thirdparty/libssh2/include" } - else - excludes { "src/modules/vfs/src/eepp/vfs/sftp/**.cpp" } - end - if _OPTIONS["with-static-eepp"] then - defines { "EE_STATIC" } - end - if not is_vs() then - buildoptions{ "-std=c++20" } - else - buildoptions{ "/std:c++20" } - end - build_base_cpp_configuration( "eepp-vfs" ) + build_vfs_module( false ) + if _OPTIONS["with-shared-vfs"] then + build_vfs_module( true ) + end project "languages-syntax-highlighting-static" kind "StaticLib" @@ -2297,6 +2336,7 @@ solution "eepp" links { "util" } end files { "src/tests/unit_tests/*.cpp", + "src/tools/eefm/location.cpp", "src/tools/eproc/process_info.cpp", "src/tools/eproc/process_model.cpp", "src/tools/eproc/process_table_state.cpp", @@ -2313,6 +2353,7 @@ solution "eepp" links { "Cocoa.framework" } end eepp_module_vfs_add() + links { "efsw-static" } eepp_module_backward_add( false ) configuration { "linux" } diff --git a/premake5.lua b/premake5.lua index 487319eef..a2552f7a7 100644 --- a/premake5.lua +++ b/premake5.lua @@ -51,6 +51,10 @@ newoption { description = "Set the shared data directory", } newoption { trigger = "with-static-cpp", description = "Builds statically libstdc++" } +newoption { trigger = "with-shared-vfs", description = "Builds and links the VFS module as a shared library instead of a static library." } +if _OPTIONS["with-static-eepp"] and _OPTIONS["with-shared-vfs"] then + error("--with-shared-vfs requires eepp-shared; it cannot be combined with --with-static-eepp") +end newoption { trigger = "without-vfs-sftp", description = "Disables sftp:// support in the eepp-vfs module (otherwise the vendored libssh2 is built)." } newoption { trigger = "with-lto", description = "Enables Link-Time Optimization for release builds." } @@ -1006,8 +1010,8 @@ function vfs_sftp_enabled() return _OPTIONS["without-vfs-sftp"] == nil and not os.istarget("emscripten") end -function eepp_module_vfs_add() - links { "eepp-vfs-static", "efsw-static" } +function eepp_module_vfs_dependencies() + links { "efsw-static" } incdirs { "src/modules/vfs/include/", "src/thirdparty/efsw/include" } -- The ssh_config evaluator (always built) resolves host names through Winsock. filter "system:windows" @@ -1033,6 +1037,57 @@ function eepp_module_vfs_add() end end +function eepp_module_vfs_add() + incdirs { "src/modules/vfs/include/", "src/thirdparty/efsw/include" } + if vfs_sftp_enabled() then + defines { "EE_VFS_SFTP" } + end + if _OPTIONS["with-shared-vfs"] then + links { "eepp-vfs" } + else + links { "eepp-vfs-static" } + defines { "EE_VFS_STATIC" } + eepp_module_vfs_dependencies() + end +end + +function build_vfs_module( shared ) + local name = shared and "eepp-vfs" or "eepp-vfs-static" + project(name) + kind(shared and "SharedLib" or "StaticLib") + language "C++" + cppdialect "C++20" + incdirs { "include", "src/modules/vfs/include/", "src/modules/vfs/src/", "src/thirdparty/efsw/include" } + files { "src/modules/vfs/src/**.cpp" } + defines { shared and "EE_VFS_EXPORTS" or "EE_VFS_STATIC" } + if shared then + eepp_module_vfs_dependencies() + links { "eepp-shared" } + end + if vfs_sftp_enabled() then + defines { "EE_VFS_SFTP" } + incdirs { "src/thirdparty/libssh2/include" } + else + excludes { "src/modules/vfs/src/eepp/vfs/sftp/**.cpp" } + end + if _OPTIONS["with-static-eepp"] then + defines { "EE_STATIC" } + end + build_base_cpp_configuration( name ) + target_dir_lib("") + filter "action:not vs*" + buildoptions { "-Wall" } + filter { "action:export-compile-commands", "system:macosx" } + buildoptions { "-std=c++20" } + + if shared then + postsymlinklib_arch( "eepp-vfs" ) + filter { "system:windows", "action:not vs*" } + links { "winpthread" } + filter {} + end +end + function eepp_module_backward_add( ecode_mode ) files { "src/modules/backward/src/backward.cpp" } incdirs { "src/thirdparty" } @@ -1202,7 +1257,7 @@ function postsymlinklib_arch(name) postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/arm64/", _MAIN_SCRIPT_DIR .. "/bin/", name, "options:arch=arm64" ) postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/arm64/", _MAIN_SCRIPT_DIR .. "/bin/", name, "options:arch=aarch64" ) postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/universal/", _MAIN_SCRIPT_DIR .. "/bin/", name, "architecture:universal" ) - if name == "eepp" then + if name == "eepp" or name == "eepp-vfs" then postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/x86/", _MAIN_SCRIPT_DIR .. "/bin/unit_tests/", name, "architecture:x86" ) postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/x86_64/", _MAIN_SCRIPT_DIR .. "/bin/unit_tests/", name, "architecture:x86_64" ) postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/arm/", _MAIN_SCRIPT_DIR .. "/bin/unit_tests/", name, "architecture:ARM" ) @@ -1781,27 +1836,10 @@ workspace "eepp" links { "CoreFoundation.framework", "CoreServices.framework" } filter {} - project "eepp-vfs-static" - kind "StaticLib" - language "C++" - cppdialect "C++20" - incdirs { "include", "src/modules/vfs/include/", "src/modules/vfs/src/", "src/thirdparty/efsw/include" } - files { "src/modules/vfs/src/**.cpp" } - if vfs_sftp_enabled() then - defines { "EE_VFS_SFTP" } - incdirs { "src/thirdparty/libssh2/include" } - else - excludes { "src/modules/vfs/src/eepp/vfs/sftp/**.cpp" } - end - if _OPTIONS["with-static-eepp"] then - defines { "EE_STATIC" } - end - build_base_cpp_configuration( "eepp-vfs" ) - target_dir_lib("") - filter "action:not vs*" - buildoptions { "-Wall" } - filter { "action:export-compile-commands", "system:macosx" } - buildoptions { "-std=c++20" } + build_vfs_module( false ) + if _OPTIONS["with-shared-vfs"] then + build_vfs_module( true ) + end project "languages-syntax-highlighting-static" kind "StaticLib" @@ -2289,6 +2327,7 @@ workspace "eepp" incdirs { "src/modules/eterm/include/", "src/thirdparty", "src/modules/vfs/src/" } language "C++" files { "src/tests/unit_tests/*.cpp", + "src/tools/eefm/location.cpp", "src/tools/eproc/process_info.cpp", "src/tools/eproc/process_model.cpp", "src/tools/eproc/process_table_state.cpp", @@ -2317,6 +2356,7 @@ workspace "eepp" links { "util" } filter {} eepp_module_vfs_add() + links { "efsw-static" } eepp_module_backward_add( false ) filter { "system:linux" } diff --git a/projects/mingw32/make.sh b/projects/mingw32/make.sh index aa58e6915..3065cf694 100755 --- a/projects/mingw32/make.sh +++ b/projects/mingw32/make.sh @@ -3,6 +3,7 @@ cd "$(dirname "$0")" || exit ARCH=32 BACKEND=sdl2 +VFS_SHARED_ARG= MAKE_ARGS=() for i in "$@"; do case $i in @@ -10,6 +11,9 @@ for i in "$@"; do CONFIG="${i#*=}" MAKE_ARGS+=("$i") ;; + --with-shared-vfs) + VFS_SHARED_ARG="--with-shared-vfs" + ;; --backend=*) BACKEND="${i#*=}" ;; @@ -78,7 +82,7 @@ detect_backend_name() fi } -PREMAKE5_ARGS="--file=../../premake5.lua --os=windows --cc=gcc --windows-mingw-build --with-backend=$(detect_backend_arg) $PREMAKE5_ARCH gmake" +PREMAKE5_ARGS="--file=../../premake5.lua --os=windows --cc=gcc --windows-mingw-build --with-backend=$(detect_backend_arg) $PREMAKE5_ARCH $VFS_SHARED_ARG gmake" if command -v premake5 &> /dev/null then diff --git a/src/eepp/system/iostreamzip.cpp b/src/eepp/system/iostreamzip.cpp index 119124f14..c76d87363 100644 --- a/src/eepp/system/iostreamzip.cpp +++ b/src/eepp/system/iostreamzip.cpp @@ -1,7 +1,10 @@ #include + +#include +#include #include #include -#include +#include namespace EE { namespace System { @@ -10,61 +13,62 @@ IOStreamZip* IOStreamZip::New( Zip* pack, const std::string& path ) { } IOStreamZip::IOStreamZip( Zip* pack, const std::string& path ) : - mPath( path ), mZip( pack->getZip() ), mFile( NULL ), mPos( 0 ) { - struct zip_stat zs; - int err = zip_stat( mZip, path.c_str(), 0, &zs ); + IOStreamZip( pack, static_cast( pack->exists( path ) ) ) {} - if ( !err ) { - mFile = zip_fopen_index( mZip, zs.index, 0 ); +IOStreamZip::IOStreamZip( Zip* pack, Uint64 index ) : + mZip( pack->getZip() ), mFile( nullptr ), mIndex( index ), mSize( 0 ), mPos( 0 ) { + struct zip_stat info; + zip_stat_init( &info ); + if ( mZip && zip_stat_index( mZip, index, 0, &info ) == 0 && + info.size <= static_cast( std::numeric_limits::max() ) ) { + mSize = static_cast( info.size ); + mFile = zip_fopen_index( mZip, index, 0 ); } } IOStreamZip::~IOStreamZip() { - if ( isOpen() ) { + if ( mFile ) zip_fclose( mFile ); - } } ios_size IOStreamZip::read( char* data, ios_size size ) { - int res = -1; - - if ( isOpen() ) { - res = zip_fread( mFile, reinterpret_cast( &data[0] ), size ); - - if ( -1 != res ) { - mPos += size; + if ( !mFile || size < 0 ) + return -1; + auto count = zip_fread( mFile, data, static_cast( size ) ); + if ( count > 0 ) { + mPos += count; + // libzip validates the CRC only on the read past EOF. Validate before reporting the + // last bytes: offset-based callers may never request another read on this stream. + if ( mPos >= mSize ) { + char extra; + if ( mPos != mSize || zip_fread( mFile, &extra, 1 ) != 0 ) + return -1; } } - - return -1 != res ? res : 0; + return count; } -ios_size IOStreamZip::write( const char* data, ios_size size ) { - return 0; +ios_size IOStreamZip::write( const char*, ios_size ) { + return -1; } ios_size IOStreamZip::seek( ios_size position ) { - if ( isOpen() && mPos != position ) { + if ( !mFile || position < 0 || position > mSize ) + return -1; + if ( position < mPos ) { zip_fclose( mFile ); - - struct zip_stat zs; - int err = zip_stat( mZip, mPath.c_str(), 0, &zs ); - - if ( !err ) { - mFile = zip_fopen_index( mZip, zs.index, 0 ); - - if ( 0 != position ) { - ScopedBuffer ptr( position ); - read( (char*)ptr.get(), position ); - } - - mPos = position; - - return position; - } + mFile = zip_fopen_index( mZip, mIndex, 0 ); + mPos = 0; + if ( !mFile ) + return -1; } - - return 0; + std::array scratch; + while ( mPos < position ) { + const auto count = std::min( position - mPos, scratch.size() ); + if ( read( scratch.data(), count ) <= 0 ) + return -1; + } + return mPos; } ios_size IOStreamZip::tell() { @@ -72,13 +76,11 @@ ios_size IOStreamZip::tell() { } ios_size IOStreamZip::getSize() { - struct zip_stat zs; - int err = zip_stat( mZip, mPath.c_str(), 0, &zs ); - return !err ? zs.size : 0; + return mSize; } bool IOStreamZip::isOpen() { - return NULL != mFile; + return mFile != nullptr; } }} // namespace EE::System diff --git a/src/eepp/system/pack.cpp b/src/eepp/system/pack.cpp index 5663caded..798e3a694 100644 --- a/src/eepp/system/pack.cpp +++ b/src/eepp/system/pack.cpp @@ -1,15 +1,19 @@ #include -#include #include +#include namespace EE { namespace System { -Pack::Pack() : Mutex(), mIsOpen( false ) { - PackRegistry::instance()->add( this ); +Pack::Pack() : Pack( true ) {} + +Pack::Pack( bool registerResource ) : mIsOpen( false ), mRegisterResource( registerResource ) { + if ( mRegisterResource ) + PackRegistry::instance()->add( this ); } Pack::~Pack() { - PackRegistry::instance()->remove( this ); + if ( mRegisterResource ) + PackRegistry::instance()->remove( this ); } bool Pack::isOpen() const { @@ -17,11 +21,13 @@ bool Pack::isOpen() const { } void Pack::onPackOpened() { - PackFileIndex::instance()->onResourceAdd( this ); + if ( mRegisterResource ) + PackFileIndex::instance()->onResourceAdd( this ); } void Pack::onPackClosed() { - PackFileIndex::instance()->onResourceRemove( this ); + if ( mRegisterResource ) + PackFileIndex::instance()->onResourceRemove( this ); } }} // namespace EE::System diff --git a/src/eepp/system/zip.cpp b/src/eepp/system/zip.cpp index c952d01f3..aa8c244b8 100644 --- a/src/eepp/system/zip.cpp +++ b/src/eepp/system/zip.cpp @@ -1,8 +1,18 @@ +#include + +#include +#include #include #include -#include +#include #include -#include +#include + +#if EE_PLATFORM == EE_PLATFORM_WIN +#include +#else +#include +#endif namespace EE { namespace System { @@ -10,56 +20,61 @@ Zip* Zip::New() { return eeNew( Zip, () ); } -Zip::Zip() : mZip( NULL ) {} +Zip::Zip() : Zip( true ) {} + +Zip::Zip( bool registerResource ) : Pack( registerResource ), mZip( nullptr ) {} Zip::~Zip() { close(); } bool Zip::create( const std::string& path ) { - if ( !FileSystem::fileExists( path ) ) { - int err; - mZip = zip_open( path.c_str(), ZIP_CREATE, &err ); - - if ( 0 == checkPack() ) { - mZipPath = path; - - mIsOpen = true; - - onPackOpened(); - - return true; - } - } else { - return open( path ); + if ( !close() ) + return false; + mError = Error::None; + mErrorMessage.clear(); + mOpenMode = OpenMode::ReadWrite; + int error = ZIP_ER_OK; + mZip = zip_open( path.c_str(), ZIP_CREATE, &error ); + if ( !mZip ) { + setError( error, errno ); + return false; } - - return false; + mZipPath = path; + mIsOpen = true; + onPackOpened(); + return true; } bool Zip::open( const std::string& path ) { - if ( FileSystem::fileExists( path ) ) { - int err; - mZip = zip_open( path.c_str(), 0, &err ); + return open( path, OpenMode::ReadWrite ); +} - if ( 0 == checkPack() ) { - mZipPath = path; - - mIsOpen = true; - - onPackOpened(); - - return true; - } +bool Zip::open( const std::string& path, OpenMode mode ) { + if ( !close() ) + return false; + mError = Error::None; + mErrorMessage.clear(); + mOpenMode = mode; + if ( mode == OpenMode::ReadOnly ) { + if ( !openReadOnly( path ) ) + return false; } else { - return create( path ); + int error = ZIP_ER_OK; + mZip = zip_open( path.c_str(), 0, &error ); + if ( !mZip ) { + setError( error, errno ); + return false; + } } - - return false; + mZipPath = path; + mIsOpen = true; + onPackOpened(); + return true; } bool Zip::close() { - if ( 0 == checkPack() ) { + if ( mZip ) { // zip_close() writes the pending changes. When that fails the archive is left allocated // (and unchanged on disk): release it and report the failure. bool written = zip_close( mZip ) == 0; @@ -81,42 +96,50 @@ bool Zip::close() { return written; } - return false; + return true; } bool Zip::addFile( const std::string& path, const std::string& inpack ) { ScopedBuffer file; - FileSystem::fileGet( path, file ); + if ( !writable() || !FileSystem::fileGet( path, file ) ) + return false; return addFile( file.get(), file.length(), inpack ); } bool Zip::addFile( const Uint8* data, const Uint32& dataSize, const std::string& inpack ) { + if ( !writable() ) + return false; if ( 0 == checkPack() ) { struct zip_source* zs = zip_source_buffer( mZip, (const void*)data, dataSize, 0 ); if ( NULL != zs ) { Int32 Result = (Int32)zip_add( mZip, inpack.c_str(), zs ); + if ( Result < 0 ) { + zip_source_free( zs ); + return false; + } std::string path = mZipPath; bool written = close(); - open( path ); + bool reopened = open( path ); - if ( -1 != Result && written ) + if ( written && reopened ) return true; - } else - zip_source_free( zs ); + } } return false; } bool Zip::addFile( std::vector& data, const std::string& inpack ) { - return addFile( reinterpret_cast( &data[0] ), (Uint32)data.size(), inpack ); + return addFile( data.data(), (Uint32)data.size(), inpack ); } bool Zip::addFiles( std::map paths ) { + if ( !writable() ) + return false; for ( std::map::iterator itr = paths.begin(); itr != paths.end(); ++itr ) if ( !addFile( itr->first, itr->second ) ) @@ -125,6 +148,8 @@ bool Zip::addFiles( std::map paths ) { } bool Zip::eraseFile( const std::string& path ) { + if ( !writable() ) + return false; std::vector tmpv; tmpv.push_back( path ); @@ -132,6 +157,8 @@ bool Zip::eraseFile( const std::string& path ) { } bool Zip::eraseFiles( const std::vector& paths ) { + if ( !writable() ) + return false; Int32 Ex; Uint32 i = 0; @@ -150,87 +177,53 @@ bool Zip::eraseFiles( const std::vector& paths ) { } bool Zip::extractFile( const std::string& path, const std::string& dest ) { - lock(); - - bool Ret; - ScopedBuffer data; - - Ret = extractFileToMemory( path, data ); - - if ( Ret ) - FileSystem::fileWrite( dest, data.get(), data.length() ); - - unlock(); - - return Ret; + return extractFileToMemory( path, data ) && + FileSystem::fileWrite( dest, data.get(), data.length() ); } bool Zip::extractFileToMemory( const std::string& path, std::vector& data ) { - lock(); - - bool Ret = false; - Int32 Pos = exists( path ); - - if ( 0 == checkPack() && -1 != Pos ) { - - data.clear(); - - struct zip_stat zs; - int err = zip_stat( mZip, path.c_str(), 0, &zs ); - - if ( !err ) { - struct zip_file* zf = zip_fopen_index( mZip, zs.index, 0 ); - - if ( NULL != zf ) { - data.resize( zs.size ); - - int Result = - (Int32)zip_fread( zf, reinterpret_cast( &data[0] ), data.size() ); - - zip_fclose( zf ); - - if ( -1 != Result ) - Ret = true; - } - } - } - - unlock(); - - return Ret; + Lock guard( *this ); + const auto index = exists( path ); + if ( index < 0 ) + return false; + IOStreamZip stream( this, index ); + if ( !stream.isOpen() || static_cast( stream.getSize() ) > data.max_size() ) + return false; + data.resize( static_cast( stream.getSize() ) ); + return extractStream( stream, data.data(), data.size() ); } bool Zip::extractFileToMemory( const std::string& path, ScopedBuffer& data ) { - lock(); + Lock guard( *this ); + const auto index = exists( path ); + if ( index < 0 ) + return false; + IOStreamZip stream( this, index ); + if ( !stream.isOpen() || + static_cast( stream.getSize() ) > std::numeric_limits::max() ) + return false; + data.reset( static_cast( stream.getSize() ) ); + return extractStream( stream, data.get(), data.size() ); +} - bool Ret = false; - Int32 Pos = exists( path ); - Int32 Result = 0; - - if ( 0 == checkPack() && -1 != Pos ) { - struct zip_stat zs; - int err = zip_stat( mZip, path.c_str(), 0, &zs ); - - if ( !err ) { - struct zip_file* zf = zip_fopen_index( mZip, zs.index, 0 ); - - if ( NULL != zf ) { - data.reset( zs.size ); - - Result = (Int32)zip_fread( zf, (void*)data.get(), data.length() ); - - zip_fclose( zf ); - - if ( -1 != Result ) - Ret = true; - } +bool Zip::extractStream( IOStream& stream, Uint8* data, size_t size ) { + size_t done = 0; + while ( done < size ) { + const auto count = stream.read( reinterpret_cast( data + done ), + std::min( size - done, 65536 ) ); + if ( count <= 0 ) { + setError( ZIP_ER_READ ); + return false; } + done += static_cast( count ); } - - unlock(); - - return Ret; + char extra; + if ( stream.read( &extra, 1 ) != 0 ) { + setError( ZIP_ER_CRC ); + return false; + } + return true; } Int32 Zip::exists( const std::string& path ) { @@ -245,21 +238,15 @@ Int8 Zip::checkPack() { } std::vector Zip::getFileList() { - std::vector tmpv; - - Int32 numfiles = zip_get_num_files( mZip ); - - for ( Int32 i = 0; i < numfiles; i++ ) { - struct zip_stat zs; - - if ( -1 != zip_stat_index( mZip, i, 0, &zs ) ) { - if ( zs.size > 0 ) { - tmpv.push_back( std::string( zs.name ) ); - } - } + std::vector files; + const auto count = entryCount(); + files.reserve( static_cast( count ) ); + Entry info; + for ( Uint64 i = 0; i < count; ++i ) { + if ( entry( i, info ) && !info.name.ends_with( '/' ) ) + files.push_back( std::move( info.name ) ); } - - return tmpv; + return files; } /** @return The file path of the opened package */ @@ -275,4 +262,112 @@ zip* Zip::getZip() { return mZip; } +bool Zip::openReadOnly( const std::string& path ) { + // fopenUtf8 preserves Windows Unicode paths. zip_fdopen consumes a duplicated descriptor, + // so the FILE owned here can always be closed normally. + FILE* file = FileSystem::fopenUtf8( path, "rb" ); + if ( !file ) { + setError( errno == ENOENT ? ZIP_ER_NOENT : ZIP_ER_OPEN, errno ); + return false; + } +#if EE_PLATFORM == EE_PLATFORM_WIN + int descriptor = _dup( _fileno( file ) ); +#else + int descriptor = dup( fileno( file ) ); +#endif + const int duplicateError = errno; + std::fclose( file ); + if ( descriptor < 0 ) { + setError( ZIP_ER_OPEN, duplicateError ); + return false; + } + int error = ZIP_ER_OK; + mZip = zip_fdopen( descriptor, 0, &error ); + if ( !mZip ) { + // This vendored libzip consumes descriptor after fdopen succeeds. Only its two early + // ZIP_ER_OPEN failures (dup/fdopen) leave it for the caller to close. + const int systemError = errno; + if ( error == ZIP_ER_OPEN ) { +#if EE_PLATFORM == EE_PLATFORM_WIN + _close( descriptor ); +#else + ::close( descriptor ); +#endif + } + setError( error, systemError ); + return false; + } + return true; +} + +Uint64 Zip::entryCount() const { + if ( !mZip ) + return 0; + const auto count = zip_get_num_entries( mZip, 0 ); + return count == std::numeric_limits::max() ? 0 : count; +} + +bool Zip::entry( Uint64 index, Entry& result ) { + struct zip_stat info; + zip_stat_init( &info ); + if ( !mZip || zip_stat_index( mZip, index, 0, &info ) != 0 ) { + setError( ZIP_ER_INCONS ); + return false; + } + result.name = info.name; + result.size = info.size; + result.modified = static_cast( info.mtime ); + result.crc = info.crc; + return true; +} + +std::unique_ptr Zip::openStream( Uint64 index ) { + struct zip_stat info; + zip_stat_init( &info ); + if ( !mZip || zip_stat_index( mZip, index, 0, &info ) != 0 ) { + setError( ZIP_ER_INCONS ); + return {}; + } + if ( info.size > static_cast( std::numeric_limits::max() ) ) { + setError( ZIP_ER_COMPNOTSUPP ); + return {}; + } + auto stream = std::make_unique( this, index ); + if ( !stream->isOpen() ) { + int code = 0; + int systemCode = 0; + zip_error_get( mZip, &code, &systemCode ); + setError( code, systemCode ); + return {}; + } + return stream; +} + +void Zip::setError( int code, int systemCode ) { + if ( code == ZIP_ER_NOENT ) + mError = Error::NotFound; + else if ( zip_error_get_sys_type( code ) == ZIP_ET_SYS && + ( systemCode == EACCES || systemCode == EPERM ) ) + mError = Error::PermissionDenied; + else if ( code == ZIP_ER_NOZIP || code == ZIP_ER_INCONS || code == ZIP_ER_EOF ) + mError = Error::InvalidArchive; + else if ( code == ZIP_ER_COMPNOTSUPP || code == ZIP_ER_MULTIDISK || + code == ZIP_ER_ENCRNOTSUPP || code == ZIP_ER_NOPASSWD || code == ZIP_ER_WRONGPASSWD ) + mError = Error::Unsupported; + else + mError = Error::IO; + char message[256]; + zip_error_to_str( message, sizeof( message ), code, systemCode ); + mErrorMessage = message; +} + +bool Zip::writable() { + if ( mOpenMode == OpenMode::ReadOnly ) { + mError = Error::ReadOnly; + mErrorMessage = "ZIP archive is read-only"; + return false; + } + return mZip != nullptr; +} + }} // namespace EE::System diff --git a/src/modules/vfs/include/eepp/vfs.hpp b/src/modules/vfs/include/eepp/vfs.hpp index 73cd0f1ac..0ef053f1a 100644 --- a/src/modules/vfs/include/eepp/vfs.hpp +++ b/src/modules/vfs/include/eepp/vfs.hpp @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include diff --git a/src/modules/vfs/include/eepp/vfs/capabilities.hpp b/src/modules/vfs/include/eepp/vfs/capabilities.hpp index 506417d7b..87c0dfea6 100644 --- a/src/modules/vfs/include/eepp/vfs/capabilities.hpp +++ b/src/modules/vfs/include/eepp/vfs/capabilities.hpp @@ -1,7 +1,7 @@ #ifndef EE_VFS_CAPABILITIES_HPP #define EE_VFS_CAPABILITIES_HPP -#include +#include namespace EE { namespace VFS { @@ -36,7 +36,7 @@ enum class VFSCapability : Uint32 { /** Set of VFSCapability flags. Capabilities may vary per URI (for example per SFTP server), so * providers report them per URI. */ -class ProviderCapabilities { +class EE_VFS_API ProviderCapabilities { public: constexpr ProviderCapabilities() noexcept = default; diff --git a/src/modules/vfs/include/eepp/vfs/config.hpp b/src/modules/vfs/include/eepp/vfs/config.hpp new file mode 100644 index 000000000..72a042457 --- /dev/null +++ b/src/modules/vfs/include/eepp/vfs/config.hpp @@ -0,0 +1,22 @@ +#ifndef EE_VFS_CONFIG_HPP +#define EE_VFS_CONFIG_HPP + +#include + +#ifndef EE_VFS_API +#if defined( EE_VFS_STATIC ) +#define EE_VFS_API +#elif EE_PLATFORM == EE_PLATFORM_WIN +#if defined( EE_VFS_EXPORTS ) +#define EE_VFS_API __declspec( dllexport ) +#else +#define EE_VFS_API __declspec( dllimport ) +#endif +#elif defined( __GNUC__ ) && __GNUC__ >= 4 +#define EE_VFS_API __attribute__( ( visibility( "default" ) ) ) +#else +#define EE_VFS_API +#endif +#endif + +#endif diff --git a/src/modules/vfs/include/eepp/vfs/directoryhandle.hpp b/src/modules/vfs/include/eepp/vfs/directoryhandle.hpp index 502818816..a07c1eb02 100644 --- a/src/modules/vfs/include/eepp/vfs/directoryhandle.hpp +++ b/src/modules/vfs/include/eepp/vfs/directoryhandle.hpp @@ -2,6 +2,7 @@ #define EE_VFS_DIRECTORYHANDLE_HPP #include +#include #include #include #include @@ -14,7 +15,7 @@ namespace EE { namespace VFS { * endOfDirectory is explicit: an empty batch does not imply EOF, and the final batch may carry * entries. "." and ".." are never reported. */ -struct DirectoryBatch { +struct EE_VFS_API DirectoryBatch { std::vector entries; bool endOfDirectory{ false }; }; @@ -23,7 +24,7 @@ struct DirectoryBatch { * a read is pending and must release native resources in their destructor if close() was never * called. Calls are serialized by the handle's single consumer; implementations need not support * concurrent read() calls. */ -class DirectoryHandleImpl { +class EE_VFS_API DirectoryHandleImpl { public: virtual ~DirectoryHandleImpl() = default; @@ -37,7 +38,7 @@ class DirectoryHandleImpl { * Move-only. Destroying an open handle releases it without waiting. read() after EOF returns an * empty batch with endOfDirectory set. */ -class DirectoryHandle { +class EE_VFS_API DirectoryHandle { public: DirectoryHandle() = default; diff --git a/src/modules/vfs/include/eepp/vfs/error.hpp b/src/modules/vfs/include/eepp/vfs/error.hpp index 7df6664a0..c6a1e0b7e 100644 --- a/src/modules/vfs/include/eepp/vfs/error.hpp +++ b/src/modules/vfs/include/eepp/vfs/error.hpp @@ -1,10 +1,10 @@ #ifndef EE_VFS_ERROR_HPP #define EE_VFS_ERROR_HPP -#include #include #include #include +#include #include #include @@ -38,13 +38,13 @@ enum class VFSErrorCode : Uint8 { }; /** @return A stable, human readable identifier for code (for example "NotFound"). */ -const char* toString( VFSErrorCode code ); +EE_VFS_API const char* toString( VFSErrorCode code ); /** @brief Typed VFS failure. * * nativeCode keeps the provider error (errno, Win32 error, SFTP status) for diagnostics only. */ -struct VFSError { +struct EE_VFS_API VFSError { URI uri; std::string message; Int64 nativeCode{ 0 }; diff --git a/src/modules/vfs/include/eepp/vfs/filehandle.hpp b/src/modules/vfs/include/eepp/vfs/filehandle.hpp index 883f92b9d..c82cadcb8 100644 --- a/src/modules/vfs/include/eepp/vfs/filehandle.hpp +++ b/src/modules/vfs/include/eepp/vfs/filehandle.hpp @@ -3,6 +3,7 @@ #include #include +#include #include #include #include @@ -16,7 +17,7 @@ namespace EE { namespace VFS { * task completes. Implementations must release native resources in their destructor if close() * was never called. */ -class FileHandleImpl { +class EE_VFS_API FileHandleImpl { public: virtual ~FileHandleImpl() = default; @@ -44,7 +45,7 @@ class FileHandleImpl { * Offset-based operations avoid a shared mutable seek position, so independent reads can be * pipelined. read() returning 0 bytes for a non-empty buffer means end of file. */ -class FileHandle { +class EE_VFS_API FileHandle { public: FileHandle() = default; diff --git a/src/modules/vfs/include/eepp/vfs/fileinfo.hpp b/src/modules/vfs/include/eepp/vfs/fileinfo.hpp index 7a48dfce5..85529dc55 100644 --- a/src/modules/vfs/include/eepp/vfs/fileinfo.hpp +++ b/src/modules/vfs/include/eepp/vfs/fileinfo.hpp @@ -3,8 +3,8 @@ #include #include -#include #include +#include #include #include #include @@ -22,7 +22,7 @@ enum class FileType : Uint8 { Socket, }; -const char* toString( FileType type ); +EE_VFS_API const char* toString( FileType type ); /** @brief Opaque, provider-defined revision token. * @@ -33,7 +33,7 @@ const char* toString( FileType type ); * * Stored inline (no heap) because it travels inside every VFSFileInfo. */ -class FileRevision { +class EE_VFS_API FileRevision { public: static constexpr size_t Capacity = 32; @@ -92,7 +92,7 @@ class FileRevision { * The entry does not store its URI: listings hold thousands of entries and the caller always * knows the URI it listed or stat'ed. Use uriIn( directory ) for an entry of a listing. */ -class VFSFileInfo { +class EE_VFS_API VFSFileInfo { public: /** Last path segment (verbatim, not percent-encoded). */ std::string name; diff --git a/src/modules/vfs/include/eepp/vfs/options.hpp b/src/modules/vfs/include/eepp/vfs/options.hpp index 1b0367a1e..5e3dfd694 100644 --- a/src/modules/vfs/include/eepp/vfs/options.hpp +++ b/src/modules/vfs/include/eepp/vfs/options.hpp @@ -1,13 +1,13 @@ #ifndef EE_VFS_OPTIONS_HPP #define EE_VFS_OPTIONS_HPP -#include +#include #include #include namespace EE { namespace VFS { -struct StatOptions { +struct EE_VFS_API StatOptions { /** When false the entry itself is described (lstat semantics). */ bool followSymlinks{ true }; }; @@ -20,7 +20,7 @@ enum class DirectoryMetadata : Uint8 { Full, }; -struct DirectoryOpenOptions { +struct EE_VFS_API DirectoryOpenOptions { DirectoryMetadata metadata{ DirectoryMetadata::Full }; /** Resolve the target type of symlinks (VFSFileInfo::targetType). May cost a stat per link. */ bool resolveSymlinkTargets{ true }; @@ -52,7 +52,7 @@ constexpr bool hasFlag( OpenFlags flags, OpenFlags flag ) noexcept { return ( static_cast( flags ) & static_cast( flag ) ) != 0; } -struct OpenOptions { +struct EE_VFS_API OpenOptions { OpenFlags flags{ OpenFlags::Read }; /** Permissions for newly created files; provider default (0666 & ~umask) when unset. */ std::optional createPermissions; @@ -62,21 +62,21 @@ struct OpenOptions { OpenOptions( OpenFlags flags ) : flags( flags ) {} }; -struct MkDirOptions { +struct EE_VFS_API MkDirOptions { std::optional permissions; }; -struct RemoveOptions { +struct EE_VFS_API RemoveOptions { /** VFS-level only: remove directory contents recursively. Providers receive false. */ bool recursive{ false }; }; -struct RenameOptions { +struct EE_VFS_API RenameOptions { /** Replace an existing destination. Atomic when the provider reports AtomicReplace. */ bool overwrite{ false }; }; -struct WriteOptions { +struct EE_VFS_API WriteOptions { /** Write a sibling temporary file and atomically rename it over the destination. */ bool atomic{ true }; /** When atomic replacement is unsupported, fall back to an in-place write instead of failing @@ -102,7 +102,7 @@ enum class OverwritePolicy : Uint8 { Skip, }; -struct CopyOptions { +struct EE_VFS_API CopyOptions { OverwritePolicy overwrite{ OverwritePolicy::Fail }; bool recursive{ true }; /** Copy permission bits when both sides support them. */ diff --git a/src/modules/vfs/include/eepp/vfs/provider.hpp b/src/modules/vfs/include/eepp/vfs/provider.hpp index e43052197..9094ad0d6 100644 --- a/src/modules/vfs/include/eepp/vfs/provider.hpp +++ b/src/modules/vfs/include/eepp/vfs/provider.hpp @@ -3,6 +3,7 @@ #include #include +#include #include #include #include @@ -12,7 +13,7 @@ namespace EE { namespace VFS { -struct SpaceInfo { +struct EE_VFS_API SpaceInfo { Uint64 capacity{ 0 }; Uint64 free{ 0 }; /** Free space available to the current (non-privileged) user. */ @@ -33,7 +34,7 @@ struct SpaceInfo { * - Cancellation is cooperative: a cancelled operation completes with VFSErrorCode::Cancelled. * - Providers never touch UI or model state. */ -class VFSProvider { +class EE_VFS_API VFSProvider { public: virtual ~VFSProvider() = default; diff --git a/src/modules/vfs/include/eepp/vfs/providers/localioscheduler.hpp b/src/modules/vfs/include/eepp/vfs/providers/localioscheduler.hpp index e0d01bea0..866b76987 100644 --- a/src/modules/vfs/include/eepp/vfs/providers/localioscheduler.hpp +++ b/src/modules/vfs/include/eepp/vfs/providers/localioscheduler.hpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -28,10 +29,10 @@ class MountWatchdog; * The kernel cannot interrupt a call on an unreachable hard NFS mount, so its thread stays blocked; * nobody waits for it anymore, and its result is discarded if it ever arrives. */ -class EE_API LocalIOScheduler { +class EE_VFS_API LocalIOScheduler { public: /** A file system mounted from another machine. */ - struct NetworkMount { + struct EE_VFS_API NetworkMount { /** Where it is mounted (a native path). */ std::string root; /** What is mounted, for messages ("server:/export", "//server/share"). */ @@ -40,7 +41,7 @@ class EE_API LocalIOScheduler { /** Where the calls of a path run: its network mount's lane, or the shared pool. Resolve it * once for a handle (an open file or directory) and reuse it for the handle's calls. */ - class Route { + class EE_VFS_API Route { public: bool isNetworkMount() const { return mLane != nullptr; } @@ -128,7 +129,7 @@ class EE_API LocalIOScheduler { /** A call queued on a lane: run once on the lane's thread, or failed by the watchdog, and * only the first of the two completes its task. */ - class Call { + class EE_VFS_API Call { public: virtual ~Call() = default; @@ -154,7 +155,7 @@ class EE_API LocalIOScheduler { /** What the routing needs, shared with the calls that resolve their route on the pool (which * can outlive the scheduler). */ - struct Routing { + struct EE_VFS_API Routing { MountLookup lookup; std::shared_ptr watchdog; diff --git a/src/modules/vfs/include/eepp/vfs/providers/localprovider.hpp b/src/modules/vfs/include/eepp/vfs/providers/localprovider.hpp index 3a9b18189..4df69e8b0 100644 --- a/src/modules/vfs/include/eepp/vfs/providers/localprovider.hpp +++ b/src/modules/vfs/include/eepp/vfs/providers/localprovider.hpp @@ -1,6 +1,7 @@ #ifndef EE_VFS_PROVIDERS_LOCALPROVIDER_HPP #define EE_VFS_PROVIDERS_LOCALPROVIDER_HPP +#include #include #include #include @@ -19,7 +20,7 @@ namespace EE { namespace VFS { * Revisions are derived from device, inode (file index), modification time and size, so both * in-place edits and atomic replacements are detected. */ -class LocalProvider : public VFSProvider { +class EE_VFS_API LocalProvider : public VFSProvider { public: explicit LocalProvider( std::shared_ptr scheduler ); diff --git a/src/modules/vfs/include/eepp/vfs/providers/memoryprovider.hpp b/src/modules/vfs/include/eepp/vfs/providers/memoryprovider.hpp index a80369b67..c082c74ae 100644 --- a/src/modules/vfs/include/eepp/vfs/providers/memoryprovider.hpp +++ b/src/modules/vfs/include/eepp/vfs/providers/memoryprovider.hpp @@ -2,6 +2,7 @@ #define EE_VFS_PROVIDERS_MEMORYPROVIDER_HPP #include +#include #include #include #include @@ -21,7 +22,7 @@ namespace EE { namespace VFS { * * Open file handles keep their data alive when the file is removed or replaced, like POSIX inodes. */ -class MemoryProvider : public VFSProvider { +class EE_VFS_API MemoryProvider : public VFSProvider { public: explicit MemoryProvider( std::string scheme = "mem", Executor* completionExecutor = nullptr ); diff --git a/src/modules/vfs/include/eepp/vfs/providers/resourceprovider.hpp b/src/modules/vfs/include/eepp/vfs/providers/resourceprovider.hpp index 162ac459e..b67e8a975 100644 --- a/src/modules/vfs/include/eepp/vfs/providers/resourceprovider.hpp +++ b/src/modules/vfs/include/eepp/vfs/providers/resourceprovider.hpp @@ -2,6 +2,7 @@ #define EE_VFS_PROVIDERS_RESOURCEPROVIDER_HPP #include +#include #include #include @@ -16,7 +17,7 @@ namespace EE { namespace VFS { * Revisions change whenever a pack is opened or closed. Sizes are reported once a file is opened * (packs expose no cheap size query), so listing and stat leave them unknown. */ -class ResourceProvider : public VFSProvider { +class EE_VFS_API ResourceProvider : public VFSProvider { public: explicit ResourceProvider( std::shared_ptr pool ); diff --git a/src/modules/vfs/include/eepp/vfs/providers/zipprovider.hpp b/src/modules/vfs/include/eepp/vfs/providers/zipprovider.hpp new file mode 100644 index 000000000..87c77f940 --- /dev/null +++ b/src/modules/vfs/include/eepp/vfs/providers/zipprovider.hpp @@ -0,0 +1,66 @@ +#ifndef EE_VFS_PROVIDERS_ZIPPROVIDER_HPP +#define EE_VFS_PROVIDERS_ZIPPROVIDER_HPP + +#include +#include +#include +#include + +namespace EE { namespace VFS { + +class ZipProviderImpl; + +/** Read-only zip:// filesystem over individual local archives. Use makeURI() to identify an + * archive and a member path: zip:///path/archive.zip!/member. Literal bangs in the backing + * path are doubled to distinguish them from the archive boundary. + * Metadata is indexed without decompressing entries. Open handles retain their archive snapshot; + * new operations notice changes to the backing file. At most eight archive indexes are cached. + * Archives on network mounts use LocalIOScheduler's lanes and buffer-safe timeout delivery. */ +class EE_VFS_API ZipProvider : public VFSProvider { + public: + explicit ZipProvider( std::shared_ptr scheduler ); + + ~ZipProvider() override; + + /** Builds an archive location. member is an unescaped, '/'-separated path inside it. Only + * file:// backing archives are currently supported; other inputs return an empty URI. + * Root is "/". */ + static URI makeURI( const URI& archive, std::string_view member = "/" ); + + /** Decodes and validates the backing file:// URI of a zip:// location. */ + static VFSResult archiveURI( const URI& uri ); + + /** Validates a ZIP location and returns its normalized path within the archive. */ + static VFSResult memberPath( const URI& uri ); + + std::string_view scheme() const override { return "zip"; } + + ProviderCapabilities capabilities( const URI& uri ) const override; + + VFSTask stat( const URI& uri, const StatOptions& options, + CancellationToken token ) override; + + VFSTask openDirectory( const URI& uri, const DirectoryOpenOptions& options, + CancellationToken token ) override; + + VFSTask open( const URI& uri, const OpenOptions& options, + CancellationToken token ) override; + + VFSTask mkdir( const URI& uri, const MkDirOptions& options, + CancellationToken token ) override; + + VFSTask remove( const URI& uri, const RemoveOptions& options, + CancellationToken token ) override; + + VFSTask rename( const URI& from, const URI& to, const RenameOptions& options, + CancellationToken token ) override; + + VFSTask canonicalize( const URI& uri, CancellationToken token ) override; + + private: + std::shared_ptr mImpl; +}; + +}} // namespace EE::VFS + +#endif diff --git a/src/modules/vfs/include/eepp/vfs/sftp/sftpprovider.hpp b/src/modules/vfs/include/eepp/vfs/sftp/sftpprovider.hpp index 66c5b32b9..67df9b551 100644 --- a/src/modules/vfs/include/eepp/vfs/sftp/sftpprovider.hpp +++ b/src/modules/vfs/include/eepp/vfs/sftp/sftpprovider.hpp @@ -2,6 +2,7 @@ #define EE_VFS_SFTP_SFTPPROVIDER_HPP #include +#include #include #include #include @@ -17,7 +18,7 @@ class VirtualFileSystem; /** Unknown host key presented for a user decision. Changed (mismatching) host keys are never * offered: they always fail with VFSErrorCode::HostKeyRejected. */ -struct SSHHostKeyInfo { +struct EE_VFS_API SSHHostKeyInfo { std::string host; /** Key algorithm, for example "ecdsa-sha2-nistp256". */ std::string keyType; @@ -37,7 +38,7 @@ enum class SSHHostKeyDecision : Uint8 { }; /** Application policy for unknown host keys (for example a UI dialog). */ -class SSHHostKeyVerifier { +class EE_VFS_API SSHHostKeyVerifier { public: virtual ~SSHHostKeyVerifier() = default; @@ -45,12 +46,12 @@ class SSHHostKeyVerifier { virtual Task verify( const SSHHostKeyInfo& info ) = 0; }; -struct SSHPrompt { +struct EE_VFS_API SSHPrompt { std::string text; bool echo{ false }; }; -struct SSHAuthenticationRequest { +struct EE_VFS_API SSHAuthenticationRequest { enum class Kind : Uint8 { Password, /** Passphrase of an encrypted identity file. */ @@ -74,7 +75,7 @@ struct SSHAuthenticationRequest { /** Supplies secrets on demand. Secrets are used for one authentication attempt and never stored * by the VFS (and never appear in URIs). */ -class SSHAuthenticationHandler { +class EE_VFS_API SSHAuthenticationHandler { public: virtual ~SSHAuthenticationHandler() = default; @@ -83,7 +84,7 @@ class SSHAuthenticationHandler { request( const SSHAuthenticationRequest& request ) = 0; }; -struct SFTPProviderConfig { +struct EE_VFS_API SFTPProviderConfig { std::shared_ptr hostKeyVerifier; std::shared_ptr authenticationHandler; /** Builds the OpenSSH configuration context for a host. Defaults to @@ -115,7 +116,7 @@ class SFTPProviderImpl; * connection too. All SSH sessions are driven by one I/O thread in non-blocking mode; * provider methods never block the caller. */ -class SFTPProvider : public VFSProvider { +class EE_VFS_API SFTPProvider : public VFSProvider { public: explicit SFTPProvider( SFTPProviderConfig config = {} ); diff --git a/src/modules/vfs/include/eepp/vfs/ssh/sshconfig.hpp b/src/modules/vfs/include/eepp/vfs/ssh/sshconfig.hpp index d2d9acec2..d78aa61ec 100644 --- a/src/modules/vfs/include/eepp/vfs/ssh/sshconfig.hpp +++ b/src/modules/vfs/include/eepp/vfs/ssh/sshconfig.hpp @@ -1,8 +1,8 @@ #ifndef EE_VFS_SSH_SSHCONFIG_HPP #define EE_VFS_SSH_SSHCONFIG_HPP -#include #include +#include #include #include #include @@ -18,7 +18,7 @@ namespace EE { namespace VFS { using EE::System::Time; /** @brief OpenSSH-style pattern matching (ssh_config(5) "PATTERNS"). */ -class SSHPattern { +class EE_VFS_API SSHPattern { public: enum class ListMatch : Int8 { /** A negated pattern matched: the whole list is rejected. */ @@ -38,7 +38,7 @@ class SSHPattern { }; /** One "Keyword arguments" line of an ssh_config file. */ -struct SSHConfigLine { +struct EE_VFS_API SSHConfigLine { /** Lowercased keyword. */ std::string keyword; /** Arguments split with OpenSSH quoting rules. */ @@ -53,7 +53,7 @@ struct SSHConfigLine { * Parsing only tokenizes: directives are kept in file order so the evaluator can apply OpenSSH's * ordering, first-value-wins, Host/Match activation and Include semantics. */ -class SSHConfigFile { +class EE_VFS_API SSHConfigFile { public: static SSHConfigFile parse( std::string_view text, std::string path = {} ); @@ -79,7 +79,7 @@ enum class SSHStrictHostKeyChecking : Uint8 { No, }; -struct SSHJumpHost { +struct EE_VFS_API SSHJumpHost { std::string user; std::string host; /** 0 when not specified (the jump host's own config / default applies). */ @@ -93,7 +93,7 @@ struct SSHJumpHost { * Paths are fully expanded (tilde, tokens, environment). The raw evaluated option values are kept * in `options` for diagnostics and `ssh -G` style dumps. */ -struct SSHResolvedConfig { +struct EE_VFS_API SSHResolvedConfig { std::string originalHost; /** Effective connection host (HostName applied, canonicalized when enabled). */ std::string hostName; @@ -157,7 +157,7 @@ struct SSHResolvedConfig { /** Environment of an evaluation. Everything that touches the system is injectable so tests stay * deterministic and never execute commands or resolve names. */ -struct SSHConfigContext { +struct EE_VFS_API SSHConfigContext { /** Host as given by the user (URI host). */ std::string originalHost; /** Command-line style overrides (from the URI): they win over every config file. */ @@ -211,12 +211,16 @@ struct SSHConfigContext { * cycle/depth protection, hostname canonicalization and the canonical/final re-parse pass, and * context-aware token and environment expansion. */ -class SSHConfigEvaluator { +class EE_VFS_API SSHConfigEvaluator { public: explicit SSHConfigEvaluator( SSHConfigContext context ); ~SSHConfigEvaluator(); + SSHConfigEvaluator( const SSHConfigEvaluator& ) = delete; + + SSHConfigEvaluator& operator=( const SSHConfigEvaluator& ) = delete; + /** @return The resolved configuration, or an error message for fatal problems (for example * canonicalization failure with CanonicalizeFallbackLocal no). */ std::variant evaluate(); diff --git a/src/modules/vfs/include/eepp/vfs/transfer.hpp b/src/modules/vfs/include/eepp/vfs/transfer.hpp index 842489815..a495b59de 100644 --- a/src/modules/vfs/include/eepp/vfs/transfer.hpp +++ b/src/modules/vfs/include/eepp/vfs/transfer.hpp @@ -4,13 +4,14 @@ #include #include #include +#include #include #include #include namespace EE { namespace VFS { -struct TransferProgress { +struct EE_VFS_API TransferProgress { URI currentSource; URI currentDestination; Uint64 bytesDone{ 0 }; @@ -19,7 +20,7 @@ struct TransferProgress { std::optional filesTotal; }; -struct TransferResult { +struct EE_VFS_API TransferResult { Uint64 bytesTransferred{ 0 }; Uint64 filesTransferred{ 0 }; Uint64 filesSkipped{ 0 }; @@ -37,7 +38,7 @@ class TransferState; } /** @brief RAII handle of a progress subscription. */ -class TransferProgressSubscription { +class EE_VFS_API TransferProgressSubscription { public: TransferProgressSubscription() = default; @@ -69,7 +70,7 @@ class TransferProgressSubscription { * coalesced: a slow subscriber receives the latest snapshot rather than every intermediate one. * Transfers keep running when the Transfer object is destroyed; call cancel() to stop them. */ -class Transfer { +class EE_VFS_API Transfer { public: Transfer() = default; diff --git a/src/modules/vfs/include/eepp/vfs/ui/vfsdirectorymodel.hpp b/src/modules/vfs/include/eepp/vfs/ui/vfsdirectorymodel.hpp index f48a697dd..e4cf21068 100644 --- a/src/modules/vfs/include/eepp/vfs/ui/vfsdirectorymodel.hpp +++ b/src/modules/vfs/include/eepp/vfs/ui/vfsdirectorymodel.hpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -39,7 +40,8 @@ using UI::Models::Variant; * All public methods must be called on the UI thread, which must be the thread the executor runs * functions on. */ -class VFSDirectoryModel : public Model, public std::enable_shared_from_this { +class EE_VFS_API VFSDirectoryModel : public Model, + public std::enable_shared_from_this { public: enum Column : Uint8 { Name, Size, Modified, Type, Permissions, Owner, Group, ColumnCount }; @@ -83,6 +85,10 @@ class VFSDirectoryModel : public Model, public std::enable_shared_from_this +#include #include #include #include @@ -31,7 +32,7 @@ using EE::System::ThreadPool; * Provider work never runs on that pool, so waits there cannot starve provider I/O. Copies and * moves run on a pool of their own: long transfers never delay the other operations. */ -class VirtualFileSystem { +class EE_VFS_API VirtualFileSystem { public: struct Config { /** Worker threads for blocking local I/O (LocalProvider and similar). */ @@ -45,7 +46,7 @@ class VirtualFileSystem { * counts as not responding: the call fails with TimedOut, and so do further calls until * the server answers again (see LocalIOScheduler). */ System::Time networkMountTimeout{ System::Seconds( 15 ) }; - /** Register file:// and res:// (and sftp:// when built with SFTP support). */ + /** Register file://, zip:// and res:// (and sftp:// when built with SFTP support). */ bool registerDefaultProviders{ true }; }; diff --git a/src/modules/vfs/include/eepp/vfs/watch.hpp b/src/modules/vfs/include/eepp/vfs/watch.hpp index 855e9a3e3..cb3c4c181 100644 --- a/src/modules/vfs/include/eepp/vfs/watch.hpp +++ b/src/modules/vfs/include/eepp/vfs/watch.hpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -29,11 +30,11 @@ enum class WatchEventType : Uint8 { RescanRequired, }; -const char* toString( WatchEventType type ); +EE_VFS_API const char* toString( WatchEventType type ); /** One change, relative to the watched root ("dir/file.txt", '/' separated, never leading '/'). */ -struct WatchEvent { +struct EE_VFS_API WatchEvent { std::string relativePath; std::string oldRelativePath; WatchEventType type{ WatchEventType::Modified }; @@ -52,16 +53,16 @@ enum class WatchStatus : Uint8 { Failed, }; -const char* toString( WatchStatus status ); +EE_VFS_API const char* toString( WatchStatus status ); /** A coalesced group of events for one watch. Events are in arrival order. */ -struct WatchEventBatch { +struct EE_VFS_API WatchEventBatch { URI root; std::vector events; WatchStatus status{ WatchStatus::Active }; }; -struct WatchOptions { +struct EE_VFS_API WatchOptions { bool recursive{ false }; bool followSymlinks{ false }; bool allowOutOfScopeLinks{ false }; @@ -84,7 +85,7 @@ class WatchSubscriptionState; * Thread-safe. Events are accumulated and delivered as one batch per executor dispatch, which * coalesces bursts without adding a latency timer. */ -class WatchSink { +class EE_VFS_API WatchSink { public: virtual ~WatchSink() = default; @@ -101,14 +102,14 @@ class WatchSink { }; /** Backend-side registration of one logical watch. Destroying it stops native watching. */ -class WatchRegistration { +class EE_VFS_API WatchRegistration { public: virtual ~WatchRegistration() = default; }; /** @brief Notification engine for a set of URIs. Backends adapt efsw (local), a remote * efsw-agent or polling into URI space; they never mutate UI or model state. */ -class WatchBackend { +class EE_VFS_API WatchBackend { public: virtual ~WatchBackend() = default; @@ -125,7 +126,7 @@ class WatchBackend { * that the callback is not running and will not run again (a callback may destroy its own * handle). */ -class WatchHandle { +class EE_VFS_API WatchHandle { public: WatchHandle() = default; @@ -167,7 +168,7 @@ class WatchHandle { * Backends are registered per scheme. A scheme without a dedicated backend uses the fallback * backend (normally polling through the VFS) when WatchOptions::allowPolling is set. */ -class WatchManager { +class EE_VFS_API WatchManager { public: WatchManager(); diff --git a/src/modules/vfs/src/eepp/vfs/providers/zipprovider.cpp b/src/modules/vfs/src/eepp/vfs/providers/zipprovider.cpp new file mode 100644 index 000000000..820576c8b --- /dev/null +++ b/src/modules/vfs/src/eepp/vfs/providers/zipprovider.cpp @@ -0,0 +1,614 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace EE::System; + +namespace EE { namespace VFS { + +namespace { + +constexpr ProviderCapabilities + ZipCapabilities( static_cast( VFSCapability::Read ) | + static_cast( VFSCapability::List ) | + static_cast( VFSCapability::Seek ) | + static_cast( VFSCapability::CaseSensitive ) ); +constexpr Uint64 NoEntry = std::numeric_limits::max(); +constexpr size_t MaxEntries = 262144; + +VFSError readOnly( const URI& uri ) { + return VFSError( VFSErrorCode::ReadOnly, uri, "ZIP archives are read-only" ); +} + +VFSError readerError( const Zip& reader, const URI& uri ) { + VFSErrorCode code = VFSErrorCode::IO; + switch ( reader.error() ) { + case Zip::Error::NotFound: + code = VFSErrorCode::NotFound; + break; + case Zip::Error::PermissionDenied: + code = VFSErrorCode::PermissionDenied; + break; + case Zip::Error::InvalidArchive: + code = VFSErrorCode::InvalidOperation; + break; + case Zip::Error::Unsupported: + code = VFSErrorCode::Unsupported; + break; + default: + break; + } + return VFSError( code, uri, reader.errorMessage() ); +} + +/** Archive paths are untrusted. Reject names that could leave a copy destination or turn into + * native Windows paths/streams; do not resolve these names into aliases of safe entries. */ +bool safeEntry( std::string_view name ) { + if ( name.empty() || name.size() > 4096 || name.front() == '/' || + name.find_first_of( std::string_view( "\\:\0", 3 ) ) != std::string_view::npos ) + return false; + size_t start = 0; + while ( start < name.size() ) { + size_t end = name.find( '/', start ); + if ( end == std::string_view::npos ) + end = name.size(); + auto part = name.substr( start, end - start ); + // Windows trims trailing dots/spaces; an all-dot/space segment can become a parent. + if ( part.empty() || part.find_first_not_of( ". " ) == std::string_view::npos ) + return false; + start = end + 1; + } + return true; +} + +/** Parsed archive identity and normalized path within it. */ +struct ZipAddress { + URI archive; + std::string member; + + static VFSResult parse( const URI& uri ) { + if ( uri.getScheme() != "zip" || uri.getAuthority() != uri.getHost() || + !uri.getFragment().empty() || uri.getPath().find( '\0' ) != std::string::npos ) + return Err( VFSError( VFSErrorCode::InvalidURI, uri, "Invalid ZIP location" ) ); + if ( !uri.getRawQuery().empty() ) + return Err( VFSError( VFSErrorCode::Unsupported, uri, "Unsupported ZIP backing URI" ) ); + const auto& path = uri.getPath(); + if ( path.empty() || path.front() != '/' ) + return Err( VFSError( VFSErrorCode::InvalidURI, uri, "ZIP path must be absolute" ) ); + std::string backing; + backing.reserve( path.size() ); + size_t boundary = std::string::npos; + for ( size_t i = 0; i < path.size(); ++i ) { + if ( path[i] == '!' ) { + if ( i + 1 < path.size() && path[i + 1] == '!' ) { + backing += '!'; + ++i; + continue; + } + if ( i + 1 == path.size() || path[i + 1] == '/' ) { + boundary = i + 1; + break; + } + } + backing += path[i]; + } + if ( boundary == std::string::npos || backing.empty() || backing.back() == '/' ) + return Err( + VFSError( VFSErrorCode::InvalidURI, uri, "ZIP path needs archive.zip!/member" ) ); + ZipAddress result; + result.archive.setScheme( "file" ); + result.archive.setHost( uri.getHost() ); + result.archive.setRawPath( std::move( backing ) ); + URI member; + member.setScheme( "zip" ); + member.setRawPath( boundary == path.size() ? "/" : path.substr( boundary ) ); + member.normalize(); + result.member = member.getPath(); + return Ok( std::move( result ) ); + } +}; + +struct ZipNode { + VFSFileInfo info; + std::vector children; + Uint64 entry{ NoEntry }; +}; + +/** Immutable directory index. Entry streams open independent readers, so their I/O and teardown + * never wait for another handle on the same archive. */ +struct ZipArchive { + std::string path; + std::vector nodes; + UnorderedMap paths; + FileRevision revision; + LocalIOScheduler::Route route; + Uint64 stamp{ 0 }; + + VFSFileInfo info( std::string name, bool directory, Uint64 entry = NoEntry, + const Zip::Entry* metadata = nullptr ) const { + VFSFileInfo value; + value.name = std::move( name ); + value.type = directory ? FileType::Directory : FileType::Regular; + value.hidden = !value.name.empty() && value.name[0] == '.'; + // Read-only is a provider capability, not the stored entry's mode. The archive API does + // not expose Unix permissions: leave them unknown so copies use writable native defaults. + if ( metadata ) { + value.setModifiedTime( metadata->modified ); + if ( !directory ) + value.setSize( metadata->size ); + } + value.revision = FileRevision::fromValues( + { stamp, entry, metadata ? metadata->crc : 0, metadata ? metadata->size : 0 } ); + return value; + } + + VFSResult index( Zip& reader, const URI& uri, const CancellationToken& token ) { + const auto count = reader.entryCount(); + if ( count > MaxEntries ) + return Err( + VFSError( VFSErrorCode::InvalidOperation, uri, "ZIP has too many entries" ) ); + nodes.push_back( { info( "/", true ), {}, NoEntry } ); + paths.emplace( "", 0 ); + Zip::Entry metadata; + for ( Uint64 entry = 0; entry < count; ++entry ) { + if ( token.isCancellationRequested() ) + return Err( cancelledError( uri ) ); + if ( !reader.entry( entry, metadata ) ) + return Err( readerError( reader, uri ) ); + std::string_view name( metadata.name ); + if ( !safeEntry( name ) ) + continue; + const bool directory = name.back() == '/'; + if ( directory ) + name.remove_suffix( 1 ); + size_t start = 0; + size_t parent = 0; + while ( start < name.size() ) { + size_t end = name.find( '/', start ); + if ( end == std::string_view::npos ) + end = name.size(); + const bool last = end == name.size(); + const bool folder = !last || directory; + std::string path( name.substr( 0, end ) ); + auto found = paths.find( path ); + size_t node; + if ( found == paths.end() ) { + if ( nodes.size() >= MaxEntries ) { + return Err( VFSError( VFSErrorCode::InvalidOperation, uri, + "ZIP has too many directories and entries" ) ); + } + node = nodes.size(); + paths.emplace( std::move( path ), node ); + nodes[parent].children.push_back( node ); + nodes.push_back( + { info( std::string( name.substr( start, end - start ) ), folder ), + {}, + NoEntry } ); + } else { + node = found->second; + } + ZipNode& value = nodes[node]; + // A directory wins a file/directory collision so that every listed child remains + // reachable. Duplicate file names resolve to their first central-directory entry. + if ( folder && value.info.type != FileType::Directory ) { + value.info = info( std::move( value.info.name ), true ); + value.entry = NoEntry; + } + if ( last && ( folder || value.info.type != FileType::Directory ) && + value.entry == NoEntry ) { + value.info = info( std::move( value.info.name ), folder, entry, &metadata ); + value.entry = entry; + } + parent = node; + start = end + 1; + } + } + return Ok(); + } + + VFSResult find( const std::string& member, const URI& uri ) const { + auto found = paths.find( member ); + if ( found != paths.end() ) { + if ( uri.getPath().ends_with( '/' ) && + nodes[found->second].info.type != FileType::Directory ) + return Err( VFSError( VFSErrorCode::NotDirectory, uri ) ); + return Ok( found->second ); + } + // Distinguish a missing member from a path through a regular file. + size_t slash = member.find( '/' ); + while ( slash != std::string::npos ) { + auto parent = paths.find( member.substr( 0, slash ) ); + if ( parent != paths.end() && nodes[parent->second].info.type != FileType::Directory ) + return Err( VFSError( VFSErrorCode::NotDirectory, uri ) ); + slash = member.find( '/', slash + 1 ); + } + return Err( VFSError( VFSErrorCode::NotFound, uri ) ); + } +}; + +class ZipFileImpl : public FileHandleImpl, public std::enable_shared_from_this { + template auto guarded( CancellationToken token, F&& fn ) { + return [self = shared_from_this(), token = std::move( token ), + fn = std::forward( fn )]() mutable -> VFSResult { + if ( token.isCancellationRequested() ) + return Err( cancelledError( self->mUri ) ); + std::lock_guard lock( self->mMutex ); + if ( !self->mStream ) + return Err( VFSError( VFSErrorCode::InvalidOperation, self->mUri, "File closed" ) ); + return fn( *self, token ); + }; + } + + public: + ZipFileImpl( std::shared_ptr scheduler, std::shared_ptr archive, + std::unique_ptr reader, std::unique_ptr stream, URI uri, + size_t node ) : + mScheduler( std::move( scheduler ) ), + mArchive( std::move( archive ) ), + mReader( std::move( reader ) ), + mStream( std::move( stream ) ), + mUri( std::move( uri ) ), + mNode( node ) {} + + VFSTask read( Uint64 offset, std::span data, CancellationToken token ) override { + if ( !mArchive->route.isNetworkMount() ) { + return mScheduler->submit( + mArchive->route, guarded( std::move( token ), + [offset, data]( ZipFileImpl& self, + const CancellationToken& token ) { + return self.readInto( offset, data, token ); + } ) ); + } + // A mount's read can outlive its timeout. Only the winning completion copies into data. + return mScheduler->submitCommitted( + mArchive->route, + guarded>( + std::move( token ), + [offset, size = data.size()]( ZipFileImpl& self, const CancellationToken& token ) + -> VFSResult> { + std::vector buffer( size ); + auto result = self.readInto( offset, buffer, token ); + if ( !result ) + return Err( std::move( result ).error() ); + buffer.resize( result.value() ); + return Ok( std::move( buffer ) ); + } ), + [data]( VFSResult>&& result ) -> VFSResult { + if ( !result ) + return Err( std::move( result ).error() ); + std::copy( result.value().begin(), result.value().end(), data.begin() ); + return Ok( result.value().size() ); + } ); + } + + VFSTask write( Uint64, std::span, CancellationToken ) override { + return readyError( readOnly( mUri ) ); + } + + VFSTask truncate( Uint64 ) override { return readyError( readOnly( mUri ) ); } + + VFSTask flush() override { return readyValue(); } + + VFSTask stat() override { + return readyValue( mArchive->nodes[mNode].info ); + } + + VFSTask close() override { + return mScheduler->submit( mArchive->route, + [self = shared_from_this()]() -> VFSResult { + std::lock_guard lock( self->mMutex ); + self->mStream.reset(); + self->mReader.reset(); + return Ok(); + } ); + } + + ProviderCapabilities capabilities() const override { return ZipCapabilities; } + + private: + VFSResult readInto( Uint64 offset, std::span data, + const CancellationToken& token ) { + const auto& node = mArchive->nodes[mNode]; + if ( offset >= node.info.size().value_or( 0 ) ) + return Ok( size_t( 0 ) ); + if ( offset < static_cast( mStream->tell() ) ) { + auto stream = mReader->openStream( node.entry ); + if ( !stream ) + return Err( readerError( *mReader, mUri ) ); + mStream = std::move( stream ); + } + std::array scratch; + while ( static_cast( mStream->tell() ) < offset ) { + if ( token.isCancellationRequested() ) + return Err( cancelledError( mUri ) ); + const auto count = std::min( offset - mStream->tell(), scratch.size() ); + if ( mStream->read( scratch.data(), count ) <= 0 ) + return Err( VFSError( VFSErrorCode::IO, mUri, "Could not seek ZIP entry" ) ); + } + size_t done = 0; + while ( done < data.size() ) { + if ( token.isCancellationRequested() ) + return Err( cancelledError( mUri ) ); + const auto count = std::min( data.size() - done, 65536 ); + const auto got = mStream->read( reinterpret_cast( data.data() + done ), count ); + if ( got < 0 ) + return Err( VFSError( VFSErrorCode::IO, mUri, + "Could not read ZIP entry (corrupt data or CRC)" ) ); + if ( got == 0 ) + break; + done += static_cast( got ); + } + return Ok( done ); + } + + std::shared_ptr mScheduler; + // Streams borrow the reader and are destroyed before it. + std::shared_ptr mArchive; + std::unique_ptr mReader; + std::unique_ptr mStream; + std::mutex mMutex; + URI mUri; + size_t mNode; +}; + +} // namespace + +class ZipProviderImpl : public std::enable_shared_from_this { + public: + explicit ZipProviderImpl( std::shared_ptr scheduler ) : + scheduler( std::move( scheduler ) ) {} + + template + VFSTask run( const URI& uri, CancellationToken token, F&& fn ) { + if ( token.isCancellationRequested() ) + return readyError( cancelledError( uri ) ); + auto address = ZipAddress::parse( uri ); + if ( !address ) + return readyError( std::move( address ).error() ); + auto native = LocalProvider::toNativePath( address.value().archive ); + if ( !native ) + return readyError( std::move( native ).error() ); + std::string member = std::move( address ).value().member; + while ( !member.empty() && member.front() == '/' ) + member.erase( member.begin() ); + while ( !member.empty() && member.back() == '/' ) + member.pop_back(); + std::string path = std::move( native ).value(); + return scheduler->submitResolved( + path, true, + [self = shared_from_this(), uri, path, member = std::move( member ), + token = std::move( token ), fn = std::forward( fn )]() mutable -> VFSResult { + if ( token.isCancellationRequested() ) + return Err( cancelledError( uri ) ); + auto archive = self->load( path, uri, token ); + if ( !archive ) + return Err( std::move( archive ).error() ); + return fn( std::move( archive ).value(), member, uri ); + } ); + } + + std::shared_ptr scheduler; + + private: + struct Cached { + std::string path; + std::shared_ptr archive; + Uint64 used{ 0 }; + }; + + VFSResult> load( const std::string& path, const URI& uri, + const CancellationToken& token ) { + const URI backing = URI::fromFSPath( path ); + auto metadata = LocalProvider::statSync( backing, path, true ); + if ( !metadata ) + return Err( std::move( metadata ).error() ); + if ( !metadata.value().isRegularFile() ) + return Err( VFSError( VFSErrorCode::InvalidOperation, uri, + "ZIP backing file is not a regular file" ) ); + const auto revision = metadata.value().revision; + { + std::lock_guard lock( mMutex ); + for ( auto& cached : mCache ) { + if ( cached.path == path && cached.archive && + cached.archive->revision == revision ) { + cached.used = ++mUsed; + return Ok( cached.archive ); + } + } + } + auto archive = std::make_shared(); + archive->revision = revision; + archive->path = path; + archive->stamp = std::hash{}( + std::string_view( reinterpret_cast( revision.data() ), revision.size() ) ); + archive->route = scheduler->resolvedRouteFor( path, true ); + Zip reader( false ); + if ( !reader.open( path, Zip::OpenMode::ReadOnly ) ) + return Err( readerError( reader, uri ) ); + auto indexed = archive->index( reader, uri, token ); + if ( !indexed ) + return Err( std::move( indexed ).error() ); + auto after = LocalProvider::statSync( backing, path, true ); + if ( !after || after.value().revision != revision ) + return Err( VFSError( VFSErrorCode::ChangedExternally, uri, + "Archive changed while it was opened" ) ); + std::shared_ptr retired; + { + std::lock_guard lock( mMutex ); + auto slot = mCache.begin(); + for ( auto it = mCache.begin(); it != mCache.end(); ++it ) { + if ( it->path == path ) { + slot = it; + break; + } + if ( it->used < slot->used ) + slot = it; + } + retired = std::move( slot->archive ); + slot->path = path; + slot->archive = archive; + slot->used = ++mUsed; + } + // Free a retired directory index outside the cache lock. + return Ok( std::move( archive ) ); + } + + std::mutex mMutex; + std::array mCache; + Uint64 mUsed{ 0 }; +}; + +ZipProvider::ZipProvider( std::shared_ptr scheduler ) : + mImpl( std::make_shared( std::move( scheduler ) ) ) {} + +ZipProvider::~ZipProvider() = default; + +URI ZipProvider::makeURI( const URI& archive, std::string_view member ) { + if ( archive.getScheme() != "file" || !archive.getRawQuery().empty() || + !archive.getFragment().empty() || archive.getAuthority() != archive.getHost() || + archive.getPath().empty() || archive.getPath().front() != '/' || + archive.getPath().find( '\0' ) != std::string::npos ) + return {}; + URI uri; + uri.setScheme( "zip" ); + uri.setHost( archive.getHost() ); + std::string path; + path.reserve( archive.getPath().size() + member.size() + 2 ); + // Escape literal bangs in the backing path, so even a directory named "packages!" is + // unambiguous. Only the first unescaped "!/" separates the archive from its member. + for ( char c : archive.getPath() ) { + path += c; + if ( c == '!' ) + path += '!'; + } + URI entry; + entry.setScheme( "zip" ); + entry.setRawPath( member.empty() || member.front() != '/' ? "/" + std::string( member ) + : std::string( member ) ); + entry.normalize(); + path += '!'; + path += entry.getPath(); + uri.setRawPath( std::move( path ) ); + return uri; +} + +VFSResult ZipProvider::archiveURI( const URI& uri ) { + auto address = ZipAddress::parse( uri ); + if ( !address ) + return Err( std::move( address ).error() ); + return Ok( std::move( address ).value().archive ); +} + +VFSResult ZipProvider::memberPath( const URI& uri ) { + auto address = ZipAddress::parse( uri ); + if ( !address ) + return Err( std::move( address ).error() ); + return Ok( std::move( address ).value().member ); +} + +ProviderCapabilities ZipProvider::capabilities( const URI& ) const { + return ZipCapabilities; +} + +VFSTask ZipProvider::stat( const URI& uri, const StatOptions&, + CancellationToken token ) { + return mImpl->run( uri, std::move( token ), + []( const std::shared_ptr& archive, + const std::string& member, + const URI& uri ) -> VFSResult { + auto node = archive->find( member, uri ); + if ( !node ) + return Err( std::move( node ).error() ); + return Ok( archive->nodes[node.value()].info ); + } ); +} + +VFSTask ZipProvider::openDirectory( const URI& uri, const DirectoryOpenOptions&, + CancellationToken token ) { + return mImpl->run( + uri, std::move( token ), + []( const std::shared_ptr& archive, const std::string& member, + const URI& uri ) -> VFSResult { + auto node = archive->find( member, uri ); + if ( !node ) + return Err( std::move( node ).error() ); + const auto& folder = archive->nodes[node.value()]; + if ( folder.info.type != FileType::Directory ) + return Err( VFSError( VFSErrorCode::NotDirectory, uri ) ); + std::vector entries; + entries.reserve( folder.children.size() ); + for ( auto child : folder.children ) + entries.push_back( archive->nodes[child].info ); + return Ok( DirectoryHandle( + uri, std::make_shared( std::move( entries ) ) ) ); + } ); +} + +VFSTask ZipProvider::open( const URI& uri, const OpenOptions& options, + CancellationToken token ) { + if ( hasFlag( options.flags, OpenFlags::Write ) || + hasFlag( options.flags, OpenFlags::Create ) || + hasFlag( options.flags, OpenFlags::Truncate ) ) + return readyError( readOnly( uri ) ); + return mImpl->run( + uri, std::move( token ), + [scheduler = mImpl->scheduler]( std::shared_ptr archive, + const std::string& member, + const URI& uri ) -> VFSResult { + auto node = archive->find( member, uri ); + if ( !node ) + return Err( std::move( node ).error() ); + const auto& info = archive->nodes[node.value()]; + if ( info.info.type == FileType::Directory ) + return Err( VFSError( VFSErrorCode::IsDirectory, uri ) ); + auto reader = std::make_unique( false ); + if ( !reader->open( archive->path, Zip::OpenMode::ReadOnly ) ) + return Err( readerError( *reader, uri ) ); + auto current = + LocalProvider::statSync( URI::fromFSPath( archive->path ), archive->path, true ); + if ( !current || current.value().revision != archive->revision ) + return Err( VFSError( VFSErrorCode::ChangedExternally, uri, + "Archive changed while opening the entry" ) ); + auto stream = reader->openStream( info.entry ); + if ( !stream ) + return Err( readerError( *reader, uri ) ); + return Ok( FileHandle( uri, std::make_shared( + scheduler, std::move( archive ), std::move( reader ), + std::move( stream ), uri, node.value() ) ) ); + } ); +} + +VFSTask ZipProvider::mkdir( const URI& uri, const MkDirOptions&, CancellationToken ) { + return readyError( readOnly( uri ) ); +} + +VFSTask ZipProvider::remove( const URI& uri, const RemoveOptions&, CancellationToken ) { + return readyError( readOnly( uri ) ); +} + +VFSTask ZipProvider::rename( const URI& from, const URI&, const RenameOptions&, + CancellationToken ) { + return readyError( readOnly( from ) ); +} + +VFSTask ZipProvider::canonicalize( const URI& uri, CancellationToken token ) { + return mImpl->run( uri, std::move( token ), + []( const std::shared_ptr& archive, + const std::string& member, const URI& uri ) -> VFSResult { + auto found = archive->find( member, uri ); + if ( !found ) + return Err( std::move( found ).error() ); + auto backing = ZipProvider::archiveURI( uri ); + URI normalized = ZipProvider::makeURI( backing.value(), member ); + return Ok( std::move( normalized ) ); + } ); +} + +}} // namespace EE::VFS diff --git a/src/modules/vfs/src/eepp/vfs/sftp/sshagent.hpp b/src/modules/vfs/src/eepp/vfs/sftp/sshagent.hpp index 93a775a9e..617a4d1d2 100644 --- a/src/modules/vfs/src/eepp/vfs/sftp/sshagent.hpp +++ b/src/modules/vfs/src/eepp/vfs/sftp/sshagent.hpp @@ -18,7 +18,7 @@ namespace EE { namespace VFS { namespace sftp { * time. Speaks the agent protocol over Unix-domain sockets; elsewhere (Windows named pipes, * Pageant) supported() is false and libssh2's own agent support is used. */ -class SSHAgent { +class EE_VFS_API SSHAgent { public: struct Identity { /** Public key blob (SSH wire format). */ diff --git a/src/modules/vfs/src/eepp/vfs/virtualfilesystem.cpp b/src/modules/vfs/src/eepp/vfs/virtualfilesystem.cpp index 2fe3304fe..1566fa8b5 100644 --- a/src/modules/vfs/src/eepp/vfs/virtualfilesystem.cpp +++ b/src/modules/vfs/src/eepp/vfs/virtualfilesystem.cpp @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -69,6 +70,7 @@ VirtualFileSystem::~VirtualFileSystem() { void VirtualFileSystem::registerDefaultProviders() { registerProvider( std::make_shared( mLocalIO ) ); + registerProvider( std::make_shared( mLocalIO ) ); registerProvider( std::make_shared( mIOPool ) ); // Network mounts are polled through the VFS (whose calls time out), not by efsw. auto polling = std::make_shared( *this ); diff --git a/src/modules/vfs/src/eepp/vfs/watchers/localwatchbackend.hpp b/src/modules/vfs/src/eepp/vfs/watchers/localwatchbackend.hpp index 84f9abb9a..26bfd8c1d 100644 --- a/src/modules/vfs/src/eepp/vfs/watchers/localwatchbackend.hpp +++ b/src/modules/vfs/src/eepp/vfs/watchers/localwatchbackend.hpp @@ -26,7 +26,7 @@ class LocalWatchBackendImpl; * Directories on a network mount go to networkBackend instead (the VFS polling backend): efsw's * own threads would block on an unreachable server, while VFS calls time out. */ -class LocalWatchBackend : public WatchBackend { +class EE_VFS_API LocalWatchBackend : public WatchBackend { public: /** @param pool Where watches are registered and removed. * @param networkBackend Watches directories on network mounts; without it efsw does. diff --git a/src/modules/vfs/src/eepp/vfs/watchers/pollingwatchbackend.cpp b/src/modules/vfs/src/eepp/vfs/watchers/pollingwatchbackend.cpp index e4c25ea64..a3b00fbe5 100644 --- a/src/modules/vfs/src/eepp/vfs/watchers/pollingwatchbackend.cpp +++ b/src/modules/vfs/src/eepp/vfs/watchers/pollingwatchbackend.cpp @@ -12,7 +12,7 @@ namespace EE { namespace VFS { namespace detail { -class PollingWatchBackendImpl { +class PollingWatchBackendImpl : public std::enable_shared_from_this { public: struct Snapshot { struct Entry { @@ -52,8 +52,12 @@ class PollingWatchBackendImpl { mStopping = true; } mWake.notify_all(); - if ( mThread.joinable() ) - mThread.join(); + if ( mThread.joinable() ) { + if ( mThread.get_id() == std::this_thread::get_id() ) + mThread.detach(); + else + mThread.join(); + } } /** Takes the initial snapshot on the polling thread and resolves once the watch is active. @@ -88,7 +92,9 @@ class PollingWatchBackendImpl { void ensureThread() { if ( !mThread.joinable() ) - mThread = std::thread( [this] { loop(); } ); + // A synchronous watch callback may release the backend on this thread. Keep its + // state alive until the loop observes stop(), without joining the current thread. + mThread = std::thread( [self = shared_from_this()] { self->loop(); } ); } void loop() { @@ -101,7 +107,9 @@ class PollingWatchBackendImpl { auto snapshot = take( *pending.watch ); lock.lock(); if ( !snapshot ) { + lock.unlock(); pending.resolve( Err( std::move( snapshot ).error() ) ); + lock.lock(); continue; } pending.watch->snapshot = std::move( snapshot ).value(); @@ -139,8 +147,10 @@ class PollingWatchBackendImpl { while ( !mPendingAdds.empty() ) { auto pending = std::move( mPendingAdds.front() ); mPendingAdds.pop_front(); + lock.unlock(); pending.resolve( Err( VFSError( VFSErrorCode::InvalidOperation, pending.watch->root, "Watch backend shutting down" ) ) ); + lock.lock(); } } diff --git a/src/modules/vfs/src/eepp/vfs/watchers/pollingwatchbackend.hpp b/src/modules/vfs/src/eepp/vfs/watchers/pollingwatchbackend.hpp index 8baabf35f..79d8ee84b 100644 --- a/src/modules/vfs/src/eepp/vfs/watchers/pollingwatchbackend.hpp +++ b/src/modules/vfs/src/eepp/vfs/watchers/pollingwatchbackend.hpp @@ -21,7 +21,7 @@ class PollingWatchBackendImpl; * successful poll emits RescanRequired. Recursive polling is limited to MaxEntries entries per * watch; larger trees degrade to the top directories that fit. */ -class PollingWatchBackend : public WatchBackend { +class EE_VFS_API PollingWatchBackend : public WatchBackend { public: static constexpr size_t MaxEntries = 100000; diff --git a/src/tests/unit_tests/vfs_tests.cpp b/src/tests/unit_tests/vfs_tests.cpp index cb1a855e5..9424542d3 100644 --- a/src/tests/unit_tests/vfs_tests.cpp +++ b/src/tests/unit_tests/vfs_tests.cpp @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -460,12 +461,15 @@ UTEST( VFSResource, zipPackWithSeveralFiles ) { EXPECT_TRUE( zip.addFile( a, "vfs-zip/a.txt" ) ); EXPECT_TRUE( zip.addFile( b, "vfs-zip/sub/b.txt" ) ); EXPECT_TRUE( zip.addFile( c, "vfs-zip/sub/c.txt" ) ); + std::vector empty; + EXPECT_TRUE( zip.addFile( empty, "vfs-zip/empty.txt" ) ); + EXPECT_TRUE( zip.addFile( empty, "vfs-zip/empty-dir/" ) ); EXPECT_TRUE( zip.close() ); } Zip zip; ASSERT_TRUE( zip.open( zipPath ) ); - EXPECT_EQ( 3u, zip.getFileList().size() ); + EXPECT_EQ( 4u, zip.getFileList().size() ); std::vector data; EXPECT_TRUE( zip.extractFileToMemory( "vfs-zip/sub/c.txt", data ) ); EXPECT_STDSTREQ( "c3", std::string( data.begin(), data.end() ) ); @@ -473,6 +477,8 @@ UTEST( VFSResource, zipPackWithSeveralFiles ) { VirtualFileSystem vfs; EXPECT_STDSTREQ( "a1", readString( vfs, URI( "res:///vfs-zip/a.txt" ) ) ); EXPECT_STDSTREQ( "b2", readString( vfs, URI( "res:///vfs-zip/sub/b.txt" ) ) ); + EXPECT_STDSTREQ( "", readString( vfs, URI( "res:///vfs-zip/empty.txt" ) ) ); + EXPECT_FALSE( PackFileIndex::instance()->fileExists( "vfs-zip/empty-dir" ) ); auto listed = waitFor( vfs.listAll( URI( "res:///vfs-zip/sub" ) ) ); ASSERT_TRUE( listed.hasValue() ); EXPECT_EQ( 2u, listed.value().size() ); @@ -1290,3 +1296,45 @@ UTEST( LocalIOScheduler, linksIntoAMountRunOnItsLane ) { gate->open(); } #endif + +namespace { + +class ReleasePollingBackendSink : public WatchSink { + public: + URI uri; + std::mutex gate; + std::shared_ptr backend; + + void push( WatchEvent&& ) override {} + + void push( std::vector&& ) override {} + + void rescanRequired() override {} + + void setStatus( WatchStatus ) override { + std::lock_guard lock( gate ); + backend.reset(); + } + + const URI& root() const override { return uri; } +}; + +} // namespace + +UTEST( VFSWatch, pollingCallbackCanReleaseTheLastBackendOwner ) { + VirtualFileSystem vfs; + auto memory = std::make_shared(); + vfs.registerProvider( memory ); + const URI root( "mem://polling-teardown/dir" ); + memory->putFile( root.withAppendedSegment( "file" ), "contents" ); + auto sink = std::make_shared(); + sink->uri = root; + sink->backend = std::make_shared( vfs ); + std::weak_ptr weak = sink->backend; + std::unique_lock hold( sink->gate ); + auto task = sink->backend->watch( root, {}, sink ); + hold.unlock(); + ASSERT_EQ( TaskState::Ready, task.waitFor( std::chrono::seconds( 5 ) ) ); + ASSERT_TRUE( waitFor( std::move( task ) ).hasValue() ); + EXPECT_TRUE( weak.expired() ); +} diff --git a/src/tests/unit_tests/zipprovider_tests.cpp b/src/tests/unit_tests/zipprovider_tests.cpp new file mode 100644 index 000000000..534b6ea39 --- /dev/null +++ b/src/tests/unit_tests/zipprovider_tests.cpp @@ -0,0 +1,440 @@ +#include "../../tools/eefm/location.hpp" +#include "utest.hpp" +#include "vfs_test_utils.hpp" +#include +#include +#include +#include +#include +#include +#include + +using namespace EE; +using namespace EE::System; +using namespace EE::VFS; +using namespace vfstest; + +namespace { + +bool makeZip( const std::string& path, + std::initializer_list> files ) { + Zip archive; + if ( !archive.create( path ) ) + return false; + for ( const auto& [name, data] : files ) { + if ( !archive.addFile( reinterpret_cast( data.data() ), data.size(), name ) ) + return false; + } + return archive.close(); +} + +} // namespace + +UTEST( ZipProvider, archiveURIsPreserveIdentityAndMemberNames ) { + TempDir temp; + const URI file = URI::fromFSPath( temp.path + "Mixed Case !!#%?.zip" ); + const URI root = ZipProvider::makeURI( file ); + const URI member = root.withAppendedSegment( "dir" ).withAppendedSegment( "Name #%?.txt" ); + const URI roundTrip( member.toString() ); + EXPECT_TRUE( member == roundTrip ); + auto backing = ZipProvider::archiveURI( roundTrip ); + ASSERT_TRUE( backing.hasValue() ); + EXPECT_TRUE( file == backing.value() ); + EXPECT_TRUE( file == ZipProvider::archiveURI( member.getParent().getParent() ).value() ); + EXPECT_FALSE( ZipProvider::archiveURI( root.getParent() ).hasValue() ); + EXPECT_FALSE( root == ZipProvider::makeURI( URI::fromFSPath( temp.path + "other.zip" ) ) ); + EXPECT_TRUE( member.isSameOrChildOf( root ) ); + const URI bangArchive = URI::fromFSPath( temp.path + "packages!/archive!.zip" ); + const URI bangMember = ZipProvider::makeURI( bangArchive, "/folder!/name!.txt" ); + EXPECT_TRUE( bangArchive == ZipProvider::archiveURI( URI( bangMember.toString() ) ).value() ); + EXPECT_STDSTREQ( "/folder!/name!.txt", ZipProvider::memberPath( bangMember ).value() ); + const URI clamped = ZipProvider::makeURI( file, "/../../inside" ); + EXPECT_TRUE( file == ZipProvider::archiveURI( clamped ).value() ); + EXPECT_STDSTREQ( "/inside", ZipProvider::memberPath( clamped ).value() ); + EXPECT_EQ( VFSErrorCode::InvalidURI, + codeOf( ZipProvider::archiveURI( URI( "zip://nothex/" ) ) ) ); + EXPECT_EQ( + VFSErrorCode::InvalidURI, + codeOf( ZipProvider::archiveURI( ZipProvider::makeURI( URI( "sftp://host/a.zip" ) ) ) ) ); +} + +UTEST( ZipProvider, listsImplicitAndEmptyDirectoriesWithMetadata ) { + TempDir temp; + const std::string path = temp.path + "archive.zip"; + ASSERT_TRUE( makeZip( path, { { "top.txt", "hello" }, + { "dir/leaf.txt", "leaf" }, + { "empty/", "" }, + { ".hidden", "secret" }, + { "dir/Upper #%.txt", "mixed" } } ) ); + VirtualFileSystem vfs; + const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) ); + auto listed = waitFor( vfs.listAll( root ) ); + ASSERT_TRUE( listed.hasValue() ); + EXPECT_TRUE( sortedNames( listed.value() ) == + std::vector( { ".hidden", "dir", "empty", "top.txt" } ) ); + auto folder = waitFor( vfs.stat( root.withAppendedSegment( "dir" ) ) ); + ASSERT_TRUE( folder.hasValue() ); + EXPECT_TRUE( folder.value().isDirectory() ); + auto file = waitFor( vfs.stat( root.withAppendedSegment( "top.txt" ) ) ); + ASSERT_TRUE( file.hasValue() ); + EXPECT_EQ( 5u, file.value().size().value_or( 0 ) ); + EXPECT_TRUE( file.value().modifiedTime().has_value() ); + EXPECT_FALSE( file.value().revision.empty() ); + EXPECT_FALSE( file.value().permissions().has_value() ); + EXPECT_FALSE( folder.value().permissions().has_value() ); + auto empty = waitFor( vfs.listAll( root.withAppendedSegment( "empty" ) ) ); + ASSERT_TRUE( empty.hasValue() ); + EXPECT_TRUE( empty.value().empty() ); + EXPECT_STDSTREQ( "mixed", + readString( vfs, root.withAppendedSegment( "dir" ).withAppendedSegment( + "Upper #%.txt" ) ) ); + auto opened = waitFor( vfs.openDirectory( root ) ); + ASSERT_TRUE( opened.hasValue() ); + auto handle = std::move( opened ).value(); + size_t count = 0; + bool end = false; + while ( !end ) { + auto batch = waitFor( handle.read( 1 ) ); + ASSERT_TRUE( batch.hasValue() ); + count += batch.value().entries.size(); + end = batch.value().endOfDirectory; + } + EXPECT_EQ( 4u, count ); + EXPECT_TRUE( waitFor( handle.close() ).hasValue() ); +} + +UTEST( ZipProvider, streamingReadsSeekBackwardsAndCopyAcrossProviders ) { + TempDir temp; + std::string content( 300000, 'x' ); + Uint32 random = 1; + for ( auto& byte : content ) { + random = random * 1664525u + 1013904223u; + byte = static_cast( random >> 24 ); + } + const std::string path = temp.path + "archive.zip"; + ASSERT_TRUE( makeZip( + path, { { "dir/large.txt", content }, { "dir/second.txt", "second" }, { "zero", "" } } ) ); + VirtualFileSystem vfs; + const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) ); + const URI file = root.withAppendedSegment( "dir" ).withAppendedSegment( "large.txt" ); + auto opened = waitFor( vfs.open( file ) ); + ASSERT_TRUE( opened.hasValue() ); + auto handle = std::move( opened ).value(); + std::array buffer; + for ( Uint64 offset : { 0ull, 100000ull, 15ull, 299980ull, 300000ull } ) { + auto result = waitFor( handle.read( offset, buffer ) ); + ASSERT_TRUE( result.hasValue() ); + const std::string expected = content.substr( offset, buffer.size() ); + EXPECT_EQ( expected.size(), result.value() ); + EXPECT_TRUE( expected == std::string( buffer.begin(), buffer.begin() + result.value() ) ); + } + EXPECT_TRUE( waitFor( handle.close() ).hasValue() ); + EXPECT_EQ( VFSErrorCode::InvalidOperation, codeOf( waitFor( handle.read( 0, buffer ) ) ) ); + EXPECT_STDSTREQ( "", readString( vfs, root.withAppendedSegment( "zero" ) ) ); + EXPECT_TRUE( content == readString( vfs, file ) ); + const URI destination = URI::fromFSPath( temp.path + "extracted" ); + auto copied = + waitFor( vfs.copy( root.withAppendedSegment( "dir" ), destination ).completion() ); + ASSERT_TRUE( copied.hasValue() ); + EXPECT_TRUE( content == readString( vfs, destination.withAppendedSegment( "large.txt" ) ) ); + EXPECT_STDSTREQ( "second", readString( vfs, destination.withAppendedSegment( "second.txt" ) ) ); + EXPECT_TRUE( waitFor( vfs.readAll( file, {}, 100 ) ).hasError() ); + // An in-place edit can break a backward seek. Keep the old stream alive when reopening + // fails: this is an I/O failure, and must not turn every later call into "File closed". + auto reopened = waitFor( vfs.open( file ) ); + ASSERT_TRUE( reopened.hasValue() ); + auto changing = std::move( reopened ).value(); + ASSERT_TRUE( waitFor( changing.read( 200000, buffer ) ).hasValue() ); + ASSERT_TRUE( FileSystem::fileWrite( path, std::string{} ) ); + EXPECT_TRUE( waitFor( changing.read( 0, buffer ) ).hasError() ); + auto eof = waitFor( changing.read( content.size(), buffer ) ); + ASSERT_TRUE( eof.hasValue() ); + EXPECT_EQ( 0u, eof.value() ); +} + +UTEST( ZipProvider, readOnlyErrorsMissingFilesAndCancellation ) { + TempDir temp; + const std::string path = temp.path + "archive.zip"; + ASSERT_TRUE( makeZip( path, { { "file", "contents" } } ) ); + VirtualFileSystem vfs; + const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) ); + const URI file = root.withAppendedSegment( "file" ); + EXPECT_EQ( VFSErrorCode::ReadOnly, codeOf( waitFor( vfs.open( file, OpenFlags::Write ) ) ) ); + EXPECT_EQ( VFSErrorCode::ReadOnly, + codeOf( waitFor( vfs.makeDir( root.withAppendedSegment( "new" ) ) ) ) ); + EXPECT_EQ( VFSErrorCode::ReadOnly, codeOf( waitFor( vfs.remove( file ) ) ) ); + EXPECT_EQ( VFSErrorCode::ReadOnly, + codeOf( waitFor( vfs.rename( file, root.withAppendedSegment( "renamed" ) ) ) ) ); + EXPECT_EQ( VFSErrorCode::IsDirectory, codeOf( waitFor( vfs.open( root ) ) ) ); + EXPECT_EQ( VFSErrorCode::NotDirectory, codeOf( waitFor( vfs.openDirectory( file ) ) ) ); + const URI fileWithSlash = ZipProvider::makeURI( URI::fromFSPath( path ), "/file/" ); + EXPECT_EQ( VFSErrorCode::NotDirectory, codeOf( waitFor( vfs.stat( fileWithSlash ) ) ) ); + EXPECT_EQ( VFSErrorCode::NotDirectory, codeOf( waitFor( vfs.open( fileWithSlash ) ) ) ); + EXPECT_EQ( VFSErrorCode::NotDirectory, + codeOf( waitFor( vfs.stat( file.withAppendedSegment( "child" ) ) ) ) ); + EXPECT_EQ( VFSErrorCode::NotFound, + codeOf( waitFor( vfs.stat( root.withAppendedSegment( "missing" ) ) ) ) ); + CancellationSource cancelled; + cancelled.cancel(); + EXPECT_EQ( VFSErrorCode::Cancelled, + codeOf( waitFor( vfs.open( file, {}, cancelled.token() ) ) ) ); + const URI missing = ZipProvider::makeURI( URI::fromFSPath( temp.path + "missing.zip" ) ); + EXPECT_EQ( VFSErrorCode::NotFound, codeOf( waitFor( vfs.stat( missing ) ) ) ); + EXPECT_FALSE( FileSystem::fileExists( temp.path + "missing.zip" ) ); + ASSERT_TRUE( FileSystem::fileWrite( temp.path + "bad.zip", "this is not a ZIP" ) ); + EXPECT_EQ( VFSErrorCode::InvalidOperation, + codeOf( waitFor( vfs.stat( + ZipProvider::makeURI( URI::fromFSPath( temp.path + "bad.zip" ) ) ) ) ) ); + EXPECT_FALSE( vfs.capabilities( file ).has( VFSCapability::Write ) ); + auto opened = waitFor( vfs.open( file ) ); + ASSERT_TRUE( opened.hasValue() ); + auto handle = std::move( opened ).value(); + const std::array byte{ 'x' }; + EXPECT_EQ( VFSErrorCode::ReadOnly, codeOf( waitFor( handle.write( 0, byte ) ) ) ); + EXPECT_EQ( VFSErrorCode::ReadOnly, codeOf( waitFor( handle.truncate( 0 ) ) ) ); +} + +UTEST( ZipProvider, browsingNeverRegistersResourcesAndNoticesArchiveReplacement ) { + TempDir temp; + const std::string path = temp.path + "archive.zip"; + ASSERT_TRUE( makeZip( path, { { "zip-provider-only.txt", "old" } } ) ); + const Uint64 generation = PackFileIndex::instance()->getGeneration(); + VirtualFileSystem vfs; + const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) ); + EXPECT_STDSTREQ( "old", + readString( vfs, root.withAppendedSegment( "zip-provider-only.txt" ) ) ); + EXPECT_EQ( generation, PackFileIndex::instance()->getGeneration() ); + EXPECT_FALSE( PackFileIndex::instance()->fileExists( "zip-provider-only.txt" ) ); + auto oldInfo = waitFor( vfs.stat( root.withAppendedSegment( "zip-provider-only.txt" ) ) ); + ASSERT_TRUE( oldInfo.hasValue() ); + // Replace only after handles have closed (portable to Windows, where an open file can + // prevent replacement). A fresh operation must discard the cached directory index. + ASSERT_TRUE( FileSystem::fileRemove( path ) ); + ASSERT_TRUE( makeZip( path, { { "new.txt", "new contents" } } ) ); + auto listing = waitFor( vfs.listAll( root ) ); + ASSERT_TRUE( listing.hasValue() ); + EXPECT_TRUE( sortedNames( listing.value() ) == std::vector( { "new.txt" } ) ); + EXPECT_STDSTREQ( "new contents", readString( vfs, root.withAppendedSegment( "new.txt" ) ) ); + EXPECT_EQ( + VFSErrorCode::NotFound, + codeOf( waitFor( vfs.stat( root.withAppendedSegment( "zip-provider-only.txt" ) ) ) ) ); +} + +UTEST( Zip, standaloneReaderIsReadOnlyAndHasBoundedSeek ) { + TempDir temp; + const auto path = temp.path + "archive.zip"; + const std::string content( 200000, 'a' ); + ASSERT_TRUE( makeZip( path, { { "large", content } } ) ); + std::string original; + ASSERT_TRUE( FileSystem::fileGet( path, original ) ); + { + Zip reader( false ); + ASSERT_TRUE( reader.open( path, Zip::OpenMode::ReadOnly ) ); + EXPECT_TRUE( reader.isOpen() ); + const std::array byte{ 'x' }; + EXPECT_FALSE( reader.addFile( byte.data(), byte.size(), "forbidden" ) ); + EXPECT_TRUE( reader.error() == Zip::Error::ReadOnly ); + EXPECT_FALSE( reader.eraseFile( "large" ) ); + EXPECT_EQ( 1u, reader.entryCount() ); + Zip::Entry entry; + ASSERT_TRUE( reader.entry( 0, entry ) ); + EXPECT_EQ( content.size(), entry.size ); + auto stream = reader.openStream( 0 ); + ASSERT_TRUE( stream != nullptr ); + EXPECT_EQ( 199999, stream->seek( 199999 ) ); + char value = 0; + EXPECT_EQ( 1, stream->read( &value, 1 ) ); + EXPECT_EQ( int( 'a' ), int( value ) ); + EXPECT_EQ( 0, stream->seek( 0 ) ); + EXPECT_EQ( 1, stream->read( &value, 1 ) ); + } + std::string after; + ASSERT_TRUE( FileSystem::fileGet( path, after ) ); + EXPECT_TRUE( original == after ); + Zip missing( false ); + EXPECT_FALSE( missing.open( temp.path + "missing.zip", Zip::OpenMode::ReadOnly ) ); + EXPECT_TRUE( missing.error() == Zip::Error::NotFound ); + EXPECT_FALSE( FileSystem::fileExists( temp.path + "missing.zip" ) ); + EXPECT_FALSE( missing.open( temp.path + "missing.zip" ) ); + EXPECT_FALSE( FileSystem::fileExists( temp.path + "missing.zip" ) ); + ASSERT_TRUE( FileSystem::fileWrite( temp.path + "invalid.zip", "not an archive" ) ); + errno = EACCES; + EXPECT_FALSE( missing.open( temp.path + "invalid.zip", Zip::OpenMode::ReadOnly ) ); + EXPECT_TRUE( missing.error() == Zip::Error::InvalidArchive ); + errno = EACCES; + EXPECT_FALSE( missing.open( temp.path + "invalid.zip" ) ); + EXPECT_TRUE( missing.error() == Zip::Error::InvalidArchive ); +} + +UTEST( ZipLocation, readableAddressesRoundTripAndRootReturnsToContainingFolder ) { + TempDir temp; + const URI archive = URI::fromFSPath( temp.path + "Mixed # %!?.zip" ); + const URI root = ZipProvider::makeURI( archive ); + const URI member = root.withAppendedSegment( "a # %.txt" ); + EXPECT_TRUE( eefm::Location::parse( eefm::Location::text( root ).toUtf8() ) == root ); + EXPECT_TRUE( eefm::Location::parse( eefm::Location::text( member ).toUtf8() ) == member ); + EXPECT_TRUE( URI( eefm::Location::text( member ).toUtf8() ) == member ); + EXPECT_TRUE( eefm::Location::text( root ).toUtf8().starts_with( "zip:///" ) ); + EXPECT_TRUE( ZipProvider::archiveURI( URI( "zip:///tmp/archive.zip!/docs" ) ).hasValue() ); + EXPECT_TRUE( eefm::Location::parent( root ) == archive.getParent() ); + EXPECT_TRUE( eefm::Location::parent( member ) == root ); + EXPECT_STDSTREQ( archive.getLastPathSegment(), eefm::Location::entryName( root ) ); + EXPECT_STDSTREQ( archive.getLastPathSegment(), eefm::Location::displayName( root ).toUtf8() ); +} + +UTEST( ZipProvider, unsafeNamesAreExcludedAndCollisionsAreDeterministic ) { + TempDir temp; + // Stored ZIP with traversal/absolute/drive names, duplicate entries, and a file/directory + // collision. Kept as a fixture because Zip::addFile refuses duplicate names. + const std::string encoded = "UEsDBBQAAAAAADIIUV2PKKQfBAAAAAQAAAAIAAAAc2FmZS50eHRzYWZlUEsDBBQAAA" + "AAADIIUV37OSuCAwAAAAMAAAAN" + "AAAALi4vZXNjYXBlLnR4dGJhZFBLAwQUAAAAAAAyCFFd+" + "zkrggMAAAADAAAADQAAAC9hYnNvbHV0ZS50eHRiYWRQSwME" + "FAAAAAAAMghRXfs5K4IDAAAAAwAAABQAAABkaXIvLi4vLi4vZXNjYXBlLnR4dGJhZF" + "BLAwQUAAAAAAAyCFFd+zkrggMA" + "AAADAAAACwAAAEM6ZHJpdmUudHh0YmFkUEsDBBQAAAAAADIIUV37OSuCAwAAAAMAAA" + "AOAAAAZGlyXGVzY2FwZS50eHRi" + "YWRQSwMEFAAAAAAAMghRXfs5K4IDAAAAAwAAAA4AAAAuLiAvZXNjYXBlLnR4dGJhZF" + "BLAwQUAAAAAAAyCFFdAAAAAAAA" + "AAAAAAAABgAAAGVtcHR5L1BLAwQUAAAAAAAyCFFdEDafjAQAAAAEAAAACQAAAGNvbG" + "xpc2lvbmZpbGVQSwMEFAAAAAAA" + "MghRXSlUsyIFAAAABQAAAA8AAABjb2xsaXNpb24vY2hpbGRjaGlsZFBLAwQUAAAAAA" + "AyCFFdV+5xkgUAAAAFAAAACQAA" + "AGR1cGxpY2F0ZWZpcnN0UEsDBBQAAAAAADIIUV1pER+" + "2BgAAAAYAAAAJAAAAZHVwbGljYXRlc2Vjb25kUEsBAhQDFAAA" + "AAAAMghRXY8opB8EAAAABAAAAAgAAAAAAAAAAAAAAIABAAAAAHNhZmUudHh0UEsBAh" + "QDFAAAAAAAMghRXfs5K4IDAAAA" + "AwAAAA0AAAAAAAAAAAAAAIABKgAAAC4uL2VzY2FwZS50eHRQSwECFAMUAAAAAAAyCF" + "Fd+zkrggMAAAADAAAADQAAAAAA" + "AAAAAAAAgAFYAAAAL2Fic29sdXRlLnR4dFBLAQIUAxQAAAAAADIIUV37OSuCAwAAAA" + "MAAAAUAAAAAAAAAAAAAACAAYYA" + "AABkaXIvLi4vLi4vZXNjYXBlLnR4dFBLAQIUAxQAAAAAADIIUV37OSuCAwAAAAMAAA" + "ALAAAAAAAAAAAAAACAAbsAAABD" + "OmRyaXZlLnR4dFBLAQIUAxQAAAAAADIIUV37OSuCAwAAAAMAAAAOAAAAAAAAAAAAAA" + "CAAecAAABkaXJcZXNjYXBlLnR4" + "dFBLAQIUAxQAAAAAADIIUV37OSuCAwAAAAMAAAAOAAAAAAAAAAAAAACAARYBAAAuLi" + "AvZXNjYXBlLnR4dFBLAQIUAxQA" + "AAAAADIIUV0AAAAAAAAAAAAAAAAGAAAAAAAAAAAAEAD9QUUBAABlbXB0eS9QSwECFA" + "MUAAAAAAAyCFFdEDafjAQAAAAE" + "AAAACQAAAAAAAAAAAAAAgAFpAQAAY29sbGlzaW9uUEsBAhQDFAAAAAAAMghRXSlUsy" + "IFAAAABQAAAA8AAAAAAAAAAAAA" + "AIABlAEAAGNvbGxpc2lvbi9jaGlsZFBLAQIUAxQAAAAAADIIUV1X7nGSBQAAAAUAAA" + "AJAAAAAAAAAAAAAACAAcYBAABk" + "dXBsaWNhdGVQSwECFAMUAAAAAAAyCFFdaREftgYAAAAGAAAACQAAAAAAAAAAAAAAgA" + "HyAQAAZHVwbGljYXRlUEsFBgAA" + "AAAMAAwAtQIAAB8CAAAAAA=="; + std::string bytes; + ASSERT_NE( size_t( -1 ), Base64::decode( encoded, bytes ) ); + ASSERT_TRUE( FileSystem::fileWrite( temp.path + "unsafe.zip", bytes ) ); + VirtualFileSystem vfs; + const URI root = ZipProvider::makeURI( URI::fromFSPath( temp.path + "unsafe.zip" ) ); + auto listed = waitFor( vfs.listAll( root ) ); + ASSERT_TRUE( listed.hasValue() ); + EXPECT_TRUE( sortedNames( listed.value() ) == + std::vector( { "collision", "duplicate", "empty", "safe.txt" } ) ); + EXPECT_STDSTREQ( "first", readString( vfs, root.withAppendedSegment( "duplicate" ) ) ); + EXPECT_STDSTREQ( + "child", + readString( vfs, root.withAppendedSegment( "collision" ).withAppendedSegment( "child" ) ) ); + const auto destination = URI::fromFSPath( temp.path + "out" ); + ASSERT_TRUE( waitFor( vfs.copy( root, destination ).completion() ).hasValue() ); + EXPECT_FALSE( FileSystem::fileExists( temp.path + "escape.txt" ) ); + EXPECT_STDSTREQ( "safe", readString( vfs, destination.withAppendedSegment( "safe.txt" ) ) ); +} + +UTEST( ZipProvider, crcFailureIsReportedEvenForAnExactSizedRead ) { + TempDir temp; + const std::string encoded = + "UEsDBBQAAAAAADIIUV0Ks4YUDwAAAA8AAAALAAAAY2hlY2tlZC50eHRjaGVja2VkIHBheWxvYWRQSwECFAMUAAAAAA" + "Ay" + "CFFdCrOGFA8AAAAPAAAACwAAAAAAAAAAAAAAgAEAAAAAY2hlY2tlZC50eHRQSwUGAAAAAAEAAQA5AAAAOAAAAAAA"; + std::string bytes; + ASSERT_NE( size_t( -1 ), Base64::decode( encoded, bytes ) ); + const auto payload = bytes.find( "checked payload" ); + ASSERT_NE( std::string::npos, payload ); + bytes[payload] ^= 1; + ASSERT_TRUE( FileSystem::fileWrite( temp.path + "corrupt.zip", bytes ) ); + VirtualFileSystem vfs; + const URI root = ZipProvider::makeURI( URI::fromFSPath( temp.path + "corrupt.zip" ) ); + auto opened = waitFor( vfs.open( root.withAppendedSegment( "checked.txt" ) ) ); + ASSERT_TRUE( opened.hasValue() ); + auto handle = std::move( opened ).value(); + std::array data; + EXPECT_EQ( VFSErrorCode::IO, codeOf( waitFor( handle.read( 0, data ) ) ) ); +} + +UTEST( ZipProvider, openHandlesSurviveCacheEvictionAndProviderRemoval ) { + TempDir temp; + const auto path = temp.path + "original.zip"; + ASSERT_TRUE( makeZip( path, { { "file", "original" }, { "other", "second" } } ) ); + VirtualFileSystem vfs; + const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) ); + auto first = waitFor( vfs.open( root.withAppendedSegment( "file" ) ) ); + auto second = waitFor( vfs.open( root.withAppendedSegment( "other" ) ) ); + ASSERT_TRUE( first.hasValue() ); + ASSERT_TRUE( second.hasValue() ); + auto a = std::move( first ).value(); + auto b = std::move( second ).value(); + for ( int i = 0; i < 9; ++i ) { + const auto other = temp.path + String::toString( i ) + ".zip"; + ASSERT_TRUE( makeZip( other, { { "x", "x" } } ) ); + ASSERT_TRUE( + waitFor( vfs.stat( ZipProvider::makeURI( URI::fromFSPath( other ) ) ) ).hasValue() ); + } + vfs.unregisterProvider( "zip" ); + std::array left; + std::array right; + auto readingA = a.read( 0, left ); + auto readingB = b.read( 0, right ); + ASSERT_TRUE( waitFor( std::move( readingA ) ).hasValue() ); + ASSERT_TRUE( waitFor( std::move( readingB ) ).hasValue() ); + EXPECT_TRUE( std::string( left.begin(), left.end() ) == "original" ); + EXPECT_TRUE( std::string( right.begin(), right.end() ) == "second" ); + EXPECT_TRUE( waitFor( a.close() ).hasValue() ); + EXPECT_TRUE( waitFor( b.close() ).hasValue() ); +} + +UTEST( ZipProvider, emptyArchivesAndUnicodeBackingFiles ) { + TempDir temp; + const std::string empty = std::string( "PK\x05\x06", 4 ) + std::string( 18, '\0' ); + const auto path = temp.path + "empty.zip"; + ASSERT_TRUE( FileSystem::fileWrite( path, empty ) ); + VirtualFileSystem vfs; + auto listing = waitFor( vfs.listAll( ZipProvider::makeURI( URI::fromFSPath( path ) ) ) ); + ASSERT_TRUE( listing.hasValue() ); + EXPECT_TRUE( listing.value().empty() ); + const auto unicode = temp.path + "archivo-ñ-雪.zip"; + // Build at an ASCII path with the legacy writer, then rename through the portable VFS. + const auto source = temp.path + "source.zip"; + ASSERT_TRUE( makeZip( + source, { { "file", "unicode archive" }, { "directorio-ñ/雪.txt", "unicode member" } } ) ); + ASSERT_TRUE( + waitFor( vfs.rename( URI::fromFSPath( source ), URI::fromFSPath( unicode ) ) ).hasValue() ); + EXPECT_STDSTREQ( "unicode archive", + readString( vfs, ZipProvider::makeURI( URI::fromFSPath( unicode ) ) + .withAppendedSegment( "file" ) ) ); + const URI member = ZipProvider::makeURI( URI::fromFSPath( unicode ), "/directorio-ñ/雪.txt" ); + EXPECT_STDSTREQ( "unicode member", readString( vfs, URI( member.toString() ) ) ); +} + +UTEST( ZipProvider, pollingWatchNoticesBackingArchiveReplacement ) { + TempDir temp; + const auto path = temp.path + "watched.zip"; + ASSERT_TRUE( makeZip( path, { { "before", "old" } } ) ); + VirtualFileSystem vfs; + const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) ); + WatchOptions options; + options.pollingInterval = Milliseconds( 50 ); + auto opened = waitFor( vfs.watch( root, options ) ); + ASSERT_TRUE( opened.hasValue() ); + WatchHandle watch( std::move( opened ).value() ); + EXPECT_EQ( WatchStatus::Polling, watch.status() ); + TestLoop loop; + EventCollector collector; + watch.onEvents( loop, collector.callback() ); + ASSERT_TRUE( FileSystem::fileRemove( path ) ); + ASSERT_TRUE( makeZip( path, { { "after", "new contents" } } ) ); + EXPECT_TRUE( loop.runUntil( [&] { return collector.has( WatchEventType::Added, "after" ); } ) ); + EXPECT_TRUE( collector.has( WatchEventType::Removed, "before" ) ); + EXPECT_STDSTREQ( "new contents", readString( vfs, root.withAppendedSegment( "after" ) ) ); +} diff --git a/src/thirdparty/libzip/zip_open.c b/src/thirdparty/libzip/zip_open.c index 05821c1d0..b32ae9e97 100644 --- a/src/thirdparty/libzip/zip_open.c +++ b/src/thirdparty/libzip/zip_open.c @@ -257,6 +257,8 @@ _zip_readcdir(FILE *fp, off_t buf_offset, unsigned char *buf, unsigned char *eoc left = cd->size; i=0; + if (cd->nentry == 0 && left == 0) + return cd; do { if ((_zip_dirent_read(cd->entry+i, fp, bufp, &left, 0, error)) < 0) { cd->nentry = i; diff --git a/src/tools/eefm/fileoperations.cpp b/src/tools/eefm/fileoperations.cpp index d2df8b810..61ef4f1b2 100644 --- a/src/tools/eefm/fileoperations.cpp +++ b/src/tools/eefm/fileoperations.cpp @@ -8,6 +8,7 @@ #include "placescontroller.hpp" #include "trash.hpp" #include +#include namespace eefm { @@ -70,6 +71,12 @@ void FileOperations::openRow( FileTab* tab, size_t row ) { mApp.panes().navigate( tab, tab->model->entryURI( row ) ); return; } + const URI entry = tab->model->entryURI( row ); + if ( entry.getScheme() == "file" && + String::toLower( FileSystem::fileExtension( tab->model->entry( row ).name ) ) == "zip" ) { + mApp.panes().navigate( tab, ZipProvider::makeURI( entry ) ); + return; + } // Enter / double click on a multiple selection opens all the selected files. auto rows = tab->selectedRows(); if ( std::find( rows.begin(), rows.end(), row ) == rows.end() ) diff --git a/src/tools/eefm/location.cpp b/src/tools/eefm/location.cpp index a8b16dbe6..ddb46b1e4 100644 --- a/src/tools/eefm/location.cpp +++ b/src/tools/eefm/location.cpp @@ -1,14 +1,16 @@ #include "location.hpp" #include +#include using namespace EE; using namespace EE::System; +using namespace EE::VFS; namespace eefm { URI Location::parse( const std::string& text ) { std::string trimmed = String::trim( text ); - if ( trimmed.find( "://" ) != std::string::npos ) { + if ( trimmed.find( "://" ) != std::string::npos || trimmed.starts_with( "zip:" ) ) { URI uri( trimmed ); if ( uri.getPath().empty() ) uri.setRawPath( "/" ); @@ -17,14 +19,46 @@ URI Location::parse( const std::string& text ) { return URI::fromFSPath( FileSystem::expandTilde( trimmed ) ); } +URI Location::parent( const URI& uri ) { + if ( uri.getScheme() == "zip" ) { + auto archive = ZipProvider::archiveURI( uri ); + auto member = ZipProvider::memberPath( uri ); + if ( archive && member ) { + if ( member.value() == "/" ) + return archive.value().getParent(); + URI entry; + entry.setRawPath( std::move( member ).value() ); + return ZipProvider::makeURI( archive.value(), entry.getParent().getPath() ); + } + } + return uri.getParent(); +} + +std::string Location::entryName( const URI& uri ) { + if ( uri.getScheme() == "zip" ) { + auto archive = ZipProvider::archiveURI( uri ); + auto member = ZipProvider::memberPath( uri ); + if ( archive && member && member.value() == "/" ) + return archive.value().getLastPathSegment(); + } + return uri.getLastPathSegment(); +} + String Location::displayName( const URI& uri ) { - std::string name = uri.getLastPathSegment(); + std::string name = entryName( uri ); + if ( uri.getScheme() == "zip" && !name.empty() ) + return String::fromUtf8( name ); if ( name.empty() || uri.getPath() == "/" ) name = uri.getScheme() == "file" ? std::string( "/" ) : uri.getHost(); return String::fromUtf8( name ); } String Location::text( const URI& uri ) { + if ( uri.getScheme() == "zip" ) { + // URI::toString() omits an empty authority for non-file schemes. Spell it explicitly + // in the location bar; both spellings parse into the same provider URI. + return String::fromUtf8( uri.getSchemeAndAuthority() + uri.getPathEtc() ); + } if ( uri.getScheme() == "file" ) { std::string path = uri.getFSPath(); FileSystem::dirAddSlashAtEnd( path ); diff --git a/src/tools/eefm/location.hpp b/src/tools/eefm/location.hpp index af0f0205d..2ca1aec19 100644 --- a/src/tools/eefm/location.hpp +++ b/src/tools/eefm/location.hpp @@ -7,21 +7,29 @@ namespace eefm { +using namespace EE; + /** How locations are typed, shown and copied. */ class Location { public: /** A typed location: a URI, or a local path ("~" expands to the home folder). */ - static EE::URI parse( const std::string& text ); + static URI parse( const std::string& text ); + + /** Parent location; leaving an archive root returns to the archive's containing folder. */ + static URI parent( const URI& uri ); + + /** Entry to select when leaving this location (the archive's name at its root). */ + static std::string entryName( const URI& uri ); /** The name of a folder for titles: its last segment, "/" or the host for roots. */ - static EE::String displayName( const EE::URI& uri ); + static String displayName( const URI& uri ); /** The location bar text: a local path with a trailing slash, the URI otherwise. */ - static EE::String text( const EE::URI& uri ); + static String text( const URI& uri ); /** A location for Copy Location: a plain path (no trailing slash) for local files and * folders, the URI otherwise. Copying files (Ctrl+C) puts URIs in the clipboard instead. */ - static std::string clipboardText( const EE::URI& uri ); + static std::string clipboardText( const URI& uri ); }; } // namespace eefm diff --git a/src/tools/eefm/panemanager.cpp b/src/tools/eefm/panemanager.cpp index 48ed1648b..1100eef60 100644 --- a/src/tools/eefm/panemanager.cpp +++ b/src/tools/eefm/panemanager.cpp @@ -568,7 +568,7 @@ void PaneManager::navigate( FileTab* tab, URI uri, bool recordHistory ) { void PaneManager::goBack() { auto* tab = currentTab(); if ( tab && tab->historyIndex > 0 ) { - tab->selectLater( tab->model->uri().getLastPathSegment() ); + tab->selectLater( Location::entryName( tab->model->uri() ) ); --tab->historyIndex; navigate( tab, tab->history[tab->historyIndex], false ); } @@ -586,10 +586,10 @@ void PaneManager::goUp() { auto* tab = currentTab(); if ( !tab ) return; - URI parent = tab->model->uri().getParent(); + URI parent = Location::parent( tab->model->uri() ); if ( parent != tab->model->uri() ) { // Like Dolphin, the folder we leave stays selected in its parent. - tab->selectLater( tab->model->uri().getLastPathSegment() ); + tab->selectLater( Location::entryName( tab->model->uri() ) ); navigate( tab, parent ); } } @@ -649,7 +649,7 @@ void PaneManager::updateChrome( Pane& pane ) { return; mUI->find( "back" )->setEnabled( tab->historyIndex > 0 ); mUI->find( "forward" )->setEnabled( tab->historyIndex + 1 < tab->history.size() ); - mUI->find( "up" )->setEnabled( uri.getParent() != uri ); + mUI->find( "up" )->setEnabled( Location::parent( uri ) != uri ); mApp.window()->setTitle( Location::text( uri ) + " — eefm" ); if ( auto* panel = mApp.places().panel() ) panel->setCurrent( uri );