feat(vfs): add ZIP browsing and optional shared module builds

- Add a read-only ZipProvider with streaming reads, cached metadata,
  change notifications, and zip:///archive.zip!/member URIs.
- Support ZIP navigation and copying files out of archives in eefm.
- Extend Zip with explicit read-only access, optional resource
  registration, and improved stream, CRC, and error handling.
- Add EE_VFS_API exports and --with-shared-vfs support; reject
  combining shared VFS with static eepp.
- Fix polling watcher teardown from its own thread.
- Add regression tests and document ZIP and shared module behavior.
This commit is contained in:
Martín Lucas Golini
2026-10-17 01:50:55 -03:00
parent aeca4cd4ce
commit 5f2c9677d6
43 changed files with 1896 additions and 312 deletions
+63 -1
View File
@@ -2,12 +2,13 @@
## Introduction
`eepp-vfs` is an optional static module that gives applications one asynchronous, URI-based API
`eepp-vfs` is an optional module that gives applications one asynchronous, URI-based API
for local files, packed resources and remote servers:
| Scheme | Provider | Notes |
| --- | --- | --- |
| `file://` | `LocalProvider` | Native filesystem (POSIX and Win32). |
| `zip://` | `ZipProvider` | Read-only browsing and streaming reads of individual local ZIP archives. |
| `res://` | `ResourceProvider` | Read-only view of every open `Pack` (`System::PackFileIndex`). |
| `sftp://` | `SFTPProvider` | Built by default (`--without-vfs-sftp` disables it), libssh2 based, honors `~/.ssh/config`. |
| `mem://` | `MemoryProvider` | In-memory filesystem, mainly for tests. |
@@ -26,6 +27,12 @@ are useful on their own:
Link the module with `eepp_module_vfs_add()` in Premake and include `<eepp/vfs.hpp>`. The UI model
lives in `<eepp/vfs/ui/vfsdirectorymodel.hpp>` and is not part of the aggregate header.
The default is `eepp-vfs-static`; generate with `--with-shared-vfs` to build and link `eepp-vfs`
as a shared library. Public types and non-inline functions use `EE_VFS_API`, controlled by
`EE_VFS_EXPORTS` for the DLL and `EE_VFS_STATIC` for static builds, independently of core eepp
exports. The Premake helper sets the consumer flags and dependencies. The MinGW helper accepts
`--with-shared-vfs` as well. Combining it with `--with-static-eepp` is rejected: a shared VFS
must use the same eepp DLL as the application, including its resource and UI singletons.
## Tasks, results and threads
@@ -256,6 +263,61 @@ on the server's `PATH`, or set `SFTPProviderConfig::watchAgentCommand`. The eepp
same connection.
- The efsw-agent is not deployed automatically.
## ZIP archives (`zip://`)
`ZipProvider` is registered by default. Build archive locations with `makeURI()` rather than
concatenating paths: the backing file URI and the path inside the archive are separate identities.
Locations use `zip:///path/archive.zip!/member`. The host, when present, identifies a local UNC
archive. Literal `!` characters in the backing path are doubled, so a directory named `packages!`
uses `packages!!/` before the archive boundary. Member names use ordinary URI escaping.
`withAppendedSegment()` appends a member; eefm handles crossing the archive root with Up.
```cpp
URI archive = URI::fromFSPath( "/home/me/downloads/package.zip" );
URI root = ZipProvider::makeURI( archive );
URI folder = root.withAppendedSegment( "docs" );
vfs.listAll( folder ).then( ui, []( VFSResult<std::vector<VFSFileInfo>>&& entries ) { ... } );
vfs.readAll( folder.withAppendedSegment( "README.txt" ) );
vfs.copy( folder, URI::fromFSPath( "/home/me/extracted-docs" ) );
```
Directories exist both for explicit directory entries (including empty directories) and implicitly
for the parents of files. Listings include uncompressed file sizes and entry timestamps. Permission bits are unspecified
(the archive API does not expose them), so copies receive the destination filesystem's defaults. Browsing
never registers an archive in `PackRegistry` or `PackFileIndex`, so archive contents do not become
application resources. Opening a missing archive reports `NotFound` and never creates a file.
The provider is read-only: creating, replacing, renaming or deleting entries returns `ReadOnly`.
Files stream through independent readers without extracting the entire entry into memory.
Sequential reads retain the decompression position; seeking backwards replays the compressed
prefix with bounded scratch memory. `Seek` is advertised; `RandomAccess` is not. CRC failures are
reported as `IO`, including when the final read requests exactly the remaining file size.
Changes to the backing file invalidate its cached index for new operations. Open handles keep
their original reader and metadata; modifying an archive in place can cause their reads to fail.
The normal VFS polling backend detects archive changes. Archives on network mounts use the local
I/O scheduler's mount isolation and timeouts.
Only `file://` backing archives are supported initially. Password-protected entries and unsupported
compression methods report `Unsupported`. ZIP symlink attributes are not followed: their stored
data is exposed as a regular file. Unsafe names (absolute paths, traversal components, backslashes
or drive/stream colons) are excluded. Indexes are limited to 262,144 entries including implicit
directories, with at most eight cached indexes; existing handles can retain older indexes.
In eefm, Enter or double-click on a local `.zip` file enters it. Files inside open through the usual
application flow, and copying entries out uses VFS transfers. Up at the archive root returns to its
containing folder and selects the archive. History and saved tabs retain archive locations. The
location bar displays a readable address such as
`zip:///home/me/downloads/package.zip!/docs`, which can also be typed or pasted there.
For lower-level archive access, construct `Zip archive(false)` to opt out of the resource registry
and call `archive.open(path, Zip::OpenMode::ReadOnly)`. `entryCount()`, `entry()` and `openStream()`
provide metadata and streaming access. Streams borrow the archive and must be destroyed before it;
callers sharing an archive across threads must serialize access. `open()` never creates a missing
file; writing starts explicitly with `create()`. The default constructor retains resource-pack
registration for the engine and other existing pack consumers.
## Resources (`res://`)
`System::PackFileIndex` (formerly `System::VirtualFileSystem`) indexes the files of every open `Pack`. `res:///fonts/a.ttf` maps to the pack path
+5 -2
View File
@@ -16,12 +16,14 @@ class EE_API IOStreamZip : public IOStream {
public:
static IOStreamZip* New( Zip* pack, const std::string& path );
/** @brief Open a file from a zip file
/** @brief Open a file from a zip file. The pack must outlive the stream.
** @param pack Pack to open from path
** @param path Path of the file in the pack file
**/
IOStreamZip( Zip* pack, const std::string& path );
IOStreamZip( Zip* pack, Uint64 index );
virtual ~IOStreamZip();
ios_size read( char* data, ios_size size );
@@ -37,9 +39,10 @@ class EE_API IOStreamZip : public IOStream {
bool isOpen();
protected:
std::string mPath;
struct zip* mZip;
struct zip_file* mFile;
Uint64 mIndex;
ios_size mSize;
ios_size mPos;
};
+4
View File
@@ -14,6 +14,9 @@ class EE_API Pack : protected Mutex {
public:
Pack();
/** Standalone archives can opt out of PackRegistry and PackFileIndex. */
explicit Pack( bool registerResource );
virtual ~Pack();
/** Creates a new pack file */
@@ -81,6 +84,7 @@ class EE_API Pack : protected Mutex {
protected:
bool mIsOpen;
bool mRegisterResource;
void onPackOpened();
+54 -2
View File
@@ -2,26 +2,66 @@
#define EE_SYSTEMCZIP_HPP
#include <eepp/system/pack.hpp>
#include <memory>
struct zip;
namespace EE { namespace System {
/** @brief Zip files package manager. */
/** ZIP archive reader/writer and optional resource pack. Use Zip(false) and OpenMode::ReadOnly
* for standalone browsing. Entry streams borrow the archive and must be destroyed before it.
* Serialize access to an archive and its streams when sharing them between threads. */
class EE_API Zip : public Pack {
public:
static Zip* New();
struct Entry {
std::string name;
Uint64 size{ 0 };
Int64 modified{ 0 };
Uint32 crc{ 0 };
};
enum class OpenMode : Uint8 { ReadOnly, ReadWrite };
enum class Error : Uint8 {
None,
NotFound,
PermissionDenied,
InvalidArchive,
Unsupported,
ReadOnly,
IO
};
Zip();
explicit Zip( bool registerResource );
~Zip();
/** Creates a new pack file */
bool create( const std::string& path );
/** Open a pack file */
/** Opens an existing archive for reading and writing. Missing files are never created;
* use create() explicitly. */
bool open( const std::string& path );
/** ReadOnly supports Unicode native paths and cannot modify or create an archive. */
bool open( const std::string& path, OpenMode mode );
Uint64 entryCount() const;
bool entry( Uint64 index, Entry& result );
/** Opens an entry without extracting it into memory. Backward seeks replay the prefix;
* scratch space is bounded. Reads/seeks return -1 on error, including CRC failures. */
std::unique_ptr<IOStream> openStream( Uint64 index );
Error error() const { return mError; }
const std::string& errorMessage() const { return mErrorMessage; }
/** Close the pack file */
bool close();
@@ -80,8 +120,20 @@ class EE_API Zip : public Pack {
struct zip* mZip;
std::string mZipPath;
std::string mErrorMessage;
OpenMode mOpenMode{ OpenMode::ReadWrite };
Error mError{ Error::None };
struct zip* getZip();
private:
bool openReadOnly( const std::string& path );
void setError( int code, int systemCode = 0 );
bool writable();
bool extractStream( IOStream& stream, Uint8* data, size_t size );
};
}} // namespace EE::System
+64 -23
View File
@@ -193,6 +193,10 @@ newoption {
description = "Set the shared data directory (default: /usr/share/ecode)",
}
newoption { trigger = "with-static-cpp", description = "Builds statically libstdc++" }
newoption { trigger = "with-shared-vfs", description = "Builds and links the VFS module as a shared library instead of a static library." }
if _OPTIONS["with-static-eepp"] and _OPTIONS["with-shared-vfs"] then
error("--with-shared-vfs requires eepp-shared; it cannot be combined with --with-static-eepp")
end
newoption { trigger = "without-vfs-sftp", description = "Disables sftp:// support in the eepp-vfs module (otherwise the vendored libssh2 is built)." }
newoption { trigger = "with-lto", description = "Enables Link-Time Optimization for release builds." }
@@ -1085,8 +1089,8 @@ function vfs_sftp_enabled()
return _OPTIONS["without-vfs-sftp"] == nil and not os.is_real("emscripten")
end
function eepp_module_vfs_add()
links { "eepp-vfs-static", "efsw-static" }
function eepp_module_vfs_dependencies()
links { "efsw-static" }
includedirs { "src/modules/vfs/include/", "src/thirdparty/efsw/include" }
-- The ssh_config evaluator (always built) resolves host names through Winsock.
if os.is_real("windows") or os.is_real("mingw32") or os.is_real("mingw64") then
@@ -1111,6 +1115,58 @@ function eepp_module_vfs_add()
end
end
function eepp_module_vfs_add()
includedirs { "src/modules/vfs/include/", "src/thirdparty/efsw/include" }
if vfs_sftp_enabled() then
defines { "EE_VFS_SFTP" }
end
if _OPTIONS["with-shared-vfs"] then
links { "eepp-vfs" }
else
links { "eepp-vfs-static" }
defines { "EE_VFS_STATIC" }
eepp_module_vfs_dependencies()
end
end
function build_vfs_module( shared )
local name = shared and "eepp-vfs" or "eepp-vfs-static"
project(name)
kind(shared and "SharedLib" or "StaticLib")
language "C++"
set_targetdir("libs/" .. os.get_real() .. "/")
includedirs { "include", "src/modules/vfs/include/", "src/modules/vfs/src/", "src/thirdparty/efsw/include" }
files { "src/modules/vfs/src/**.cpp" }
defines { shared and "EE_VFS_EXPORTS" or "EE_VFS_STATIC" }
if shared then
eepp_module_vfs_dependencies()
links { "eepp-shared" }
end
if vfs_sftp_enabled() then
defines { "EE_VFS_SFTP" }
includedirs { "src/thirdparty/libssh2/include" }
else
excludes { "src/modules/vfs/src/eepp/vfs/sftp/**.cpp" }
end
if _OPTIONS["with-static-eepp"] then
defines { "EE_STATIC" }
end
if not is_vs() then
buildoptions{ "-std=c++20" }
else
buildoptions{ "/std:c++20" }
end
build_base_cpp_configuration( name )
if shared then
postsymlinklib("../libs/" .. os.get_real() .. "/", "../../bin/", "eepp-vfs" )
postsymlinklib("../libs/" .. os.get_real() .. "/", "../../bin/unit_tests/", "eepp-vfs" )
if os.is_real("mingw32") or os.is_real("mingw64") or (os.is("windows") and not is_vs()) then
links { "winpthread" }
end
end
end
function eepp_module_backward_add( ecode_mode )
files { "src/modules/backward/src/backward.cpp" }
includedirs { "src/thirdparty" }
@@ -1761,27 +1817,10 @@ solution "eepp"
end
build_base_cpp_configuration( "efsw-agent" )
project "eepp-vfs-static"
kind "StaticLib"
language "C++"
set_targetdir("libs/" .. os.get_real() .. "/")
includedirs { "include", "src/modules/vfs/include/", "src/modules/vfs/src/", "src/thirdparty/efsw/include" }
files { "src/modules/vfs/src/**.cpp" }
if vfs_sftp_enabled() then
defines { "EE_VFS_SFTP" }
includedirs { "src/thirdparty/libssh2/include" }
else
excludes { "src/modules/vfs/src/eepp/vfs/sftp/**.cpp" }
end
if _OPTIONS["with-static-eepp"] then
defines { "EE_STATIC" }
end
if not is_vs() then
buildoptions{ "-std=c++20" }
else
buildoptions{ "/std:c++20" }
end
build_base_cpp_configuration( "eepp-vfs" )
build_vfs_module( false )
if _OPTIONS["with-shared-vfs"] then
build_vfs_module( true )
end
project "languages-syntax-highlighting-static"
kind "StaticLib"
@@ -2297,6 +2336,7 @@ solution "eepp"
links { "util" }
end
files { "src/tests/unit_tests/*.cpp",
"src/tools/eefm/location.cpp",
"src/tools/eproc/process_info.cpp",
"src/tools/eproc/process_model.cpp",
"src/tools/eproc/process_table_state.cpp",
@@ -2313,6 +2353,7 @@ solution "eepp"
links { "Cocoa.framework" }
end
eepp_module_vfs_add()
links { "efsw-static" }
eepp_module_backward_add( false )
configuration { "linux" }
+64 -24
View File
@@ -51,6 +51,10 @@ newoption {
description = "Set the shared data directory",
}
newoption { trigger = "with-static-cpp", description = "Builds statically libstdc++" }
newoption { trigger = "with-shared-vfs", description = "Builds and links the VFS module as a shared library instead of a static library." }
if _OPTIONS["with-static-eepp"] and _OPTIONS["with-shared-vfs"] then
error("--with-shared-vfs requires eepp-shared; it cannot be combined with --with-static-eepp")
end
newoption { trigger = "without-vfs-sftp", description = "Disables sftp:// support in the eepp-vfs module (otherwise the vendored libssh2 is built)." }
newoption { trigger = "with-lto", description = "Enables Link-Time Optimization for release builds." }
@@ -1006,8 +1010,8 @@ function vfs_sftp_enabled()
return _OPTIONS["without-vfs-sftp"] == nil and not os.istarget("emscripten")
end
function eepp_module_vfs_add()
links { "eepp-vfs-static", "efsw-static" }
function eepp_module_vfs_dependencies()
links { "efsw-static" }
incdirs { "src/modules/vfs/include/", "src/thirdparty/efsw/include" }
-- The ssh_config evaluator (always built) resolves host names through Winsock.
filter "system:windows"
@@ -1033,6 +1037,57 @@ function eepp_module_vfs_add()
end
end
function eepp_module_vfs_add()
incdirs { "src/modules/vfs/include/", "src/thirdparty/efsw/include" }
if vfs_sftp_enabled() then
defines { "EE_VFS_SFTP" }
end
if _OPTIONS["with-shared-vfs"] then
links { "eepp-vfs" }
else
links { "eepp-vfs-static" }
defines { "EE_VFS_STATIC" }
eepp_module_vfs_dependencies()
end
end
function build_vfs_module( shared )
local name = shared and "eepp-vfs" or "eepp-vfs-static"
project(name)
kind(shared and "SharedLib" or "StaticLib")
language "C++"
cppdialect "C++20"
incdirs { "include", "src/modules/vfs/include/", "src/modules/vfs/src/", "src/thirdparty/efsw/include" }
files { "src/modules/vfs/src/**.cpp" }
defines { shared and "EE_VFS_EXPORTS" or "EE_VFS_STATIC" }
if shared then
eepp_module_vfs_dependencies()
links { "eepp-shared" }
end
if vfs_sftp_enabled() then
defines { "EE_VFS_SFTP" }
incdirs { "src/thirdparty/libssh2/include" }
else
excludes { "src/modules/vfs/src/eepp/vfs/sftp/**.cpp" }
end
if _OPTIONS["with-static-eepp"] then
defines { "EE_STATIC" }
end
build_base_cpp_configuration( name )
target_dir_lib("")
filter "action:not vs*"
buildoptions { "-Wall" }
filter { "action:export-compile-commands", "system:macosx" }
buildoptions { "-std=c++20" }
if shared then
postsymlinklib_arch( "eepp-vfs" )
filter { "system:windows", "action:not vs*" }
links { "winpthread" }
filter {}
end
end
function eepp_module_backward_add( ecode_mode )
files { "src/modules/backward/src/backward.cpp" }
incdirs { "src/thirdparty" }
@@ -1202,7 +1257,7 @@ function postsymlinklib_arch(name)
postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/arm64/", _MAIN_SCRIPT_DIR .. "/bin/", name, "options:arch=arm64" )
postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/arm64/", _MAIN_SCRIPT_DIR .. "/bin/", name, "options:arch=aarch64" )
postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/universal/", _MAIN_SCRIPT_DIR .. "/bin/", name, "architecture:universal" )
if name == "eepp" then
if name == "eepp" or name == "eepp-vfs" then
postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/x86/", _MAIN_SCRIPT_DIR .. "/bin/unit_tests/", name, "architecture:x86" )
postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/x86_64/", _MAIN_SCRIPT_DIR .. "/bin/unit_tests/", name, "architecture:x86_64" )
postsymlinklib( _MAIN_SCRIPT_DIR .. "/libs/" .. os.target() .. "/arm/", _MAIN_SCRIPT_DIR .. "/bin/unit_tests/", name, "architecture:ARM" )
@@ -1781,27 +1836,10 @@ workspace "eepp"
links { "CoreFoundation.framework", "CoreServices.framework" }
filter {}
project "eepp-vfs-static"
kind "StaticLib"
language "C++"
cppdialect "C++20"
incdirs { "include", "src/modules/vfs/include/", "src/modules/vfs/src/", "src/thirdparty/efsw/include" }
files { "src/modules/vfs/src/**.cpp" }
if vfs_sftp_enabled() then
defines { "EE_VFS_SFTP" }
incdirs { "src/thirdparty/libssh2/include" }
else
excludes { "src/modules/vfs/src/eepp/vfs/sftp/**.cpp" }
end
if _OPTIONS["with-static-eepp"] then
defines { "EE_STATIC" }
end
build_base_cpp_configuration( "eepp-vfs" )
target_dir_lib("")
filter "action:not vs*"
buildoptions { "-Wall" }
filter { "action:export-compile-commands", "system:macosx" }
buildoptions { "-std=c++20" }
build_vfs_module( false )
if _OPTIONS["with-shared-vfs"] then
build_vfs_module( true )
end
project "languages-syntax-highlighting-static"
kind "StaticLib"
@@ -2289,6 +2327,7 @@ workspace "eepp"
incdirs { "src/modules/eterm/include/", "src/thirdparty", "src/modules/vfs/src/" }
language "C++"
files { "src/tests/unit_tests/*.cpp",
"src/tools/eefm/location.cpp",
"src/tools/eproc/process_info.cpp",
"src/tools/eproc/process_model.cpp",
"src/tools/eproc/process_table_state.cpp",
@@ -2317,6 +2356,7 @@ workspace "eepp"
links { "util" }
filter {}
eepp_module_vfs_add()
links { "efsw-static" }
eepp_module_backward_add( false )
filter { "system:linux" }
+5 -1
View File
@@ -3,6 +3,7 @@ cd "$(dirname "$0")" || exit
ARCH=32
BACKEND=sdl2
VFS_SHARED_ARG=
MAKE_ARGS=()
for i in "$@"; do
case $i in
@@ -10,6 +11,9 @@ for i in "$@"; do
CONFIG="${i#*=}"
MAKE_ARGS+=("$i")
;;
--with-shared-vfs)
VFS_SHARED_ARG="--with-shared-vfs"
;;
--backend=*)
BACKEND="${i#*=}"
;;
@@ -78,7 +82,7 @@ detect_backend_name()
fi
}
PREMAKE5_ARGS="--file=../../premake5.lua --os=windows --cc=gcc --windows-mingw-build --with-backend=$(detect_backend_arg) $PREMAKE5_ARCH gmake"
PREMAKE5_ARGS="--file=../../premake5.lua --os=windows --cc=gcc --windows-mingw-build --with-backend=$(detect_backend_arg) $PREMAKE5_ARCH $VFS_SHARED_ARG gmake"
if command -v premake5 &> /dev/null
then
+44 -42
View File
@@ -1,7 +1,10 @@
#include <eepp/system/iostreamzip.hpp>
#include <algorithm>
#include <array>
#include <eepp/system/zip.hpp>
#include <libzip/zip.h>
#include <libzip/zipint.h>
#include <limits>
namespace EE { namespace System {
@@ -10,61 +13,62 @@ IOStreamZip* IOStreamZip::New( Zip* pack, const std::string& path ) {
}
IOStreamZip::IOStreamZip( Zip* pack, const std::string& path ) :
mPath( path ), mZip( pack->getZip() ), mFile( NULL ), mPos( 0 ) {
struct zip_stat zs;
int err = zip_stat( mZip, path.c_str(), 0, &zs );
IOStreamZip( pack, static_cast<Uint64>( pack->exists( path ) ) ) {}
if ( !err ) {
mFile = zip_fopen_index( mZip, zs.index, 0 );
IOStreamZip::IOStreamZip( Zip* pack, Uint64 index ) :
mZip( pack->getZip() ), mFile( nullptr ), mIndex( index ), mSize( 0 ), mPos( 0 ) {
struct zip_stat info;
zip_stat_init( &info );
if ( mZip && zip_stat_index( mZip, index, 0, &info ) == 0 &&
info.size <= static_cast<Uint64>( std::numeric_limits<ios_size>::max() ) ) {
mSize = static_cast<ios_size>( info.size );
mFile = zip_fopen_index( mZip, index, 0 );
}
}
IOStreamZip::~IOStreamZip() {
if ( isOpen() ) {
if ( mFile )
zip_fclose( mFile );
}
}
ios_size IOStreamZip::read( char* data, ios_size size ) {
int res = -1;
if ( isOpen() ) {
res = zip_fread( mFile, reinterpret_cast<void*>( &data[0] ), size );
if ( -1 != res ) {
mPos += size;
if ( !mFile || size < 0 )
return -1;
auto count = zip_fread( mFile, data, static_cast<Uint64>( size ) );
if ( count > 0 ) {
mPos += count;
// libzip validates the CRC only on the read past EOF. Validate before reporting the
// last bytes: offset-based callers may never request another read on this stream.
if ( mPos >= mSize ) {
char extra;
if ( mPos != mSize || zip_fread( mFile, &extra, 1 ) != 0 )
return -1;
}
}
return -1 != res ? res : 0;
return count;
}
ios_size IOStreamZip::write( const char* data, ios_size size ) {
return 0;
ios_size IOStreamZip::write( const char*, ios_size ) {
return -1;
}
ios_size IOStreamZip::seek( ios_size position ) {
if ( isOpen() && mPos != position ) {
if ( !mFile || position < 0 || position > mSize )
return -1;
if ( position < mPos ) {
zip_fclose( mFile );
struct zip_stat zs;
int err = zip_stat( mZip, mPath.c_str(), 0, &zs );
if ( !err ) {
mFile = zip_fopen_index( mZip, zs.index, 0 );
if ( 0 != position ) {
ScopedBuffer ptr( position );
read( (char*)ptr.get(), position );
}
mPos = position;
return position;
}
mFile = zip_fopen_index( mZip, mIndex, 0 );
mPos = 0;
if ( !mFile )
return -1;
}
return 0;
std::array<char, 16384> scratch;
while ( mPos < position ) {
const auto count = std::min<ios_size>( position - mPos, scratch.size() );
if ( read( scratch.data(), count ) <= 0 )
return -1;
}
return mPos;
}
ios_size IOStreamZip::tell() {
@@ -72,13 +76,11 @@ ios_size IOStreamZip::tell() {
}
ios_size IOStreamZip::getSize() {
struct zip_stat zs;
int err = zip_stat( mZip, mPath.c_str(), 0, &zs );
return !err ? zs.size : 0;
return mSize;
}
bool IOStreamZip::isOpen() {
return NULL != mFile;
return mFile != nullptr;
}
}} // namespace EE::System
+12 -6
View File
@@ -1,15 +1,19 @@
#include <eepp/system/pack.hpp>
#include <eepp/system/packregistry.hpp>
#include <eepp/system/packfileindex.hpp>
#include <eepp/system/packregistry.hpp>
namespace EE { namespace System {
Pack::Pack() : Mutex(), mIsOpen( false ) {
PackRegistry::instance()->add( this );
Pack::Pack() : Pack( true ) {}
Pack::Pack( bool registerResource ) : mIsOpen( false ), mRegisterResource( registerResource ) {
if ( mRegisterResource )
PackRegistry::instance()->add( this );
}
Pack::~Pack() {
PackRegistry::instance()->remove( this );
if ( mRegisterResource )
PackRegistry::instance()->remove( this );
}
bool Pack::isOpen() const {
@@ -17,11 +21,13 @@ bool Pack::isOpen() const {
}
void Pack::onPackOpened() {
PackFileIndex::instance()->onResourceAdd( this );
if ( mRegisterResource )
PackFileIndex::instance()->onResourceAdd( this );
}
void Pack::onPackClosed() {
PackFileIndex::instance()->onResourceRemove( this );
if ( mRegisterResource )
PackFileIndex::instance()->onResourceRemove( this );
}
}} // namespace EE::System
+223 -128
View File
@@ -1,8 +1,18 @@
#include <eepp/system/zip.hpp>
#include <algorithm>
#include <cerrno>
#include <eepp/system/filesystem.hpp>
#include <eepp/system/iostreamzip.hpp>
#include <eepp/system/zip.hpp>
#include <eepp/system/lock.hpp>
#include <libzip/zip.h>
#include <libzip/zipint.h>
#include <limits>
#if EE_PLATFORM == EE_PLATFORM_WIN
#include <io.h>
#else
#include <unistd.h>
#endif
namespace EE { namespace System {
@@ -10,56 +20,61 @@ Zip* Zip::New() {
return eeNew( Zip, () );
}
Zip::Zip() : mZip( NULL ) {}
Zip::Zip() : Zip( true ) {}
Zip::Zip( bool registerResource ) : Pack( registerResource ), mZip( nullptr ) {}
Zip::~Zip() {
close();
}
bool Zip::create( const std::string& path ) {
if ( !FileSystem::fileExists( path ) ) {
int err;
mZip = zip_open( path.c_str(), ZIP_CREATE, &err );
if ( 0 == checkPack() ) {
mZipPath = path;
mIsOpen = true;
onPackOpened();
return true;
}
} else {
return open( path );
if ( !close() )
return false;
mError = Error::None;
mErrorMessage.clear();
mOpenMode = OpenMode::ReadWrite;
int error = ZIP_ER_OK;
mZip = zip_open( path.c_str(), ZIP_CREATE, &error );
if ( !mZip ) {
setError( error, errno );
return false;
}
return false;
mZipPath = path;
mIsOpen = true;
onPackOpened();
return true;
}
bool Zip::open( const std::string& path ) {
if ( FileSystem::fileExists( path ) ) {
int err;
mZip = zip_open( path.c_str(), 0, &err );
return open( path, OpenMode::ReadWrite );
}
if ( 0 == checkPack() ) {
mZipPath = path;
mIsOpen = true;
onPackOpened();
return true;
}
bool Zip::open( const std::string& path, OpenMode mode ) {
if ( !close() )
return false;
mError = Error::None;
mErrorMessage.clear();
mOpenMode = mode;
if ( mode == OpenMode::ReadOnly ) {
if ( !openReadOnly( path ) )
return false;
} else {
return create( path );
int error = ZIP_ER_OK;
mZip = zip_open( path.c_str(), 0, &error );
if ( !mZip ) {
setError( error, errno );
return false;
}
}
return false;
mZipPath = path;
mIsOpen = true;
onPackOpened();
return true;
}
bool Zip::close() {
if ( 0 == checkPack() ) {
if ( mZip ) {
// zip_close() writes the pending changes. When that fails the archive is left allocated
// (and unchanged on disk): release it and report the failure.
bool written = zip_close( mZip ) == 0;
@@ -81,42 +96,50 @@ bool Zip::close() {
return written;
}
return false;
return true;
}
bool Zip::addFile( const std::string& path, const std::string& inpack ) {
ScopedBuffer file;
FileSystem::fileGet( path, file );
if ( !writable() || !FileSystem::fileGet( path, file ) )
return false;
return addFile( file.get(), file.length(), inpack );
}
bool Zip::addFile( const Uint8* data, const Uint32& dataSize, const std::string& inpack ) {
if ( !writable() )
return false;
if ( 0 == checkPack() ) {
struct zip_source* zs = zip_source_buffer( mZip, (const void*)data, dataSize, 0 );
if ( NULL != zs ) {
Int32 Result = (Int32)zip_add( mZip, inpack.c_str(), zs );
if ( Result < 0 ) {
zip_source_free( zs );
return false;
}
std::string path = mZipPath;
bool written = close();
open( path );
bool reopened = open( path );
if ( -1 != Result && written )
if ( written && reopened )
return true;
} else
zip_source_free( zs );
}
}
return false;
}
bool Zip::addFile( std::vector<Uint8>& data, const std::string& inpack ) {
return addFile( reinterpret_cast<const Uint8*>( &data[0] ), (Uint32)data.size(), inpack );
return addFile( data.data(), (Uint32)data.size(), inpack );
}
bool Zip::addFiles( std::map<std::string, std::string> paths ) {
if ( !writable() )
return false;
for ( std::map<std::string, std::string>::iterator itr = paths.begin(); itr != paths.end();
++itr )
if ( !addFile( itr->first, itr->second ) )
@@ -125,6 +148,8 @@ bool Zip::addFiles( std::map<std::string, std::string> paths ) {
}
bool Zip::eraseFile( const std::string& path ) {
if ( !writable() )
return false;
std::vector<std::string> tmpv;
tmpv.push_back( path );
@@ -132,6 +157,8 @@ bool Zip::eraseFile( const std::string& path ) {
}
bool Zip::eraseFiles( const std::vector<std::string>& paths ) {
if ( !writable() )
return false;
Int32 Ex;
Uint32 i = 0;
@@ -150,87 +177,53 @@ bool Zip::eraseFiles( const std::vector<std::string>& paths ) {
}
bool Zip::extractFile( const std::string& path, const std::string& dest ) {
lock();
bool Ret;
ScopedBuffer data;
Ret = extractFileToMemory( path, data );
if ( Ret )
FileSystem::fileWrite( dest, data.get(), data.length() );
unlock();
return Ret;
return extractFileToMemory( path, data ) &&
FileSystem::fileWrite( dest, data.get(), data.length() );
}
bool Zip::extractFileToMemory( const std::string& path, std::vector<Uint8>& data ) {
lock();
bool Ret = false;
Int32 Pos = exists( path );
if ( 0 == checkPack() && -1 != Pos ) {
data.clear();
struct zip_stat zs;
int err = zip_stat( mZip, path.c_str(), 0, &zs );
if ( !err ) {
struct zip_file* zf = zip_fopen_index( mZip, zs.index, 0 );
if ( NULL != zf ) {
data.resize( zs.size );
int Result =
(Int32)zip_fread( zf, reinterpret_cast<void*>( &data[0] ), data.size() );
zip_fclose( zf );
if ( -1 != Result )
Ret = true;
}
}
}
unlock();
return Ret;
Lock guard( *this );
const auto index = exists( path );
if ( index < 0 )
return false;
IOStreamZip stream( this, index );
if ( !stream.isOpen() || static_cast<Uint64>( stream.getSize() ) > data.max_size() )
return false;
data.resize( static_cast<size_t>( stream.getSize() ) );
return extractStream( stream, data.data(), data.size() );
}
bool Zip::extractFileToMemory( const std::string& path, ScopedBuffer& data ) {
lock();
Lock guard( *this );
const auto index = exists( path );
if ( index < 0 )
return false;
IOStreamZip stream( this, index );
if ( !stream.isOpen() ||
static_cast<Uint64>( stream.getSize() ) > std::numeric_limits<size_t>::max() )
return false;
data.reset( static_cast<size_t>( stream.getSize() ) );
return extractStream( stream, data.get(), data.size() );
}
bool Ret = false;
Int32 Pos = exists( path );
Int32 Result = 0;
if ( 0 == checkPack() && -1 != Pos ) {
struct zip_stat zs;
int err = zip_stat( mZip, path.c_str(), 0, &zs );
if ( !err ) {
struct zip_file* zf = zip_fopen_index( mZip, zs.index, 0 );
if ( NULL != zf ) {
data.reset( zs.size );
Result = (Int32)zip_fread( zf, (void*)data.get(), data.length() );
zip_fclose( zf );
if ( -1 != Result )
Ret = true;
}
bool Zip::extractStream( IOStream& stream, Uint8* data, size_t size ) {
size_t done = 0;
while ( done < size ) {
const auto count = stream.read( reinterpret_cast<char*>( data + done ),
std::min<size_t>( size - done, 65536 ) );
if ( count <= 0 ) {
setError( ZIP_ER_READ );
return false;
}
done += static_cast<size_t>( count );
}
unlock();
return Ret;
char extra;
if ( stream.read( &extra, 1 ) != 0 ) {
setError( ZIP_ER_CRC );
return false;
}
return true;
}
Int32 Zip::exists( const std::string& path ) {
@@ -245,21 +238,15 @@ Int8 Zip::checkPack() {
}
std::vector<std::string> Zip::getFileList() {
std::vector<std::string> tmpv;
Int32 numfiles = zip_get_num_files( mZip );
for ( Int32 i = 0; i < numfiles; i++ ) {
struct zip_stat zs;
if ( -1 != zip_stat_index( mZip, i, 0, &zs ) ) {
if ( zs.size > 0 ) {
tmpv.push_back( std::string( zs.name ) );
}
}
std::vector<std::string> files;
const auto count = entryCount();
files.reserve( static_cast<size_t>( count ) );
Entry info;
for ( Uint64 i = 0; i < count; ++i ) {
if ( entry( i, info ) && !info.name.ends_with( '/' ) )
files.push_back( std::move( info.name ) );
}
return tmpv;
return files;
}
/** @return The file path of the opened package */
@@ -275,4 +262,112 @@ zip* Zip::getZip() {
return mZip;
}
bool Zip::openReadOnly( const std::string& path ) {
// fopenUtf8 preserves Windows Unicode paths. zip_fdopen consumes a duplicated descriptor,
// so the FILE owned here can always be closed normally.
FILE* file = FileSystem::fopenUtf8( path, "rb" );
if ( !file ) {
setError( errno == ENOENT ? ZIP_ER_NOENT : ZIP_ER_OPEN, errno );
return false;
}
#if EE_PLATFORM == EE_PLATFORM_WIN
int descriptor = _dup( _fileno( file ) );
#else
int descriptor = dup( fileno( file ) );
#endif
const int duplicateError = errno;
std::fclose( file );
if ( descriptor < 0 ) {
setError( ZIP_ER_OPEN, duplicateError );
return false;
}
int error = ZIP_ER_OK;
mZip = zip_fdopen( descriptor, 0, &error );
if ( !mZip ) {
// This vendored libzip consumes descriptor after fdopen succeeds. Only its two early
// ZIP_ER_OPEN failures (dup/fdopen) leave it for the caller to close.
const int systemError = errno;
if ( error == ZIP_ER_OPEN ) {
#if EE_PLATFORM == EE_PLATFORM_WIN
_close( descriptor );
#else
::close( descriptor );
#endif
}
setError( error, systemError );
return false;
}
return true;
}
Uint64 Zip::entryCount() const {
if ( !mZip )
return 0;
const auto count = zip_get_num_entries( mZip, 0 );
return count == std::numeric_limits<Uint64>::max() ? 0 : count;
}
bool Zip::entry( Uint64 index, Entry& result ) {
struct zip_stat info;
zip_stat_init( &info );
if ( !mZip || zip_stat_index( mZip, index, 0, &info ) != 0 ) {
setError( ZIP_ER_INCONS );
return false;
}
result.name = info.name;
result.size = info.size;
result.modified = static_cast<Int64>( info.mtime );
result.crc = info.crc;
return true;
}
std::unique_ptr<IOStream> Zip::openStream( Uint64 index ) {
struct zip_stat info;
zip_stat_init( &info );
if ( !mZip || zip_stat_index( mZip, index, 0, &info ) != 0 ) {
setError( ZIP_ER_INCONS );
return {};
}
if ( info.size > static_cast<Uint64>( std::numeric_limits<ios_size>::max() ) ) {
setError( ZIP_ER_COMPNOTSUPP );
return {};
}
auto stream = std::make_unique<IOStreamZip>( this, index );
if ( !stream->isOpen() ) {
int code = 0;
int systemCode = 0;
zip_error_get( mZip, &code, &systemCode );
setError( code, systemCode );
return {};
}
return stream;
}
void Zip::setError( int code, int systemCode ) {
if ( code == ZIP_ER_NOENT )
mError = Error::NotFound;
else if ( zip_error_get_sys_type( code ) == ZIP_ET_SYS &&
( systemCode == EACCES || systemCode == EPERM ) )
mError = Error::PermissionDenied;
else if ( code == ZIP_ER_NOZIP || code == ZIP_ER_INCONS || code == ZIP_ER_EOF )
mError = Error::InvalidArchive;
else if ( code == ZIP_ER_COMPNOTSUPP || code == ZIP_ER_MULTIDISK ||
code == ZIP_ER_ENCRNOTSUPP || code == ZIP_ER_NOPASSWD || code == ZIP_ER_WRONGPASSWD )
mError = Error::Unsupported;
else
mError = Error::IO;
char message[256];
zip_error_to_str( message, sizeof( message ), code, systemCode );
mErrorMessage = message;
}
bool Zip::writable() {
if ( mOpenMode == OpenMode::ReadOnly ) {
mError = Error::ReadOnly;
mErrorMessage = "ZIP archive is read-only";
return false;
}
return mZip != nullptr;
}
}} // namespace EE::System
+1
View File
@@ -11,6 +11,7 @@
#include <eepp/vfs/providers/localprovider.hpp>
#include <eepp/vfs/providers/memoryprovider.hpp>
#include <eepp/vfs/providers/resourceprovider.hpp>
#include <eepp/vfs/providers/zipprovider.hpp>
#include <eepp/vfs/transfer.hpp>
#include <eepp/vfs/virtualfilesystem.hpp>
#include <eepp/vfs/watch.hpp>
@@ -1,7 +1,7 @@
#ifndef EE_VFS_CAPABILITIES_HPP
#define EE_VFS_CAPABILITIES_HPP
#include <eepp/config.hpp>
#include <eepp/vfs/config.hpp>
namespace EE { namespace VFS {
@@ -36,7 +36,7 @@ enum class VFSCapability : Uint32 {
/** Set of VFSCapability flags. Capabilities may vary per URI (for example per SFTP server), so
* providers report them per URI. */
class ProviderCapabilities {
class EE_VFS_API ProviderCapabilities {
public:
constexpr ProviderCapabilities() noexcept = default;
@@ -0,0 +1,22 @@
#ifndef EE_VFS_CONFIG_HPP
#define EE_VFS_CONFIG_HPP
#include <eepp/config.hpp>
#ifndef EE_VFS_API
#if defined( EE_VFS_STATIC )
#define EE_VFS_API
#elif EE_PLATFORM == EE_PLATFORM_WIN
#if defined( EE_VFS_EXPORTS )
#define EE_VFS_API __declspec( dllexport )
#else
#define EE_VFS_API __declspec( dllimport )
#endif
#elif defined( __GNUC__ ) && __GNUC__ >= 4
#define EE_VFS_API __attribute__( ( visibility( "default" ) ) )
#else
#define EE_VFS_API
#endif
#endif
#endif
@@ -2,6 +2,7 @@
#define EE_VFS_DIRECTORYHANDLE_HPP
#include <eepp/core/cancellationtoken.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/error.hpp>
#include <eepp/vfs/fileinfo.hpp>
#include <memory>
@@ -14,7 +15,7 @@ namespace EE { namespace VFS {
* endOfDirectory is explicit: an empty batch does not imply EOF, and the final batch may carry
* entries. "." and ".." are never reported.
*/
struct DirectoryBatch {
struct EE_VFS_API DirectoryBatch {
std::vector<VFSFileInfo> entries;
bool endOfDirectory{ false };
};
@@ -23,7 +24,7 @@ struct DirectoryBatch {
* a read is pending and must release native resources in their destructor if close() was never
* called. Calls are serialized by the handle's single consumer; implementations need not support
* concurrent read() calls. */
class DirectoryHandleImpl {
class EE_VFS_API DirectoryHandleImpl {
public:
virtual ~DirectoryHandleImpl() = default;
@@ -37,7 +38,7 @@ class DirectoryHandleImpl {
* Move-only. Destroying an open handle releases it without waiting. read() after EOF returns an
* empty batch with endOfDirectory set.
*/
class DirectoryHandle {
class EE_VFS_API DirectoryHandle {
public:
DirectoryHandle() = default;
+3 -3
View File
@@ -1,10 +1,10 @@
#ifndef EE_VFS_ERROR_HPP
#define EE_VFS_ERROR_HPP
#include <eepp/config.hpp>
#include <eepp/core/result.hpp>
#include <eepp/core/task.hpp>
#include <eepp/core/uri.hpp>
#include <eepp/vfs/config.hpp>
#include <string>
#include <string_view>
@@ -38,13 +38,13 @@ enum class VFSErrorCode : Uint8 {
};
/** @return A stable, human readable identifier for code (for example "NotFound"). */
const char* toString( VFSErrorCode code );
EE_VFS_API const char* toString( VFSErrorCode code );
/** @brief Typed VFS failure.
*
* nativeCode keeps the provider error (errno, Win32 error, SFTP status) for diagnostics only.
*/
struct VFSError {
struct EE_VFS_API VFSError {
URI uri;
std::string message;
Int64 nativeCode{ 0 };
@@ -3,6 +3,7 @@
#include <eepp/core/cancellationtoken.hpp>
#include <eepp/vfs/capabilities.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/error.hpp>
#include <eepp/vfs/fileinfo.hpp>
#include <memory>
@@ -16,7 +17,7 @@ namespace EE { namespace VFS {
* task completes. Implementations must release native resources in their destructor if close()
* was never called.
*/
class FileHandleImpl {
class EE_VFS_API FileHandleImpl {
public:
virtual ~FileHandleImpl() = default;
@@ -44,7 +45,7 @@ class FileHandleImpl {
* Offset-based operations avoid a shared mutable seek position, so independent reads can be
* pipelined. read() returning 0 bytes for a non-empty buffer means end of file.
*/
class FileHandle {
class EE_VFS_API FileHandle {
public:
FileHandle() = default;
@@ -3,8 +3,8 @@
#include <array>
#include <cstring>
#include <eepp/config.hpp>
#include <eepp/core/uri.hpp>
#include <eepp/vfs/config.hpp>
#include <initializer_list>
#include <optional>
#include <string>
@@ -22,7 +22,7 @@ enum class FileType : Uint8 {
Socket,
};
const char* toString( FileType type );
EE_VFS_API const char* toString( FileType type );
/** @brief Opaque, provider-defined revision token.
*
@@ -33,7 +33,7 @@ const char* toString( FileType type );
*
* Stored inline (no heap) because it travels inside every VFSFileInfo.
*/
class FileRevision {
class EE_VFS_API FileRevision {
public:
static constexpr size_t Capacity = 32;
@@ -92,7 +92,7 @@ class FileRevision {
* The entry does not store its URI: listings hold thousands of entries and the caller always
* knows the URI it listed or stat'ed. Use uriIn( directory ) for an entry of a listing.
*/
class VFSFileInfo {
class EE_VFS_API VFSFileInfo {
public:
/** Last path segment (verbatim, not percent-encoded). */
std::string name;
+9 -9
View File
@@ -1,13 +1,13 @@
#ifndef EE_VFS_OPTIONS_HPP
#define EE_VFS_OPTIONS_HPP
#include <eepp/config.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/fileinfo.hpp>
#include <optional>
namespace EE { namespace VFS {
struct StatOptions {
struct EE_VFS_API StatOptions {
/** When false the entry itself is described (lstat semantics). */
bool followSymlinks{ true };
};
@@ -20,7 +20,7 @@ enum class DirectoryMetadata : Uint8 {
Full,
};
struct DirectoryOpenOptions {
struct EE_VFS_API DirectoryOpenOptions {
DirectoryMetadata metadata{ DirectoryMetadata::Full };
/** Resolve the target type of symlinks (VFSFileInfo::targetType). May cost a stat per link. */
bool resolveSymlinkTargets{ true };
@@ -52,7 +52,7 @@ constexpr bool hasFlag( OpenFlags flags, OpenFlags flag ) noexcept {
return ( static_cast<Uint8>( flags ) & static_cast<Uint8>( flag ) ) != 0;
}
struct OpenOptions {
struct EE_VFS_API OpenOptions {
OpenFlags flags{ OpenFlags::Read };
/** Permissions for newly created files; provider default (0666 & ~umask) when unset. */
std::optional<Uint32> createPermissions;
@@ -62,21 +62,21 @@ struct OpenOptions {
OpenOptions( OpenFlags flags ) : flags( flags ) {}
};
struct MkDirOptions {
struct EE_VFS_API MkDirOptions {
std::optional<Uint32> permissions;
};
struct RemoveOptions {
struct EE_VFS_API RemoveOptions {
/** VFS-level only: remove directory contents recursively. Providers receive false. */
bool recursive{ false };
};
struct RenameOptions {
struct EE_VFS_API RenameOptions {
/** Replace an existing destination. Atomic when the provider reports AtomicReplace. */
bool overwrite{ false };
};
struct WriteOptions {
struct EE_VFS_API WriteOptions {
/** Write a sibling temporary file and atomically rename it over the destination. */
bool atomic{ true };
/** When atomic replacement is unsupported, fall back to an in-place write instead of failing
@@ -102,7 +102,7 @@ enum class OverwritePolicy : Uint8 {
Skip,
};
struct CopyOptions {
struct EE_VFS_API CopyOptions {
OverwritePolicy overwrite{ OverwritePolicy::Fail };
bool recursive{ true };
/** Copy permission bits when both sides support them. */
@@ -3,6 +3,7 @@
#include <eepp/core/cancellationtoken.hpp>
#include <eepp/vfs/capabilities.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/directoryhandle.hpp>
#include <eepp/vfs/error.hpp>
#include <eepp/vfs/filehandle.hpp>
@@ -12,7 +13,7 @@
namespace EE { namespace VFS {
struct SpaceInfo {
struct EE_VFS_API SpaceInfo {
Uint64 capacity{ 0 };
Uint64 free{ 0 };
/** Free space available to the current (non-privileged) user. */
@@ -33,7 +34,7 @@ struct SpaceInfo {
* - Cancellation is cooperative: a cancelled operation completes with VFSErrorCode::Cancelled.
* - Providers never touch UI or model state.
*/
class VFSProvider {
class EE_VFS_API VFSProvider {
public:
virtual ~VFSProvider() = default;
@@ -5,6 +5,7 @@
#include <eepp/core/task.hpp>
#include <eepp/system/threadpool.hpp>
#include <eepp/system/time.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/error.hpp>
#include <functional>
#include <memory>
@@ -28,10 +29,10 @@ class MountWatchdog;
* The kernel cannot interrupt a call on an unreachable hard NFS mount, so its thread stays blocked;
* nobody waits for it anymore, and its result is discarded if it ever arrives.
*/
class EE_API LocalIOScheduler {
class EE_VFS_API LocalIOScheduler {
public:
/** A file system mounted from another machine. */
struct NetworkMount {
struct EE_VFS_API NetworkMount {
/** Where it is mounted (a native path). */
std::string root;
/** What is mounted, for messages ("server:/export", "//server/share"). */
@@ -40,7 +41,7 @@ class EE_API LocalIOScheduler {
/** Where the calls of a path run: its network mount's lane, or the shared pool. Resolve it
* once for a handle (an open file or directory) and reuse it for the handle's calls. */
class Route {
class EE_VFS_API Route {
public:
bool isNetworkMount() const { return mLane != nullptr; }
@@ -128,7 +129,7 @@ class EE_API LocalIOScheduler {
/** A call queued on a lane: run once on the lane's thread, or failed by the watchdog, and
* only the first of the two completes its task. */
class Call {
class EE_VFS_API Call {
public:
virtual ~Call() = default;
@@ -154,7 +155,7 @@ class EE_API LocalIOScheduler {
/** What the routing needs, shared with the calls that resolve their route on the pool (which
* can outlive the scheduler). */
struct Routing {
struct EE_VFS_API Routing {
MountLookup lookup;
std::shared_ptr<MountWatchdog> watchdog;
@@ -1,6 +1,7 @@
#ifndef EE_VFS_PROVIDERS_LOCALPROVIDER_HPP
#define EE_VFS_PROVIDERS_LOCALPROVIDER_HPP
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/provider.hpp>
#include <eepp/vfs/providers/localioscheduler.hpp>
#include <memory>
@@ -19,7 +20,7 @@ namespace EE { namespace VFS {
* Revisions are derived from device, inode (file index), modification time and size, so both
* in-place edits and atomic replacements are detected.
*/
class LocalProvider : public VFSProvider {
class EE_VFS_API LocalProvider : public VFSProvider {
public:
explicit LocalProvider( std::shared_ptr<LocalIOScheduler> scheduler );
@@ -2,6 +2,7 @@
#define EE_VFS_PROVIDERS_MEMORYPROVIDER_HPP
#include <eepp/core/executor.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/provider.hpp>
#include <map>
#include <memory>
@@ -21,7 +22,7 @@ namespace EE { namespace VFS {
*
* Open file handles keep their data alive when the file is removed or replaced, like POSIX inodes.
*/
class MemoryProvider : public VFSProvider {
class EE_VFS_API MemoryProvider : public VFSProvider {
public:
explicit MemoryProvider( std::string scheme = "mem", Executor* completionExecutor = nullptr );
@@ -2,6 +2,7 @@
#define EE_VFS_PROVIDERS_RESOURCEPROVIDER_HPP
#include <eepp/system/threadpool.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/provider.hpp>
#include <memory>
@@ -16,7 +17,7 @@ namespace EE { namespace VFS {
* Revisions change whenever a pack is opened or closed. Sizes are reported once a file is opened
* (packs expose no cheap size query), so listing and stat leave them unknown.
*/
class ResourceProvider : public VFSProvider {
class EE_VFS_API ResourceProvider : public VFSProvider {
public:
explicit ResourceProvider( std::shared_ptr<System::ThreadPool> pool );
@@ -0,0 +1,66 @@
#ifndef EE_VFS_PROVIDERS_ZIPPROVIDER_HPP
#define EE_VFS_PROVIDERS_ZIPPROVIDER_HPP
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/provider.hpp>
#include <eepp/vfs/providers/localioscheduler.hpp>
#include <memory>
namespace EE { namespace VFS {
class ZipProviderImpl;
/** Read-only zip:// filesystem over individual local archives. Use makeURI() to identify an
* archive and a member path: zip:///path/archive.zip!/member. Literal bangs in the backing
* path are doubled to distinguish them from the archive boundary.
* Metadata is indexed without decompressing entries. Open handles retain their archive snapshot;
* new operations notice changes to the backing file. At most eight archive indexes are cached.
* Archives on network mounts use LocalIOScheduler's lanes and buffer-safe timeout delivery. */
class EE_VFS_API ZipProvider : public VFSProvider {
public:
explicit ZipProvider( std::shared_ptr<LocalIOScheduler> scheduler );
~ZipProvider() override;
/** Builds an archive location. member is an unescaped, '/'-separated path inside it. Only
* file:// backing archives are currently supported; other inputs return an empty URI.
* Root is "/". */
static URI makeURI( const URI& archive, std::string_view member = "/" );
/** Decodes and validates the backing file:// URI of a zip:// location. */
static VFSResult<URI> archiveURI( const URI& uri );
/** Validates a ZIP location and returns its normalized path within the archive. */
static VFSResult<std::string> memberPath( const URI& uri );
std::string_view scheme() const override { return "zip"; }
ProviderCapabilities capabilities( const URI& uri ) const override;
VFSTask<VFSFileInfo> stat( const URI& uri, const StatOptions& options,
CancellationToken token ) override;
VFSTask<DirectoryHandle> openDirectory( const URI& uri, const DirectoryOpenOptions& options,
CancellationToken token ) override;
VFSTask<FileHandle> open( const URI& uri, const OpenOptions& options,
CancellationToken token ) override;
VFSTask<void> mkdir( const URI& uri, const MkDirOptions& options,
CancellationToken token ) override;
VFSTask<void> remove( const URI& uri, const RemoveOptions& options,
CancellationToken token ) override;
VFSTask<void> rename( const URI& from, const URI& to, const RenameOptions& options,
CancellationToken token ) override;
VFSTask<URI> canonicalize( const URI& uri, CancellationToken token ) override;
private:
std::shared_ptr<ZipProviderImpl> mImpl;
};
}} // namespace EE::VFS
#endif
@@ -2,6 +2,7 @@
#define EE_VFS_SFTP_SFTPPROVIDER_HPP
#include <eepp/core/task.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/provider.hpp>
#include <eepp/vfs/ssh/sshconfig.hpp>
#include <eepp/vfs/watch.hpp>
@@ -17,7 +18,7 @@ class VirtualFileSystem;
/** Unknown host key presented for a user decision. Changed (mismatching) host keys are never
* offered: they always fail with VFSErrorCode::HostKeyRejected. */
struct SSHHostKeyInfo {
struct EE_VFS_API SSHHostKeyInfo {
std::string host;
/** Key algorithm, for example "ecdsa-sha2-nistp256". */
std::string keyType;
@@ -37,7 +38,7 @@ enum class SSHHostKeyDecision : Uint8 {
};
/** Application policy for unknown host keys (for example a UI dialog). */
class SSHHostKeyVerifier {
class EE_VFS_API SSHHostKeyVerifier {
public:
virtual ~SSHHostKeyVerifier() = default;
@@ -45,12 +46,12 @@ class SSHHostKeyVerifier {
virtual Task<SSHHostKeyDecision> verify( const SSHHostKeyInfo& info ) = 0;
};
struct SSHPrompt {
struct EE_VFS_API SSHPrompt {
std::string text;
bool echo{ false };
};
struct SSHAuthenticationRequest {
struct EE_VFS_API SSHAuthenticationRequest {
enum class Kind : Uint8 {
Password,
/** Passphrase of an encrypted identity file. */
@@ -74,7 +75,7 @@ struct SSHAuthenticationRequest {
/** Supplies secrets on demand. Secrets are used for one authentication attempt and never stored
* by the VFS (and never appear in URIs). */
class SSHAuthenticationHandler {
class EE_VFS_API SSHAuthenticationHandler {
public:
virtual ~SSHAuthenticationHandler() = default;
@@ -83,7 +84,7 @@ class SSHAuthenticationHandler {
request( const SSHAuthenticationRequest& request ) = 0;
};
struct SFTPProviderConfig {
struct EE_VFS_API SFTPProviderConfig {
std::shared_ptr<SSHHostKeyVerifier> hostKeyVerifier;
std::shared_ptr<SSHAuthenticationHandler> authenticationHandler;
/** Builds the OpenSSH configuration context for a host. Defaults to
@@ -115,7 +116,7 @@ class SFTPProviderImpl;
* connection too. All SSH sessions are driven by one I/O thread in non-blocking mode;
* provider methods never block the caller.
*/
class SFTPProvider : public VFSProvider {
class EE_VFS_API SFTPProvider : public VFSProvider {
public:
explicit SFTPProvider( SFTPProviderConfig config = {} );
@@ -1,8 +1,8 @@
#ifndef EE_VFS_SSH_SSHCONFIG_HPP
#define EE_VFS_SSH_SSHCONFIG_HPP
#include <eepp/config.hpp>
#include <eepp/system/time.hpp>
#include <eepp/vfs/config.hpp>
#include <functional>
#include <map>
#include <memory>
@@ -18,7 +18,7 @@ namespace EE { namespace VFS {
using EE::System::Time;
/** @brief OpenSSH-style pattern matching (ssh_config(5) "PATTERNS"). */
class SSHPattern {
class EE_VFS_API SSHPattern {
public:
enum class ListMatch : Int8 {
/** A negated pattern matched: the whole list is rejected. */
@@ -38,7 +38,7 @@ class SSHPattern {
};
/** One "Keyword arguments" line of an ssh_config file. */
struct SSHConfigLine {
struct EE_VFS_API SSHConfigLine {
/** Lowercased keyword. */
std::string keyword;
/** Arguments split with OpenSSH quoting rules. */
@@ -53,7 +53,7 @@ struct SSHConfigLine {
* Parsing only tokenizes: directives are kept in file order so the evaluator can apply OpenSSH's
* ordering, first-value-wins, Host/Match activation and Include semantics.
*/
class SSHConfigFile {
class EE_VFS_API SSHConfigFile {
public:
static SSHConfigFile parse( std::string_view text, std::string path = {} );
@@ -79,7 +79,7 @@ enum class SSHStrictHostKeyChecking : Uint8 {
No,
};
struct SSHJumpHost {
struct EE_VFS_API SSHJumpHost {
std::string user;
std::string host;
/** 0 when not specified (the jump host's own config / default applies). */
@@ -93,7 +93,7 @@ struct SSHJumpHost {
* Paths are fully expanded (tilde, tokens, environment). The raw evaluated option values are kept
* in `options` for diagnostics and `ssh -G` style dumps.
*/
struct SSHResolvedConfig {
struct EE_VFS_API SSHResolvedConfig {
std::string originalHost;
/** Effective connection host (HostName applied, canonicalized when enabled). */
std::string hostName;
@@ -157,7 +157,7 @@ struct SSHResolvedConfig {
/** Environment of an evaluation. Everything that touches the system is injectable so tests stay
* deterministic and never execute commands or resolve names. */
struct SSHConfigContext {
struct EE_VFS_API SSHConfigContext {
/** Host as given by the user (URI host). */
std::string originalHost;
/** Command-line style overrides (from the URI): they win over every config file. */
@@ -211,12 +211,16 @@ struct SSHConfigContext {
* cycle/depth protection, hostname canonicalization and the canonical/final re-parse pass, and
* context-aware token and environment expansion.
*/
class SSHConfigEvaluator {
class EE_VFS_API SSHConfigEvaluator {
public:
explicit SSHConfigEvaluator( SSHConfigContext context );
~SSHConfigEvaluator();
SSHConfigEvaluator( const SSHConfigEvaluator& ) = delete;
SSHConfigEvaluator& operator=( const SSHConfigEvaluator& ) = delete;
/** @return The resolved configuration, or an error message for fatal problems (for example
* canonicalization failure with CanonicalizeFallbackLocal no). */
std::variant<SSHResolvedConfig, std::string> evaluate();
@@ -4,13 +4,14 @@
#include <eepp/core/cancellationtoken.hpp>
#include <eepp/core/executor.hpp>
#include <eepp/core/small_unique_function.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/error.hpp>
#include <memory>
#include <optional>
namespace EE { namespace VFS {
struct TransferProgress {
struct EE_VFS_API TransferProgress {
URI currentSource;
URI currentDestination;
Uint64 bytesDone{ 0 };
@@ -19,7 +20,7 @@ struct TransferProgress {
std::optional<Uint64> filesTotal;
};
struct TransferResult {
struct EE_VFS_API TransferResult {
Uint64 bytesTransferred{ 0 };
Uint64 filesTransferred{ 0 };
Uint64 filesSkipped{ 0 };
@@ -37,7 +38,7 @@ class TransferState;
}
/** @brief RAII handle of a progress subscription. */
class TransferProgressSubscription {
class EE_VFS_API TransferProgressSubscription {
public:
TransferProgressSubscription() = default;
@@ -69,7 +70,7 @@ class TransferProgressSubscription {
* coalesced: a slow subscriber receives the latest snapshot rather than every intermediate one.
* Transfers keep running when the Transfer object is destroyed; call cancel() to stop them.
*/
class Transfer {
class EE_VFS_API Transfer {
public:
Transfer() = default;
@@ -6,6 +6,7 @@
#include <eepp/system/regex.hpp>
#include <eepp/ui/models/model.hpp>
#include <eepp/ui/models/variant.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/virtualfilesystem.hpp>
#include <functional>
#include <memory>
@@ -39,7 +40,8 @@ using UI::Models::Variant;
* All public methods must be called on the UI thread, which must be the thread the executor runs
* functions on.
*/
class VFSDirectoryModel : public Model, public std::enable_shared_from_this<VFSDirectoryModel> {
class EE_VFS_API VFSDirectoryModel : public Model,
public std::enable_shared_from_this<VFSDirectoryModel> {
public:
enum Column : Uint8 { Name, Size, Modified, Type, Permissions, Owner, Group, ColumnCount };
@@ -83,6 +85,10 @@ class VFSDirectoryModel : public Model, public std::enable_shared_from_this<VFSD
~VFSDirectoryModel() override;
VFSDirectoryModel( const VFSDirectoryModel& ) = delete;
VFSDirectoryModel& operator=( const VFSDirectoryModel& ) = delete;
/** Navigates to a directory. */
void setURI( URI uri );
@@ -2,6 +2,7 @@
#define EE_VFS_VIRTUALFILESYSTEM_HPP
#include <eepp/system/threadpool.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/provider.hpp>
#include <eepp/vfs/providers/localioscheduler.hpp>
#include <eepp/vfs/transfer.hpp>
@@ -31,7 +32,7 @@ using EE::System::ThreadPool;
* Provider work never runs on that pool, so waits there cannot starve provider I/O. Copies and
* moves run on a pool of their own: long transfers never delay the other operations.
*/
class VirtualFileSystem {
class EE_VFS_API VirtualFileSystem {
public:
struct Config {
/** Worker threads for blocking local I/O (LocalProvider and similar). */
@@ -45,7 +46,7 @@ class VirtualFileSystem {
* counts as not responding: the call fails with TimedOut, and so do further calls until
* the server answers again (see LocalIOScheduler). */
System::Time networkMountTimeout{ System::Seconds( 15 ) };
/** Register file:// and res:// (and sftp:// when built with SFTP support). */
/** Register file://, zip:// and res:// (and sftp:// when built with SFTP support). */
bool registerDefaultProviders{ true };
};
+11 -10
View File
@@ -5,6 +5,7 @@
#include <eepp/core/executor.hpp>
#include <eepp/core/small_unique_function.hpp>
#include <eepp/system/time.hpp>
#include <eepp/vfs/config.hpp>
#include <eepp/vfs/error.hpp>
#include <memory>
#include <mutex>
@@ -29,11 +30,11 @@ enum class WatchEventType : Uint8 {
RescanRequired,
};
const char* toString( WatchEventType type );
EE_VFS_API const char* toString( WatchEventType type );
/** One change, relative to the watched root ("dir/file.txt", '/' separated, never leading '/').
*/
struct WatchEvent {
struct EE_VFS_API WatchEvent {
std::string relativePath;
std::string oldRelativePath;
WatchEventType type{ WatchEventType::Modified };
@@ -52,16 +53,16 @@ enum class WatchStatus : Uint8 {
Failed,
};
const char* toString( WatchStatus status );
EE_VFS_API const char* toString( WatchStatus status );
/** A coalesced group of events for one watch. Events are in arrival order. */
struct WatchEventBatch {
struct EE_VFS_API WatchEventBatch {
URI root;
std::vector<WatchEvent> events;
WatchStatus status{ WatchStatus::Active };
};
struct WatchOptions {
struct EE_VFS_API WatchOptions {
bool recursive{ false };
bool followSymlinks{ false };
bool allowOutOfScopeLinks{ false };
@@ -84,7 +85,7 @@ class WatchSubscriptionState;
* Thread-safe. Events are accumulated and delivered as one batch per executor dispatch, which
* coalesces bursts without adding a latency timer.
*/
class WatchSink {
class EE_VFS_API WatchSink {
public:
virtual ~WatchSink() = default;
@@ -101,14 +102,14 @@ class WatchSink {
};
/** Backend-side registration of one logical watch. Destroying it stops native watching. */
class WatchRegistration {
class EE_VFS_API WatchRegistration {
public:
virtual ~WatchRegistration() = default;
};
/** @brief Notification engine for a set of URIs. Backends adapt efsw (local), a remote
* efsw-agent or polling into URI space; they never mutate UI or model state. */
class WatchBackend {
class EE_VFS_API WatchBackend {
public:
virtual ~WatchBackend() = default;
@@ -125,7 +126,7 @@ class WatchBackend {
* that the callback is not running and will not run again (a callback may destroy its own
* handle).
*/
class WatchHandle {
class EE_VFS_API WatchHandle {
public:
WatchHandle() = default;
@@ -167,7 +168,7 @@ class WatchHandle {
* Backends are registered per scheme. A scheme without a dedicated backend uses the fallback
* backend (normally polling through the VFS) when WatchOptions::allowPolling is set.
*/
class WatchManager {
class EE_VFS_API WatchManager {
public:
WatchManager();
@@ -0,0 +1,614 @@
#include <eepp/vfs/providers/zipprovider.hpp>
#include <algorithm>
#include <array>
#include <eepp/core/containers.hpp>
#include <eepp/system/zip.hpp>
#include <eepp/vfs/detail/memoryhandles.hpp>
#include <eepp/vfs/providers/localprovider.hpp>
#include <limits>
#include <mutex>
using namespace EE::System;
namespace EE { namespace VFS {
namespace {
constexpr ProviderCapabilities
ZipCapabilities( static_cast<Uint32>( VFSCapability::Read ) |
static_cast<Uint32>( VFSCapability::List ) |
static_cast<Uint32>( VFSCapability::Seek ) |
static_cast<Uint32>( VFSCapability::CaseSensitive ) );
constexpr Uint64 NoEntry = std::numeric_limits<Uint64>::max();
constexpr size_t MaxEntries = 262144;
VFSError readOnly( const URI& uri ) {
return VFSError( VFSErrorCode::ReadOnly, uri, "ZIP archives are read-only" );
}
VFSError readerError( const Zip& reader, const URI& uri ) {
VFSErrorCode code = VFSErrorCode::IO;
switch ( reader.error() ) {
case Zip::Error::NotFound:
code = VFSErrorCode::NotFound;
break;
case Zip::Error::PermissionDenied:
code = VFSErrorCode::PermissionDenied;
break;
case Zip::Error::InvalidArchive:
code = VFSErrorCode::InvalidOperation;
break;
case Zip::Error::Unsupported:
code = VFSErrorCode::Unsupported;
break;
default:
break;
}
return VFSError( code, uri, reader.errorMessage() );
}
/** Archive paths are untrusted. Reject names that could leave a copy destination or turn into
* native Windows paths/streams; do not resolve these names into aliases of safe entries. */
bool safeEntry( std::string_view name ) {
if ( name.empty() || name.size() > 4096 || name.front() == '/' ||
name.find_first_of( std::string_view( "\\:\0", 3 ) ) != std::string_view::npos )
return false;
size_t start = 0;
while ( start < name.size() ) {
size_t end = name.find( '/', start );
if ( end == std::string_view::npos )
end = name.size();
auto part = name.substr( start, end - start );
// Windows trims trailing dots/spaces; an all-dot/space segment can become a parent.
if ( part.empty() || part.find_first_not_of( ". " ) == std::string_view::npos )
return false;
start = end + 1;
}
return true;
}
/** Parsed archive identity and normalized path within it. */
struct ZipAddress {
URI archive;
std::string member;
static VFSResult<ZipAddress> parse( const URI& uri ) {
if ( uri.getScheme() != "zip" || uri.getAuthority() != uri.getHost() ||
!uri.getFragment().empty() || uri.getPath().find( '\0' ) != std::string::npos )
return Err( VFSError( VFSErrorCode::InvalidURI, uri, "Invalid ZIP location" ) );
if ( !uri.getRawQuery().empty() )
return Err( VFSError( VFSErrorCode::Unsupported, uri, "Unsupported ZIP backing URI" ) );
const auto& path = uri.getPath();
if ( path.empty() || path.front() != '/' )
return Err( VFSError( VFSErrorCode::InvalidURI, uri, "ZIP path must be absolute" ) );
std::string backing;
backing.reserve( path.size() );
size_t boundary = std::string::npos;
for ( size_t i = 0; i < path.size(); ++i ) {
if ( path[i] == '!' ) {
if ( i + 1 < path.size() && path[i + 1] == '!' ) {
backing += '!';
++i;
continue;
}
if ( i + 1 == path.size() || path[i + 1] == '/' ) {
boundary = i + 1;
break;
}
}
backing += path[i];
}
if ( boundary == std::string::npos || backing.empty() || backing.back() == '/' )
return Err(
VFSError( VFSErrorCode::InvalidURI, uri, "ZIP path needs archive.zip!/member" ) );
ZipAddress result;
result.archive.setScheme( "file" );
result.archive.setHost( uri.getHost() );
result.archive.setRawPath( std::move( backing ) );
URI member;
member.setScheme( "zip" );
member.setRawPath( boundary == path.size() ? "/" : path.substr( boundary ) );
member.normalize();
result.member = member.getPath();
return Ok( std::move( result ) );
}
};
struct ZipNode {
VFSFileInfo info;
std::vector<size_t> children;
Uint64 entry{ NoEntry };
};
/** Immutable directory index. Entry streams open independent readers, so their I/O and teardown
* never wait for another handle on the same archive. */
struct ZipArchive {
std::string path;
std::vector<ZipNode> nodes;
UnorderedMap<std::string, size_t> paths;
FileRevision revision;
LocalIOScheduler::Route route;
Uint64 stamp{ 0 };
VFSFileInfo info( std::string name, bool directory, Uint64 entry = NoEntry,
const Zip::Entry* metadata = nullptr ) const {
VFSFileInfo value;
value.name = std::move( name );
value.type = directory ? FileType::Directory : FileType::Regular;
value.hidden = !value.name.empty() && value.name[0] == '.';
// Read-only is a provider capability, not the stored entry's mode. The archive API does
// not expose Unix permissions: leave them unknown so copies use writable native defaults.
if ( metadata ) {
value.setModifiedTime( metadata->modified );
if ( !directory )
value.setSize( metadata->size );
}
value.revision = FileRevision::fromValues(
{ stamp, entry, metadata ? metadata->crc : 0, metadata ? metadata->size : 0 } );
return value;
}
VFSResult<void> index( Zip& reader, const URI& uri, const CancellationToken& token ) {
const auto count = reader.entryCount();
if ( count > MaxEntries )
return Err(
VFSError( VFSErrorCode::InvalidOperation, uri, "ZIP has too many entries" ) );
nodes.push_back( { info( "/", true ), {}, NoEntry } );
paths.emplace( "", 0 );
Zip::Entry metadata;
for ( Uint64 entry = 0; entry < count; ++entry ) {
if ( token.isCancellationRequested() )
return Err( cancelledError( uri ) );
if ( !reader.entry( entry, metadata ) )
return Err( readerError( reader, uri ) );
std::string_view name( metadata.name );
if ( !safeEntry( name ) )
continue;
const bool directory = name.back() == '/';
if ( directory )
name.remove_suffix( 1 );
size_t start = 0;
size_t parent = 0;
while ( start < name.size() ) {
size_t end = name.find( '/', start );
if ( end == std::string_view::npos )
end = name.size();
const bool last = end == name.size();
const bool folder = !last || directory;
std::string path( name.substr( 0, end ) );
auto found = paths.find( path );
size_t node;
if ( found == paths.end() ) {
if ( nodes.size() >= MaxEntries ) {
return Err( VFSError( VFSErrorCode::InvalidOperation, uri,
"ZIP has too many directories and entries" ) );
}
node = nodes.size();
paths.emplace( std::move( path ), node );
nodes[parent].children.push_back( node );
nodes.push_back(
{ info( std::string( name.substr( start, end - start ) ), folder ),
{},
NoEntry } );
} else {
node = found->second;
}
ZipNode& value = nodes[node];
// A directory wins a file/directory collision so that every listed child remains
// reachable. Duplicate file names resolve to their first central-directory entry.
if ( folder && value.info.type != FileType::Directory ) {
value.info = info( std::move( value.info.name ), true );
value.entry = NoEntry;
}
if ( last && ( folder || value.info.type != FileType::Directory ) &&
value.entry == NoEntry ) {
value.info = info( std::move( value.info.name ), folder, entry, &metadata );
value.entry = entry;
}
parent = node;
start = end + 1;
}
}
return Ok();
}
VFSResult<size_t> find( const std::string& member, const URI& uri ) const {
auto found = paths.find( member );
if ( found != paths.end() ) {
if ( uri.getPath().ends_with( '/' ) &&
nodes[found->second].info.type != FileType::Directory )
return Err( VFSError( VFSErrorCode::NotDirectory, uri ) );
return Ok( found->second );
}
// Distinguish a missing member from a path through a regular file.
size_t slash = member.find( '/' );
while ( slash != std::string::npos ) {
auto parent = paths.find( member.substr( 0, slash ) );
if ( parent != paths.end() && nodes[parent->second].info.type != FileType::Directory )
return Err( VFSError( VFSErrorCode::NotDirectory, uri ) );
slash = member.find( '/', slash + 1 );
}
return Err( VFSError( VFSErrorCode::NotFound, uri ) );
}
};
class ZipFileImpl : public FileHandleImpl, public std::enable_shared_from_this<ZipFileImpl> {
template <typename T, typename F> auto guarded( CancellationToken token, F&& fn ) {
return [self = shared_from_this(), token = std::move( token ),
fn = std::forward<F>( fn )]() mutable -> VFSResult<T> {
if ( token.isCancellationRequested() )
return Err( cancelledError( self->mUri ) );
std::lock_guard<std::mutex> lock( self->mMutex );
if ( !self->mStream )
return Err( VFSError( VFSErrorCode::InvalidOperation, self->mUri, "File closed" ) );
return fn( *self, token );
};
}
public:
ZipFileImpl( std::shared_ptr<LocalIOScheduler> scheduler, std::shared_ptr<ZipArchive> archive,
std::unique_ptr<Zip> reader, std::unique_ptr<IOStream> stream, URI uri,
size_t node ) :
mScheduler( std::move( scheduler ) ),
mArchive( std::move( archive ) ),
mReader( std::move( reader ) ),
mStream( std::move( stream ) ),
mUri( std::move( uri ) ),
mNode( node ) {}
VFSTask<size_t> read( Uint64 offset, std::span<Uint8> data, CancellationToken token ) override {
if ( !mArchive->route.isNetworkMount() ) {
return mScheduler->submit<size_t>(
mArchive->route, guarded<size_t>( std::move( token ),
[offset, data]( ZipFileImpl& self,
const CancellationToken& token ) {
return self.readInto( offset, data, token );
} ) );
}
// A mount's read can outlive its timeout. Only the winning completion copies into data.
return mScheduler->submitCommitted<size_t>(
mArchive->route,
guarded<std::vector<Uint8>>(
std::move( token ),
[offset, size = data.size()]( ZipFileImpl& self, const CancellationToken& token )
-> VFSResult<std::vector<Uint8>> {
std::vector<Uint8> buffer( size );
auto result = self.readInto( offset, buffer, token );
if ( !result )
return Err( std::move( result ).error() );
buffer.resize( result.value() );
return Ok( std::move( buffer ) );
} ),
[data]( VFSResult<std::vector<Uint8>>&& result ) -> VFSResult<size_t> {
if ( !result )
return Err( std::move( result ).error() );
std::copy( result.value().begin(), result.value().end(), data.begin() );
return Ok( result.value().size() );
} );
}
VFSTask<size_t> write( Uint64, std::span<const Uint8>, CancellationToken ) override {
return readyError<size_t>( readOnly( mUri ) );
}
VFSTask<void> truncate( Uint64 ) override { return readyError<void>( readOnly( mUri ) ); }
VFSTask<void> flush() override { return readyValue<void>(); }
VFSTask<VFSFileInfo> stat() override {
return readyValue<VFSFileInfo>( mArchive->nodes[mNode].info );
}
VFSTask<void> close() override {
return mScheduler->submit<void>( mArchive->route,
[self = shared_from_this()]() -> VFSResult<void> {
std::lock_guard<std::mutex> lock( self->mMutex );
self->mStream.reset();
self->mReader.reset();
return Ok();
} );
}
ProviderCapabilities capabilities() const override { return ZipCapabilities; }
private:
VFSResult<size_t> readInto( Uint64 offset, std::span<Uint8> data,
const CancellationToken& token ) {
const auto& node = mArchive->nodes[mNode];
if ( offset >= node.info.size().value_or( 0 ) )
return Ok( size_t( 0 ) );
if ( offset < static_cast<Uint64>( mStream->tell() ) ) {
auto stream = mReader->openStream( node.entry );
if ( !stream )
return Err( readerError( *mReader, mUri ) );
mStream = std::move( stream );
}
std::array<char, 16384> scratch;
while ( static_cast<Uint64>( mStream->tell() ) < offset ) {
if ( token.isCancellationRequested() )
return Err( cancelledError( mUri ) );
const auto count = std::min<Uint64>( offset - mStream->tell(), scratch.size() );
if ( mStream->read( scratch.data(), count ) <= 0 )
return Err( VFSError( VFSErrorCode::IO, mUri, "Could not seek ZIP entry" ) );
}
size_t done = 0;
while ( done < data.size() ) {
if ( token.isCancellationRequested() )
return Err( cancelledError( mUri ) );
const auto count = std::min<size_t>( data.size() - done, 65536 );
const auto got = mStream->read( reinterpret_cast<char*>( data.data() + done ), count );
if ( got < 0 )
return Err( VFSError( VFSErrorCode::IO, mUri,
"Could not read ZIP entry (corrupt data or CRC)" ) );
if ( got == 0 )
break;
done += static_cast<size_t>( got );
}
return Ok( done );
}
std::shared_ptr<LocalIOScheduler> mScheduler;
// Streams borrow the reader and are destroyed before it.
std::shared_ptr<ZipArchive> mArchive;
std::unique_ptr<Zip> mReader;
std::unique_ptr<IOStream> mStream;
std::mutex mMutex;
URI mUri;
size_t mNode;
};
} // namespace
class ZipProviderImpl : public std::enable_shared_from_this<ZipProviderImpl> {
public:
explicit ZipProviderImpl( std::shared_ptr<LocalIOScheduler> scheduler ) :
scheduler( std::move( scheduler ) ) {}
template <typename T, typename F>
VFSTask<T> run( const URI& uri, CancellationToken token, F&& fn ) {
if ( token.isCancellationRequested() )
return readyError<T>( cancelledError( uri ) );
auto address = ZipAddress::parse( uri );
if ( !address )
return readyError<T>( std::move( address ).error() );
auto native = LocalProvider::toNativePath( address.value().archive );
if ( !native )
return readyError<T>( std::move( native ).error() );
std::string member = std::move( address ).value().member;
while ( !member.empty() && member.front() == '/' )
member.erase( member.begin() );
while ( !member.empty() && member.back() == '/' )
member.pop_back();
std::string path = std::move( native ).value();
return scheduler->submitResolved<T>(
path, true,
[self = shared_from_this(), uri, path, member = std::move( member ),
token = std::move( token ), fn = std::forward<F>( fn )]() mutable -> VFSResult<T> {
if ( token.isCancellationRequested() )
return Err( cancelledError( uri ) );
auto archive = self->load( path, uri, token );
if ( !archive )
return Err( std::move( archive ).error() );
return fn( std::move( archive ).value(), member, uri );
} );
}
std::shared_ptr<LocalIOScheduler> scheduler;
private:
struct Cached {
std::string path;
std::shared_ptr<ZipArchive> archive;
Uint64 used{ 0 };
};
VFSResult<std::shared_ptr<ZipArchive>> load( const std::string& path, const URI& uri,
const CancellationToken& token ) {
const URI backing = URI::fromFSPath( path );
auto metadata = LocalProvider::statSync( backing, path, true );
if ( !metadata )
return Err( std::move( metadata ).error() );
if ( !metadata.value().isRegularFile() )
return Err( VFSError( VFSErrorCode::InvalidOperation, uri,
"ZIP backing file is not a regular file" ) );
const auto revision = metadata.value().revision;
{
std::lock_guard<std::mutex> lock( mMutex );
for ( auto& cached : mCache ) {
if ( cached.path == path && cached.archive &&
cached.archive->revision == revision ) {
cached.used = ++mUsed;
return Ok( cached.archive );
}
}
}
auto archive = std::make_shared<ZipArchive>();
archive->revision = revision;
archive->path = path;
archive->stamp = std::hash<std::string_view>{}(
std::string_view( reinterpret_cast<const char*>( revision.data() ), revision.size() ) );
archive->route = scheduler->resolvedRouteFor( path, true );
Zip reader( false );
if ( !reader.open( path, Zip::OpenMode::ReadOnly ) )
return Err( readerError( reader, uri ) );
auto indexed = archive->index( reader, uri, token );
if ( !indexed )
return Err( std::move( indexed ).error() );
auto after = LocalProvider::statSync( backing, path, true );
if ( !after || after.value().revision != revision )
return Err( VFSError( VFSErrorCode::ChangedExternally, uri,
"Archive changed while it was opened" ) );
std::shared_ptr<ZipArchive> retired;
{
std::lock_guard<std::mutex> lock( mMutex );
auto slot = mCache.begin();
for ( auto it = mCache.begin(); it != mCache.end(); ++it ) {
if ( it->path == path ) {
slot = it;
break;
}
if ( it->used < slot->used )
slot = it;
}
retired = std::move( slot->archive );
slot->path = path;
slot->archive = archive;
slot->used = ++mUsed;
}
// Free a retired directory index outside the cache lock.
return Ok( std::move( archive ) );
}
std::mutex mMutex;
std::array<Cached, 8> mCache;
Uint64 mUsed{ 0 };
};
ZipProvider::ZipProvider( std::shared_ptr<LocalIOScheduler> scheduler ) :
mImpl( std::make_shared<ZipProviderImpl>( std::move( scheduler ) ) ) {}
ZipProvider::~ZipProvider() = default;
URI ZipProvider::makeURI( const URI& archive, std::string_view member ) {
if ( archive.getScheme() != "file" || !archive.getRawQuery().empty() ||
!archive.getFragment().empty() || archive.getAuthority() != archive.getHost() ||
archive.getPath().empty() || archive.getPath().front() != '/' ||
archive.getPath().find( '\0' ) != std::string::npos )
return {};
URI uri;
uri.setScheme( "zip" );
uri.setHost( archive.getHost() );
std::string path;
path.reserve( archive.getPath().size() + member.size() + 2 );
// Escape literal bangs in the backing path, so even a directory named "packages!" is
// unambiguous. Only the first unescaped "!/" separates the archive from its member.
for ( char c : archive.getPath() ) {
path += c;
if ( c == '!' )
path += '!';
}
URI entry;
entry.setScheme( "zip" );
entry.setRawPath( member.empty() || member.front() != '/' ? "/" + std::string( member )
: std::string( member ) );
entry.normalize();
path += '!';
path += entry.getPath();
uri.setRawPath( std::move( path ) );
return uri;
}
VFSResult<URI> ZipProvider::archiveURI( const URI& uri ) {
auto address = ZipAddress::parse( uri );
if ( !address )
return Err( std::move( address ).error() );
return Ok( std::move( address ).value().archive );
}
VFSResult<std::string> ZipProvider::memberPath( const URI& uri ) {
auto address = ZipAddress::parse( uri );
if ( !address )
return Err( std::move( address ).error() );
return Ok( std::move( address ).value().member );
}
ProviderCapabilities ZipProvider::capabilities( const URI& ) const {
return ZipCapabilities;
}
VFSTask<VFSFileInfo> ZipProvider::stat( const URI& uri, const StatOptions&,
CancellationToken token ) {
return mImpl->run<VFSFileInfo>( uri, std::move( token ),
[]( const std::shared_ptr<ZipArchive>& archive,
const std::string& member,
const URI& uri ) -> VFSResult<VFSFileInfo> {
auto node = archive->find( member, uri );
if ( !node )
return Err( std::move( node ).error() );
return Ok( archive->nodes[node.value()].info );
} );
}
VFSTask<DirectoryHandle> ZipProvider::openDirectory( const URI& uri, const DirectoryOpenOptions&,
CancellationToken token ) {
return mImpl->run<DirectoryHandle>(
uri, std::move( token ),
[]( const std::shared_ptr<ZipArchive>& archive, const std::string& member,
const URI& uri ) -> VFSResult<DirectoryHandle> {
auto node = archive->find( member, uri );
if ( !node )
return Err( std::move( node ).error() );
const auto& folder = archive->nodes[node.value()];
if ( folder.info.type != FileType::Directory )
return Err( VFSError( VFSErrorCode::NotDirectory, uri ) );
std::vector<VFSFileInfo> entries;
entries.reserve( folder.children.size() );
for ( auto child : folder.children )
entries.push_back( archive->nodes[child].info );
return Ok( DirectoryHandle(
uri, std::make_shared<detail::VectorDirectoryImpl>( std::move( entries ) ) ) );
} );
}
VFSTask<FileHandle> ZipProvider::open( const URI& uri, const OpenOptions& options,
CancellationToken token ) {
if ( hasFlag( options.flags, OpenFlags::Write ) ||
hasFlag( options.flags, OpenFlags::Create ) ||
hasFlag( options.flags, OpenFlags::Truncate ) )
return readyError<FileHandle>( readOnly( uri ) );
return mImpl->run<FileHandle>(
uri, std::move( token ),
[scheduler = mImpl->scheduler]( std::shared_ptr<ZipArchive> archive,
const std::string& member,
const URI& uri ) -> VFSResult<FileHandle> {
auto node = archive->find( member, uri );
if ( !node )
return Err( std::move( node ).error() );
const auto& info = archive->nodes[node.value()];
if ( info.info.type == FileType::Directory )
return Err( VFSError( VFSErrorCode::IsDirectory, uri ) );
auto reader = std::make_unique<Zip>( false );
if ( !reader->open( archive->path, Zip::OpenMode::ReadOnly ) )
return Err( readerError( *reader, uri ) );
auto current =
LocalProvider::statSync( URI::fromFSPath( archive->path ), archive->path, true );
if ( !current || current.value().revision != archive->revision )
return Err( VFSError( VFSErrorCode::ChangedExternally, uri,
"Archive changed while opening the entry" ) );
auto stream = reader->openStream( info.entry );
if ( !stream )
return Err( readerError( *reader, uri ) );
return Ok( FileHandle( uri, std::make_shared<ZipFileImpl>(
scheduler, std::move( archive ), std::move( reader ),
std::move( stream ), uri, node.value() ) ) );
} );
}
VFSTask<void> ZipProvider::mkdir( const URI& uri, const MkDirOptions&, CancellationToken ) {
return readyError<void>( readOnly( uri ) );
}
VFSTask<void> ZipProvider::remove( const URI& uri, const RemoveOptions&, CancellationToken ) {
return readyError<void>( readOnly( uri ) );
}
VFSTask<void> ZipProvider::rename( const URI& from, const URI&, const RenameOptions&,
CancellationToken ) {
return readyError<void>( readOnly( from ) );
}
VFSTask<URI> ZipProvider::canonicalize( const URI& uri, CancellationToken token ) {
return mImpl->run<URI>( uri, std::move( token ),
[]( const std::shared_ptr<ZipArchive>& archive,
const std::string& member, const URI& uri ) -> VFSResult<URI> {
auto found = archive->find( member, uri );
if ( !found )
return Err( std::move( found ).error() );
auto backing = ZipProvider::archiveURI( uri );
URI normalized = ZipProvider::makeURI( backing.value(), member );
return Ok( std::move( normalized ) );
} );
}
}} // namespace EE::VFS
@@ -18,7 +18,7 @@ namespace EE { namespace VFS { namespace sftp {
* time. Speaks the agent protocol over Unix-domain sockets; elsewhere (Windows named pipes,
* Pageant) supported() is false and libssh2's own agent support is used.
*/
class SSHAgent {
class EE_VFS_API SSHAgent {
public:
struct Identity {
/** Public key blob (SSH wire format). */
@@ -3,6 +3,7 @@
#include <eepp/vfs/detail/filewriter.hpp>
#include <eepp/vfs/providers/localprovider.hpp>
#include <eepp/vfs/providers/resourceprovider.hpp>
#include <eepp/vfs/providers/zipprovider.hpp>
#include <eepp/vfs/transferengine.hpp>
#include <eepp/vfs/virtualfilesystem.hpp>
#include <eepp/vfs/watchers/localwatchbackend.hpp>
@@ -69,6 +70,7 @@ VirtualFileSystem::~VirtualFileSystem() {
void VirtualFileSystem::registerDefaultProviders() {
registerProvider( std::make_shared<LocalProvider>( mLocalIO ) );
registerProvider( std::make_shared<ZipProvider>( mLocalIO ) );
registerProvider( std::make_shared<ResourceProvider>( mIOPool ) );
// Network mounts are polled through the VFS (whose calls time out), not by efsw.
auto polling = std::make_shared<PollingWatchBackend>( *this );
@@ -26,7 +26,7 @@ class LocalWatchBackendImpl;
* Directories on a network mount go to networkBackend instead (the VFS polling backend): efsw's
* own threads would block on an unreachable server, while VFS calls time out.
*/
class LocalWatchBackend : public WatchBackend {
class EE_VFS_API LocalWatchBackend : public WatchBackend {
public:
/** @param pool Where watches are registered and removed.
* @param networkBackend Watches directories on network mounts; without it efsw does.
@@ -12,7 +12,7 @@ namespace EE { namespace VFS {
namespace detail {
class PollingWatchBackendImpl {
class PollingWatchBackendImpl : public std::enable_shared_from_this<PollingWatchBackendImpl> {
public:
struct Snapshot {
struct Entry {
@@ -52,8 +52,12 @@ class PollingWatchBackendImpl {
mStopping = true;
}
mWake.notify_all();
if ( mThread.joinable() )
mThread.join();
if ( mThread.joinable() ) {
if ( mThread.get_id() == std::this_thread::get_id() )
mThread.detach();
else
mThread.join();
}
}
/** Takes the initial snapshot on the polling thread and resolves once the watch is active.
@@ -88,7 +92,9 @@ class PollingWatchBackendImpl {
void ensureThread() {
if ( !mThread.joinable() )
mThread = std::thread( [this] { loop(); } );
// A synchronous watch callback may release the backend on this thread. Keep its
// state alive until the loop observes stop(), without joining the current thread.
mThread = std::thread( [self = shared_from_this()] { self->loop(); } );
}
void loop() {
@@ -101,7 +107,9 @@ class PollingWatchBackendImpl {
auto snapshot = take( *pending.watch );
lock.lock();
if ( !snapshot ) {
lock.unlock();
pending.resolve( Err( std::move( snapshot ).error() ) );
lock.lock();
continue;
}
pending.watch->snapshot = std::move( snapshot ).value();
@@ -139,8 +147,10 @@ class PollingWatchBackendImpl {
while ( !mPendingAdds.empty() ) {
auto pending = std::move( mPendingAdds.front() );
mPendingAdds.pop_front();
lock.unlock();
pending.resolve( Err( VFSError( VFSErrorCode::InvalidOperation, pending.watch->root,
"Watch backend shutting down" ) ) );
lock.lock();
}
}
@@ -21,7 +21,7 @@ class PollingWatchBackendImpl;
* successful poll emits RescanRequired. Recursive polling is limited to MaxEntries entries per
* watch; larger trees degrade to the top directories that fit.
*/
class PollingWatchBackend : public WatchBackend {
class EE_VFS_API PollingWatchBackend : public WatchBackend {
public:
static constexpr size_t MaxEntries = 100000;
+49 -1
View File
@@ -12,6 +12,7 @@
#include <eepp/vfs/providers/localioscheduler.hpp>
#include <eepp/vfs/providers/localprovider.hpp>
#include <eepp/vfs/watchers/localwatchbackend.hpp>
#include <eepp/vfs/watchers/pollingwatchbackend.hpp>
#include <mutex>
#include <optional>
#include <thread>
@@ -460,12 +461,15 @@ UTEST( VFSResource, zipPackWithSeveralFiles ) {
EXPECT_TRUE( zip.addFile( a, "vfs-zip/a.txt" ) );
EXPECT_TRUE( zip.addFile( b, "vfs-zip/sub/b.txt" ) );
EXPECT_TRUE( zip.addFile( c, "vfs-zip/sub/c.txt" ) );
std::vector<Uint8> empty;
EXPECT_TRUE( zip.addFile( empty, "vfs-zip/empty.txt" ) );
EXPECT_TRUE( zip.addFile( empty, "vfs-zip/empty-dir/" ) );
EXPECT_TRUE( zip.close() );
}
Zip zip;
ASSERT_TRUE( zip.open( zipPath ) );
EXPECT_EQ( 3u, zip.getFileList().size() );
EXPECT_EQ( 4u, zip.getFileList().size() );
std::vector<Uint8> data;
EXPECT_TRUE( zip.extractFileToMemory( "vfs-zip/sub/c.txt", data ) );
EXPECT_STDSTREQ( "c3", std::string( data.begin(), data.end() ) );
@@ -473,6 +477,8 @@ UTEST( VFSResource, zipPackWithSeveralFiles ) {
VirtualFileSystem vfs;
EXPECT_STDSTREQ( "a1", readString( vfs, URI( "res:///vfs-zip/a.txt" ) ) );
EXPECT_STDSTREQ( "b2", readString( vfs, URI( "res:///vfs-zip/sub/b.txt" ) ) );
EXPECT_STDSTREQ( "", readString( vfs, URI( "res:///vfs-zip/empty.txt" ) ) );
EXPECT_FALSE( PackFileIndex::instance()->fileExists( "vfs-zip/empty-dir" ) );
auto listed = waitFor( vfs.listAll( URI( "res:///vfs-zip/sub" ) ) );
ASSERT_TRUE( listed.hasValue() );
EXPECT_EQ( 2u, listed.value().size() );
@@ -1290,3 +1296,45 @@ UTEST( LocalIOScheduler, linksIntoAMountRunOnItsLane ) {
gate->open();
}
#endif
namespace {
class ReleasePollingBackendSink : public WatchSink {
public:
URI uri;
std::mutex gate;
std::shared_ptr<PollingWatchBackend> backend;
void push( WatchEvent&& ) override {}
void push( std::vector<WatchEvent>&& ) override {}
void rescanRequired() override {}
void setStatus( WatchStatus ) override {
std::lock_guard<std::mutex> lock( gate );
backend.reset();
}
const URI& root() const override { return uri; }
};
} // namespace
UTEST( VFSWatch, pollingCallbackCanReleaseTheLastBackendOwner ) {
VirtualFileSystem vfs;
auto memory = std::make_shared<MemoryProvider>();
vfs.registerProvider( memory );
const URI root( "mem://polling-teardown/dir" );
memory->putFile( root.withAppendedSegment( "file" ), "contents" );
auto sink = std::make_shared<ReleasePollingBackendSink>();
sink->uri = root;
sink->backend = std::make_shared<PollingWatchBackend>( vfs );
std::weak_ptr<PollingWatchBackend> weak = sink->backend;
std::unique_lock<std::mutex> hold( sink->gate );
auto task = sink->backend->watch( root, {}, sink );
hold.unlock();
ASSERT_EQ( TaskState::Ready, task.waitFor( std::chrono::seconds( 5 ) ) );
ASSERT_TRUE( waitFor( std::move( task ) ).hasValue() );
EXPECT_TRUE( weak.expired() );
}
+440
View File
@@ -0,0 +1,440 @@
#include "../../tools/eefm/location.hpp"
#include "utest.hpp"
#include "vfs_test_utils.hpp"
#include <array>
#include <cerrno>
#include <eepp/system/base64.hpp>
#include <eepp/system/packfileindex.hpp>
#include <eepp/system/zip.hpp>
#include <eepp/vfs/providers/zipprovider.hpp>
#include <fstream>
using namespace EE;
using namespace EE::System;
using namespace EE::VFS;
using namespace vfstest;
namespace {
bool makeZip( const std::string& path,
std::initializer_list<std::pair<std::string, std::string>> files ) {
Zip archive;
if ( !archive.create( path ) )
return false;
for ( const auto& [name, data] : files ) {
if ( !archive.addFile( reinterpret_cast<const Uint8*>( data.data() ), data.size(), name ) )
return false;
}
return archive.close();
}
} // namespace
UTEST( ZipProvider, archiveURIsPreserveIdentityAndMemberNames ) {
TempDir temp;
const URI file = URI::fromFSPath( temp.path + "Mixed Case !!#%?.zip" );
const URI root = ZipProvider::makeURI( file );
const URI member = root.withAppendedSegment( "dir" ).withAppendedSegment( "Name #%?.txt" );
const URI roundTrip( member.toString() );
EXPECT_TRUE( member == roundTrip );
auto backing = ZipProvider::archiveURI( roundTrip );
ASSERT_TRUE( backing.hasValue() );
EXPECT_TRUE( file == backing.value() );
EXPECT_TRUE( file == ZipProvider::archiveURI( member.getParent().getParent() ).value() );
EXPECT_FALSE( ZipProvider::archiveURI( root.getParent() ).hasValue() );
EXPECT_FALSE( root == ZipProvider::makeURI( URI::fromFSPath( temp.path + "other.zip" ) ) );
EXPECT_TRUE( member.isSameOrChildOf( root ) );
const URI bangArchive = URI::fromFSPath( temp.path + "packages!/archive!.zip" );
const URI bangMember = ZipProvider::makeURI( bangArchive, "/folder!/name!.txt" );
EXPECT_TRUE( bangArchive == ZipProvider::archiveURI( URI( bangMember.toString() ) ).value() );
EXPECT_STDSTREQ( "/folder!/name!.txt", ZipProvider::memberPath( bangMember ).value() );
const URI clamped = ZipProvider::makeURI( file, "/../../inside" );
EXPECT_TRUE( file == ZipProvider::archiveURI( clamped ).value() );
EXPECT_STDSTREQ( "/inside", ZipProvider::memberPath( clamped ).value() );
EXPECT_EQ( VFSErrorCode::InvalidURI,
codeOf( ZipProvider::archiveURI( URI( "zip://nothex/" ) ) ) );
EXPECT_EQ(
VFSErrorCode::InvalidURI,
codeOf( ZipProvider::archiveURI( ZipProvider::makeURI( URI( "sftp://host/a.zip" ) ) ) ) );
}
UTEST( ZipProvider, listsImplicitAndEmptyDirectoriesWithMetadata ) {
TempDir temp;
const std::string path = temp.path + "archive.zip";
ASSERT_TRUE( makeZip( path, { { "top.txt", "hello" },
{ "dir/leaf.txt", "leaf" },
{ "empty/", "" },
{ ".hidden", "secret" },
{ "dir/Upper #%.txt", "mixed" } } ) );
VirtualFileSystem vfs;
const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) );
auto listed = waitFor( vfs.listAll( root ) );
ASSERT_TRUE( listed.hasValue() );
EXPECT_TRUE( sortedNames( listed.value() ) ==
std::vector<std::string>( { ".hidden", "dir", "empty", "top.txt" } ) );
auto folder = waitFor( vfs.stat( root.withAppendedSegment( "dir" ) ) );
ASSERT_TRUE( folder.hasValue() );
EXPECT_TRUE( folder.value().isDirectory() );
auto file = waitFor( vfs.stat( root.withAppendedSegment( "top.txt" ) ) );
ASSERT_TRUE( file.hasValue() );
EXPECT_EQ( 5u, file.value().size().value_or( 0 ) );
EXPECT_TRUE( file.value().modifiedTime().has_value() );
EXPECT_FALSE( file.value().revision.empty() );
EXPECT_FALSE( file.value().permissions().has_value() );
EXPECT_FALSE( folder.value().permissions().has_value() );
auto empty = waitFor( vfs.listAll( root.withAppendedSegment( "empty" ) ) );
ASSERT_TRUE( empty.hasValue() );
EXPECT_TRUE( empty.value().empty() );
EXPECT_STDSTREQ( "mixed",
readString( vfs, root.withAppendedSegment( "dir" ).withAppendedSegment(
"Upper #%.txt" ) ) );
auto opened = waitFor( vfs.openDirectory( root ) );
ASSERT_TRUE( opened.hasValue() );
auto handle = std::move( opened ).value();
size_t count = 0;
bool end = false;
while ( !end ) {
auto batch = waitFor( handle.read( 1 ) );
ASSERT_TRUE( batch.hasValue() );
count += batch.value().entries.size();
end = batch.value().endOfDirectory;
}
EXPECT_EQ( 4u, count );
EXPECT_TRUE( waitFor( handle.close() ).hasValue() );
}
UTEST( ZipProvider, streamingReadsSeekBackwardsAndCopyAcrossProviders ) {
TempDir temp;
std::string content( 300000, 'x' );
Uint32 random = 1;
for ( auto& byte : content ) {
random = random * 1664525u + 1013904223u;
byte = static_cast<char>( random >> 24 );
}
const std::string path = temp.path + "archive.zip";
ASSERT_TRUE( makeZip(
path, { { "dir/large.txt", content }, { "dir/second.txt", "second" }, { "zero", "" } } ) );
VirtualFileSystem vfs;
const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) );
const URI file = root.withAppendedSegment( "dir" ).withAppendedSegment( "large.txt" );
auto opened = waitFor( vfs.open( file ) );
ASSERT_TRUE( opened.hasValue() );
auto handle = std::move( opened ).value();
std::array<Uint8, 31> buffer;
for ( Uint64 offset : { 0ull, 100000ull, 15ull, 299980ull, 300000ull } ) {
auto result = waitFor( handle.read( offset, buffer ) );
ASSERT_TRUE( result.hasValue() );
const std::string expected = content.substr( offset, buffer.size() );
EXPECT_EQ( expected.size(), result.value() );
EXPECT_TRUE( expected == std::string( buffer.begin(), buffer.begin() + result.value() ) );
}
EXPECT_TRUE( waitFor( handle.close() ).hasValue() );
EXPECT_EQ( VFSErrorCode::InvalidOperation, codeOf( waitFor( handle.read( 0, buffer ) ) ) );
EXPECT_STDSTREQ( "", readString( vfs, root.withAppendedSegment( "zero" ) ) );
EXPECT_TRUE( content == readString( vfs, file ) );
const URI destination = URI::fromFSPath( temp.path + "extracted" );
auto copied =
waitFor( vfs.copy( root.withAppendedSegment( "dir" ), destination ).completion() );
ASSERT_TRUE( copied.hasValue() );
EXPECT_TRUE( content == readString( vfs, destination.withAppendedSegment( "large.txt" ) ) );
EXPECT_STDSTREQ( "second", readString( vfs, destination.withAppendedSegment( "second.txt" ) ) );
EXPECT_TRUE( waitFor( vfs.readAll( file, {}, 100 ) ).hasError() );
// An in-place edit can break a backward seek. Keep the old stream alive when reopening
// fails: this is an I/O failure, and must not turn every later call into "File closed".
auto reopened = waitFor( vfs.open( file ) );
ASSERT_TRUE( reopened.hasValue() );
auto changing = std::move( reopened ).value();
ASSERT_TRUE( waitFor( changing.read( 200000, buffer ) ).hasValue() );
ASSERT_TRUE( FileSystem::fileWrite( path, std::string{} ) );
EXPECT_TRUE( waitFor( changing.read( 0, buffer ) ).hasError() );
auto eof = waitFor( changing.read( content.size(), buffer ) );
ASSERT_TRUE( eof.hasValue() );
EXPECT_EQ( 0u, eof.value() );
}
UTEST( ZipProvider, readOnlyErrorsMissingFilesAndCancellation ) {
TempDir temp;
const std::string path = temp.path + "archive.zip";
ASSERT_TRUE( makeZip( path, { { "file", "contents" } } ) );
VirtualFileSystem vfs;
const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) );
const URI file = root.withAppendedSegment( "file" );
EXPECT_EQ( VFSErrorCode::ReadOnly, codeOf( waitFor( vfs.open( file, OpenFlags::Write ) ) ) );
EXPECT_EQ( VFSErrorCode::ReadOnly,
codeOf( waitFor( vfs.makeDir( root.withAppendedSegment( "new" ) ) ) ) );
EXPECT_EQ( VFSErrorCode::ReadOnly, codeOf( waitFor( vfs.remove( file ) ) ) );
EXPECT_EQ( VFSErrorCode::ReadOnly,
codeOf( waitFor( vfs.rename( file, root.withAppendedSegment( "renamed" ) ) ) ) );
EXPECT_EQ( VFSErrorCode::IsDirectory, codeOf( waitFor( vfs.open( root ) ) ) );
EXPECT_EQ( VFSErrorCode::NotDirectory, codeOf( waitFor( vfs.openDirectory( file ) ) ) );
const URI fileWithSlash = ZipProvider::makeURI( URI::fromFSPath( path ), "/file/" );
EXPECT_EQ( VFSErrorCode::NotDirectory, codeOf( waitFor( vfs.stat( fileWithSlash ) ) ) );
EXPECT_EQ( VFSErrorCode::NotDirectory, codeOf( waitFor( vfs.open( fileWithSlash ) ) ) );
EXPECT_EQ( VFSErrorCode::NotDirectory,
codeOf( waitFor( vfs.stat( file.withAppendedSegment( "child" ) ) ) ) );
EXPECT_EQ( VFSErrorCode::NotFound,
codeOf( waitFor( vfs.stat( root.withAppendedSegment( "missing" ) ) ) ) );
CancellationSource cancelled;
cancelled.cancel();
EXPECT_EQ( VFSErrorCode::Cancelled,
codeOf( waitFor( vfs.open( file, {}, cancelled.token() ) ) ) );
const URI missing = ZipProvider::makeURI( URI::fromFSPath( temp.path + "missing.zip" ) );
EXPECT_EQ( VFSErrorCode::NotFound, codeOf( waitFor( vfs.stat( missing ) ) ) );
EXPECT_FALSE( FileSystem::fileExists( temp.path + "missing.zip" ) );
ASSERT_TRUE( FileSystem::fileWrite( temp.path + "bad.zip", "this is not a ZIP" ) );
EXPECT_EQ( VFSErrorCode::InvalidOperation,
codeOf( waitFor( vfs.stat(
ZipProvider::makeURI( URI::fromFSPath( temp.path + "bad.zip" ) ) ) ) ) );
EXPECT_FALSE( vfs.capabilities( file ).has( VFSCapability::Write ) );
auto opened = waitFor( vfs.open( file ) );
ASSERT_TRUE( opened.hasValue() );
auto handle = std::move( opened ).value();
const std::array<Uint8, 1> byte{ 'x' };
EXPECT_EQ( VFSErrorCode::ReadOnly, codeOf( waitFor( handle.write( 0, byte ) ) ) );
EXPECT_EQ( VFSErrorCode::ReadOnly, codeOf( waitFor( handle.truncate( 0 ) ) ) );
}
UTEST( ZipProvider, browsingNeverRegistersResourcesAndNoticesArchiveReplacement ) {
TempDir temp;
const std::string path = temp.path + "archive.zip";
ASSERT_TRUE( makeZip( path, { { "zip-provider-only.txt", "old" } } ) );
const Uint64 generation = PackFileIndex::instance()->getGeneration();
VirtualFileSystem vfs;
const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) );
EXPECT_STDSTREQ( "old",
readString( vfs, root.withAppendedSegment( "zip-provider-only.txt" ) ) );
EXPECT_EQ( generation, PackFileIndex::instance()->getGeneration() );
EXPECT_FALSE( PackFileIndex::instance()->fileExists( "zip-provider-only.txt" ) );
auto oldInfo = waitFor( vfs.stat( root.withAppendedSegment( "zip-provider-only.txt" ) ) );
ASSERT_TRUE( oldInfo.hasValue() );
// Replace only after handles have closed (portable to Windows, where an open file can
// prevent replacement). A fresh operation must discard the cached directory index.
ASSERT_TRUE( FileSystem::fileRemove( path ) );
ASSERT_TRUE( makeZip( path, { { "new.txt", "new contents" } } ) );
auto listing = waitFor( vfs.listAll( root ) );
ASSERT_TRUE( listing.hasValue() );
EXPECT_TRUE( sortedNames( listing.value() ) == std::vector<std::string>( { "new.txt" } ) );
EXPECT_STDSTREQ( "new contents", readString( vfs, root.withAppendedSegment( "new.txt" ) ) );
EXPECT_EQ(
VFSErrorCode::NotFound,
codeOf( waitFor( vfs.stat( root.withAppendedSegment( "zip-provider-only.txt" ) ) ) ) );
}
UTEST( Zip, standaloneReaderIsReadOnlyAndHasBoundedSeek ) {
TempDir temp;
const auto path = temp.path + "archive.zip";
const std::string content( 200000, 'a' );
ASSERT_TRUE( makeZip( path, { { "large", content } } ) );
std::string original;
ASSERT_TRUE( FileSystem::fileGet( path, original ) );
{
Zip reader( false );
ASSERT_TRUE( reader.open( path, Zip::OpenMode::ReadOnly ) );
EXPECT_TRUE( reader.isOpen() );
const std::array<Uint8, 1> byte{ 'x' };
EXPECT_FALSE( reader.addFile( byte.data(), byte.size(), "forbidden" ) );
EXPECT_TRUE( reader.error() == Zip::Error::ReadOnly );
EXPECT_FALSE( reader.eraseFile( "large" ) );
EXPECT_EQ( 1u, reader.entryCount() );
Zip::Entry entry;
ASSERT_TRUE( reader.entry( 0, entry ) );
EXPECT_EQ( content.size(), entry.size );
auto stream = reader.openStream( 0 );
ASSERT_TRUE( stream != nullptr );
EXPECT_EQ( 199999, stream->seek( 199999 ) );
char value = 0;
EXPECT_EQ( 1, stream->read( &value, 1 ) );
EXPECT_EQ( int( 'a' ), int( value ) );
EXPECT_EQ( 0, stream->seek( 0 ) );
EXPECT_EQ( 1, stream->read( &value, 1 ) );
}
std::string after;
ASSERT_TRUE( FileSystem::fileGet( path, after ) );
EXPECT_TRUE( original == after );
Zip missing( false );
EXPECT_FALSE( missing.open( temp.path + "missing.zip", Zip::OpenMode::ReadOnly ) );
EXPECT_TRUE( missing.error() == Zip::Error::NotFound );
EXPECT_FALSE( FileSystem::fileExists( temp.path + "missing.zip" ) );
EXPECT_FALSE( missing.open( temp.path + "missing.zip" ) );
EXPECT_FALSE( FileSystem::fileExists( temp.path + "missing.zip" ) );
ASSERT_TRUE( FileSystem::fileWrite( temp.path + "invalid.zip", "not an archive" ) );
errno = EACCES;
EXPECT_FALSE( missing.open( temp.path + "invalid.zip", Zip::OpenMode::ReadOnly ) );
EXPECT_TRUE( missing.error() == Zip::Error::InvalidArchive );
errno = EACCES;
EXPECT_FALSE( missing.open( temp.path + "invalid.zip" ) );
EXPECT_TRUE( missing.error() == Zip::Error::InvalidArchive );
}
UTEST( ZipLocation, readableAddressesRoundTripAndRootReturnsToContainingFolder ) {
TempDir temp;
const URI archive = URI::fromFSPath( temp.path + "Mixed # %!?.zip" );
const URI root = ZipProvider::makeURI( archive );
const URI member = root.withAppendedSegment( "a # %.txt" );
EXPECT_TRUE( eefm::Location::parse( eefm::Location::text( root ).toUtf8() ) == root );
EXPECT_TRUE( eefm::Location::parse( eefm::Location::text( member ).toUtf8() ) == member );
EXPECT_TRUE( URI( eefm::Location::text( member ).toUtf8() ) == member );
EXPECT_TRUE( eefm::Location::text( root ).toUtf8().starts_with( "zip:///" ) );
EXPECT_TRUE( ZipProvider::archiveURI( URI( "zip:///tmp/archive.zip!/docs" ) ).hasValue() );
EXPECT_TRUE( eefm::Location::parent( root ) == archive.getParent() );
EXPECT_TRUE( eefm::Location::parent( member ) == root );
EXPECT_STDSTREQ( archive.getLastPathSegment(), eefm::Location::entryName( root ) );
EXPECT_STDSTREQ( archive.getLastPathSegment(), eefm::Location::displayName( root ).toUtf8() );
}
UTEST( ZipProvider, unsafeNamesAreExcludedAndCollisionsAreDeterministic ) {
TempDir temp;
// Stored ZIP with traversal/absolute/drive names, duplicate entries, and a file/directory
// collision. Kept as a fixture because Zip::addFile refuses duplicate names.
const std::string encoded = "UEsDBBQAAAAAADIIUV2PKKQfBAAAAAQAAAAIAAAAc2FmZS50eHRzYWZlUEsDBBQAAA"
"AAADIIUV37OSuCAwAAAAMAAAAN"
"AAAALi4vZXNjYXBlLnR4dGJhZFBLAwQUAAAAAAAyCFFd+"
"zkrggMAAAADAAAADQAAAC9hYnNvbHV0ZS50eHRiYWRQSwME"
"FAAAAAAAMghRXfs5K4IDAAAAAwAAABQAAABkaXIvLi4vLi4vZXNjYXBlLnR4dGJhZF"
"BLAwQUAAAAAAAyCFFd+zkrggMA"
"AAADAAAACwAAAEM6ZHJpdmUudHh0YmFkUEsDBBQAAAAAADIIUV37OSuCAwAAAAMAAA"
"AOAAAAZGlyXGVzY2FwZS50eHRi"
"YWRQSwMEFAAAAAAAMghRXfs5K4IDAAAAAwAAAA4AAAAuLiAvZXNjYXBlLnR4dGJhZF"
"BLAwQUAAAAAAAyCFFdAAAAAAAA"
"AAAAAAAABgAAAGVtcHR5L1BLAwQUAAAAAAAyCFFdEDafjAQAAAAEAAAACQAAAGNvbG"
"xpc2lvbmZpbGVQSwMEFAAAAAAA"
"MghRXSlUsyIFAAAABQAAAA8AAABjb2xsaXNpb24vY2hpbGRjaGlsZFBLAwQUAAAAAA"
"AyCFFdV+5xkgUAAAAFAAAACQAA"
"AGR1cGxpY2F0ZWZpcnN0UEsDBBQAAAAAADIIUV1pER+"
"2BgAAAAYAAAAJAAAAZHVwbGljYXRlc2Vjb25kUEsBAhQDFAAA"
"AAAAMghRXY8opB8EAAAABAAAAAgAAAAAAAAAAAAAAIABAAAAAHNhZmUudHh0UEsBAh"
"QDFAAAAAAAMghRXfs5K4IDAAAA"
"AwAAAA0AAAAAAAAAAAAAAIABKgAAAC4uL2VzY2FwZS50eHRQSwECFAMUAAAAAAAyCF"
"Fd+zkrggMAAAADAAAADQAAAAAA"
"AAAAAAAAgAFYAAAAL2Fic29sdXRlLnR4dFBLAQIUAxQAAAAAADIIUV37OSuCAwAAAA"
"MAAAAUAAAAAAAAAAAAAACAAYYA"
"AABkaXIvLi4vLi4vZXNjYXBlLnR4dFBLAQIUAxQAAAAAADIIUV37OSuCAwAAAAMAAA"
"ALAAAAAAAAAAAAAACAAbsAAABD"
"OmRyaXZlLnR4dFBLAQIUAxQAAAAAADIIUV37OSuCAwAAAAMAAAAOAAAAAAAAAAAAAA"
"CAAecAAABkaXJcZXNjYXBlLnR4"
"dFBLAQIUAxQAAAAAADIIUV37OSuCAwAAAAMAAAAOAAAAAAAAAAAAAACAARYBAAAuLi"
"AvZXNjYXBlLnR4dFBLAQIUAxQA"
"AAAAADIIUV0AAAAAAAAAAAAAAAAGAAAAAAAAAAAAEAD9QUUBAABlbXB0eS9QSwECFA"
"MUAAAAAAAyCFFdEDafjAQAAAAE"
"AAAACQAAAAAAAAAAAAAAgAFpAQAAY29sbGlzaW9uUEsBAhQDFAAAAAAAMghRXSlUsy"
"IFAAAABQAAAA8AAAAAAAAAAAAA"
"AIABlAEAAGNvbGxpc2lvbi9jaGlsZFBLAQIUAxQAAAAAADIIUV1X7nGSBQAAAAUAAA"
"AJAAAAAAAAAAAAAACAAcYBAABk"
"dXBsaWNhdGVQSwECFAMUAAAAAAAyCFFdaREftgYAAAAGAAAACQAAAAAAAAAAAAAAgA"
"HyAQAAZHVwbGljYXRlUEsFBgAA"
"AAAMAAwAtQIAAB8CAAAAAA==";
std::string bytes;
ASSERT_NE( size_t( -1 ), Base64::decode( encoded, bytes ) );
ASSERT_TRUE( FileSystem::fileWrite( temp.path + "unsafe.zip", bytes ) );
VirtualFileSystem vfs;
const URI root = ZipProvider::makeURI( URI::fromFSPath( temp.path + "unsafe.zip" ) );
auto listed = waitFor( vfs.listAll( root ) );
ASSERT_TRUE( listed.hasValue() );
EXPECT_TRUE( sortedNames( listed.value() ) ==
std::vector<std::string>( { "collision", "duplicate", "empty", "safe.txt" } ) );
EXPECT_STDSTREQ( "first", readString( vfs, root.withAppendedSegment( "duplicate" ) ) );
EXPECT_STDSTREQ(
"child",
readString( vfs, root.withAppendedSegment( "collision" ).withAppendedSegment( "child" ) ) );
const auto destination = URI::fromFSPath( temp.path + "out" );
ASSERT_TRUE( waitFor( vfs.copy( root, destination ).completion() ).hasValue() );
EXPECT_FALSE( FileSystem::fileExists( temp.path + "escape.txt" ) );
EXPECT_STDSTREQ( "safe", readString( vfs, destination.withAppendedSegment( "safe.txt" ) ) );
}
UTEST( ZipProvider, crcFailureIsReportedEvenForAnExactSizedRead ) {
TempDir temp;
const std::string encoded =
"UEsDBBQAAAAAADIIUV0Ks4YUDwAAAA8AAAALAAAAY2hlY2tlZC50eHRjaGVja2VkIHBheWxvYWRQSwECFAMUAAAAAA"
"Ay"
"CFFdCrOGFA8AAAAPAAAACwAAAAAAAAAAAAAAgAEAAAAAY2hlY2tlZC50eHRQSwUGAAAAAAEAAQA5AAAAOAAAAAAA";
std::string bytes;
ASSERT_NE( size_t( -1 ), Base64::decode( encoded, bytes ) );
const auto payload = bytes.find( "checked payload" );
ASSERT_NE( std::string::npos, payload );
bytes[payload] ^= 1;
ASSERT_TRUE( FileSystem::fileWrite( temp.path + "corrupt.zip", bytes ) );
VirtualFileSystem vfs;
const URI root = ZipProvider::makeURI( URI::fromFSPath( temp.path + "corrupt.zip" ) );
auto opened = waitFor( vfs.open( root.withAppendedSegment( "checked.txt" ) ) );
ASSERT_TRUE( opened.hasValue() );
auto handle = std::move( opened ).value();
std::array<Uint8, 15> data;
EXPECT_EQ( VFSErrorCode::IO, codeOf( waitFor( handle.read( 0, data ) ) ) );
}
UTEST( ZipProvider, openHandlesSurviveCacheEvictionAndProviderRemoval ) {
TempDir temp;
const auto path = temp.path + "original.zip";
ASSERT_TRUE( makeZip( path, { { "file", "original" }, { "other", "second" } } ) );
VirtualFileSystem vfs;
const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) );
auto first = waitFor( vfs.open( root.withAppendedSegment( "file" ) ) );
auto second = waitFor( vfs.open( root.withAppendedSegment( "other" ) ) );
ASSERT_TRUE( first.hasValue() );
ASSERT_TRUE( second.hasValue() );
auto a = std::move( first ).value();
auto b = std::move( second ).value();
for ( int i = 0; i < 9; ++i ) {
const auto other = temp.path + String::toString( i ) + ".zip";
ASSERT_TRUE( makeZip( other, { { "x", "x" } } ) );
ASSERT_TRUE(
waitFor( vfs.stat( ZipProvider::makeURI( URI::fromFSPath( other ) ) ) ).hasValue() );
}
vfs.unregisterProvider( "zip" );
std::array<Uint8, 8> left;
std::array<Uint8, 6> right;
auto readingA = a.read( 0, left );
auto readingB = b.read( 0, right );
ASSERT_TRUE( waitFor( std::move( readingA ) ).hasValue() );
ASSERT_TRUE( waitFor( std::move( readingB ) ).hasValue() );
EXPECT_TRUE( std::string( left.begin(), left.end() ) == "original" );
EXPECT_TRUE( std::string( right.begin(), right.end() ) == "second" );
EXPECT_TRUE( waitFor( a.close() ).hasValue() );
EXPECT_TRUE( waitFor( b.close() ).hasValue() );
}
UTEST( ZipProvider, emptyArchivesAndUnicodeBackingFiles ) {
TempDir temp;
const std::string empty = std::string( "PK\x05\x06", 4 ) + std::string( 18, '\0' );
const auto path = temp.path + "empty.zip";
ASSERT_TRUE( FileSystem::fileWrite( path, empty ) );
VirtualFileSystem vfs;
auto listing = waitFor( vfs.listAll( ZipProvider::makeURI( URI::fromFSPath( path ) ) ) );
ASSERT_TRUE( listing.hasValue() );
EXPECT_TRUE( listing.value().empty() );
const auto unicode = temp.path + "archivo-ñ-雪.zip";
// Build at an ASCII path with the legacy writer, then rename through the portable VFS.
const auto source = temp.path + "source.zip";
ASSERT_TRUE( makeZip(
source, { { "file", "unicode archive" }, { "directorio-ñ/雪.txt", "unicode member" } } ) );
ASSERT_TRUE(
waitFor( vfs.rename( URI::fromFSPath( source ), URI::fromFSPath( unicode ) ) ).hasValue() );
EXPECT_STDSTREQ( "unicode archive",
readString( vfs, ZipProvider::makeURI( URI::fromFSPath( unicode ) )
.withAppendedSegment( "file" ) ) );
const URI member = ZipProvider::makeURI( URI::fromFSPath( unicode ), "/directorio-ñ/雪.txt" );
EXPECT_STDSTREQ( "unicode member", readString( vfs, URI( member.toString() ) ) );
}
UTEST( ZipProvider, pollingWatchNoticesBackingArchiveReplacement ) {
TempDir temp;
const auto path = temp.path + "watched.zip";
ASSERT_TRUE( makeZip( path, { { "before", "old" } } ) );
VirtualFileSystem vfs;
const URI root = ZipProvider::makeURI( URI::fromFSPath( path ) );
WatchOptions options;
options.pollingInterval = Milliseconds( 50 );
auto opened = waitFor( vfs.watch( root, options ) );
ASSERT_TRUE( opened.hasValue() );
WatchHandle watch( std::move( opened ).value() );
EXPECT_EQ( WatchStatus::Polling, watch.status() );
TestLoop loop;
EventCollector collector;
watch.onEvents( loop, collector.callback() );
ASSERT_TRUE( FileSystem::fileRemove( path ) );
ASSERT_TRUE( makeZip( path, { { "after", "new contents" } } ) );
EXPECT_TRUE( loop.runUntil( [&] { return collector.has( WatchEventType::Added, "after" ); } ) );
EXPECT_TRUE( collector.has( WatchEventType::Removed, "before" ) );
EXPECT_STDSTREQ( "new contents", readString( vfs, root.withAppendedSegment( "after" ) ) );
}
+2
View File
@@ -257,6 +257,8 @@ _zip_readcdir(FILE *fp, off_t buf_offset, unsigned char *buf, unsigned char *eoc
left = cd->size;
i=0;
if (cd->nentry == 0 && left == 0)
return cd;
do {
if ((_zip_dirent_read(cd->entry+i, fp, bufp, &left, 0, error)) < 0) {
cd->nentry = i;
+7
View File
@@ -8,6 +8,7 @@
#include "placescontroller.hpp"
#include "trash.hpp"
#include <eepp/system/sha1.hpp>
#include <eepp/vfs/providers/zipprovider.hpp>
namespace eefm {
@@ -70,6 +71,12 @@ void FileOperations::openRow( FileTab* tab, size_t row ) {
mApp.panes().navigate( tab, tab->model->entryURI( row ) );
return;
}
const URI entry = tab->model->entryURI( row );
if ( entry.getScheme() == "file" &&
String::toLower( FileSystem::fileExtension( tab->model->entry( row ).name ) ) == "zip" ) {
mApp.panes().navigate( tab, ZipProvider::makeURI( entry ) );
return;
}
// Enter / double click on a multiple selection opens all the selected files.
auto rows = tab->selectedRows();
if ( std::find( rows.begin(), rows.end(), row ) == rows.end() )
+36 -2
View File
@@ -1,14 +1,16 @@
#include "location.hpp"
#include <eepp/system/filesystem.hpp>
#include <eepp/vfs/providers/zipprovider.hpp>
using namespace EE;
using namespace EE::System;
using namespace EE::VFS;
namespace eefm {
URI Location::parse( const std::string& text ) {
std::string trimmed = String::trim( text );
if ( trimmed.find( "://" ) != std::string::npos ) {
if ( trimmed.find( "://" ) != std::string::npos || trimmed.starts_with( "zip:" ) ) {
URI uri( trimmed );
if ( uri.getPath().empty() )
uri.setRawPath( "/" );
@@ -17,14 +19,46 @@ URI Location::parse( const std::string& text ) {
return URI::fromFSPath( FileSystem::expandTilde( trimmed ) );
}
URI Location::parent( const URI& uri ) {
if ( uri.getScheme() == "zip" ) {
auto archive = ZipProvider::archiveURI( uri );
auto member = ZipProvider::memberPath( uri );
if ( archive && member ) {
if ( member.value() == "/" )
return archive.value().getParent();
URI entry;
entry.setRawPath( std::move( member ).value() );
return ZipProvider::makeURI( archive.value(), entry.getParent().getPath() );
}
}
return uri.getParent();
}
std::string Location::entryName( const URI& uri ) {
if ( uri.getScheme() == "zip" ) {
auto archive = ZipProvider::archiveURI( uri );
auto member = ZipProvider::memberPath( uri );
if ( archive && member && member.value() == "/" )
return archive.value().getLastPathSegment();
}
return uri.getLastPathSegment();
}
String Location::displayName( const URI& uri ) {
std::string name = uri.getLastPathSegment();
std::string name = entryName( uri );
if ( uri.getScheme() == "zip" && !name.empty() )
return String::fromUtf8( name );
if ( name.empty() || uri.getPath() == "/" )
name = uri.getScheme() == "file" ? std::string( "/" ) : uri.getHost();
return String::fromUtf8( name );
}
String Location::text( const URI& uri ) {
if ( uri.getScheme() == "zip" ) {
// URI::toString() omits an empty authority for non-file schemes. Spell it explicitly
// in the location bar; both spellings parse into the same provider URI.
return String::fromUtf8( uri.getSchemeAndAuthority() + uri.getPathEtc() );
}
if ( uri.getScheme() == "file" ) {
std::string path = uri.getFSPath();
FileSystem::dirAddSlashAtEnd( path );
+12 -4
View File
@@ -7,21 +7,29 @@
namespace eefm {
using namespace EE;
/** How locations are typed, shown and copied. */
class Location {
public:
/** A typed location: a URI, or a local path ("~" expands to the home folder). */
static EE::URI parse( const std::string& text );
static URI parse( const std::string& text );
/** Parent location; leaving an archive root returns to the archive's containing folder. */
static URI parent( const URI& uri );
/** Entry to select when leaving this location (the archive's name at its root). */
static std::string entryName( const URI& uri );
/** The name of a folder for titles: its last segment, "/" or the host for roots. */
static EE::String displayName( const EE::URI& uri );
static String displayName( const URI& uri );
/** The location bar text: a local path with a trailing slash, the URI otherwise. */
static EE::String text( const EE::URI& uri );
static String text( const URI& uri );
/** A location for Copy Location: a plain path (no trailing slash) for local files and
* folders, the URI otherwise. Copying files (Ctrl+C) puts URIs in the clipboard instead. */
static std::string clipboardText( const EE::URI& uri );
static std::string clipboardText( const URI& uri );
};
} // namespace eefm
+4 -4
View File
@@ -568,7 +568,7 @@ void PaneManager::navigate( FileTab* tab, URI uri, bool recordHistory ) {
void PaneManager::goBack() {
auto* tab = currentTab();
if ( tab && tab->historyIndex > 0 ) {
tab->selectLater( tab->model->uri().getLastPathSegment() );
tab->selectLater( Location::entryName( tab->model->uri() ) );
--tab->historyIndex;
navigate( tab, tab->history[tab->historyIndex], false );
}
@@ -586,10 +586,10 @@ void PaneManager::goUp() {
auto* tab = currentTab();
if ( !tab )
return;
URI parent = tab->model->uri().getParent();
URI parent = Location::parent( tab->model->uri() );
if ( parent != tab->model->uri() ) {
// Like Dolphin, the folder we leave stays selected in its parent.
tab->selectLater( tab->model->uri().getLastPathSegment() );
tab->selectLater( Location::entryName( tab->model->uri() ) );
navigate( tab, parent );
}
}
@@ -649,7 +649,7 @@ void PaneManager::updateChrome( Pane& pane ) {
return;
mUI->find<UIWidget>( "back" )->setEnabled( tab->historyIndex > 0 );
mUI->find<UIWidget>( "forward" )->setEnabled( tab->historyIndex + 1 < tab->history.size() );
mUI->find<UIWidget>( "up" )->setEnabled( uri.getParent() != uri );
mUI->find<UIWidget>( "up" )->setEnabled( Location::parent( uri ) != uri );
mApp.window()->setTitle( Location::text( uri ) + " — eefm" );
if ( auto* panel = mApp.places().panel() )
panel->setCurrent( uri );