eepp/docs/articles/vfs.md

Virtual File System (eepp-vfs)

Introduction

eepp-vfs is an optional static module that gives applications one asynchronous, URI-based API for local files, packed resources and remote servers:

Scheme	Provider	Notes
file://	LocalProvider	Native filesystem (POSIX and Win32).
res://	ResourceProvider	Read-only view of every open Pack (System::PackFileIndex).
sftp://	SFTPProvider	Built by default (--without-vfs-sftp disables it), libssh2 based, honors ~/.ssh/config.
mem://	MemoryProvider	In-memory filesystem, mainly for tests.

The module builds on the async core, generic primitives that live in eepp core and are useful on their own:

Header	Type
<eepp/core/result.hpp>	Result<T, E>, Ok(), Err()
<eepp/core/task.hpp>	Task<T>, TaskSource<T>, makeReadyTask()
<eepp/core/cancellationtoken.hpp>	CancellationSource, CancellationToken, CancellationRegistration
<eepp/core/executor.hpp>	Executor, InlineExecutor (ThreadPool is an Executor and has submit())
<eepp/scene/mainthreadexecutor.hpp>	MainThreadExecutor: runs continuations on the UI thread
<eepp/core/uri.hpp>	URI (formerly Network::URI, which remains an alias)

Link the module with eepp_module_vfs_add() in Premake and include <eepp/vfs.hpp>. The UI model lives in <eepp/vfs/ui/vfsdirectorymodel.hpp> and is not part of the aggregate header.

Tasks, results and threads

Every VFS operation returns VFSTask<T>, an alias of Task<Result<T, VFSError>>:

auto& vfs = EE::VFS::instance();
MainThreadExecutor ui( sceneNode );

vfs.stat( URI( "sftp://makyo/home/black/notes.txt" ) )
	.thenOn( ui, []( VFSResult<FileInfo>&& info ) {
		if ( info )
			label->setText( info.value().name );
		else
			label->setText( info.error().toString() );
	} );

Rules:

• VFS methods never block and may be called from any thread.
• Tasks resolve on VFS threads. then() runs where the task resolves; thenOn( executor, ... ) always goes through the executor. Touch UI or model state only from thenOn( uiExecutor, ... ).
• A Task has one consumer: continuations consume it (std::move( task ).then( ... )). A continuation returning another task is flattened.
• If a producer is destroyed without resolving, its task chain becomes Abandoned instead of hanging. That is an infrastructure bug (reported by TaskDiagnostics), never a domain outcome.
• Cancellation is cooperative: pass CancellationTokens; cancelled operations complete with VFSErrorCode::Cancelled.
• Task::wait()/get() block. They exist for worker-thread adapters (for example IOStreamVFS) and tests; never use them on the UI thread.

Errors are typed (VFSErrorCode): callers branch on codes such as NotFound, PermissionDenied, ChangedExternally or HostKeyRejected, never on errno or libssh2 values.

Files and directories

// Incremental listing: batches with an explicit end-of-directory flag.
vfs.openDirectory( uri );    // VFSTask<DirectoryHandle>; then handle.read( 256 ) per batch

vfs.listAll( uri );          // convenience: the whole directory
vfs.readAll( uri );          // VFSTask<std::vector<Uint8>>
vfs.writeAll( uri, data );   // atomic by default, returns the new FileInfo
vfs.makeDirs( uri );
vfs.remove( uri, RemoveOptions{ true } );   // recursive
vfs.rename( from, to, RenameOptions{ true } );
vfs.canonicalize( uri );     // resolves symlinks/aliases; URIs are never canonicalized implicitly

FileHandle uses explicit offsets (read( offset, span ), write( offset, span )); FileCursor adds a sequential position. IOStreamVFS exposes a VFS file as a blocking System::IOStream for existing decoders running on worker threads.

FileInfo is a plain value. Unknown metadata stays unknown (std::optional accessors). It does not store its own URI, because listings hold many entries and the caller always knows the URI it listed: use info.uriIn( directory ) for a listed entry (VFSDirectoryModel::entryURI( row ) in the model). FileInfo::revision is an opaque token used for conflict detection:

WriteOptions options;
options.expectedRevision = lastKnown.revision;   // from a previous stat/writeAll
vfs.writeAll( uri, text, options );              // fails with ChangedExternally if modified

Atomic writes go to a sibling temporary file, are flushed and renamed over the destination; a symlinked destination is written through to its target. Providers report what they can do through capabilities( uri ) (for example AtomicReplace, Fsync, Permissions).

Copy and move

Recursive and cross-provider transfers are orchestrated by the VFS, not by providers:

Transfer transfer = vfs.copy( URI( "sftp://makyo/srv/www" ), URI::fromFSPath( "/tmp/www" ) );
auto progress = transfer.onProgress( []( const TransferProgress& p ) { ... }, &uiExecutor );
transfer.completion().thenOn( uiExecutor, []( VFSResult<TransferResult>&& r ) { ... } );
transfer.cancel();

Moves use a native rename when both sides are on the same provider/authority and fall back to copy plus removal of the entries that were completely copied; a source is never removed after a partial transfer.

File copies are written to a temporary sibling before publication. Fail and Skip also apply when a destination appears while the copy is running. The Overwrite policy requires AtomicReplace; otherwise the transfer returns Unsupported and preserves the old contents. Creating a file with Fail or Skip requires Rename. Moving a symlink through the copy fallback is unsupported; native same-filesystem renames still move the link itself.

Watching

vfs.watch( uri, WatchOptions{ /*recursive*/ true } )
	.thenOn( ui, [this]( VFSResult<WatchHandle>&& watch ) {
		mWatch = std::move( watch ).value();
		mWatch.onEvents( ui, [this]( WatchEventBatch&& batch ) { ... } );
	} );

Events are relative to the watched root and arrive in coalesced batches through the executor you choose. RescanRequired means continuity was lost (missed native events, reconnection, overflow): re-list the directory. WatchHandle::status() reports Active, Polling, Reconnecting or Failed, so a broken remote watch never looks healthy.

Backends:

• file://: efsw native watching; directories efsw reports as remote (NFS, SMB) use efsw's generic polling watcher.
• sftp://: efsw-agent --stdio runs on the server through an exec channel of the same SSH connection (one agent per connection, many watches). When the agent dies or the connection drops, watches report Reconnecting, re-register and emit RescanRequired. If the agent is not installed they fall back to polling over SFTP. SFTP itself never depends on the agent.
• Anything else: polling through the VFS (PollingWatchBackend).

To establish a consistent view, watch first, list second, then apply the events that arrived during the listing; VFSDirectoryModel does exactly that.

VFSDirectoryModel

A UI::Models::Model for one directory of any scheme:

auto model = VFSDirectoryModel::New( uiExecutor );
tableView->setModel( model );
model->setURI( URI( "sftp://makyo/home/black" ) );

It enumerates incrementally (the first rows appear after the first batch), discards results of previous navigations, stays live through watches (row inserts/deletes keep the selection), and exposes loadState(), error() and watchStatus().

Columns are Name, Size, Modified, Type, Permissions, Owner and Group; cells carry a per-column CSS class (vfs_size, ...). Applications customize the text with setTypeProvider(), setTimeFormatter() and setItemCountFormatter(). With Config::countDirectoryItems (or setCountDirectoryItems()), directories are counted in the background and the Size column shows "N items". setNameFilter( pattern, mode, caseSensitive ) filters rows by name with unanchored PlainText, Glob or RegEx patterns, like Dolphin's filter bar.

SFTP

SFTP is built by default (disable it with --without-vfs-sftp, Premake 4 or 5; it is never built for emscripten). This compiles the vendored libssh2 (src/thirdparty/libssh2) against eepp's mbedTLS, or OpenSSL when --with-openssl is used.

sftp://host/path resolves host through OpenSSH configuration exactly like ssh host would: ~/.ssh/config and the system ssh_config, Host/Match (all client criteria), Include, first-value-wins, HostName, Port, User, IdentityFile, IdentityAgent, IdentitiesOnly, ProxyJump, ProxyCommand, UserKnownHostsFile, StrictHostKeyChecking, HostKeyAlias, algorithm lists, hostname canonicalization and the final re-parse pass. A user or port in the URI overrides the configuration. Passwords are rejected in URIs. SSHConfigEvaluator is usable on its own and is tested against ssh -G.

Ed25519 identity files and host keys and the curve25519-sha256 key exchange are supported with both crypto backends. mbedTLS does not implement them, so the vendored libssh2 gets them from Monocypher (src/thirdparty/monocypher, src/thirdparty/libssh2/src/mbedtls_ed25519.c).

Host keys follow StrictHostKeyChecking. Unless HostKeyAlgorithms is configured, the host key algorithms that known_hosts already has keys for are preferred, as OpenSSH does. Changed keys are always refused. For unknown keys with ask, the provider asks an application SSHHostKeyVerifier; secrets (passwords, passphrases, keyboard-interactive answers) come from an SSHAuthenticationHandler. Both return tasks, so the SSH I/O thread never blocks while the user answers. Accepted keys are appended atomically to the user's known_hosts file.

Authentication order: ssh-agent, identity files (asking for passphrases), keyboard-interactive, password, filtered by PreferredAuthentications and the *Authentication options.

Connections are pooled per effective configuration and shared by every operation and watch. All sessions are driven by one non-blocking I/O thread. posix-rename@openssh.com (atomic replace), fsync@openssh.com and statvfs@openssh.com are detected per server.

efsw-agent

efsw-agent is part of efsw (src/agent/efsw-agent.cpp, protocol in include/efsw/agent_protocol.hpp). It has no eepp dependency and only reports changes. Install it on the server's PATH, or set SFTPProviderConfig::watchAgentCommand. The eepp build produces bin/efsw-agent for the build host.

Known limitations

• FIDO security-key identity files (ssh-keygen -t ed25519-sk / ecdsa-sk) are not supported with either crypto backend: signing needs the hardware token (libfido2), which libssh2 leaves to the application. Such keys can be used through ssh-agent, which signs on the token's behalf (untested).
• Match localnetwork is not implemented on Windows.
• Jump hosts use their own ssh_config entries, but their own ProxyJump/ProxyCommand are ignored (the chain comes from the destination).
• Operations on one SSH connection are serialized; a long transfer delays metadata requests on the same connection.
• The efsw-agent is not deployed automatically.

Resources (res://)

System::PackFileIndex (formerly System::VirtualFileSystem, still available as a deprecated alias) indexes the files of every open Pack. res:///fonts/a.ttf maps to the pack path fonts/a.ttf; directories are implicit. The provider is read-only and extracts files to memory when they are opened.