fix(vfs): data safety, SSH policy and liveness fixes from the pre-merge review

Data safety (VFS):
- Copies write a temporary sibling and replace the destination only
  once it is complete (as writeAll() already did): a failed overwrite
  kept nothing of the old destination, and copying onto a hard link to
  the source truncated the source itself. A destination symlink is
  written through, and one that resolves to the source fails as the
  same file.
- Moving a symlink to another file system is refused: the fallback copy
  followed it and then removed the copied entries through the link,
  emptying its target.
- Saves keep the exact mode of the file they replace: the temporary was
  created with it, but filtered through the umask.

Deadlocks and blocking:
- writeAll( createParents ) waited on makeDirs() queued on its own
  orchestration pool: one thread deadlocked at once, two with two
  concurrent saves. Orchestration jobs now create parents directly.
- Copies and moves run on their own pool (Config::transferThreads), so
  long transfers no longer delay saves, reads and recursive removals.
- Local watches are registered and removed on the I/O pool: efsw and the
  directory checks touch the filesystem, which blocked the UI thread on
  slow mounts.

SSH security:
- "@revoked" known_hosts keys are rejected; they were skipped as
  unknown, so StrictHostKeyChecking accept-new trusted them.
- Sessions are shared only between configurations with the same
  security policy (known hosts, host key checking, algorithms,
  authentication settings), not just the same destination: a strict
  alias reused a session a permissive one had accepted.
- A configured algorithm restriction that permits nothing this libssh2
  supports fails the connection instead of using libssh2's defaults.

SSH liveness:
- Operations submitted to a failed or closed connection fail with
  ConnectionLost instead of staying pending forever.
- ssh-agent requests run on a worker thread (SSHAgent, Unix-domain
  sockets), with libssh2_userauth_publickey() and a sign callback that
  waits without blocking: an agent waiting for a hardware key touch no
  longer stalls every other session. Windows keeps libssh2's agent.
- Connecting no longer holds a helper thread while the user answers a
  host key or password prompt: the setup continues from the session's
  ready task, one ProxyJump hop at a time. Jump hosts only need to
  authenticate, not to offer SFTP.
- Idle connections close after SFTPProviderConfig::idleTimeout. The old
  check compared use_count() with 1 and never passed: connections now
  track their users (open handles, pumps, jump children).
- A remote watch agent that never answers falls back to polling after
  5 seconds; the directory listing waited for it indefinitely.

Core:
- CancellationSource::cancel() keeps the shared state alive while its
  callbacks run: a callback destroying the source freed it mid-cancel
  (heap use after free under AddressSanitizer).
- MainThreadExecutor::shutdown() detaches the executor from its
  dispatcher node and drops queued and later functions.
- String::icompare(): case-insensitive comparison without allocations.

VFSDirectoryModel (UI thread):
- Names compare with String::icompare() instead of lowercased copies,
  which made every sort, merge and insertion allocate (about 2x slower
  when sorting).
- Sorting by type asks the type provider once per entry instead of
  twice per comparison.
- The case-insensitive plain text filter keeps a search table instead
  of lowercasing every name on each row rebuild.
- Watch events update the visible rows in place instead of rebuilding
  them from every entry per event.

Drag and drop:
- The X11 drag source showed the pending badge over the source window
  for every application, which broke the X11 drag and drop integration
  tests in CI. DragOffer::sourceHandlesDrops now asks for it; eefm sets
  it, since it completes those drops as its own drag.

eefm:
- Shutdown destroyed the main thread executor before the VFS and the
  thread pool while continuations still held it: closing during a
  transfer could dispatch through freed memory. Transfers are cancelled,
  the VFS is destroyed, the executor is detached, and only then are the
  scene, its pool and the executor destroyed.
- Remote files opened locally were downloaded to
  /tmp/eefm/<scheme>/<host>/<name>: files with the same name from other
  folders, users or ports overwrote each other, in a directory shared by
  every user. They now go to a per-file folder (a hash of the complete
  URI) in the user's cache.

Every fix comes with a regression test.
This commit is contained in:
Martín Lucas Golini
2026-10-15 20:33:13 -03:00
parent 8827f8de86
commit 0e7e12ba76
38 changed files with 2448 additions and 500 deletions
+3 -1
View File
@@ -216,7 +216,9 @@ including one the user switches to with Alt+Tab mid-drag. Internal drag feedback
the source window still receives the pointer. Ignore the internal drag's release while the native
session is active and resolve it from the completion instead. On X11, a release over the source
window ends the session `Cancelled` with `DragResult::releasedOverSource` set, so the application
completes it as its internal drop. Elsewhere, such a drop arrives as `FileDropped` events.
completes it as its internal drop. Elsewhere, such a drop arrives as `FileDropped` events. Set
`DragOffer::sourceHandlesDrops` so the preview shows a pending drop, not a rejected one, while the
pointer is over the source window.
Applications that install their own `SDL_SetX11EventHook` must coordinate its use: SDL3 has a single
hook slot, which eepp uses during an active native drag and clears afterwards.
+6
View File
@@ -110,6 +110,12 @@ Moves use a native rename when both sides are on the same provider/authority and
plus removal of the entries that were completely copied; a source is never removed after a partial
transfer.
File copies are written to a temporary sibling before publication. `Fail` and `Skip` also apply
when a destination appears while the copy is running. The `Overwrite` policy requires
`AtomicReplace`; otherwise the transfer returns `Unsupported` and preserves the old contents.
Creating a file with `Fail` or `Skip` requires `Rename`. Moving a symlink through the copy fallback
is unsupported; native same-filesystem renames still move the link itself.
## Watching
```cpp
+4
View File
@@ -539,6 +539,10 @@ class EE_API String {
static bool iequals( String::View str1, String::View str2 );
/** Compares ignoring ASCII case, without allocating (as comparing String::toLower() copies
* would). @return Negative, zero or positive, like std::string::compare(). */
static int icompare( std::string_view str1, std::string_view str2 );
/** Creates a random string using the dictionary characters. */
static std::string randString(
size_t len,
+8 -2
View File
@@ -15,8 +15,9 @@ class Node;
* deferred to the next scene update, even when execute() is called from the main thread, which is
* what Task::thenOn() requires.
*
* The dispatcher node must outlive the executor. Destroying the executor drops queued functions
* that have not run yet (releasing what they own, for example abandoning task sources).
* The dispatcher node must outlive the executor, or shutdown() must be called before the node is
* destroyed. Destroying the executor drops queued functions that have not run yet (releasing what
* they own, for example abandoning task sources).
*/
class EE_API MainThreadExecutor : public Executor {
public:
@@ -30,6 +31,11 @@ class EE_API MainThreadExecutor : public Executor {
void execute( Function function ) override;
/** Detaches the executor from its dispatcher node and drops the queued functions. Functions
* executed afterwards are dropped too. Call it before the dispatcher node is destroyed when
* work may still be executed on this executor (it must then outlive that work). */
void shutdown();
/** @return Number of functions waiting for the main thread. */
size_t pending() const;
+7
View File
@@ -42,6 +42,13 @@ class EE_API Sys {
/** @return A storage path for config files for every platform */
static std::string getConfigPath( const std::string& appname );
/** @return A storage path for cache files (data the application can recreate) for every
* platform, private to the user: $XDG_CACHE_HOME/appname (~/.cache/appname) on Linux and BSD,
* ~/Library/Caches/appname on macOS and iOS, %LOCALAPPDATA%\appname on Windows,
* ~/config/cache/appname on Haiku. Elsewhere, or without those variables, a directory in the
* temporary path. The directory is not created. */
static std::string getCachePath( const std::string& appname );
/** @return The path of the directory designated for temporary files. */
static std::string getTempPath();
+3
View File
@@ -47,6 +47,9 @@ struct DragOffer {
SmallVector<DragFile, 1> files;
DragActions actions{ static_cast<DragActions>( DragAction::Copy ) };
DragPreview preview;
/** The application completes drops on its own window (DragResult::releasedOverSource on
* X11): the preview shows a pending drop there instead of a rejected one. */
bool sourceHandlesDrops{ false };
};
struct DragResult {
+4 -1
View File
@@ -107,7 +107,10 @@ CancellationSource::CancellationSource() :
mState( std::make_shared<detail::CancellationState>() ) {}
bool CancellationSource::cancel() {
return mState && mState->cancel();
// A callback may destroy this source (and with it the last reference to the state) while the
// state is still running callbacks: keep the state alive until cancel() returns.
std::shared_ptr<detail::CancellationState> state( mState );
return state && state->cancel();
}
} // namespace EE
+11
View File
@@ -2828,6 +2828,17 @@ bool String::iequals( std::string_view str1, std::string_view str2 ) {
} );
}
int String::icompare( std::string_view str1, std::string_view str2 ) {
const size_t length = std::min( str1.size(), str2.size() );
for ( size_t i = 0; i < length; ++i ) {
const int c1 = std::tolower( static_cast<unsigned char>( str1[i] ) );
const int c2 = std::tolower( static_cast<unsigned char>( str2[i] ) );
if ( c1 != c2 )
return c1 < c2 ? -1 : 1;
}
return str1.size() < str2.size() ? -1 : ( str1.size() > str2.size() ? 1 : 0 );
}
bool String::iequals( String::View str1, String::View str2 ) {
return str1.length() == str2.length() &&
std::equal( str1.begin(), str1.end(), str2.begin(),
+4
View File
@@ -28,6 +28,10 @@ MainThreadExecutor::MainThreadExecutor( Node* dispatcher ) : mState( std::make_s
}
MainThreadExecutor::~MainThreadExecutor() {
shutdown();
}
void MainThreadExecutor::shutdown() {
std::vector<Function> dropped;
{
std::lock_guard<std::mutex> lock( mState->mutex );
+34
View File
@@ -807,6 +807,40 @@ std::string Sys::getConfigPath( const std::string& appname ) {
return std::string( path );
}
std::string Sys::getCachePath( const std::string& appname ) {
auto under = []( std::string base, const char* subdirectory, const std::string& appname ) {
FileSystem::dirAddSlashAtEnd( base );
return base + subdirectory + appname;
};
#if EE_PLATFORM == EE_PLATFORM_WIN
std::string local = getEnv( "LOCALAPPDATA" );
if ( !local.empty() )
return under( std::move( local ), "", appname );
#elif EE_PLATFORM == EE_PLATFORM_MACOS || EE_PLATFORM == EE_PLATFORM_IOS
// On iOS HOME is the application's container.
std::string home = getEnv( "HOME" );
if ( !home.empty() )
return under( std::move( home ), "Library/Caches/", appname );
#elif EE_PLATFORM == EE_PLATFORM_HAIKU
std::string home = getEnv( "HOME" );
if ( !home.empty() )
return under( std::move( home ), "config/cache/", appname );
#elif EE_PLATFORM == EE_PLATFORM_LINUX || EE_PLATFORM == EE_PLATFORM_BSD || \
EE_PLATFORM == EE_PLATFORM_SOLARIS
// The XDG Base Directory specification ignores empty and relative values.
std::string cache = getEnv( "XDG_CACHE_HOME" );
if ( !cache.empty() && cache[0] == '/' )
return under( std::move( cache ), "", appname );
std::string home = getEnv( "HOME" );
if ( !home.empty() )
return under( std::move( home ), ".cache/", appname );
#elif EE_PLATFORM == EE_PLATFORM_ANDROID
return under( Window::Engine::instance()->getPlatformHelper()->getInternalStoragePath(),
"cache/", appname );
#endif
return getTempPath() + appname;
}
std::string Sys::getTempPath() {
char path[EE_MAX_CFG_PATH_LEN + 1];
@@ -140,6 +140,7 @@ class X11DragSource : public DragSource {
!( mask & Button1Mask ) )
return { "A left-button pointer gesture is required", DragStartStatus::InvalidGesture };
preview = NativeDragAccess::createNativeDragPreview( *owner, offer, true );
sourceHandlesDrops = offer.sourceHandlesDrops;
if ( !preview )
return { "Could not create the drag preview", DragStartStatus::Failed };
XErrorScope setup( x11, display );
@@ -481,9 +482,10 @@ class X11DragSource : public DragSource {
accepted = false;
waitingStatus = false;
uriRequested = false;
// Over the source window the application resolves the drop itself.
preview->setFeedback( target || overOrigin ? DragFeedback::Pending
: DragFeedback::Rejected );
// Over the source window, an application that handles its own drops resolves it.
preview->setFeedback( target || ( overOrigin && sourceHandlesDrops )
? DragFeedback::Pending
: DragFeedback::Rejected );
if ( target )
send( atoms[Enter], static_cast<long>( version ) << 24, atoms[UriList] );
}
@@ -728,6 +730,8 @@ class X11DragSource : public DragSource {
bool pointerGrabbed{ false };
/** The pointer is over the source window (never a target), as of the last position(). */
bool overOrigin{ false };
/** DragOffer::sourceHandlesDrops of the session. */
bool sourceHandlesDrops{ false };
bool dirty{ true };
bool released{ false };
bool waitingStatus{ false };
@@ -146,9 +146,11 @@ struct SSHResolvedConfig {
/** Non-fatal problems (unknown directives, failed expansions, unreadable includes). */
std::vector<std::string> warnings;
/** @return A key identifying meaningfully different SSH sessions (host, port, user, proxy
* chain, authentication profile, host key alias). Never contains a filesystem path of the
* remote side. */
/** @return A key identifying meaningfully different SSH sessions: destination (host, port,
* user, proxy chain, host key alias) and the security policy that verified and
* authenticated it (known hosts, host key checking, algorithms, authentication settings).
* Sessions are only shared between configurations with equal keys. Never contains a
* filesystem path of the remote side. */
std::string connectionKey() const;
};
@@ -229,11 +229,22 @@ class VFSDirectoryModel : public Model, public std::enable_shared_from_this<VFSD
void rebuildRows();
/** Updates mRows for an entry inserted at entryIndex (already in mEntries). */
void rowsInserted( size_t entryIndex, bool visible );
/** Updates mRows for the entry erased from entryIndex. */
void rowsErased( size_t entryIndex );
/** Sorts entries by the current order (stable). */
void sortEntries( std::vector<FileInfo>& entries ) const;
void setLoadState( LoadState state, std::optional<VFSError> error = std::nullopt );
void notifyChanged();
bool less( const FileInfo& a, const FileInfo& b ) const;
/** The sort order. aType and bType are the entries' types when already known. */
bool less( const FileInfo& a, const FileInfo& b, const std::string* aType = nullptr,
const std::string* bType = nullptr ) const;
bool isVisible( const FileInfo& info ) const;
@@ -282,6 +293,8 @@ class VFSDirectoryModel : public Model, public std::enable_shared_from_this<VFSD
bool mCountRefreshPending{ false };
std::string mFilterPattern;
std::string mFilterGlob;
/** Case-insensitive plain text filter: the pattern's search table. */
String::BMH::OccTable mFilterOccurrences;
std::unique_ptr<System::RegEx> mFilterRegEx;
NameFilterMode mFilterMode{ NameFilterMode::Glob };
bool mFilterCaseSensitive{ false };
@@ -27,15 +27,19 @@ using EE::System::ThreadPool;
* - Tasks resolve on VFS or provider threads; use thenOn() with a UI executor before touching UI
* or model state.
* - Multi-step operations run on a dedicated orchestration pool that waits on provider tasks.
* Provider work never runs on that pool, so waits there cannot starve provider I/O.
* Provider work never runs on that pool, so waits there cannot starve provider I/O. Copies and
* moves run on a pool of their own: long transfers never delay the other operations.
*/
class VirtualFileSystem {
public:
struct Config {
/** Worker threads for blocking local I/O (LocalProvider and similar). */
Uint32 ioThreads{ 4 };
/** Worker threads for multi-step orchestration (transfers, recursive operations). */
/** Worker threads for multi-step orchestration (whole-file reads and writes, recursive
* operations). */
Uint32 orchestrationThreads{ 2 };
/** Copies and moves running at the same time; more wait for one to finish. */
Uint32 transferThreads{ 2 };
/** Register file:// and res:// (and sftp:// when built with SFTP support). */
bool registerDefaultProviders{ true };
};
@@ -154,6 +158,7 @@ class VirtualFileSystem {
std::unordered_map<std::string, std::shared_ptr<VFSProvider>> mProviders;
std::shared_ptr<ThreadPool> mIOPool;
std::shared_ptr<ThreadPool> mOrchestrationPool;
std::shared_ptr<ThreadPool> mTransferPool;
std::unique_ptr<WatchManager> mWatchManager;
};
@@ -1,6 +1,7 @@
#ifndef EE_VFS_WATCHERS_LOCALWATCHBACKEND_HPP
#define EE_VFS_WATCHERS_LOCALWATCHBACKEND_HPP
#include <eepp/system/threadpool.hpp>
#include <eepp/vfs/watch.hpp>
#include <memory>
@@ -18,11 +19,16 @@ class LocalWatchBackendImpl;
* (efsw handleMissedFileActions) become RescanRequired.
*
* Several logical watches on the same directory share one efsw watch.
*
* Registration and removal touch the filesystem (the generic watcher takes a snapshot of the
* tree), so they run on the I/O pool, never on the caller's thread.
*/
class LocalWatchBackend : public WatchBackend {
public:
/** @param genericPollingInterval Polling interval for the generic efsw watcher. */
explicit LocalWatchBackend( Time genericPollingInterval = System::Seconds( 1 ) );
/** @param pool Where watches are registered and removed.
* @param genericPollingInterval Polling interval for the generic efsw watcher. */
explicit LocalWatchBackend( std::shared_ptr<System::ThreadPool> pool,
Time genericPollingInterval = System::Seconds( 1 ) );
~LocalWatchBackend() override;
@@ -30,6 +36,7 @@ class LocalWatchBackend : public WatchBackend {
std::shared_ptr<WatchSink> sink ) override;
private:
std::shared_ptr<System::ThreadPool> mPool;
std::shared_ptr<detail::LocalWatchBackendImpl> mImpl;
};
@@ -58,10 +58,7 @@ class AgentClient : public SSHChannelPump, public std::enable_shared_from_this<A
/** Thread-safe: queue a frame for the agent. */
void send( std::vector<uint8_t> frame ) {
auto self = shared_from_this();
auto context = ioContext();
if ( !context )
return;
context->post( [self, frame = std::move( frame )]() mutable {
mConnection->ioContext()->post( [self, frame = std::move( frame )]() mutable {
if ( self->mState.load() == State::Dead )
return;
self->mOut.insert( self->mOut.end(), frame.begin(), frame.end() );
@@ -72,9 +69,14 @@ class AgentClient : public SSHChannelPump, public std::enable_shared_from_this<A
void connectionLost( const VFSError& error ) override { die( error.message ); }
private:
std::shared_ptr<SSHIOContext> ioContext() const;
/** Thread-safe: gives up on the agent (as if its channel closed). */
void stop( std::string reason ) {
auto self = shared_from_this();
mConnection->ioContext()->post(
[self, reason = std::move( reason )] { self->die( reason ); } );
}
private:
void die( const std::string& reason );
void handle( const agent::Frame& frame );
@@ -148,6 +150,10 @@ class RemoteWatchBackendImpl : public std::enable_shared_from_this<RemoteWatchBa
static constexpr Uint32 MaxReconnectDelaySeconds = 30;
/** How long an agent may take to confirm a watch (its handshake included) before the watch
* falls back to polling: the directory listing waits for the watch. */
static constexpr auto EstablishTimeout = std::chrono::seconds( 5 );
RemoteWatchBackendImpl( std::shared_ptr<SFTPProviderImpl> provider, VirtualFileSystem& vfs ) :
mProvider( provider ),
mPolling( std::make_shared<PollingWatchBackend>( vfs ) ),
@@ -157,11 +163,17 @@ class RemoteWatchBackendImpl : public std::enable_shared_from_this<RemoteWatchBa
void shutdown() {
std::map<Uint32, std::shared_ptr<RemoteWatch>> watches;
std::map<std::string, AgentEntry> agents;
{
std::lock_guard<std::mutex> lock( mMutex );
watches.swap( mWatches );
mAgents.clear();
agents.swap( mAgents );
for ( auto& watch : watches )
watch.second->mode = RemoteWatch::Mode::Closed;
}
for ( auto& entry : agents )
if ( entry.second.agent )
entry.second.agent->stop( "Watch backend closed" );
for ( auto& watch : watches )
watch.second->polling.reset();
}
@@ -210,6 +222,7 @@ class RemoteWatchBackendImpl : public std::enable_shared_from_this<RemoteWatchBa
w.u32( id );
watch->agent->send( w.frame( agent::MessageUnwatch ) );
}
stopUnusedAgent( watch->agent );
watch->polling.reset();
}
@@ -350,11 +363,6 @@ class RemoteWatchBackendImpl : public std::enable_shared_from_this<RemoteWatchBa
sink->rescanRequired();
}
std::shared_ptr<SSHIOContext> ioContext() const {
auto provider = mProvider.lock();
return provider ? provider->ioContext() : nullptr;
}
private:
struct AgentEntry {
std::shared_ptr<AgentClient> agent;
@@ -363,6 +371,22 @@ class RemoteWatchBackendImpl : public std::enable_shared_from_this<RemoteWatchBa
Uint32 failures{ 0 };
};
void stopUnusedAgent( const std::shared_ptr<AgentClient>& client ) {
if ( !client )
return;
{
std::lock_guard<std::mutex> lock( mMutex );
for ( const auto& watch : mWatches )
if ( watch.second->agent == client )
return;
auto it = mAgents.find( client->key() );
if ( it != mAgents.end() && it->second.agent == client )
it->second.agent.reset();
}
// An unused pump otherwise counts as an active user forever, preventing idle cleanup.
client->stop( "No watches remain" );
}
void completeEstablishment( RemoteWatch& watch, VFSResult<void>&& result ) {
std::optional<TaskSource<VFSResult<void>>> source;
{
@@ -401,13 +425,15 @@ class RemoteWatchBackendImpl : public std::enable_shared_from_this<RemoteWatchBa
"efsw-agent is not available and polling is disabled" ) ) );
return;
}
std::shared_ptr<AgentClient> previousAgent;
{
std::lock_guard<std::mutex> lock( mMutex );
if ( watch->mode == RemoteWatch::Mode::Closed )
return;
watch->mode = RemoteWatch::Mode::Polling;
watch->agent.reset();
previousAgent = std::move( watch->agent );
}
stopUnusedAgent( previousAgent );
std::weak_ptr<RemoteWatchBackendImpl> weak = weak_from_this();
std::weak_ptr<RemoteWatch> weakWatch = watch;
mPolling->watch( watch->root, watch->options, sink )
@@ -539,6 +565,35 @@ class RemoteWatchBackendImpl : public std::enable_shared_from_this<RemoteWatchBa
else if ( ready )
sendWatch( client, *watch );
// Otherwise agentReady() sends it once the handshake completes.
scheduleEstablishTimeout( watch, client );
}
/** An agent that never answers (stalled, or a command that is not the agent) must not keep
* the watch, and the listing waiting on it, pending forever. */
void scheduleEstablishTimeout( const std::shared_ptr<RemoteWatch>& watch,
const std::shared_ptr<AgentClient>& client ) {
std::weak_ptr<RemoteWatchBackendImpl> weak = weak_from_this();
std::weak_ptr<RemoteWatch> weakWatch = watch;
std::weak_ptr<AgentClient> weakClient = client;
mRunner.runAfter( EstablishTimeout, [weak, weakWatch, weakClient] {
auto self = weak.lock();
auto watch = weakWatch.lock();
auto client = weakClient.lock();
if ( !self || !watch || !client )
return;
{
std::lock_guard<std::mutex> lock( self->mMutex );
if ( watch->mode == RemoteWatch::Mode::Closed ||
watch->mode == RemoteWatch::Mode::Agent || watch->agent != client )
return;
}
if ( client->state() != AgentClient::State::Ready ) {
// No handshake: the agent is marked unavailable and its watches poll.
client->stop( "efsw-agent did not answer" );
return;
}
self->usePolling( watch );
} );
}
std::weak_ptr<SFTPProviderImpl> mProvider;
@@ -555,11 +610,6 @@ class RemoteWatchBackendImpl : public std::enable_shared_from_this<RemoteWatchBa
// Agent client
// ---------------------------------------------------------------------------------------------
std::shared_ptr<SSHIOContext> AgentClient::ioContext() const {
auto backend = mBackend.lock();
return backend ? backend->ioContext() : nullptr;
}
void AgentClient::start( const std::string& command ) {
auto self = shared_from_this();
mConnection
+187 -116
View File
@@ -506,76 +506,112 @@ Time SFTPProviderImpl::connectTimeout( const SSHResolvedConfig& config ) const {
return config.connectTimeout ? *config.connectTimeout : mConfig.defaultConnectTimeout;
}
/** Builds (blocking, on a helper thread) a ready connection for a resolved configuration,
* including its ProxyJump chain or ProxyCommand transport. */
VFSResult<std::shared_ptr<SSHConnection>>
SFTPProviderImpl::buildConnection( const SSHResolvedConfig& config ) {
std::unique_ptr<SSHTransport> transport;
if ( !config.proxyJump.empty() ) {
// Each jump host uses its own ssh_config (like `ssh -J`); the chain is the target's.
std::shared_ptr<SSHConnection> parent;
for ( size_t i = 0; i <= config.proxyJump.size(); ++i ) {
bool last = i == config.proxyJump.size();
SSHResolvedConfig hopConfig;
if ( last ) {
hopConfig = config;
} else {
const auto& jump = config.proxyJump[i];
auto resolved = resolveSync(
jump.host, jump.user.empty() ? std::nullopt : std::optional( jump.user ),
jump.port ? std::optional<Uint16>( jump.port ) : std::nullopt );
if ( !resolved )
return Err( std::move( resolved ).error() );
hopConfig = std::move( resolved ).value();
hopConfig.proxyJump.clear();
hopConfig.proxyCommand.reset();
}
std::unique_ptr<SSHTransport> hopTransport;
if ( !parent ) {
std::string error;
SocketHandle socket = connectTcp(
hopConfig.hostName, hopConfig.port,
static_cast<int>( connectTimeout( hopConfig ).asMilliseconds() ), error );
if ( socket == InvalidSocket )
return Err( VFSError( VFSErrorCode::ConnectionLost, URI(), error ) );
hopTransport = std::make_unique<SocketTransport>( socket );
} else {
auto channelTransport = std::make_unique<ChannelTransport>(
parent, hopConfig.hostName, hopConfig.port );
ChannelTransport* raw = channelTransport.get();
// Opening the channel is an operation on the parent: it is serialized with the
// parent's own SFTP traffic.
auto opened = await( parent->submit<void>(
{}, URI(), [raw]( SSHConnection& ) -> std::optional<VFSResult<void>> {
VFSError error;
if ( raw->open( error ) == Step::Again )
return std::nullopt;
if ( !raw->channel() )
return VFSResult<void>( Err( std::move( error ) ) );
return VFSResult<void>( Ok() );
} ) );
if ( !opened )
return Err( std::move( opened ).error() );
hopTransport = std::move( channelTransport );
}
auto connection = std::make_shared<SSHConnection>( mContext, hopConfig, hooks(),
std::move( hopTransport ),
connectTimeout( hopConfig ) );
mContext->post(
[context = mContext, connection] { context->addConnection( connection ); } );
auto ready = await( connection->whenReady() );
/** A connection being built, one hop of its ProxyJump chain at a time. */
struct SFTPProviderImpl::Build {
SSHResolvedConfig config;
/** The previous hop, which carries the next one's channel. */
std::shared_ptr<SSHConnection> parent;
size_t hop{ 0 };
TaskSource<VFSResult<std::shared_ptr<SSHConnection>>> source;
};
VFSTask<std::shared_ptr<SSHConnection>>
SFTPProviderImpl::buildConnection( SSHResolvedConfig config ) {
auto build = std::make_shared<Build>();
build->config = std::move( config );
auto task = build->source.task();
buildHop( std::move( build ) );
return task;
}
void SFTPProviderImpl::buildHop( std::shared_ptr<Build> build ) {
// Transports are set up on a helper thread (TCP connect blocks, bounded by ConnectTimeout).
// Waiting for the session is not: the handshake and authentication may wait on the user
// (host key, password), so it continues from the connection's ready task instead.
auto pool = mHelperPool;
if ( !pool ) {
build->source.resolve(
Err( VFSError( VFSErrorCode::ConnectionLost, URI(), "SFTP provider shutting down" ) ) );
return;
}
std::weak_ptr<SFTPProviderImpl> weak = weak_from_this();
pool->run( [weak, build = std::move( build )]() mutable {
HelperThreadScope scope;
auto self = weak.lock();
if ( !self )
return; // The source is abandoned with the build.
auto connection = self->startHop( *build );
if ( !connection ) {
build->source.resolve( Err( std::move( connection ).error() ) );
return;
}
bool last = build->hop == build->config.proxyJump.size();
std::string hopName = connection.value()->config().hostName;
connection.value()->whenReady().then( [weak, build, connection = connection.value(), last,
hopName]( VFSResult<void>&& ready ) mutable {
auto self = weak.lock();
if ( !self )
return;
if ( !ready ) {
VFSError error = std::move( ready ).error();
if ( !last )
error.message = "ProxyJump " + hopConfig.hostName + ": " + error.message;
return Err( std::move( error ) );
error.message = "ProxyJump " + hopName + ": " + error.message;
build->source.resolve( Err( std::move( error ) ) );
return;
}
parent = std::move( connection );
}
return Ok( std::move( parent ) );
if ( !last ) {
build->parent = std::move( connection );
++build->hop;
self->buildHop( std::move( build ) );
return;
}
self->probeExtensions( connection )
.then( [build, connection]( VFSResult<void>&& ) mutable {
build->source.resolve( Ok( std::move( connection ) ) );
} );
} );
} );
}
VFSResult<std::shared_ptr<SSHConnection>> SFTPProviderImpl::startHop( Build& build ) {
const SSHResolvedConfig& config = build.config;
bool last = build.hop == config.proxyJump.size();
SSHResolvedConfig hopConfig;
if ( last ) {
hopConfig = config;
} else {
// Each jump host uses its own ssh_config (like `ssh -J`); the chain is the target's.
const auto& jump = config.proxyJump[build.hop];
auto resolved =
resolveSync( jump.host, jump.user.empty() ? std::nullopt : std::optional( jump.user ),
jump.port ? std::optional<Uint16>( jump.port ) : std::nullopt );
if ( !resolved )
return Err( std::move( resolved ).error() );
hopConfig = std::move( resolved ).value();
hopConfig.proxyJump.clear();
hopConfig.proxyCommand.reset();
}
if ( config.proxyCommand ) {
std::unique_ptr<SSHTransport> transport;
if ( build.parent ) {
auto channelTransport =
std::make_unique<ChannelTransport>( build.parent, hopConfig.hostName, hopConfig.port );
ChannelTransport* raw = channelTransport.get();
// Opening the channel is an operation on the parent: it is serialized with the parent's
// own traffic. Bounded (no user interaction), so waiting here is fine.
auto opened = await( build.parent->submit<void>(
{}, URI(), [raw]( SSHConnection& ) -> std::optional<VFSResult<void>> {
VFSError error;
if ( raw->open( error ) == Step::Again )
return std::nullopt;
if ( !raw->channel() )
return VFSResult<void>( Err( std::move( error ) ) );
return VFSResult<void>( Ok() );
} ) );
if ( !opened )
return Err( std::move( opened ).error() );
transport = std::move( channelTransport );
} else if ( last && config.proxyCommand ) {
auto process = std::make_unique<ProcessTransport>( mContext, *config.proxyCommand );
if ( !process->started() )
return Err( VFSError( VFSErrorCode::ConnectionLost, URI(),
@@ -584,26 +620,28 @@ SFTPProviderImpl::buildConnection( const SSHResolvedConfig& config ) {
} else {
std::string error;
SocketHandle socket =
connectTcp( config.hostName, config.port,
static_cast<int>( connectTimeout( config ).asMilliseconds() ), error );
connectTcp( hopConfig.hostName, hopConfig.port,
static_cast<int>( connectTimeout( hopConfig ).asMilliseconds() ), error );
if ( socket == InvalidSocket )
return Err( VFSError( VFSErrorCode::ConnectionLost, URI(), error ) );
transport = std::make_unique<SocketTransport>( socket );
}
auto connection = std::make_shared<SSHConnection>(
mContext, config, hooks(), std::move( transport ), connectTimeout( config ) );
Time timeout = connectTimeout( hopConfig );
auto connection = std::make_shared<SSHConnection>( mContext, std::move( hopConfig ), hooks(),
std::move( transport ), timeout );
connection->setIdleTimeout( mConfig.idleTimeout );
connection->setSFTPRequired( last );
mContext->post( [context = mContext, connection] { context->addConnection( connection ); } );
auto ready = await( connection->whenReady() );
if ( !ready )
return Err( std::move( ready ).error() );
return Ok( std::move( connection ) );
}
/** Probes SFTP extensions once per connection: posix-rename (atomic replace), statvfs. fsync is
* detected on first use. */
void SFTPProviderImpl::probeExtensions( const std::shared_ptr<SSHConnection>& connection ) {
VFSTask<void>
SFTPProviderImpl::probeExtensions( const std::shared_ptr<SSHConnection>& connection ) {
static const std::string probe = "/.eepp-vfs-extension-probe-does-not-exist";
await( connection->submit<void>(
return connection->submit<void>(
{}, URI(), []( SSHConnection& c ) -> std::optional<VFSResult<void>> {
if ( c.posixRenameSupport < 0 ) {
int rc = libssh2_sftp_posix_rename_ex(
@@ -628,19 +666,19 @@ void SFTPProviderImpl::probeExtensions( const std::shared_ptr<SSHConnection>& co
if ( c.posixRenameSupport == 1 && c.fsyncSupport < 0 )
c.fsyncSupport = 1; // Both arrived in OpenSSH 6.5; confirmed on first use.
return VFSResult<void>( Ok() );
} ) );
} );
}
void SFTPProviderImpl::pruneIdle() {
auto now = std::chrono::steady_clock::now();
auto idle =
std::chrono::milliseconds( static_cast<Int64>( mConfig.idleTimeout.asMilliseconds() ) );
// Idle connections close themselves on the I/O thread (SSHConnection::setIdleTimeout());
// here the pool forgets the closed ones, and closes those whose timer has not fired yet.
for ( auto it = mPool.begin(); it != mPool.end(); ) {
auto& entry = it->second;
if ( entry.connection &&
( !entry.connection->isUsable() || ( entry.connection.use_count() == 1 &&
now - entry.connection->lastUse() > idle ) ) ) {
entry.connection->close();
auto connection = entry.connection.lock();
if ( connection && ( !connection->isUsable() || connection->idleExpired() ) ) {
connection->close();
it = mPool.erase( it );
} else if ( !connection && entry.waiters.empty() ) {
it = mPool.erase( it );
} else {
++it;
@@ -651,18 +689,21 @@ void SFTPProviderImpl::pruneIdle() {
VFSTask<std::shared_ptr<SSHConnection>> SFTPProviderImpl::connection( const URI& uri ) {
using Result = VFSResult<std::shared_ptr<SSHConnection>>;
std::string key = authorityKey( uri );
Uint64 generation;
TaskSource<Result> source;
auto task = source.task();
{
std::lock_guard<std::mutex> lock( mPoolMutex );
generation = mPoolGeneration;
pruneIdle();
auto it = mPool.find( key );
if ( it != mPool.end() ) {
if ( it->second.connection && it->second.connection->isUsable() ) {
source.resolve( Ok( it->second.connection ) );
auto connection = it->second.connection.lock();
if ( connection && connection->isUsable() ) {
source.resolve( Ok( std::move( connection ) ) );
return task;
}
if ( !it->second.connection ) {
if ( !connection && !it->second.waiters.empty() ) {
it->second.waiters.push_back( std::move( source ) );
return task;
}
@@ -675,7 +716,7 @@ VFSTask<std::shared_ptr<SSHConnection>> SFTPProviderImpl::connection( const URI&
return readyError<std::shared_ptr<SSHConnection>>( VFSErrorCode::ConnectionLost, uri,
"SFTP provider shutting down" );
std::weak_ptr<SFTPProviderImpl> weak = weak_from_this();
pool->run( [weak, uri, key]() {
pool->run( [weak, uri, key, generation]() {
HelperThreadScope scope;
auto self = weak.lock();
if ( !self )
@@ -691,10 +732,13 @@ VFSTask<std::shared_ptr<SSHConnection>> SFTPProviderImpl::connection( const URI&
std::shared_ptr<SSHConnection> existing;
{
std::lock_guard<std::mutex> lock( self->mPoolMutex );
if ( self->mPoolGeneration != generation )
return;
for ( auto& entry : self->mPool ) {
if ( entry.second.connection && entry.second.connection->isUsable() &&
entry.second.connection->config().connectionKey() == connectionKey ) {
existing = entry.second.connection;
auto connection = entry.second.connection.lock();
if ( connection && connection->isUsable() &&
connection->config().connectionKey() == connectionKey ) {
existing = std::move( connection );
break;
}
}
@@ -702,37 +746,51 @@ VFSTask<std::shared_ptr<SSHConnection>> SFTPProviderImpl::connection( const URI&
if ( existing ) {
result = Ok( existing );
} else {
result = self->buildConnection( config.value() );
if ( result )
self->probeExtensions( result.value() );
}
if ( !result && result.error().uri.empty() )
result.error().uri = uri;
}
std::vector<TaskSource<Result>> waiters;
{
std::lock_guard<std::mutex> lock( self->mPoolMutex );
auto it = self->mPool.find( key );
if ( it != self->mPool.end() ) {
waiters = std::move( it->second.waiters );
if ( result )
it->second.connection = result.value();
else
self->mPool.erase( it );
self->buildConnection( std::move( config ).value() )
.then( [weak, uri, key, generation]( Result&& built ) {
if ( auto self = weak.lock() )
self->finishConnection( key, uri, generation, std::move( built ) );
} );
return;
}
}
for ( auto& waiter : waiters )
waiter.resolve( Result( result ) );
self->finishConnection( key, uri, generation, std::move( result ) );
} );
return task;
}
void SFTPProviderImpl::finishConnection( const std::string& key, const URI& uri, Uint64 generation,
VFSResult<std::shared_ptr<SSHConnection>>&& result ) {
if ( !result && result.error().uri.empty() )
result.error().uri = uri;
std::vector<TaskSource<VFSResult<std::shared_ptr<SSHConnection>>>> waiters;
{
std::lock_guard<std::mutex> lock( mPoolMutex );
if ( generation != mPoolGeneration ) {
if ( result )
result.value()->close();
return;
}
auto it = mPool.find( key );
if ( it != mPool.end() ) {
waiters = std::move( it->second.waiters );
if ( result )
it->second.connection = result.value();
else
mPool.erase( it );
}
}
for ( auto& waiter : waiters )
waiter.resolve( VFSResult<std::shared_ptr<SSHConnection>>( result ) );
}
ProviderCapabilities SFTPProviderImpl::capabilities( const URI& uri ) const {
ProviderCapabilities caps( BaseCapabilities );
std::lock_guard<std::mutex> lock( mPoolMutex );
auto it = mPool.find( authorityKey( uri ) );
if ( it != mPool.end() && it->second.connection ) {
const auto& c = *it->second.connection;
auto connection = it != mPool.end() ? it->second.connection.lock() : nullptr;
if ( connection ) {
const auto& c = *connection;
caps.set( VFSCapability::AtomicRename, c.posixRenameSupport == 1 );
caps.set( VFSCapability::AtomicReplace, c.posixRenameSupport == 1 );
caps.set( VFSCapability::Fsync, c.fsyncSupport == 1 );
@@ -742,19 +800,32 @@ ProviderCapabilities SFTPProviderImpl::capabilities( const URI& uri ) const {
}
void SFTPProviderImpl::disconnectAll() {
std::lock_guard<std::mutex> lock( mPoolMutex );
for ( auto& entry : mPool )
if ( entry.second.connection )
entry.second.connection->close();
mPool.clear();
std::map<std::string, PoolEntry> removed;
{
std::lock_guard<std::mutex> lock( mPoolMutex );
++mPoolGeneration;
removed.swap( mPool );
}
// Completing waiters can run arbitrary continuations: never under the pool lock. Include
// pending connections, whose authentication may be waiting on UI that is shutting down.
mContext->closeConnections();
for ( auto& entry : removed ) {
if ( auto connection = entry.second.connection.lock() )
connection->close();
for ( auto& waiter : entry.second.waiters )
waiter.resolve(
Err( VFSError( VFSErrorCode::ConnectionLost, URI(), "Connection closed" ) ) );
}
}
size_t SFTPProviderImpl::connectionCount() const {
std::lock_guard<std::mutex> lock( mPoolMutex );
size_t count = 0;
for ( const auto& entry : mPool )
if ( entry.second.connection && entry.second.connection->isUsable() )
for ( const auto& entry : mPool ) {
if ( auto connection = entry.second.connection.lock();
connection && connection->isUsable() )
++count;
}
return count;
}
@@ -44,7 +44,9 @@ class SFTPProviderImpl : public std::enable_shared_from_this<SFTPProviderImpl> {
private:
struct PoolEntry {
std::shared_ptr<SSHConnection> connection;
/** The I/O context owns live sessions; the pool must not retain their resources after
* the context closes them. An expired entry with waiters is still being built. */
std::weak_ptr<SSHConnection> connection;
std::vector<TaskSource<VFSResult<std::shared_ptr<SSHConnection>>>> waiters;
};
@@ -52,9 +54,23 @@ class SFTPProviderImpl : public std::enable_shared_from_this<SFTPProviderImpl> {
std::optional<std::string> user,
std::optional<Uint16> port ) const;
VFSResult<std::shared_ptr<SSHConnection>> buildConnection( const SSHResolvedConfig& config );
struct Build;
void probeExtensions( const std::shared_ptr<SSHConnection>& connection );
/** Connects to config through its ProxyJump chain or ProxyCommand. The task resolves on the
* I/O thread once the session is ready (authenticated, SFTP started). */
VFSTask<std::shared_ptr<SSHConnection>> buildConnection( SSHResolvedConfig config );
/** Sets up the next hop of build on a helper thread and continues when it is ready. */
void buildHop( std::shared_ptr<Build> build );
/** Creates the transport and connection of build's current hop (helper thread). */
VFSResult<std::shared_ptr<SSHConnection>> startHop( Build& build );
VFSTask<void> probeExtensions( const std::shared_ptr<SSHConnection>& connection );
/** Hands the connection (or the failure) to every request waiting on the pool key. */
void finishConnection( const std::string& key, const URI& uri, Uint64 generation,
VFSResult<std::shared_ptr<SSHConnection>>&& result );
SSHConnectionHooks hooks() const;
@@ -69,6 +85,8 @@ class SFTPProviderImpl : public std::enable_shared_from_this<SFTPProviderImpl> {
std::shared_ptr<System::ThreadPool> mHelperPool;
mutable std::mutex mPoolMutex;
std::map<std::string, PoolEntry> mPool;
/** Invalidates connection builds still in flight when disconnectAll() is called. */
Uint64 mPoolGeneration{ 0 };
};
}}} // namespace EE::VFS::sftp
@@ -0,0 +1,275 @@
#include "sshagent.hpp"
#if EE_PLATFORM != EE_PLATFORM_WIN
#include <cerrno>
#include <cstring>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#endif
namespace EE { namespace VFS { namespace sftp {
namespace {
// draft-miller-ssh-agent message numbers and flags.
constexpr Uint8 AgentFailure = 5;
constexpr Uint8 RequestIdentities = 11;
constexpr Uint8 IdentitiesAnswer = 12;
constexpr Uint8 SignRequest = 13;
constexpr Uint8 SignResponse = 14;
constexpr Uint32 RSASHA256 = 2;
constexpr Uint32 RSASHA512 = 4;
/** Agent replies larger than this are refused (OpenSSH uses 256 KiB). */
constexpr Uint32 MaxMessage = 256 * 1024;
void putU32( std::string& out, Uint32 value ) {
out += static_cast<char>( ( value >> 24 ) & 0xFF );
out += static_cast<char>( ( value >> 16 ) & 0xFF );
out += static_cast<char>( ( value >> 8 ) & 0xFF );
out += static_cast<char>( value & 0xFF );
}
void putString( std::string& out, std::string_view value ) {
putU32( out, static_cast<Uint32>( value.size() ) );
out.append( value );
}
/** Sequential reader of SSH wire data; any overrun makes it invalid. */
class WireReader {
public:
explicit WireReader( std::string_view data ) : mData( data ) {}
bool valid() const { return mValid; }
Uint8 u8() {
if ( !need( 1 ) )
return 0;
return static_cast<Uint8>( mData[mPosition++] );
}
Uint32 u32() {
if ( !need( 4 ) )
return 0;
Uint32 value = 0;
for ( int i = 0; i < 4; ++i )
value = ( value << 8 ) | static_cast<Uint8>( mData[mPosition++] );
return value;
}
std::string_view string() {
Uint32 length = u32();
if ( !need( length ) )
return {};
std::string_view value = mData.substr( mPosition, length );
mPosition += length;
return value;
}
private:
bool need( size_t bytes ) {
if ( !mValid || mData.size() - mPosition < bytes )
mValid = false;
return mValid;
}
std::string_view mData;
size_t mPosition{ 0 };
bool mValid{ true };
};
VFSError agentError( std::string message ) {
return VFSError( VFSErrorCode::AuthenticationFailed, URI(), "ssh-agent: " + message );
}
} // namespace
bool SSHAgent::supported() {
#if EE_PLATFORM != EE_PLATFORM_WIN
return true;
#else
return false;
#endif
}
SSHAgent::SSHAgent( std::string socketPath ) : mPath( std::move( socketPath ) ) {}
SSHAgent::~SSHAgent() {
disconnect();
}
void SSHAgent::disconnect() {
std::lock_guard<std::mutex> lock( mSocketMutex );
#if EE_PLATFORM != EE_PLATFORM_WIN
if ( mSocket >= 0 )
::close( mSocket );
#endif
mSocket = -1;
}
void SSHAgent::cancel() {
std::lock_guard<std::mutex> lock( mSocketMutex );
mCancelled.store( true, std::memory_order_release );
#if EE_PLATFORM != EE_PLATFORM_WIN
// Do not close here: the worker may still use this descriptor, and closing could let a
// different thread reuse its number. shutdown wakes its blocking receive; it closes it.
if ( mSocket >= 0 )
::shutdown( mSocket, SHUT_RDWR );
#endif
}
VFSResult<std::string> SSHAgent::transact( std::string_view request ) {
#if EE_PLATFORM != EE_PLATFORM_WIN
if ( mCancelled.load( std::memory_order_acquire ) )
return Err( agentError( "request cancelled" ) );
if ( mSocket < 0 ) {
sockaddr_un address{};
if ( mPath.empty() || mPath.size() >= sizeof( address.sun_path ) )
return Err( agentError( "invalid socket path \"" + mPath + "\"" ) );
address.sun_family = AF_UNIX;
std::memcpy( address.sun_path, mPath.c_str(), mPath.size() + 1 );
{
std::lock_guard<std::mutex> lock( mSocketMutex );
if ( mCancelled.load( std::memory_order_acquire ) )
return Err( agentError( "request cancelled" ) );
mSocket = ::socket( AF_UNIX, SOCK_STREAM, 0 );
}
#ifdef SO_NOSIGPIPE
// No MSG_NOSIGNAL on Apple platforms: an agent that went away must not kill the process.
int noSigPipe = 1;
if ( mSocket >= 0 )
::setsockopt( mSocket, SOL_SOCKET, SO_NOSIGPIPE, &noSigPipe, sizeof( noSigPipe ) );
#endif
if ( mSocket < 0 || ::connect( mSocket, reinterpret_cast<sockaddr*>( &address ),
sizeof( address ) ) != 0 ) {
std::string error( std::strerror( errno ) );
disconnect();
return Err( agentError( "cannot connect to " + mPath + ": " + error ) );
}
}
if ( mCancelled.load( std::memory_order_acquire ) ) {
disconnect();
return Err( agentError( "request cancelled" ) );
}
std::string message;
message.reserve( 4 + request.size() );
putString( message, request );
size_t sent = 0;
while ( sent < message.size() ) {
#ifdef MSG_NOSIGNAL
constexpr int flags = MSG_NOSIGNAL;
#else
constexpr int flags = 0;
#endif
ssize_t rc = ::send( mSocket, message.data() + sent, message.size() - sent, flags );
if ( rc < 0 && errno == EINTR )
continue;
if ( rc <= 0 ) {
disconnect();
return Err( agentError( "write failed" ) );
}
sent += static_cast<size_t>( rc );
}
auto receive = [this]( char* buffer, size_t size ) {
size_t received = 0;
while ( received < size ) {
ssize_t rc = ::recv( mSocket, buffer + received, size - received, 0 );
if ( rc < 0 && errno == EINTR )
continue;
if ( rc <= 0 )
return false;
received += static_cast<size_t>( rc );
}
return true;
};
char header[4];
if ( !receive( header, sizeof( header ) ) ) {
disconnect();
return Err( agentError( "read failed" ) );
}
Uint32 length = WireReader( std::string_view( header, sizeof( header ) ) ).u32();
if ( length == 0 || length > MaxMessage ) {
disconnect();
return Err( agentError( "invalid reply length" ) );
}
std::string reply( length, '\0' );
if ( !receive( reply.data(), reply.size() ) ) {
disconnect();
return Err( agentError( "read failed" ) );
}
return Ok( std::move( reply ) );
#else
(void)request;
return Err( agentError( "not supported on this platform" ) );
#endif
}
VFSResult<std::vector<SSHAgent::Identity>> SSHAgent::listIdentities() {
std::string request( 1, static_cast<char>( RequestIdentities ) );
auto reply = transact( request );
if ( !reply )
return Err( std::move( reply ).error() );
WireReader reader( reply.value() );
if ( reader.u8() != IdentitiesAnswer )
return Err( agentError( "the agent refused to list its keys" ) );
Uint32 count = reader.u32();
std::vector<Identity> identities;
for ( Uint32 i = 0; i < count && reader.valid(); ++i ) {
std::string_view blob = reader.string();
std::string_view comment = reader.string();
if ( reader.valid() )
identities.push_back( { std::string( blob ), std::string( comment ) } );
}
if ( !reader.valid() )
return Err( agentError( "malformed key list" ) );
return Ok( std::move( identities ) );
}
VFSResult<std::string> SSHAgent::sign( std::string_view blob, std::string_view data,
std::string_view algorithm ) {
Uint32 flags = algorithm == "rsa-sha2-512" ? RSASHA512
: algorithm == "rsa-sha2-256" ? RSASHA256
: 0;
std::string request( 1, static_cast<char>( SignRequest ) );
putString( request, blob );
putString( request, data );
putU32( request, flags );
auto reply = transact( request );
if ( !reply )
return Err( std::move( reply ).error() );
WireReader reader( reply.value() );
Uint8 type = reader.u8();
if ( type == AgentFailure )
return Err( agentError( "signing was refused" ) );
if ( type != SignResponse )
return Err( agentError( "unexpected reply to a signing request" ) );
WireReader signature( reader.string() );
std::string_view signedWith = signature.string();
std::string_view raw = signature.string();
if ( !reader.valid() || !signature.valid() )
return Err( agentError( "malformed signature" ) );
// The server verifies the algorithm announced in the request.
if ( !algorithm.empty() && signedWith != algorithm )
return Err( agentError( "signed with " + std::string( signedWith ) + " instead of " +
std::string( algorithm ) ) );
return Ok( std::string( raw ) );
}
std::string_view SSHAgent::algorithmOfSignedRequest( std::string_view data ) {
// string session identifier, byte SSH_MSG_USERAUTH_REQUEST, string user, string service,
// string "publickey", boolean TRUE, string algorithm, string public key.
WireReader reader( data );
reader.string();
reader.u8();
reader.string();
reader.string();
std::string_view method = reader.string();
reader.u8();
std::string_view algorithm = reader.string();
if ( !reader.valid() || method != "publickey" )
return {};
return algorithm;
}
}}} // namespace EE::VFS::sftp
@@ -0,0 +1,71 @@
#ifndef EE_VFS_SFTP_SSHAGENT_HPP
#define EE_VFS_SFTP_SSHAGENT_HPP
#include <atomic>
#include <eepp/config.hpp>
#include <eepp/vfs/error.hpp>
#include <mutex>
#include <string>
#include <string_view>
#include <vector>
namespace EE { namespace VFS { namespace sftp {
/** @brief Minimal ssh-agent client: lists the agent's keys and signs with them.
*
* Every call blocks on the agent, and signing may wait for the user (a hardware key touch, a
* confirmation dialog): use it from a worker thread, never from the SSH I/O thread. One call at a
* time. Speaks the agent protocol over Unix-domain sockets; elsewhere (Windows named pipes,
* Pageant) supported() is false and libssh2's own agent support is used.
*/
class SSHAgent {
public:
struct Identity {
/** Public key blob (SSH wire format). */
std::string blob;
std::string comment;
};
/** @return True where agents are reached through Unix-domain sockets, which this client
* speaks: every POSIX platform that builds SFTP (Linux, BSD, macOS, iOS). It does not mean an
* agent runs: without one, connecting fails and authentication goes on with identity files.
* Windows agents are named pipes or Pageant, left to libssh2. */
static bool supported();
explicit SSHAgent( std::string socketPath );
~SSHAgent();
/** Interrupts a pending request and prevents further requests. Safe from any thread. */
void cancel();
SSHAgent( const SSHAgent& ) = delete;
SSHAgent& operator=( const SSHAgent& ) = delete;
VFSResult<std::vector<Identity>> listIdentities();
/** Signs data with the key of blob for the signature algorithm (for example "rsa-sha2-512",
* which selects the agent's RSA SHA-2 flags). @return The raw signature, without its
* algorithm name (as libssh2 sign callbacks return it). */
VFSResult<std::string> sign( std::string_view blob, std::string_view data,
std::string_view algorithm );
/** @return The signature algorithm of a SSH_MSG_USERAUTH_REQUEST "publickey" signing payload
* (session identifier followed by the request), empty when it cannot be parsed. */
static std::string_view algorithmOfSignedRequest( std::string_view data );
private:
VFSResult<std::string> transact( std::string_view request );
void disconnect();
std::string mPath;
std::mutex mSocketMutex;
int mSocket{ -1 };
std::atomic<bool> mCancelled{ false };
};
}}} // namespace EE::VFS::sftp
#endif
+285 -88
View File
@@ -1,8 +1,10 @@
#include "sshsession.hpp"
#include <algorithm>
#include <cerrno>
#include <cstdlib>
#include <cstring>
#include <eepp/core/string.hpp>
#include <eepp/system/base64.hpp>
#include <eepp/system/filesystem.hpp>
#include <eepp/system/process.hpp>
#include <eepp/system/sys.hpp>
@@ -80,10 +82,20 @@ int knownHostKeyBits( int type ) {
}
}
/** Decodes base64 text (as in known_hosts and .pub files), nullopt when it is not valid. */
std::optional<std::string> decodeBase64( std::string_view text ) {
std::string decoded;
if ( System::Base64::decode( text, decoded ) == static_cast<size_t>( -1 ) )
return std::nullopt;
return decoded;
}
/** Loads known_hosts line by line: libssh2_knownhost_readfile() stops at the first entry it
* cannot parse (certificate authorities, security-key types...), which would hide every later
* host. */
void loadKnownHosts( LIBSSH2_KNOWNHOSTS* hosts, const std::string& path ) {
* host. libssh2 does not know markers: the key blobs of "@revoked" lines are added to revoked
* (when given), "@cert-authority" lines are ignored. */
void loadKnownHosts( LIBSSH2_KNOWNHOSTS* hosts, const std::string& path,
std::vector<std::string>* revoked ) {
std::string data;
if ( !FileSystem::fileExists( path ) || !FileSystem::fileGet( path, data ) )
return;
@@ -97,8 +109,18 @@ void loadKnownHosts( LIBSSH2_KNOWNHOSTS* hosts, const std::string& path ) {
while ( !line.empty() && ( line.back() == '\r' || line.back() == ' ' ) )
line.remove_suffix( 1 );
size_t first = line.find_first_not_of( " \t" );
if ( first == std::string_view::npos || line[first] == '#' || line[first] == '@' )
if ( first == std::string_view::npos || line[first] == '#' )
continue;
if ( line[first] == '@' ) {
// "@revoked hosts keytype base64key [comment]": the key is never accepted, for any
// host (stricter than matching the host patterns, never weaker).
auto fields = String::split( std::string( line.substr( first ) ), " \t" );
if ( revoked && fields.size() >= 4 && fields[0] == "@revoked" ) {
if ( auto blob = decodeBase64( fields[3] ) )
revoked->push_back( std::move( *blob ) );
}
continue;
}
// libssh2 inspects the byte after the given length (expecting an optional newline), so
// hand it a NUL-terminated copy rather than a view into the file buffer.
std::string entry( line );
@@ -120,29 +142,13 @@ bool isKeyEncrypted( const std::string& path ) {
if ( begin == std::string::npos )
return false;
size_t end = data.find( "-----END", begin + Begin.size() );
std::string base64;
for ( size_t i = begin + Begin.size(); i < end && i < data.size(); ++i )
if ( !std::isspace( static_cast<unsigned char>( data[i] ) ) )
base64 += data[i];
// Decode just the header: "openssh-key-v1\0" + uint32 length + cipher name.
static constexpr char Alphabet[] =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
std::string decoded;
Uint32 buffer = 0;
int bits = 0;
for ( char c : base64 ) {
if ( c == '=' || decoded.size() > 64 )
break;
const char* pos = std::strchr( Alphabet, c );
if ( !pos )
break;
buffer = ( buffer << 6 ) | static_cast<Uint32>( pos - Alphabet );
bits += 6;
if ( bits >= 8 ) {
bits -= 8;
decoded += static_cast<char>( ( buffer >> bits ) & 0xFF );
}
}
auto decodedText = decodeBase64( std::string_view( data ).substr(
begin + Begin.size(),
end == std::string::npos ? std::string::npos : end - begin - Begin.size() ) );
if ( !decodedText )
return false;
// Only the header matters: "openssh-key-v1\0" + uint32 length + cipher name.
const std::string& decoded = *decodedText;
static constexpr size_t Magic = 15; // "openssh-key-v1" + NUL
if ( decoded.size() < Magic + 4 )
return false;
@@ -161,25 +167,9 @@ std::optional<std::string> readPublicKeyBlob( const std::string& path ) {
auto fields = String::split( data, ' ' );
if ( fields.size() < 2 )
return std::nullopt;
static constexpr char Alphabet[] =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
std::string decoded;
Uint32 buffer = 0;
int bits = 0;
for ( char c : fields[1] ) {
if ( c == '=' || c == '\n' || c == '\r' )
break;
const char* pos = std::strchr( Alphabet, c );
if ( !pos )
return std::nullopt;
buffer = ( buffer << 6 ) | static_cast<Uint32>( pos - Alphabet );
bits += 6;
if ( bits >= 8 ) {
bits -= 8;
decoded += static_cast<char>( ( buffer >> bits ) & 0xFF );
}
}
return decoded;
String::trimInPlace( fields[1], '\n' );
String::trimInPlace( fields[1], '\r' );
return decodeBase64( fields[1] );
}
/** Host key algorithms of a known_hosts key type, in libssh2 method names. */
@@ -210,32 +200,17 @@ bool isSocketError( int rc ) {
rc == LIBSSH2_ERROR_KEX_FAILURE;
}
} // namespace
/** Base64 without padding, as used by OpenSSH fingerprints. */
std::string base64Unpadded( const unsigned char* data, size_t length ) {
static constexpr char Alphabet[] =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
std::string out;
size_t i = 0;
for ( ; i + 2 < length; i += 3 ) {
Uint32 v = ( data[i] << 16 ) | ( data[i + 1] << 8 ) | data[i + 2];
out += Alphabet[( v >> 18 ) & 63];
out += Alphabet[( v >> 12 ) & 63];
out += Alphabet[( v >> 6 ) & 63];
out += Alphabet[v & 63];
}
if ( i < length ) {
Uint32 v = data[i] << 16;
if ( i + 1 < length )
v |= data[i + 1] << 8;
out += Alphabet[( v >> 18 ) & 63];
out += Alphabet[( v >> 12 ) & 63];
if ( i + 1 < length )
out += Alphabet[( v >> 6 ) & 63];
}
System::Base64::encode( std::string_view( reinterpret_cast<const char*>( data ), length ),
out );
while ( !out.empty() && out.back() == '=' )
out.pop_back();
return out;
}
/** Maps an SFTP status code to a VFS error code. */
VFSErrorCode mapSFTPStatus( unsigned long status ) {
switch ( status ) {
case LIBSSH2_FX_OK:
@@ -274,6 +249,8 @@ VFSErrorCode mapSFTPStatus( unsigned long status ) {
}
}
} // namespace
// ---------------------------------------------------------------------------------------------
// I/O context
// ---------------------------------------------------------------------------------------------
@@ -315,6 +292,8 @@ void SSHIOContext::stop() {
void SSHIOContext::post( Command command ) {
{
std::lock_guard<std::mutex> lock( mMutex );
if ( mStopping.load( std::memory_order_acquire ) )
return;
mCommands.emplace_back( std::move( command ) );
}
mWake.signal();
@@ -324,6 +303,15 @@ void SSHIOContext::addConnection( std::shared_ptr<SSHConnection> connection ) {
mConnections.push_back( std::move( connection ) );
}
void SSHIOContext::closeConnections() {
post( [this] {
for ( auto& connection : mConnections ) {
connection->fail(
VFSError( VFSErrorCode::ConnectionLost, URI(), "Connection closed" ) );
}
} );
}
void SSHIOContext::loop() {
std::vector<Command> commands;
std::vector<PollFd> fds;
@@ -341,7 +329,7 @@ void SSHIOContext::loop() {
// Drop finished connections; their owners (pool, handles) may still reference them.
mConnections.erase( std::remove_if( mConnections.begin(), mConnections.end(),
[]( const std::shared_ptr<SSHConnection>& c ) {
return !c->isUsable() && c->mQueue.empty();
return !c->isUsable() && !c->hasQueuedOperations();
} ),
mConnections.end() );
@@ -449,20 +437,43 @@ void SSHConnection::markUsed() {
}
void SSHConnection::enqueue( std::unique_ptr<SSHOperation> operation ) {
std::optional<VFSError> failure;
{
std::lock_guard<std::mutex> lock( mQueueMutex );
mLastUse = std::chrono::steady_clock::now();
mQueue.push_back( std::move( operation ) );
// A failed or closed connection is dropped by the I/O context: nobody would run the
// operation.
if ( mQueueFailure ) {
failure = mQueueFailure;
} else {
mLastUse = std::chrono::steady_clock::now();
mQueue.push_back( std::move( operation ) );
}
}
if ( failure ) {
operation->fail( *failure );
return;
}
mContext->wake();
}
bool SSHConnection::hasQueuedOperations() const {
std::lock_guard<std::mutex> lock( mQueueMutex );
return !mQueue.empty();
}
void SSHConnection::fail( VFSError error ) {
State state = mState.load();
if ( state == State::Failed || state == State::Closed )
return;
mFailure = std::move( error );
mState = State::Failed;
if ( mAgentClient )
mAgentClient->cancel();
{
// From now on enqueue() fails operations itself; drive() fails the queued ones.
std::lock_guard<std::mutex> lock( mQueueMutex );
mQueueFailure = mFailure;
}
for ( auto& waiter : mReadyWaiters )
waiter.resolve( Err( mFailure ) );
mReadyWaiters.clear();
@@ -540,9 +551,20 @@ int SSHConnection::nextTimeoutMs() const {
consider( mDeadline );
if ( state == State::Ready && mConfig.serverAliveInterval )
consider( mNextKeepalive );
if ( state == State::Ready && mIdleTimeout != Time::Zero && isIdle() )
consider( lastUse() + idleTimeout() );
return timeout;
}
std::chrono::milliseconds SSHConnection::idleTimeout() const {
return std::chrono::milliseconds( static_cast<Int64>( mIdleTimeout.asMilliseconds() ) );
}
bool SSHConnection::idleExpired() const {
return mState.load() == State::Ready && mIdleTimeout != Time::Zero && isIdle() &&
std::chrono::steady_clock::now() - lastUse() > idleTimeout();
}
void SSHConnection::drive() {
if ( mTransport && !mTransport->alive() && isUsable() )
fail( VFSError( VFSErrorCode::ConnectionLost, URI(), "Transport closed" ) );
@@ -566,8 +588,10 @@ void SSHConnection::drive() {
operation->fail( mFailure );
auto pumps = std::move( mPumps );
mPumps.clear();
for ( auto& pump : pumps )
for ( auto& pump : pumps ) {
removeUser();
pump->connectionLost( mFailure );
}
return;
}
@@ -600,6 +624,7 @@ void SSHConnection::drive() {
++i;
} else {
mPumps.erase( mPumps.begin() + i );
removeUser();
}
if ( !isUsable() ) {
drive();
@@ -607,6 +632,12 @@ void SSHConnection::drive() {
}
}
if ( idleExpired() ) {
fail( VFSError( VFSErrorCode::ConnectionLost, URI(), "Idle connection closed" ) );
mState = State::Closed;
return;
}
if ( mConfig.serverAliveInterval && std::chrono::steady_clock::now() >= mNextKeepalive ) {
int seconds = 0;
int rc = libssh2_keepalive_send( mSession, &seconds );
@@ -625,9 +656,9 @@ std::vector<std::string_view> SSHConnection::knownHostKeyAlgorithms() {
if ( !hosts )
return known;
for ( const auto& file : mConfig.userKnownHostsFiles )
loadKnownHosts( hosts, file );
loadKnownHosts( hosts, file, nullptr );
for ( const auto& file : mConfig.globalKnownHostsFiles )
loadKnownHosts( hosts, file );
loadKnownHosts( hosts, file, nullptr );
std::string checkHost = mConfig.hostKeyAlias.value_or( mConfig.hostName );
// A key that can never be stored: a "mismatch" means an entry of that type exists for the
// host. This lets libssh2 handle hashed names and [host]:port entries.
@@ -647,7 +678,7 @@ std::vector<std::string_view> SSHConnection::knownHostKeyAlgorithms() {
return known;
}
void SSHConnection::applyAlgorithmPreferences() {
VFSResult<void> SSHConnection::applyAlgorithmPreferences() {
// Like OpenSSH, unless HostKeyAlgorithms is configured, prefer the host key algorithms that
// known_hosts already has keys for, so a known host is not reported as unknown just because
// the server also offers another key type.
@@ -677,22 +708,30 @@ void SSHConnection::applyAlgorithmPreferences() {
struct Pref {
const std::optional<std::string>* spec;
int method;
const char* option;
};
const Pref prefs[] = {
{ &mConfig.kexAlgorithms, LIBSSH2_METHOD_KEX },
{ &mConfig.hostKeyAlgorithms, LIBSSH2_METHOD_HOSTKEY },
{ &mConfig.ciphers, LIBSSH2_METHOD_CRYPT_CS },
{ &mConfig.ciphers, LIBSSH2_METHOD_CRYPT_SC },
{ &mConfig.macs, LIBSSH2_METHOD_MAC_CS },
{ &mConfig.macs, LIBSSH2_METHOD_MAC_SC },
{ &mConfig.kexAlgorithms, LIBSSH2_METHOD_KEX, "KexAlgorithms" },
{ &mConfig.hostKeyAlgorithms, LIBSSH2_METHOD_HOSTKEY, "HostKeyAlgorithms" },
{ &mConfig.ciphers, LIBSSH2_METHOD_CRYPT_CS, "Ciphers" },
{ &mConfig.ciphers, LIBSSH2_METHOD_CRYPT_SC, "Ciphers" },
{ &mConfig.macs, LIBSSH2_METHOD_MAC_CS, "MACs" },
{ &mConfig.macs, LIBSSH2_METHOD_MAC_SC, "MACs" },
};
for ( const auto& pref : prefs ) {
if ( !*pref.spec )
continue;
// A configured restriction is a policy: when nothing it permits is available, the
// connection fails instead of falling back to libssh2's defaults.
auto unsupported = [&]() -> VFSResult<void> {
return Err( VFSError( VFSErrorCode::Unsupported, URI(),
std::string( "None of the configured " ) + pref.option + " (" +
**pref.spec + ") is supported" ) );
};
const char** algs = nullptr;
int count = libssh2_session_supported_algs( mSession, pref.method, &algs );
if ( count <= 0 )
continue;
return unsupported();
std::vector<std::string> supported( algs, algs + count );
libssh2_free( mSession, algs );
auto list = SSHConfigEvaluator::applyAlgorithmSpec( supported, **pref.spec );
@@ -705,9 +744,11 @@ void SSHConnection::applyAlgorithmPreferences() {
joined += ',';
joined += alg;
}
if ( !joined.empty() )
libssh2_session_method_pref( mSession, pref.method, joined.c_str() );
if ( joined.empty() ||
libssh2_session_method_pref( mSession, pref.method, joined.c_str() ) != 0 )
return unsupported();
}
return Ok();
}
Step SSHConnection::establish() {
@@ -721,7 +762,10 @@ Step SSHConnection::establish() {
libssh2_session_set_blocking( mSession, 0 );
if ( mConfig.compression )
libssh2_session_flag( mSession, LIBSSH2_FLAG_COMPRESS, 1 );
applyAlgorithmPreferences();
if ( auto preferred = applyAlgorithmPreferences(); !preferred ) {
fail( std::move( preferred ).error() );
return Step::Done;
}
mTransport->install( mSession );
mState = State::Handshake;
[[fallthrough]];
@@ -747,8 +791,9 @@ Step SSHConnection::establish() {
mState = State::SFTPInit;
[[fallthrough]];
case State::SFTPInit:
mSFTP = libssh2_sftp_init( mSession );
if ( !mSFTP ) {
// Jump hosts only forward channels: they need not offer SFTP.
mSFTP = mSFTPRequired ? libssh2_sftp_init( mSession ) : nullptr;
if ( !mSFTP && mSFTPRequired ) {
if ( libssh2_session_last_errno( mSession ) == LIBSSH2_ERROR_EAGAIN )
return Step::Again;
fail( lastError( URI(), "Unable to start the SFTP subsystem" ) );
@@ -818,10 +863,21 @@ Step SSHConnection::stepHostKey() {
std::string checkHost = mConfig.hostKeyAlias.value_or( mConfig.hostName );
int keyBits = knownHostKeyBits( keyType );
LIBSSH2_KNOWNHOSTS* hosts = libssh2_knownhost_init( mSession );
std::vector<std::string> revoked;
for ( const auto& file : mConfig.userKnownHostsFiles )
loadKnownHosts( hosts, file );
loadKnownHosts( hosts, file, &revoked );
for ( const auto& file : mConfig.globalKnownHostsFiles )
loadKnownHosts( hosts, file );
loadKnownHosts( hosts, file, &revoked );
const std::string_view presented( key, keyLength );
if ( std::find( revoked.begin(), revoked.end(), presented ) != revoked.end() ) {
// Checked before anything else: a revoked key is never accepted, even when it is also
// listed as known or StrictHostKeyChecking would accept new keys.
libssh2_knownhost_free( hosts );
fail( VFSError( VFSErrorCode::HostKeyRejected, URI(),
"The " + std::string( hostKeyTypeName( keyType ) ) + " host key of " +
checkHost + " is revoked" ) );
return Step::Done;
}
struct libssh2_knownhost* entry = nullptr;
int check = libssh2_knownhost_checkp(
hosts, checkHost.c_str(), mConfig.port, key, keyLength,
@@ -959,6 +1015,101 @@ bool SSHConnection::saveHostKey( const char* key, size_t keyLength, int keyBits,
return true;
}
void SSHConnection::startAgentListing() {
mAuthPhase = AuthPhase::AgentListing;
// Listing identities needs no user interaction. Keep the connection deadline active so an
// unresponsive agent cannot leave the connection pending forever.
mWaitingExternal = false;
auto agent = std::make_shared<SSHAgent>( *mConfig.identityAgent );
mAgentClient = agent;
std::vector<std::string> allowed;
if ( mConfig.identitiesOnly ) {
for ( const auto& file : mConfig.identityFiles )
if ( auto blob = readPublicKeyBlob( file + ".pub" ) )
allowed.push_back( std::move( *blob ) );
}
std::weak_ptr<SSHConnection> weak = weak_from_this();
std::weak_ptr<SSHIOContext> context = mContext;
// The connection cancels the agent socket on failure or shutdown, waking this worker even
// if the agent never answers. The worker owns its captures until it exits.
std::thread( [agent, weak, context, identitiesOnly = mConfig.identitiesOnly,
allowed = std::move( allowed )]() mutable {
auto keys = agent->listIdentities();
auto io = context.lock();
if ( !io )
return;
io->post( [agent = std::move( agent ), weak, identitiesOnly, allowed = std::move( allowed ),
keys = std::make_shared<decltype( keys )>( std::move( keys ) )]() mutable {
auto self = weak.lock();
if ( !self || !self->isUsable() || self->mAuthPhase != AuthPhase::AgentListing )
return;
self->mWaitingExternal = false;
self->mAgentKeys.clear();
self->mAgentKeyIndex = 0;
if ( *keys ) {
for ( auto& key : keys->value() ) {
if ( !identitiesOnly ||
std::find( allowed.begin(), allowed.end(), key.blob ) != allowed.end() )
self->mAgentKeys.push_back( std::move( key ) );
}
}
self->mAgentClient = self->mAgentKeys.empty() ? nullptr : std::move( agent );
self->mAuthPhase =
self->mAgentKeys.empty() ? AuthPhase::KeyNext : AuthPhase::AgentKeyNext;
} );
} ).detach();
}
void SSHConnection::startAgentSignature( std::string_view payload ) {
mAgentSignPayload.assign( payload );
mAgentSignature.reset();
mAgentSigning = true;
mWaitingExternal = true;
std::weak_ptr<SSHConnection> weak = weak_from_this();
std::weak_ptr<SSHIOContext> context = mContext;
std::thread( [agent = mAgentClient, weak, context, blob = mAgentKeys[mAgentKeyIndex].blob,
payload = mAgentSignPayload,
algorithm = std::string( SSHAgent::algorithmOfSignedRequest( payload ) )] {
auto signature =
std::make_shared<VFSResult<std::string>>( agent->sign( blob, payload, algorithm ) );
if ( auto io = context.lock() ) {
io->post( [weak, signature] {
auto self = weak.lock();
if ( !self || !self->isUsable() || !self->mAgentSigning )
return;
self->mAgentSigning = false;
self->mWaitingExternal = false;
self->mAgentSignature = std::move( *signature );
} );
}
} ).detach();
}
int SSHConnection::agentSign( LIBSSH2_SESSION*, unsigned char** signature, size_t* signatureLength,
const unsigned char* data, size_t dataLength, void** abstract ) {
auto* self = static_cast<SSHConnection*>( *abstract );
std::string_view payload( reinterpret_cast<const char*>( data ), dataLength );
// libssh2 calls again with the same payload after an EAGAIN.
if ( self->mAgentSignature && self->mAgentSignPayload == payload ) {
auto result = std::move( *self->mAgentSignature );
self->mAgentSignature.reset();
self->mAgentSignPayload.clear();
if ( !result )
return LIBSSH2_ERROR_AGENT_PROTOCOL;
const std::string& raw = result.value();
// Released by libssh2 with the session's (default) allocator.
*signature = static_cast<unsigned char*>( std::malloc( raw.size() ) );
if ( !*signature )
return LIBSSH2_ERROR_ALLOC;
std::memcpy( *signature, raw.data(), raw.size() );
*signatureLength = raw.size();
return 0;
}
if ( !self->mAgentSigning )
self->startAgentSignature( payload );
return LIBSSH2_ERROR_EAGAIN;
}
void SSHConnection::requestExternal( const SSHAuthenticationRequest& request ) {
mWaitingExternal = true;
mExternalAnswer.reset();
@@ -1075,6 +1226,10 @@ Step SSHConnection::stepAuthentication() {
mAuthPhase = AuthPhase::KeyNext;
continue;
}
if ( SSHAgent::supported() ) {
startAgentListing();
return Step::Again;
}
mAgent = libssh2_agent_init( mSession );
if ( mAgent ) {
libssh2_agent_set_identity_path( mAgent, mConfig.identityAgent->c_str() );
@@ -1132,6 +1287,43 @@ Step SSHConnection::stepAuthentication() {
mAuthPhase = AuthPhase::AgentNext;
continue;
}
case AuthPhase::AgentListing:
// startAgentListing() moves on once the agent answered.
return Step::Again;
case AuthPhase::AgentKeyNext: {
if ( mAgentKeyIndex >= mAgentKeys.size() ) {
mAgentClient.reset();
mAgentKeys.clear();
mAuthPhase = AuthPhase::KeyNext;
continue;
}
mAgentSignPayload.clear();
mAgentSignature.reset();
mAuthPhase = AuthPhase::AgentKeyAuth;
continue;
}
case AuthPhase::AgentKeyAuth: {
const std::string& blob = mAgentKeys[mAgentKeyIndex].blob;
void* abstract = this;
int rc = libssh2_userauth_publickey(
mSession, user.c_str(), reinterpret_cast<const unsigned char*>( blob.data() ),
blob.size(), &SSHConnection::agentSign, &abstract );
if ( rc == LIBSSH2_ERROR_EAGAIN )
return Step::Again;
if ( rc == 0 ) {
mAgentClient.reset();
mAgentKeys.clear();
mAuthPhase = AuthPhase::Done;
continue;
}
if ( isSocketError( libssh2_session_last_errno( mSession ) ) ) {
fail( lastError( URI(), "Authentication failed" ) );
return Step::Done;
}
++mAgentKeyIndex;
mAuthPhase = AuthPhase::AgentKeyNext;
continue;
}
case AuthPhase::KeyNext: {
while ( mKeyIndex < mConfig.identityFiles.size() &&
!FileSystem::fileExists( mConfig.identityFiles[mKeyIndex] ) )
@@ -1327,9 +1519,14 @@ short SocketTransport::pollEvents( short directions ) const {
ChannelTransport::ChannelTransport( std::shared_ptr<SSHConnection> parent, std::string host,
Uint16 port ) :
mParent( std::move( parent ) ), mHost( std::move( host ) ), mPort( port ) {}
mParent( std::move( parent ) ), mHost( std::move( host ) ), mPort( port ) {
// The child's traffic is invisible to the parent's queue: keep the parent from closing as
// idle while the child lives.
mParent->addUser();
}
ChannelTransport::~ChannelTransport() {
mParent->removeUser();
// The channel belongs to the parent session; libssh2_session_free() of the parent releases
// it. Freeing it here could race with the parent's I/O thread usage.
}
@@ -1,6 +1,7 @@
#ifndef EE_VFS_SFTP_SSHSESSION_HPP
#define EE_VFS_SFTP_SSHSESSION_HPP
#include "sshagent.hpp"
#include "sshsocket.hpp"
#include <algorithm>
#include <atomic>
@@ -128,6 +129,9 @@ class SSHIOContext : public std::enable_shared_from_this<SSHIOContext> {
/** Registers a connection (I/O thread only). */
void addConnection( std::shared_ptr<SSHConnection> connection );
/** Fails registered sessions, including sessions still awaiting authentication. */
void closeConnections();
private:
void loop();
@@ -221,6 +225,8 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
const SSHResolvedConfig& config() const { return mConfig; }
const std::shared_ptr<SSHIOContext>& ioContext() const { return mContext; }
/** Last activity time (any thread), for idle pooling. */
std::chrono::steady_clock::time_point lastUse() const;
@@ -234,6 +240,11 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
/** @return Milliseconds until a timer of this connection fires, -1 if none. */
int nextTimeoutMs() const;
std::chrono::milliseconds idleTimeout() const;
/** @return True when the connection stayed idle longer than its idle timeout. */
bool idleExpired() const;
LIBSSH2_SESSION* session() const { return mSession; }
LIBSSH2_SFTP* sftp() const { return mSFTP; }
@@ -253,14 +264,44 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
/** Tracks an open SFTP handle (I/O thread only). Handles still open when the connection is
* destroyed (for example because their owners died after the connection failed) are closed
* then, so libssh2 releases them. */
void trackHandle( LIBSSH2_SFTP_HANDLE* handle ) { mHandles.push_back( handle ); }
void trackHandle( LIBSSH2_SFTP_HANDLE* handle ) {
mHandles.push_back( handle );
addUser();
}
void untrackHandle( LIBSSH2_SFTP_HANDLE* handle ) {
mHandles.erase( std::remove( mHandles.begin(), mHandles.end(), handle ), mHandles.end() );
auto it = std::find( mHandles.begin(), mHandles.end(), handle );
if ( it != mHandles.end() ) {
mHandles.erase( it );
removeUser();
}
}
/** Registers a channel pump (I/O thread only, connection must be ready). */
void addPump( std::shared_ptr<SSHChannelPump> pump ) { mPumps.push_back( std::move( pump ) ); }
void addPump( std::shared_ptr<SSHChannelPump> pump ) {
mPumps.push_back( std::move( pump ) );
addUser();
}
/** Something keeps using the connection without queued operations: an open handle, a pump,
* a jump child's channel. A connection in use is never closed as idle. Any thread. */
void addUser() { mUsers.fetch_add( 1, std::memory_order_relaxed ); }
void removeUser() { mUsers.fetch_sub( 1, std::memory_order_relaxed ); }
/** @return True when nothing uses the connection (no users, no queued operations). */
bool isIdle() const {
return mUsers.load( std::memory_order_relaxed ) == 0 && !hasQueuedOperations();
}
/** Closes the connection once it stayed idle this long (zero: never). Set it before the
* connection is added to the I/O context. */
void setIdleTimeout( Time timeout ) { mIdleTimeout = timeout; }
/** Whether Ready includes a started SFTP subsystem (true by default). Jump hosts only open
* channels for the next hop and need no SFTP. Set it before the connection is added to the
* I/O context. */
void setSFTPRequired( bool required ) { mSFTPRequired = required; }
/** Opens a direct-tcpip channel for a child session (ProxyJump). I/O thread only. */
LIBSSH2_CHANNEL* openDirectTcpip( const std::string& host, Uint16 port, bool& again );
@@ -286,8 +327,11 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
friend class ChannelTransport;
friend class ProcessTransport;
/** Queues operation, or fails it right away when the connection is already terminal. */
void enqueue( std::unique_ptr<SSHOperation> operation );
bool hasQueuedOperations() const;
void fail( VFSError error );
Step establish();
@@ -302,7 +346,19 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
void requestExternal( const SSHAuthenticationRequest& request );
void applyAlgorithmPreferences();
/** Lists the agent's keys on a worker thread (AgentListing until it answers). */
void startAgentListing();
/** Asks the agent for the signature of payload on a worker thread. */
void startAgentSignature( std::string_view payload );
/** libssh2 sign callback of agent keys: EAGAIN until the agent thread answered. */
static int agentSign( LIBSSH2_SESSION* session, unsigned char** signature,
size_t* signatureLength, const unsigned char* data, size_t dataLength,
void** abstract );
/** Applies the configured algorithm restrictions; fails when one permits nothing available. */
VFSResult<void> applyAlgorithmPreferences();
/** Host key algorithms for which known_hosts holds a key of this host. */
std::vector<std::string_view> knownHostKeyAlgorithms();
@@ -319,6 +375,9 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
mutable std::mutex mQueueMutex;
std::deque<std::unique_ptr<SSHOperation>> mQueue;
/** Set (under mQueueMutex) once the connection failed or closed: new operations fail with it
* instead of being queued. */
std::optional<VFSError> mQueueFailure;
std::vector<std::shared_ptr<SSHChannelPump>> mPumps;
std::vector<LIBSSH2_SFTP_HANDLE*> mHandles;
std::vector<TaskSource<VFSResult<void>>> mReadyWaiters;
@@ -333,8 +392,13 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
List,
NextMethod,
AgentConnect,
/** libssh2's agent (where SSHAgent is not supported). */
AgentNext,
AgentAuth,
/** SSHAgent on a worker thread: listing, then each key. */
AgentListing,
AgentKeyNext,
AgentKeyAuth,
KeyNext,
KeyAuth,
KbdAuth,
@@ -349,6 +413,15 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
std::string mServerMethods;
struct libssh2_agent_publickey* mAgentIdentity{ nullptr };
std::vector<std::string> mAllowedAgentBlobs;
/** Agent calls block (signing may wait for a hardware key touch): they run on a worker
* thread while the I/O thread keeps serving the other sessions. */
std::shared_ptr<SSHAgent> mAgentClient;
std::vector<SSHAgent::Identity> mAgentKeys;
size_t mAgentKeyIndex{ 0 };
/** The signature libssh2 asked for (its payload), and the agent's answer once it arrived. */
std::string mAgentSignPayload;
std::optional<VFSResult<std::string>> mAgentSignature;
bool mAgentSigning{ false };
size_t mKeyIndex{ 0 };
std::string mPassphrase;
Uint32 mAttempts{ 0 };
@@ -365,15 +438,12 @@ class SSHConnection : public std::enable_shared_from_this<SSHConnection> {
int mPendingHostKeyBits{ 0 };
std::atomic<State> mState{ State::Starting };
std::atomic<Uint32> mUsers{ 0 };
Time mIdleTimeout;
bool mSFTPRequired{ true };
bool mWaitingExternal{ false };
};
/** Base64 without padding, as used by OpenSSH fingerprints. */
std::string base64Unpadded( const unsigned char* data, size_t length );
/** Maps an SFTP status code to a VFS error code. */
VFSErrorCode mapSFTPStatus( unsigned long status );
/** TCP transport. */
class SocketTransport : public SSHTransport {
public:
@@ -1307,8 +1307,32 @@ std::string SSHResolvedConfig::connectionKey() const {
add( identitiesOnly ? "1" : "0" );
for ( const auto& identity : identityFiles )
add( identity );
for ( const auto& certificate : certificateFiles )
add( certificate );
add( identityAgent.value_or( "" ) );
add( hostKeyAlias.value_or( "" ) );
// Security policy: a connection verified or authenticated under a weaker policy must never
// be handed to a destination that demands a stricter one.
add( "known" );
for ( const auto& file : userKnownHostsFiles )
add( file );
add( "global" );
for ( const auto& file : globalKnownHostsFiles )
add( file );
add( String::toString( static_cast<int>( strictHostKeyChecking ) ) );
add( checkHostIP ? "1" : "0" );
for ( const auto* algorithms :
{ &ciphers, &macs, &kexAlgorithms, &hostKeyAlgorithms, &pubkeyAcceptedAlgorithms } )
add( algorithms->has_value() ? "+" + **algorithms : std::string() );
add( "auth" );
for ( const auto& method : preferredAuthentications )
add( method );
add( pubkeyAuthentication ? "1" : "0" );
add( passwordAuthentication ? "1" : "0" );
add( kbdInteractiveAuthentication ? "1" : "0" );
add( String::toString( numberOfPasswordPrompts ) );
add( batchMode ? "1" : "0" );
add( compression ? "1" : "0" );
return key;
}
@@ -359,13 +359,15 @@ void VFSDirectoryModel::rescan() {
// Entry storage
// ---------------------------------------------------------------------------------------------
bool VFSDirectoryModel::less( const FileInfo& a, const FileInfo& b ) const {
bool VFSDirectoryModel::less( const FileInfo& a, const FileInfo& b, const std::string* aType,
const std::string* bType ) const {
bool aDir = a.isDirectory(), bDir = b.isDirectory();
if ( aDir != bDir )
return aDir; // Directories first, whatever the order.
int cmp = 0;
// Runs O(n log n) times per sort on the UI thread: no allocations.
auto compareNames = [&] {
int c = String::toLower( a.name ).compare( String::toLower( b.name ) );
int c = String::icompare( a.name, b.name );
return c != 0 ? c : a.name.compare( b.name );
};
auto compareNumbers = []( auto x, auto y ) { return x < y ? -1 : ( y < x ? 1 : 0 ); };
@@ -378,7 +380,7 @@ bool VFSDirectoryModel::less( const FileInfo& a, const FileInfo& b ) const {
cmp = compareNumbers( a.size().value_or( 0 ), b.size().value_or( 0 ) );
break;
case Type:
cmp = typeOf( a ).compare( typeOf( b ) );
cmp = aType && bType ? aType->compare( *bType ) : typeOf( a ).compare( typeOf( b ) );
break;
case Modified:
cmp = compareNumbers( a.modifiedTime().value_or( 0 ), b.modifiedTime().value_or( 0 ) );
@@ -400,6 +402,47 @@ bool VFSDirectoryModel::less( const FileInfo& a, const FileInfo& b ) const {
return mSortOrder == SortOrder::Descending ? cmp > 0 : cmp < 0;
}
void VFSDirectoryModel::sortEntries( std::vector<FileInfo>& entries ) const {
if ( mSortColumn != Type ) {
std::stable_sort( entries.begin(), entries.end(),
[this]( const FileInfo& a, const FileInfo& b ) { return less( a, b ); } );
return;
}
// The type provider (a std::function returning a string) runs once per entry, not twice per
// comparison.
std::vector<std::string> types;
types.reserve( entries.size() );
for ( const auto& entry : entries )
types.push_back( typeOf( entry ) );
std::vector<size_t> order( entries.size() );
for ( size_t i = 0; i < order.size(); ++i )
order[i] = i;
std::stable_sort( order.begin(), order.end(), [&]( size_t x, size_t y ) {
return less( entries[x], entries[y], &types[x], &types[y] );
} );
std::vector<FileInfo> sorted;
sorted.reserve( entries.size() );
for ( size_t index : order )
sorted.push_back( std::move( entries[index] ) );
entries.swap( sorted );
}
void VFSDirectoryModel::rowsInserted( size_t entryIndex, bool visible ) {
size_t row = visibleRowFor( entryIndex );
for ( size_t i = row; i < mRows.size(); ++i )
++mRows[i];
if ( visible )
mRows.insert( mRows.begin() + row, entryIndex );
}
void VFSDirectoryModel::rowsErased( size_t entryIndex ) {
size_t row = visibleRowFor( entryIndex );
if ( row < mRows.size() && mRows[row] == entryIndex )
mRows.erase( mRows.begin() + row );
for ( size_t i = row; i < mRows.size(); ++i )
--mRows[i];
}
void VFSDirectoryModel::rebuildRows() {
mRows.clear();
mRows.reserve( mEntries.size() );
@@ -419,7 +462,7 @@ void VFSDirectoryModel::mergeBatch( std::vector<FileInfo>&& batch ) {
for ( const auto& info : batch )
queueItemCount( info );
auto cmp = [this]( const FileInfo& a, const FileInfo& b ) { return less( a, b ); };
std::sort( batch.begin(), batch.end(), cmp );
sortEntries( batch );
if ( mEntries.empty() ) {
mEntries = std::move( batch );
} else {
@@ -446,15 +489,17 @@ void VFSDirectoryModel::removeByName( const std::string& name ) {
size_t index = static_cast<size_t>( it - mEntries.begin() );
bool visible = isVisible( *it );
mItemCounts.erase( name );
// Rows are updated in place: watch events arrive one by one, a full rebuild would cost a
// pass over every entry each time.
if ( visible ) {
int row = static_cast<int>( visibleRowFor( index ) );
beginDeleteRows( {}, row, row );
mEntries.erase( it );
rebuildRows();
rowsErased( index );
endDeleteRows();
} else {
mEntries.erase( it );
rebuildRows();
rowsErased( index );
}
onModelUpdate( Model::UpdateFlag::DontInvalidateIndexes );
notifyChanged();
@@ -491,11 +536,11 @@ void VFSDirectoryModel::upsert( FileInfo&& info ) {
int row = static_cast<int>( visibleRowFor( index ) );
beginInsertRows( {}, row, row );
mEntries.insert( pos, std::move( info ) );
rebuildRows();
rowsInserted( index, true );
endInsertRows();
} else {
mEntries.insert( pos, std::move( info ) );
rebuildRows();
rowsInserted( index, false );
}
onModelUpdate( Model::UpdateFlag::DontInvalidateIndexes );
notifyChanged();
@@ -683,8 +728,9 @@ bool VFSDirectoryModel::matchesNameFilter( const std::string& name ) const {
case NameFilterMode::PlainText:
if ( mFilterCaseSensitive )
return name.find( mFilterPattern ) != std::string::npos;
return String::toLower( name ).find( String::toLower( mFilterPattern ) ) !=
std::string::npos;
// Every entry is matched on each rebuild: no lowercase copies.
return String::BMH::find( std::string_view( name ), mFilterPattern, 0,
mFilterOccurrences, true ) != -1;
case NameFilterMode::Glob:
return String::globMatch( name, mFilterGlob, !mFilterCaseSensitive );
case NameFilterMode::RegEx:
@@ -702,6 +748,12 @@ void VFSDirectoryModel::setNameFilter( std::string pattern, NameFilterMode mode,
mFilterCaseSensitive = caseSensitive;
mFilterRegEx.reset();
mFilterValid = true;
mFilterOccurrences.clear();
if ( mode == NameFilterMode::PlainText && !caseSensitive && !mFilterPattern.empty() ) {
mFilterOccurrences = String::BMH::createOccTable(
reinterpret_cast<const unsigned char*>( mFilterPattern.data() ),
mFilterPattern.size(), true );
}
// Unanchored like Dolphin ("md" matches "notes.md"). A doubled '*' would mean "**" (any
// path) to globMatch, so only missing wildcards are added.
mFilterGlob.clear();
@@ -893,8 +945,7 @@ Variant VFSDirectoryModel::data( const ModelIndex& index, ModelRole role ) const
void VFSDirectoryModel::sort( const size_t& column, const SortOrder& order ) {
mSortColumn = column < ColumnCount ? column : Name;
mSortOrder = order == SortOrder::None ? SortOrder::Ascending : order;
std::stable_sort( mEntries.begin(), mEntries.end(),
[this]( const FileInfo& a, const FileInfo& b ) { return less( a, b ); } );
sortEntries( mEntries );
rebuildRows();
invalidate( Model::UpdateFlag::InvalidateAllIndexes );
}
+268 -204
View File
@@ -46,22 +46,142 @@ std::string temporarySiblingName( std::string_view name ) {
return tmp;
}
/** Writes every byte of data through handle, starting at offset 0. */
/** Writes every byte of data through handle, starting at offset. */
VFSResult<void> writeFully( FileHandle& handle, std::span<const Uint8> data,
const CancellationToken& token ) {
Uint64 offset = 0;
while ( offset < data.size() ) {
size_t chunk = std::min<size_t>( IOChunkSize, data.size() - offset );
auto written = await( handle.write( offset, data.subspan( offset, chunk ), token ) );
const CancellationToken& token, Uint64 offset = 0 ) {
size_t done = 0;
while ( done < data.size() ) {
size_t chunk = std::min<size_t>( IOChunkSize, data.size() - done );
auto written = await( handle.write( offset + done, data.subspan( done, chunk ), token ) );
if ( !written )
return Err( std::move( written ).error() );
if ( written.value() == 0 )
return Err( VFSError( VFSErrorCode::IO, handle.uri(), "Short write" ) );
offset += written.value();
done += written.value();
}
return Ok();
}
/** makeDirs() body. Orchestration jobs call it directly: waiting on a makeDirs() task from the
* orchestration pool would wait on a job queued behind the waiting one. */
VFSResult<void> makeDirsBlocking( VFSProvider& provider, const URI& uri,
const MkDirOptions& options, const CancellationToken& token ) {
std::vector<URI> missing;
URI current( uri );
while ( true ) {
auto info = await( provider.stat( current, StatOptions{}, token ) );
if ( info ) {
if ( !info.value().isDirectory() )
return Err( VFSError( VFSErrorCode::NotDirectory, current ) );
break;
}
if ( !info.error().is( VFSErrorCode::NotFound ) )
return Err( std::move( info ).error() );
missing.push_back( current );
URI parent( current.getParent() );
if ( parent.getPath() == current.getPath() )
break;
current = std::move( parent );
}
for ( auto it = missing.rbegin(); it != missing.rend(); ++it ) {
auto made = await( provider.mkdir( *it, options, token ) );
if ( !made ) {
if ( !made.error().is( VFSErrorCode::AlreadyExists ) )
return made;
// Lost a race with another creator: fine if it is a directory.
auto info = await( provider.stat( *it, StatOptions{}, token ) );
if ( !info || !info.value().isDirectory() )
return made;
}
}
return Ok();
}
/** The file a write to uri replaces: the target of a symlink (so the link survives the
* replacement), otherwise uri itself. */
VFSResult<URI> resolveWriteTarget( VFSProvider& provider, const URI& uri,
const CancellationToken& token ) {
auto linkInfo = await( provider.stat( uri, StatOptions{ false }, token ) );
if ( !linkInfo || !linkInfo.value().isSymlink() )
return Ok( URI( uri ) );
auto resolved = await( provider.canonicalize( uri, token ) );
// A dangling link is replaced by the new file.
if ( !resolved && resolved.error().is( VFSErrorCode::NotFound ) )
return Ok( URI( uri ) );
return resolved;
}
struct ReplaceOptions {
/** Mode of a newly created file (the provider filters it through the umask). */
std::optional<Uint32> createPermissions;
/** Exact mode to give the new version (for example the replaced file's), applied after it is
* written: creation modes are filtered through the umask. */
std::optional<Uint32> exactPermissions;
bool fsync{ false };
/** Replace through a temporary sibling. When false, target is rewritten in place. */
bool atomic{ true };
/** Without atomic replacement support, rewrite target in place instead of failing. */
bool allowInPlace{ false };
/** Fail if another entry appeared at target while the temporary was being written. */
bool overwrite{ true };
};
/** Writes a new version of target with write( FileHandle& ). The data goes to a temporary sibling
* that replaces target only once write() and beforeReplace() succeeded, so a failure never damages
* the current version, nor another name of the same file (a hard link to it). Non-atomic writes,
* and providers without atomic replacement (when allowed), rewrite target in place. */
template <typename Write, typename BeforeReplace>
VFSResult<void> replaceFile( VFSProvider& provider, const URI& target,
const ReplaceOptions& options, const CancellationToken& token,
Write&& write, BeforeReplace&& beforeReplace ) {
ProviderCapabilities caps = provider.capabilities( target );
auto writeTo = [&]( const URI& file, OpenFlags flags ) -> VFSResult<void> {
OpenOptions openOptions( flags );
openOptions.createPermissions =
options.exactPermissions ? options.exactPermissions : options.createPermissions;
auto opened = await( provider.open( file, openOptions, token ) );
if ( !opened )
return Err( std::move( opened ).error() );
FileHandle handle( std::move( opened ).value() );
VFSResult<void> written = write( handle );
if ( written && options.fsync && caps.has( VFSCapability::Fsync ) )
written = await( handle.flush() );
auto closed = await( handle.close() );
if ( !written )
return written;
if ( !closed )
return closed;
if ( options.exactPermissions && caps.has( VFSCapability::Permissions ) )
return await( provider.setPermissions( file, *options.exactPermissions, token ) );
return Ok();
};
if ( !options.atomic || !caps.has( VFSCapability::Rename ) ||
( options.overwrite && !caps.has( VFSCapability::AtomicReplace ) ) ) {
if ( options.atomic && !options.allowInPlace ) {
return Err( VFSError( VFSErrorCode::Unsupported, target,
"Atomic replacement is not supported by this provider" ) );
}
// beforeReplace() guards a replacement; in place, the old version is already gone.
return writeTo( target, OpenFlags::Write | OpenFlags::Create | OpenFlags::Truncate );
}
URI temporary( target.getParent().withAppendedSegment(
temporarySiblingName( target.getLastPathSegment() ) ) );
auto written =
writeTo( temporary, OpenFlags::Write | OpenFlags::Create | OpenFlags::Exclusive );
if ( written )
written = beforeReplace();
if ( written )
written = await(
provider.rename( temporary, target, RenameOptions{ options.overwrite }, token ) );
if ( !written ) {
// Best effort: never leave the temporary behind.
await( provider.remove( temporary, {}, CancellationToken() ) );
}
return written;
}
} // namespace
VirtualFileSystem& VirtualFileSystem::instance() {
@@ -85,6 +205,7 @@ VirtualFileSystem::VirtualFileSystem( const Config& config ) :
mIOPool( ThreadPool::createShared( std::max<Uint32>( 1, config.ioThreads ) ) ),
mOrchestrationPool(
ThreadPool::createShared( std::max<Uint32>( 1, config.orchestrationThreads ) ) ),
mTransferPool( ThreadPool::createShared( std::max<Uint32>( 1, config.transferThreads ) ) ),
mWatchManager( std::make_unique<WatchManager>() ) {
if ( config.registerDefaultProviders )
registerDefaultProviders();
@@ -92,6 +213,7 @@ VirtualFileSystem::VirtualFileSystem( const Config& config ) :
VirtualFileSystem::~VirtualFileSystem() {
// Orchestration jobs wait on provider tasks: stop them first, while providers still run.
mTransferPool.reset();
mOrchestrationPool.reset();
mWatchManager.reset();
{
@@ -104,7 +226,7 @@ VirtualFileSystem::~VirtualFileSystem() {
void VirtualFileSystem::registerDefaultProviders() {
registerProvider( std::make_shared<LocalProvider>( mIOPool ) );
registerProvider( std::make_shared<ResourceProvider>( mIOPool ) );
mWatchManager->registerBackend( "file", std::make_shared<LocalWatchBackend>() );
mWatchManager->registerBackend( "file", std::make_shared<LocalWatchBackend>( mIOPool ) );
mWatchManager->setFallbackBackend( std::make_shared<PollingWatchBackend>( *this ) );
#ifdef EE_VFS_SFTP
auto sftp = std::make_shared<SFTPProvider>();
@@ -372,103 +494,64 @@ VFSTask<FileInfo> VirtualFileSystem::writeAll( const URI& uri, std::vector<Uint8
if ( !provider )
return unsupportedScheme<FileInfo>( uri );
// The buffer is moved into the job: the caller's data must outlive the asynchronous write.
return orchestrate<FileInfo>( [this, provider, uri, data = std::move( data ), options,
token]() -> VFSResult<FileInfo> {
auto checkRevision = [&]( const VFSResult<FileInfo>& current ) -> VFSResult<void> {
if ( !options.expectedRevision )
return Ok();
if ( !current ) {
if ( current.error().is( VFSErrorCode::NotFound ) )
return orchestrate<FileInfo>(
[provider, uri, data = std::move( data ), options, token]() -> VFSResult<FileInfo> {
auto checkRevision = [&]( const VFSResult<FileInfo>& current ) -> VFSResult<void> {
if ( !options.expectedRevision )
return Ok();
if ( !current ) {
if ( current.error().is( VFSErrorCode::NotFound ) )
return Err( VFSError( VFSErrorCode::ChangedExternally, uri,
"The file was removed externally" ) );
return Err( current.error() );
}
if ( !current.value().revision.matches( *options.expectedRevision ) )
return Err( VFSError( VFSErrorCode::ChangedExternally, uri,
"The file was removed externally" ) );
return Err( current.error() );
"The file was modified externally" ) );
return Ok();
};
auto target = resolveWriteTarget( *provider, uri, token );
if ( !target )
return Err( std::move( target ).error() );
auto existing = await( provider->stat( target.value(), StatOptions{}, token ) );
if ( !existing && !existing.error().is( VFSErrorCode::NotFound ) )
return Err( std::move( existing ).error() );
if ( auto check = checkRevision( existing ); !check )
return Err( std::move( check ).error() );
if ( existing && existing.value().isDirectory() )
return Err( VFSError( VFSErrorCode::IsDirectory, uri ) );
if ( options.createParents && !existing ) {
auto parents = makeDirsBlocking( *provider, target.value().getParent(), {}, token );
if ( !parents )
return Err( std::move( parents ).error() );
}
if ( !current.value().revision.matches( *options.expectedRevision ) )
return Err( VFSError( VFSErrorCode::ChangedExternally, uri,
"The file was modified externally" ) );
return Ok();
};
// Saving through a symlink writes the link target, so the link survives the atomic
// replace.
URI target( uri );
auto linkInfo = await( provider->stat( uri, StatOptions{ false }, token ) );
if ( linkInfo && linkInfo.value().isSymlink() ) {
auto resolved = await( provider->canonicalize( uri, token ) );
if ( !resolved )
return Err( std::move( resolved ).error() );
target = std::move( resolved ).value();
}
auto existing = await( provider->stat( target, StatOptions{}, token ) );
if ( !existing && !existing.error().is( VFSErrorCode::NotFound ) )
return Err( std::move( existing ).error() );
if ( auto check = checkRevision( existing ); !check )
return Err( std::move( check ).error() );
if ( existing && existing.value().isDirectory() )
return Err( VFSError( VFSErrorCode::IsDirectory, uri ) );
if ( options.createParents && !existing ) {
auto parents = await( makeDirs( target.getParent(), {}, token ) );
if ( !parents )
return Err( std::move( parents ).error() );
}
ProviderCapabilities caps = provider->capabilities( target );
bool atomic = options.atomic && caps.has( VFSCapability::Rename ) &&
caps.has( VFSCapability::AtomicReplace );
if ( options.atomic && !atomic && !options.allowNonAtomicFallback ) {
return Err( VFSError( VFSErrorCode::Unsupported, uri,
"Atomic replacement is not supported by this provider" ) );
}
OpenOptions openOptions;
openOptions.createPermissions =
existing ? existing.value().permissions() : options.permissions;
std::span<const Uint8> bytes( data.data(), data.size() );
auto writeTo = [&]( const URI& file, OpenFlags flags ) -> VFSResult<void> {
openOptions.flags = flags;
auto opened = await( provider->open( file, openOptions, token ) );
if ( !opened )
return Err( std::move( opened ).error() );
FileHandle handle( std::move( opened ).value() );
auto written = writeFully( handle, bytes, token );
if ( written && options.fsync && caps.has( VFSCapability::Fsync ) )
written = await( handle.flush() );
auto closed = await( handle.close() );
if ( !written )
return written;
return closed;
};
if ( !atomic ) {
auto written =
writeTo( target, OpenFlags::Write | OpenFlags::Create | OpenFlags::Truncate );
ReplaceOptions replace;
if ( existing )
replace.exactPermissions = existing.value().permissions();
else
replace.createPermissions = options.permissions;
replace.fsync = options.fsync;
replace.atomic = options.atomic;
replace.allowInPlace = options.allowNonAtomicFallback;
std::span<const Uint8> bytes( data.data(), data.size() );
auto written = replaceFile(
*provider, target.value(), replace, token,
[&]( FileHandle& handle ) { return writeFully( handle, bytes, token ); },
[&]() -> VFSResult<void> {
// Narrow the race window: re-check right before replacing.
if ( !options.expectedRevision )
return Ok();
return checkRevision(
await( provider->stat( target.value(), StatOptions{}, token ) ) );
} );
if ( !written )
return Err( std::move( written ).error() );
} else {
URI temporary( target.getParent().withAppendedSegment(
temporarySiblingName( target.getLastPathSegment() ) ) );
auto written =
writeTo( temporary, OpenFlags::Write | OpenFlags::Create | OpenFlags::Exclusive );
if ( written && options.expectedRevision ) {
// Narrow the race window: re-check right before replacing.
written = checkRevision( await( provider->stat( target, StatOptions{}, token ) ) );
}
if ( written ) {
written =
await( provider->rename( temporary, target, RenameOptions{ true }, token ) );
}
if ( !written ) {
// Best effort: never leave the temporary behind.
await( provider->remove( temporary, {}, CancellationToken() ) );
return Err( std::move( written ).error() );
}
}
return await( provider->stat( target, StatOptions{}, token ) );
} );
return await( provider->stat( target.value(), StatOptions{}, token ) );
} );
}
VFSTask<void> VirtualFileSystem::makeDirs( const URI& uri, const MkDirOptions& options,
@@ -477,35 +560,7 @@ VFSTask<void> VirtualFileSystem::makeDirs( const URI& uri, const MkDirOptions& o
if ( !provider )
return unsupportedScheme<void>( uri );
return orchestrate<void>( [provider, uri, options, token]() -> VFSResult<void> {
std::vector<URI> missing;
URI current( uri );
while ( true ) {
auto info = await( provider->stat( current, StatOptions{}, token ) );
if ( info ) {
if ( !info.value().isDirectory() )
return Err( VFSError( VFSErrorCode::NotDirectory, current ) );
break;
}
if ( !info.error().is( VFSErrorCode::NotFound ) )
return Err( std::move( info ).error() );
missing.push_back( current );
URI parent( current.getParent() );
if ( parent.getPath() == current.getPath() )
break;
current = std::move( parent );
}
for ( auto it = missing.rbegin(); it != missing.rend(); ++it ) {
auto made = await( provider->mkdir( *it, options, token ) );
if ( !made ) {
if ( !made.error().is( VFSErrorCode::AlreadyExists ) )
return made;
// Lost a race with another creator: fine if it is a directory.
auto info = await( provider->stat( *it, StatOptions{}, token ) );
if ( !info || !info.value().isDirectory() )
return made;
}
}
return Ok();
return makeDirsBlocking( *provider, uri, options, token );
} );
}
@@ -600,9 +655,9 @@ class TransferEngine {
return Err( VFSError( VFSErrorCode::InvalidOperation, mSource,
"Source and destination are the same" ) );
bool sameProvider = mSrcProvider == mDstProvider && sameAuthority( mSource, mDestination );
mSameProvider = mSrcProvider == mDstProvider && sameAuthority( mSource, mDestination );
if ( mMove && sameProvider ) {
if ( mMove && mSameProvider ) {
auto renamed = await( mSrcProvider->rename(
mSource, mDestination,
RenameOptions{ mOptions.overwrite == OverwritePolicy::Overwrite }, mToken ) );
@@ -628,11 +683,23 @@ class TransferEngine {
return Err( std::move( renamed ).error() );
}
auto rootInfo = await( mSrcProvider->stat( mSource, StatOptions{ true }, mToken ) );
auto rootInfo = await( mSrcProvider->stat( mSource, StatOptions{ false }, mToken ) );
if ( !rootInfo )
return Err( std::move( rootInfo ).error() );
if ( rootInfo.value().isSymlink() ) {
// Providers cannot create links, so a copy takes the link's content. A move cannot:
// removing what was copied would empty the link's target, not move the link.
if ( mMove ) {
return Err( VFSError( VFSErrorCode::Unsupported, mSource,
"Symbolic links can only be moved within their file "
"system" ) );
}
rootInfo = await( mSrcProvider->stat( mSource, StatOptions{ true }, mToken ) );
if ( !rootInfo )
return Err( std::move( rootInfo ).error() );
}
if ( rootInfo.value().isDirectory() && sameProvider &&
if ( rootInfo.value().isDirectory() && mSameProvider &&
mDestination.isSameOrChildOf( mSource ) ) {
return Err( VFSError( VFSErrorCode::InvalidOperation, mDestination,
"Cannot copy a directory into itself" ) );
@@ -736,12 +803,30 @@ class TransferEngine {
advanceFiles( info );
return Ok( false );
}
if ( mSameProvider ) {
auto same = isSameFile( source, destination );
if ( !same )
return Err( std::move( same ).error() );
if ( same.value() ) {
return Err( VFSError( VFSErrorCode::InvalidOperation, destination,
"Source and destination are the same file" ) );
}
}
} else if ( !existing.error().is( VFSErrorCode::NotFound ) ) {
return Err( std::move( existing ).error() );
}
if ( auto copied = copyFile( info, source, destination ); !copied )
if ( auto copied = copyFile( info, source, destination ); !copied ) {
// A destination can appear while the temporary is written. Skip still applies at
// publication time, and a move must keep that source.
if ( mOptions.overwrite == OverwritePolicy::Skip &&
copied.error().is( VFSErrorCode::AlreadyExists ) ) {
++mResult.filesSkipped;
advanceFiles( info );
return Ok( false );
}
return Err( std::move( copied ).error() );
}
++mResult.filesTransferred;
advanceFiles( info, false );
@@ -805,6 +890,18 @@ class TransferEngine {
true );
}
/** @return True when both URIs name the same file through symlinks or "..". Hard links are
* not detected; replaceFile() keeps their data safe. */
VFSResult<bool> isSameFile( const URI& source, const URI& destination ) {
auto canonicalSource = await( mSrcProvider->canonicalize( source, mToken ) );
if ( !canonicalSource )
return Err( std::move( canonicalSource ).error() );
auto canonicalDestination = await( mDstProvider->canonicalize( destination, mToken ) );
if ( !canonicalDestination )
return Err( std::move( canonicalDestination ).error() );
return Ok( canonicalSource.value() == canonicalDestination.value() );
}
VFSResult<void> copyFile( const FileInfo& info, const URI& source, const URI& destination ) {
mState->updateProgress(
[&]( TransferProgress& p ) {
@@ -813,80 +910,45 @@ class TransferEngine {
},
true );
auto target = resolveWriteTarget( *mDstProvider, destination, mToken );
if ( !target )
return Err( std::move( target ).error() );
auto srcOpened = await( mSrcProvider->open( source, OpenFlags::Read, mToken ) );
if ( !srcOpened )
return Err( std::move( srcOpened ).error() );
FileHandle src( std::move( srcOpened ).value() );
OpenOptions dstOptions( OpenFlags::Write | OpenFlags::Create | OpenFlags::Truncate );
bool preserve = mOptions.preservePermissions && info.permissions() &&
mDstProvider->capabilities( destination ).has( VFSCapability::Permissions );
if ( preserve )
dstOptions.createPermissions = info.permissions();
auto dstOpened = await( mDstProvider->open( destination, dstOptions, mToken ) );
if ( !dstOpened ) {
src.close();
return Err( std::move( dstOpened ).error() );
}
FileHandle dst( std::move( dstOpened ).value() );
ReplaceOptions replace;
if ( mOptions.preservePermissions && info.permissions() )
replace.exactPermissions = info.permissions();
replace.overwrite = mOptions.overwrite == OverwritePolicy::Overwrite;
std::vector<Uint8> buffer( mOptions.chunkSize );
Uint64 offset = 0;
VFSResult<void> status = Ok();
while ( true ) {
if ( auto c = checkCancelled( source ); !c ) {
status = std::move( c );
break;
}
auto read = await( src.read( offset, buffer, mToken ) );
if ( !read ) {
status = Err( std::move( read ).error() );
break;
}
size_t got = read.value();
if ( got == 0 )
break;
if ( auto written = writeFully( dst, std::span<const Uint8>( buffer.data(), got ),
mToken, offset );
!written ) {
status = std::move( written );
break;
}
offset += got;
mResult.bytesTransferred += got;
mState->updateProgress( [got]( TransferProgress& p ) { p.bytesDone += got; } );
}
auto srcClosed = await( src.close() );
auto dstClosed = await( dst.close() );
if ( status && !dstClosed )
status = std::move( dstClosed );
(void)srcClosed;
if ( !status ) {
// Never leave a truncated file that looks complete.
await( mDstProvider->remove( destination, {}, CancellationToken() ) );
return status;
}
if ( preserve ) {
// Creation permissions are filtered by the umask; apply the exact mode.
await( mDstProvider->setPermissions( destination, *info.permissions(), mToken ) );
}
return Ok();
}
static VFSResult<void> writeFully( FileHandle& handle, std::span<const Uint8> data,
const CancellationToken& token, Uint64 baseOffset ) {
size_t done = 0;
while ( done < data.size() ) {
auto written = await( handle.write( baseOffset + done, data.subspan( done ), token ) );
if ( !written )
return Err( std::move( written ).error() );
if ( written.value() == 0 )
return Err( VFSError( VFSErrorCode::IO, handle.uri(), "Short write" ) );
done += written.value();
}
return Ok();
auto copied = replaceFile(
*mDstProvider, target.value(), replace, mToken,
[&]( FileHandle& dst ) -> VFSResult<void> {
Uint64 offset = 0;
while ( true ) {
if ( auto c = checkCancelled( source ); !c )
return c;
auto read = await( src.read( offset, buffer, mToken ) );
if ( !read )
return Err( std::move( read ).error() );
size_t got = read.value();
if ( got == 0 )
return Ok();
if ( auto written = writeFully(
dst, std::span<const Uint8>( buffer.data(), got ), mToken, offset );
!written )
return written;
offset += got;
mResult.bytesTransferred += got;
mState->updateProgress( [got]( TransferProgress& p ) { p.bytesDone += got; } );
}
},
[]() -> VFSResult<void> { return Ok(); } );
await( src.close() );
return copied;
}
VirtualFileSystem& mVfs;
@@ -899,6 +961,8 @@ class TransferEngine {
VFSProvider* mDstProvider{ nullptr };
TransferResult mResult;
bool mMove;
/** Source and destination are served by the same provider and authority. */
bool mSameProvider{ false };
};
namespace {
@@ -922,12 +986,12 @@ Transfer startTransfer( VirtualFileSystem& vfs, std::shared_ptr<ThreadPool> pool
Transfer VirtualFileSystem::copy( const URI& source, const URI& destination,
const CopyOptions& options ) {
return startTransfer( *this, mOrchestrationPool, source, destination, options, false );
return startTransfer( *this, mTransferPool, source, destination, options, false );
}
Transfer VirtualFileSystem::move( const URI& source, const URI& destination,
const CopyOptions& options ) {
return startTransfer( *this, mOrchestrationPool, source, destination, options, true );
return startTransfer( *this, mTransferPool, source, destination, options, true );
}
}} // namespace EE::VFS
@@ -331,19 +331,31 @@ class LocalWatchBackendImpl : public std::enable_shared_from_this<LocalWatchBack
class LocalWatchRegistration : public WatchRegistration {
public:
LocalWatchRegistration( std::shared_ptr<LocalWatchBackendImpl> impl, Uint64 token ) :
mImpl( std::move( impl ) ), mToken( token ) {}
LocalWatchRegistration( std::weak_ptr<System::ThreadPool> pool,
std::shared_ptr<LocalWatchBackendImpl> impl, Uint64 token ) :
mPool( std::move( pool ) ), mImpl( std::move( impl ) ), mToken( token ) {}
~LocalWatchRegistration() override { mImpl->remove( mToken ); }
~LocalWatchRegistration() override {
// Removing an efsw watch may wait on its thread: off the caller's (UI) thread. Events
// stop anyway, the handle closes its delivery right after this.
if ( auto pool = mPool.lock() ) {
pool->run( [impl = std::move( mImpl ), token = mToken] { impl->remove( token ); } );
} else {
mImpl->remove( mToken );
}
}
private:
std::weak_ptr<System::ThreadPool> mPool;
std::shared_ptr<LocalWatchBackendImpl> mImpl;
Uint64 mToken;
};
} // namespace detail
LocalWatchBackend::LocalWatchBackend( Time genericPollingInterval ) :
LocalWatchBackend::LocalWatchBackend( std::shared_ptr<System::ThreadPool> pool,
Time genericPollingInterval ) :
mPool( std::move( pool ) ),
mImpl( std::make_shared<detail::LocalWatchBackendImpl>( genericPollingInterval ) ) {}
LocalWatchBackend::~LocalWatchBackend() = default;
@@ -355,20 +367,25 @@ LocalWatchBackend::watch( const URI& root, const WatchOptions& options,
auto path = LocalProvider::toNativePath( root );
if ( !path )
return readyError<Registration>( std::move( path ).error() );
if ( !System::FileSystem::isDirectory( path.value() ) ) {
return readyError<Registration>( System::FileSystem::fileExists( path.value() )
? VFSErrorCode::NotDirectory
: VFSErrorCode::NotFound,
root );
}
// efsw registration is synchronous and fast; once it returns the watch is active, so a
// listing started afterwards cannot miss changes.
auto added = mImpl->add( path.value(), options, sink );
if ( !added )
return readyError<Registration>( std::move( added ).error() );
sink->setStatus( added.value().second ? WatchStatus::Polling : WatchStatus::Active );
return readyValue<Registration>(
std::make_unique<detail::LocalWatchRegistration>( mImpl, added.value().first ) );
std::weak_ptr<System::ThreadPool> weakPool = mPool;
// Checking the directory and registering it touch the filesystem (a slow mount blocks): on
// the I/O pool. Once the task resolves the watch is active, so a listing started afterwards
// cannot miss changes.
return mPool->submit( [impl = mImpl, weakPool, root, options, sink = std::move( sink ),
nativeRoot = std::move( path ).value()]() -> VFSResult<Registration> {
if ( !System::FileSystem::isDirectory( nativeRoot ) ) {
return Err( VFSError( System::FileSystem::fileExists( nativeRoot )
? VFSErrorCode::NotDirectory
: VFSErrorCode::NotFound,
root ) );
}
auto added = impl->add( nativeRoot, options, sink );
if ( !added )
return Err( std::move( added ).error() );
sink->setStatus( added.value().second ? WatchStatus::Polling : WatchStatus::Active );
return Ok( Registration(
std::make_unique<detail::LocalWatchRegistration>( weakPool, impl, added.value().first ) ) );
} );
}
}} // namespace EE::VFS
+16
View File
@@ -308,6 +308,22 @@ UTEST( Cancellation, callbackMayDestroyItsOwnRegistration ) {
EXPECT_FALSE( static_cast<bool>( reg ) );
}
UTEST( Cancellation, callbackMayDestroyTheSource ) {
// The state must outlive the callbacks even when one of them destroys the source that is
// cancelling (use-after-free under AddressSanitizer otherwise).
auto source = std::make_unique<CancellationSource>();
CancellationRegistration reg;
int calls = 0;
reg = source->token().onCancel( [&] {
++calls;
reg.reset();
source.reset();
} );
source->cancel();
EXPECT_EQ( 1, calls );
EXPECT_TRUE( source == nullptr );
}
UTEST( Cancellation, sourceAndTokenLifetimeIndependence ) {
CancellationToken token;
{
+454 -1
View File
@@ -3,14 +3,21 @@
#if defined( EE_VFS_SFTP ) && EE_PLATFORM != EE_PLATFORM_WIN
#include "../../modules/vfs/src/eepp/vfs/sftp/sshagent.hpp"
#include "vfs_test_utils.hpp"
#include <condition_variable>
#include <cstdlib>
#include <eepp/system/base64.hpp>
#include <eepp/system/process.hpp>
#include <eepp/vfs/sftp/sftpprovider.hpp>
#include <netinet/in.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/un.h>
#include <unistd.h>
#if defined( __linux__ )
#include <dirent.h>
#endif
using namespace vfstest;
using EE::System::Process;
@@ -211,7 +218,21 @@ class TestSSHServer {
// preferred by default: connecting needs the known-type ordering.
"Host edhost\n StrictHostKeyChecking yes\n UserKnownHostsFile " + mDir +
"kh_ed25519\n" + common +
"Host * !secret !edkey !ednopub !edsecret\n IdentityFile " + mDir + "id_ecdsa\n" +
// Every host key of the server is @revoked: never accepted, not even as new.
"Host revoked\n StrictHostKeyChecking accept-new\n UserKnownHostsFile " + mDir +
"kh_revoked\n" + common +
// Only an algorithm libssh2 does not support: the connection must fail.
"Host badalgo\n HostKeyAlgorithms sk-ssh-ed25519@openssh.com\n"
" StrictHostKeyChecking accept-new\n UserKnownHostsFile " +
mDir + "kh_badalgo\n" + common +
// Only an ssh-agent can authenticate these: their identity file does not exist.
"Host agent\n IdentityAgent " + mDir + "agent.sock\n IdentitiesOnly no\n" +
" IdentityFile " + mDir + "no-such-key\n PreferredAuthentications publickey\n" +
common + "Host stalledagent\n IdentityAgent " + mDir +
"stalled-agent.sock\n IdentitiesOnly no\n IdentityFile " + mDir +
"no-such-key\n PreferredAuthentications publickey\n" + common +
"Host * !secret !edkey !ednopub !edsecret !agent !stalledagent\n IdentityFile " +
mDir + "id_ecdsa\n" +
"Host *\n StrictHostKeyChecking accept-new\n UserKnownHostsFile " + mDir +
"known_hosts\n" + common;
FileSystem::fileWrite( mDir + "ssh_config", std::string_view( sshConfig ) );
@@ -222,6 +243,15 @@ class TestSSHServer {
FileSystem::fileWrite(
mDir + "kh_mismatch",
std::string_view( "mismatch-alias " + fields[0] + " " + fields[1] + "\n" ) );
std::string revoked;
for ( const char* key : { "host_ecdsa.pub", "host_ed25519.pub" } ) {
std::string pub;
FileSystem::fileGet( mDir + key, pub );
auto keyFields = String::split( pub, ' ' );
revoked +=
"@revoked [127.0.0.1]:" + port + " " + keyFields[0] + " " + keyFields[1] + "\n";
}
FileSystem::fileWrite( mDir + "kh_revoked", std::string_view( revoked ) );
std::string hostEd;
FileSystem::fileGet( mDir + "host_ed25519.pub", hostEd );
fields = String::split( hostEd, ' ' );
@@ -474,6 +504,30 @@ UTEST( SFTP, hostKeyPolicies ) {
}
}
UTEST( SFTP, revokedHostKeysAreRejected ) {
REQUIRE_SSHD();
SFTPFixture fx;
auto revoked = waitFor( fx.vfs.stat( server.uri( "revoked" ) ) );
EXPECT_EQ( VFSErrorCode::HostKeyRejected, codeOf( revoked ) );
}
UTEST( SFTP, unsupportedAlgorithmRestrictionFails ) {
REQUIRE_SSHD();
SFTPFixture fx;
auto connected = waitFor( fx.vfs.stat( server.uri( "badalgo" ) ) );
EXPECT_FALSE( connected.hasValue() );
}
UTEST( SFTP, reuseKeepsTheStricterHostKeyPolicy ) {
REQUIRE_SSHD();
// "testhost" accepts new keys, "strict" does not know the server: both reach the same server
// and user, but "strict" must not reuse the connection "testhost" made.
SFTPFixture fx;
REQUIRE_OK( waitFor( fx.vfs.stat( server.uri( "testhost" ) ) ) );
auto strict = waitFor( fx.vfs.stat( server.uri( "strict" ) ) );
EXPECT_EQ( VFSErrorCode::HostKeyRejected, codeOf( strict ) );
}
UTEST( SFTP, authenticationPrompts ) {
REQUIRE_SSHD();
{
@@ -584,6 +638,365 @@ UTEST( SFTP, connectionReuseReconnectAndErrors ) {
EXPECT_EQ( 1u, config.value().proxyJump.size() );
}
UTEST( SFTP, operationsOnAClosedConnectionFail ) {
REQUIRE_SSHD();
SFTPFixture fx;
URI file = server.uri( "testhost", uniqueName( "closed" ) );
REQUIRE_OK( waitFor( fx.vfs.writeAll( file, std::string_view( "data" ) ) ) );
auto opened = waitFor( fx.vfs.open( file ) );
REQUIRE_OK( opened );
FileHandle handle( std::move( opened ).value() );
fx.sftp->disconnectAll();
// Give the I/O thread time to drop the closed connection.
std::this_thread::sleep_for( std::chrono::milliseconds( 200 ) );
Uint8 buffer[8];
auto read = handle.read( 0, buffer );
ASSERT_EQ( TaskState::Ready, read.waitFor( std::chrono::seconds( 5 ) ) );
EXPECT_EQ( VFSErrorCode::ConnectionLost, codeOf( std::move( read ).takeValue() ) );
auto closed = handle.close();
ASSERT_EQ( TaskState::Ready, closed.waitFor( std::chrono::seconds( 5 ) ) );
}
UTEST( SFTP, idleConnectionsCloseButNotWhileInUse ) {
REQUIRE_SSHD();
SFTPProviderConfig config;
config.idleTimeout = System::Milliseconds( 300 );
SFTPFixture fx( config );
URI file = server.uri( "testhost", uniqueName( "idle" ) );
REQUIRE_OK( waitFor( fx.vfs.writeAll( file, std::string_view( "data" ) ) ) );
auto opened = waitFor( fx.vfs.open( file ) );
REQUIRE_OK( opened );
FileHandle handle( std::move( opened ).value() );
// An open handle keeps the connection.
std::this_thread::sleep_for( std::chrono::milliseconds( 800 ) );
EXPECT_EQ( 1u, fx.sftp->connectionCount() );
Uint8 buffer[4];
auto read = waitFor( handle.read( 0, buffer ) );
REQUIRE_OK( read );
EXPECT_EQ( 4u, read.value() );
REQUIRE_OK( waitFor( handle.close() ) );
// Unused, it closes after the idle timeout; the next operation reconnects.
bool closed = false;
for ( int i = 0; i < 50 && !closed; ++i ) {
std::this_thread::sleep_for( std::chrono::milliseconds( 100 ) );
closed = fx.sftp->connectionCount() == 0;
}
EXPECT_TRUE( closed );
EXPECT_TRUE( waitFor( fx.vfs.stat( file ) ).hasValue() );
}
#if defined( __linux__ )
UTEST( SFTP, idleConnectionReleasesItsSocket ) {
REQUIRE_SSHD();
auto connectedSockets = [&server] {
int count = 0;
DIR* fds = ::opendir( "/proc/self/fd" );
if ( !fds )
return -1;
while ( auto* entry = ::readdir( fds ) ) {
if ( entry->d_name[0] == '.' )
continue;
int fd = std::atoi( entry->d_name );
sockaddr_in peer{};
socklen_t length = sizeof( peer );
if ( ::getpeername( fd, reinterpret_cast<sockaddr*>( &peer ), &length ) == 0 &&
peer.sin_family == AF_INET && ntohs( peer.sin_port ) == server.port() )
++count;
}
::closedir( fds );
return count;
};
int before = connectedSockets();
ASSERT_GE( before, 0 );
SFTPProviderConfig config;
config.idleTimeout = System::Milliseconds( 300 );
SFTPFixture fx( config );
REQUIRE_OK( waitFor( fx.vfs.stat( server.uri( "testhost" ) ) ) );
EXPECT_EQ( before + 1, connectedSockets() );
for ( int i = 0; i < 50 && connectedSockets() != before; ++i )
std::this_thread::sleep_for( std::chrono::milliseconds( 100 ) );
EXPECT_EQ( before, connectedSockets() );
}
#endif
namespace {
/** Never answers until released: a user who leaves a password dialog open. */
class PendingAuthenticator : public SSHAuthenticationHandler {
public:
Task<std::optional<std::vector<std::string>>>
request( const SSHAuthenticationRequest& ) override {
std::lock_guard<std::mutex> lock( mMutex );
if ( mReleased )
return makeReadyTask( std::optional<std::vector<std::string>>() );
mPending.emplace_back();
return mPending.back().task();
}
size_t pending() {
std::lock_guard<std::mutex> lock( mMutex );
return mPending.size();
}
void release() {
std::vector<TaskSource<std::optional<std::vector<std::string>>>> pending;
{
std::lock_guard<std::mutex> lock( mMutex );
mReleased = true;
pending.swap( mPending );
}
// Declined: later prompts (other methods, attempts) are declined right away.
for ( auto& source : pending )
source.resolve( std::nullopt );
}
private:
std::mutex mMutex;
std::vector<TaskSource<std::optional<std::vector<std::string>>>> mPending;
bool mReleased{ false };
};
} // namespace
UTEST( SFTP, pendingPromptsDoNotBlockOtherConnections ) {
REQUIRE_SSHD();
auto prompts = std::make_shared<PendingAuthenticator>();
SFTPProviderConfig config;
config.authenticationHandler = prompts;
SFTPFixture fx( config );
// Two connections wait for the user (as many as connection helper threads).
auto pw = fx.vfs.stat( server.uri( "pw" ) );
auto secret = fx.vfs.stat( server.uri( "secret" ) );
for ( int i = 0; i < 100 && prompts->pending() < 2; ++i )
std::this_thread::sleep_for( std::chrono::milliseconds( 50 ) );
EXPECT_EQ( 2u, prompts->pending() );
auto other = fx.vfs.stat( server.uri( "testhost" ) );
ASSERT_EQ( TaskState::Ready, other.waitFor( std::chrono::seconds( 10 ) ) );
EXPECT_TRUE( std::move( other ).takeValue().hasValue() );
prompts->release();
EXPECT_EQ( TaskState::Ready, pw.waitFor( std::chrono::seconds( 10 ) ) );
EXPECT_EQ( TaskState::Ready, secret.waitFor( std::chrono::seconds( 10 ) ) );
}
UTEST( SFTP, disconnectCompletesOperationsWaitingForAPrompt ) {
REQUIRE_SSHD();
auto auth = std::make_shared<PendingAuthenticator>();
SFTPProviderConfig config;
config.authenticationHandler = auth;
SFTPFixture fx( config );
auto pending = fx.vfs.stat( server.uri( "pw" ) );
for ( int i = 0; i < 200 && auth->pending() == 0; ++i )
std::this_thread::sleep_for( std::chrono::milliseconds( 10 ) );
EXPECT_GT( auth->pending(), 0u );
fx.sftp->disconnectAll();
auto state = pending.waitFor( std::chrono::seconds( 2 ) );
auth->release();
ASSERT_EQ( TaskState::Ready, state );
EXPECT_EQ( VFSErrorCode::ConnectionLost, codeOf( std::move( pending ).takeValue() ) );
}
namespace {
/** An ssh-agent that lists one key and answers signing requests only once released (a hardware
* key waiting for a touch). */
class StalledAgent {
public:
StalledAgent( const std::string& path, std::string keyBlob ) : mKey( std::move( keyBlob ) ) {
::unlink( path.c_str() );
mListener = ::socket( AF_UNIX, SOCK_STREAM, 0 );
sockaddr_un address{};
address.sun_family = AF_UNIX;
std::strncpy( address.sun_path, path.c_str(), sizeof( address.sun_path ) - 1 );
::bind( mListener, reinterpret_cast<sockaddr*>( &address ), sizeof( address ) );
::listen( mListener, 4 );
mThread = std::thread( [this] { serve(); } );
}
~StalledAgent() {
release();
::shutdown( mListener, SHUT_RDWR );
::close( mListener );
{
std::lock_guard<std::mutex> lock( mMutex );
if ( mClient >= 0 )
::shutdown( mClient, SHUT_RDWR );
}
mThread.join();
}
bool signRequested() {
std::lock_guard<std::mutex> lock( mMutex );
return mSignRequested;
}
void release() {
std::lock_guard<std::mutex> lock( mMutex );
mReleased = true;
mChanged.notify_all();
}
private:
static void putU32( std::string& out, Uint32 value ) {
for ( int shift = 24; shift >= 0; shift -= 8 )
out += static_cast<char>( ( value >> shift ) & 0xFF );
}
static bool readAll( int fd, char* data, size_t size ) {
while ( size > 0 ) {
ssize_t rc = ::recv( fd, data, size, 0 );
if ( rc <= 0 )
return false;
data += rc;
size -= static_cast<size_t>( rc );
}
return true;
}
void serve() {
while ( true ) {
int client = ::accept( mListener, nullptr, nullptr );
if ( client < 0 )
return;
#ifdef SO_NOSIGPIPE
int noSigPipe = 1;
::setsockopt( client, SOL_SOCKET, SO_NOSIGPIPE, &noSigPipe, sizeof( noSigPipe ) );
#endif
{
std::lock_guard<std::mutex> lock( mMutex );
mClient = client;
}
while ( true ) {
unsigned char header[4];
if ( !readAll( client, reinterpret_cast<char*>( header ), 4 ) )
break;
Uint32 length =
( header[0] << 24 ) | ( header[1] << 16 ) | ( header[2] << 8 ) | header[3];
std::string request( length, '\0' );
if ( !readAll( client, request.data(), length ) )
break;
std::string body;
if ( request[0] == 11 ) { // Request identities.
body += static_cast<char>( 12 );
putU32( body, 1 );
putU32( body, static_cast<Uint32>( mKey.size() ) );
body += mKey;
putU32( body, 0 );
} else {
std::unique_lock<std::mutex> lock( mMutex );
mSignRequested = true;
mChanged.wait( lock, [this] { return mReleased; } );
body += static_cast<char>( 5 ); // Failure.
}
std::string reply;
putU32( reply, static_cast<Uint32>( body.size() ) );
reply += body;
#ifdef MSG_NOSIGNAL
::send( client, reply.data(), reply.size(), MSG_NOSIGNAL );
#else
::send( client, reply.data(), reply.size(), 0 );
#endif
}
{
std::lock_guard<std::mutex> lock( mMutex );
mClient = -1;
}
::close( client );
}
}
std::string mKey;
int mListener{ -1 };
int mClient{ -1 };
std::mutex mMutex;
std::condition_variable mChanged;
bool mSignRequested{ false };
bool mReleased{ false };
std::thread mThread;
};
std::string publicKeyBlob( const std::string& pubFile ) {
std::string text, blob;
FileSystem::fileGet( pubFile, text );
auto fields = String::split( text, ' ' );
if ( fields.size() >= 2 )
System::Base64::decode( fields[1], blob );
return blob;
}
} // namespace
UTEST( SFTP, agentAuthentication ) {
REQUIRE_SSHD();
std::string agentBinary = Sys::which( "ssh-agent" );
std::string addBinary = Sys::which( "ssh-add" );
if ( agentBinary.empty() || addBinary.empty() ) {
printf( " ssh-agent not found, skipped\n" );
return;
}
std::string socket = server.dir() + "agent.sock";
FileSystem::fileRemove( socket );
Process agent;
ASSERT_TRUE( agent.create( agentBinary, { "-D", "-a", socket } ) );
for ( int i = 0; i < 100 && !FileSystem::fileExists( socket ); ++i )
std::this_thread::sleep_for( std::chrono::milliseconds( 20 ) );
Process add;
ASSERT_TRUE( add.create( "/bin/sh", { "-c", "SSH_AUTH_SOCK='" + socket + "' '" + addBinary +
"' '" + server.dir() + "id_ecdsa'" } ) );
int code = -1;
add.join( &code );
ASSERT_EQ( 0, code );
{
// The identity file does not exist: only the agent's key can authenticate.
SFTPFixture fx;
REQUIRE_OK( waitFor( fx.vfs.stat( server.uri( "agent" ) ) ) );
}
agent.kill();
}
UTEST( SFTP, stalledAgentDoesNotBlockOtherConnections ) {
REQUIRE_SSHD();
StalledAgent agent( server.dir() + "stalled-agent.sock",
publicKeyBlob( server.dir() + "id_ecdsa.pub" ) );
SFTPFixture fx;
auto stalled = fx.vfs.stat( server.uri( "stalledagent" ) );
for ( int i = 0; i < 200 && !agent.signRequested(); ++i )
std::this_thread::sleep_for( std::chrono::milliseconds( 25 ) );
EXPECT_TRUE( agent.signRequested() );
// The agent waits for a touch: every other session keeps working meanwhile.
auto other = fx.vfs.stat( server.uri( "testhost" ) );
ASSERT_EQ( TaskState::Ready, other.waitFor( std::chrono::seconds( 10 ) ) );
EXPECT_TRUE( std::move( other ).takeValue().hasValue() );
EXPECT_TRUE( stalled.isPending() );
agent.release();
ASSERT_EQ( TaskState::Ready, stalled.waitFor( std::chrono::seconds( 10 ) ) );
EXPECT_EQ( VFSErrorCode::AuthenticationFailed, codeOf( std::move( stalled ).takeValue() ) );
}
UTEST( SFTP, cancelledAgentRequestDoesNotWaitForTheAgent ) {
REQUIRE_SSHD();
std::string path = server.dir() + "cancel-agent.sock";
StalledAgent agent( path, publicKeyBlob( server.dir() + "id_ecdsa.pub" ) );
sftp::SSHAgent client( path );
auto keys = client.listIdentities();
REQUIRE_OK( keys );
ASSERT_FALSE( keys.value().empty() );
TaskSource<VFSResult<std::string>> source;
auto task = source.task();
std::thread worker( [&client, &keys, source = std::move( source )]() mutable {
source.resolve( client.sign( keys.value()[0].blob, "payload", "ecdsa-sha2-nistp256" ) );
} );
for ( int i = 0; i < 200 && !agent.signRequested(); ++i )
std::this_thread::sleep_for( std::chrono::milliseconds( 10 ) );
EXPECT_TRUE( agent.signRequested() );
client.cancel();
auto state = task.waitFor( std::chrono::seconds( 2 ) );
// Also release on failure so a regression cannot hang the suite during thread destruction.
agent.release();
worker.join();
ASSERT_EQ( TaskState::Ready, state );
EXPECT_FALSE( std::move( task ).takeValue().hasValue() );
}
UTEST( SFTP, remoteWatchThroughAgent ) {
REQUIRE_SSHD();
std::string agent = Sys::getProcessPath();
@@ -652,4 +1065,44 @@ UTEST( SFTP, remoteWatchFallsBackToPolling ) {
loop.runUntil( [&] { return events.has( WatchEventType::Added, "polled.txt" ); } ) );
}
UTEST( SFTP, closingTheLastWatchAllowsIdleCleanup ) {
REQUIRE_SSHD();
std::string agent = Sys::getProcessPath();
FileSystem::dirAddSlashAtEnd( agent );
agent += "../efsw-agent";
if ( !FileSystem::fileExists( agent ) ) {
printf( " efsw-agent not built, remote watch test skipped\n" );
return;
}
SFTPProviderConfig config;
config.watchAgentCommand = agent + " --stdio";
config.idleTimeout = System::Milliseconds( 300 );
SFTPFixture fx( config );
auto opened = waitFor( fx.vfs.watch( server.uri( "testhost" ) ) );
REQUIRE_OK( opened );
WatchHandle watch( std::move( opened ).value() );
std::this_thread::sleep_for( std::chrono::milliseconds( 700 ) );
EXPECT_EQ( 1u, fx.sftp->connectionCount() );
watch.reset();
for ( int i = 0; i < 50 && fx.sftp->connectionCount() != 0; ++i )
std::this_thread::sleep_for( std::chrono::milliseconds( 100 ) );
EXPECT_EQ( 0u, fx.sftp->connectionCount() );
}
UTEST( SFTP, silentWatchAgentFallsBackToPolling ) {
REQUIRE_SSHD();
// The command runs but never answers the handshake: the watch (and the listing that waits
// for it) must not stay pending.
SFTPProviderConfig config;
config.watchAgentCommand = "sleep 60";
SFTPFixture fx( config );
std::string name = uniqueName( "silentwatch" );
FileSystem::makeDir( server.remoteRoot() + "/" + name );
auto task = fx.vfs.watch( server.uri( "testhost", name ) );
ASSERT_EQ( TaskState::Ready, task.waitFor( std::chrono::seconds( 20 ) ) );
auto opened = std::move( task ).takeValue();
REQUIRE_OK( opened );
EXPECT_EQ( WatchStatus::Polling, opened.value().status() );
}
#endif
+19
View File
@@ -485,6 +485,25 @@ UTEST( SSHConfig, connectionKeyIgnoresPathsAndDistinguishesProfiles ) {
EXPECT_TRUE( a.connectionKey() != c.connectionKey() );
}
UTEST( SSHConfig, connectionKeyIncludesTheSecurityPolicy ) {
// A session verified under a permissive policy must not serve a stricter configuration.
FakeSystem fake;
fake.files[UserConfig] = "Host makyo\n User black\n";
auto base = fake.resolve( "makyo" );
auto stricter = base;
stricter.strictHostKeyChecking = SSHStrictHostKeyChecking::Yes;
EXPECT_TRUE( base.connectionKey() != stricter.connectionKey() );
auto otherKnownHosts = base;
otherKnownHosts.userKnownHostsFiles = { "/different/known_hosts" };
EXPECT_TRUE( base.connectionKey() != otherKnownHosts.connectionKey() );
auto restricted = base;
restricted.hostKeyAlgorithms = std::string( "ssh-ed25519" );
EXPECT_TRUE( base.connectionKey() != restricted.connectionKey() );
auto noPassword = base;
noPassword.passwordAuthentication = false;
EXPECT_TRUE( base.connectionKey() != noPassword.connectionKey() );
}
#if EE_PLATFORM != EE_PLATFORM_WIN
namespace {
@@ -66,6 +66,16 @@ class ScopedTestDirectory {
} // namespace
UTEST( String, icompare ) {
EXPECT_EQ( 0, String::icompare( "Readme.MD", "readme.md" ) );
EXPECT_TRUE( String::icompare( "alpha", "Beta" ) < 0 );
EXPECT_TRUE( String::icompare( "Beta", "alpha" ) > 0 );
EXPECT_TRUE( String::icompare( "abc", "ABCD" ) < 0 );
EXPECT_TRUE( String::icompare( "", "a" ) < 0 );
// Bytes above ASCII compare unsigned, as std::string::compare does.
EXPECT_TRUE( String::icompare( "z", "\xC3\xA1" ) < 0 );
}
UTEST( String, countLines ) {
EXPECT_EQ( static_cast<size_t>( 0 ), String::countLines( "" ) );
EXPECT_EQ( static_cast<size_t>( 1 ), String::countLines( "A" ) );
+30
View File
@@ -1,5 +1,6 @@
#include "utest.hpp"
#include <cstdlib>
#include <eepp/system/filesystem.hpp>
#include <eepp/system/sys.hpp>
@@ -46,3 +47,32 @@ UTEST( SystemPath, workingDirectoryRoundTripsUnicode ) {
EXPECT_STDSTREQ( FileSystem::getRealPath( temp.path ),
FileSystem::getCurrentWorkingDirectory() );
}
#if EE_PLATFORM == EE_PLATFORM_LINUX || EE_PLATFORM == EE_PLATFORM_BSD
UTEST( SystemPath, cachePathFollowsXDG ) {
auto restore = []( const char* name, const char* value ) {
if ( value )
::setenv( name, value, 1 );
else
::unsetenv( name );
};
const char* xdg = std::getenv( "XDG_CACHE_HOME" );
std::string savedXdg( xdg ? xdg : "" );
const char* home = std::getenv( "HOME" );
std::string savedHome( home ? home : "" );
::setenv( "XDG_CACHE_HOME", "/xdg/cache", 1 );
::setenv( "HOME", "/home/someone", 1 );
EXPECT_STDSTREQ( "/xdg/cache/app", Sys::getCachePath( "app" ) );
// The specification ignores relative and empty values.
::setenv( "XDG_CACHE_HOME", "relative", 1 );
EXPECT_STDSTREQ( "/home/someone/.cache/app", Sys::getCachePath( "app" ) );
::unsetenv( "XDG_CACHE_HOME" );
EXPECT_STDSTREQ( "/home/someone/.cache/app", Sys::getCachePath( "app" ) );
restore( "XDG_CACHE_HOME", xdg ? savedXdg.c_str() : nullptr );
restore( "HOME", home ? savedHome.c_str() : nullptr );
}
#endif
+38
View File
@@ -196,6 +196,44 @@ UTEST( VFSDirectoryModel, staysLiveThroughWatchEvents ) {
EXPECT_EQ( 3u, model->rowCount() );
}
UTEST( VFSDirectoryModel, watchEventsKeepRowsInStepWithHiddenEntries ) {
// Rows are updated in place by watch events: hidden and filtered entries in between must
// keep the visible rows pointing at the right entries.
TempDir tmp;
VirtualFileSystem vfs;
for ( const char* name : { ".a-hidden", "b-keep.txt", "c-other.txt", ".d-hidden" } )
FileSystem::fileWrite( tmp.path + name, std::string_view( "1" ) );
TestLoop loop;
VFSDirectoryModel::Config config;
config.vfs = &vfs;
auto model = VFSDirectoryModel::New( loop, config );
model->setURI( tmp.uri() );
EXPECT_TRUE( loop.runUntil(
[&] { return model->loadState() == VFSDirectoryModel::LoadState::Ready; } ) );
model->setNameFilter( "keep", VFSDirectoryModel::NameFilterMode::PlainText );
EXPECT_EQ( 1u, model->rowCount() );
for ( const char* name : { ".0-hidden", "a-keep.txt", "bb-other.txt", "z-KEEP.txt" } )
FileSystem::fileWrite( tmp.path + name, std::string_view( "1" ) );
FileSystem::fileRemove( tmp.path + ".a-hidden" );
EXPECT_TRUE( loop.runUntil( [&] {
return model->rowOf( "a-keep.txt" ) && model->rowOf( "z-KEEP.txt" ) &&
model->rowCount() == 3;
} ) );
FileSystem::fileRemove( tmp.path + "b-keep.txt" );
EXPECT_TRUE( loop.runUntil( [&] { return !model->rowOf( "b-keep.txt" ); } ) );
auto names = modelNames( *model );
ASSERT_EQ( 2u, names.size() );
EXPECT_STDSTREQ( "a-keep.txt", names[0] );
EXPECT_STDSTREQ( "z-KEEP.txt", names[1] );
// The same rows as a full rebuild.
model->setNameFilter( "" );
model->setNameFilter( "keep", VFSDirectoryModel::NameFilterMode::PlainText );
EXPECT_EQ( 2u, model->rowCount() );
EXPECT_STDSTREQ( "a-keep.txt", model->entry( 0 ).name );
EXPECT_STDSTREQ( "z-KEEP.txt", model->entry( 1 ).name );
}
UTEST( MainThreadExecutor, runsDeferredOnSceneUpdate ) {
UIApplication app(
WindowSettings( 320, 240, "eepp - MainThreadExecutor Test", WindowStyle::Default,
+311 -1
View File
@@ -2,6 +2,7 @@
#include "vfs_test_utils.hpp"
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <eepp/system/directorypack.hpp>
#include <eepp/system/filesystem.hpp>
#include <eepp/system/packfileindex.hpp>
@@ -13,7 +14,8 @@
#include <thread>
#if EE_PLATFORM != EE_PLATFORM_WIN
#include <unistd.h> // symlink
#include <sys/stat.h> // umask
#include <unistd.h> // symlink, link
#endif
using namespace EE;
@@ -645,3 +647,311 @@ UTEST( VFSIOStream, synchronousAdapterReadsSeeksAndWrites ) {
EXPECT_FALSE( missing.isOpen() );
EXPECT_TRUE( missing.error().has_value() );
}
// ---------------------------------------------------------------------------------------------
// Data safety: copies, moves and saves never lose existing data
// ---------------------------------------------------------------------------------------------
namespace {
/** Every read fails: a copy from it fails after the destination was opened. */
class FailingReadHandle : public FileHandleImpl {
public:
VFSTask<size_t> read( Uint64, std::span<Uint8>, CancellationToken ) override {
return readyError<size_t>( VFSErrorCode::IO, URI(), "injected read failure" );
}
VFSTask<size_t> write( Uint64, std::span<const Uint8>, CancellationToken ) override {
return readyError<size_t>( VFSErrorCode::IO, URI() );
}
VFSTask<void> truncate( Uint64 ) override { return readyValue<void>(); }
VFSTask<void> flush() override { return readyValue<void>(); }
VFSTask<FileInfo> stat() override { return readyError<FileInfo>( VFSErrorCode::IO, URI() ); }
VFSTask<void> close() override { return readyValue<void>(); }
ProviderCapabilities capabilities() const override { return {}; }
};
class FailingReadProvider : public MemoryProvider {
public:
FailingReadProvider() : MemoryProvider( "fault" ) {}
VFSTask<FileHandle> open( const URI& uri, const OpenOptions&, CancellationToken ) override {
return readyValue<FileHandle>( uri, std::make_shared<FailingReadHandle>() );
}
};
class NonAtomicMemoryProvider : public MemoryProvider {
public:
ProviderCapabilities capabilities( const URI& uri ) const override {
auto caps = MemoryProvider::capabilities( uri );
return caps.set( VFSCapability::AtomicReplace, false );
}
};
/** Publishes another application's destination after the transfer's existence check. */
class RacingReadHandle : public FailingReadHandle {
public:
explicit RacingReadHandle( std::shared_ptr<MemoryProvider> destination ) :
mDestination( std::move( destination ) ) {}
VFSTask<size_t> read( Uint64 offset, std::span<Uint8> data, CancellationToken ) override {
if ( offset != 0 )
return readyValue<size_t>( 0 );
mDestination->putFile( URI( "mem://t/destination" ), "concurrently created data" );
data[0] = 'x';
return readyValue<size_t>( 1 );
}
private:
std::shared_ptr<MemoryProvider> mDestination;
};
class RacingReadProvider : public MemoryProvider {
public:
explicit RacingReadProvider( std::shared_ptr<MemoryProvider> destination ) :
MemoryProvider( "race" ), mDestination( std::move( destination ) ) {}
VFSTask<FileHandle> open( const URI& uri, const OpenOptions&, CancellationToken ) override {
return readyValue<FileHandle>( uri, std::make_shared<RacingReadHandle>( mDestination ) );
}
private:
std::shared_ptr<MemoryProvider> mDestination;
};
} // namespace
UTEST( VFSSafety, unsafeReplacementIsRefusedWithoutAtomicReplace ) {
VirtualFileSystem vfs;
auto destination = std::make_shared<NonAtomicMemoryProvider>();
auto source = std::make_shared<FailingReadProvider>();
vfs.registerProvider( destination );
vfs.registerProvider( source );
source->putFile( URI( "fault://t/source" ), "new content" );
destination->putFile( URI( "mem://t/destination" ), "valuable data" );
CopyOptions options;
options.overwrite = OverwritePolicy::Overwrite;
auto result = waitFor(
vfs.copy( URI( "fault://t/source" ), URI( "mem://t/destination" ), options ).completion() );
ASSERT_FALSE( result.hasValue() );
EXPECT_EQ( VFSErrorCode::Unsupported, result.error().code );
EXPECT_STDSTREQ( "valuable data", *destination->fileContent( URI( "mem://t/destination" ) ) );
// A new file still needs only a no-replace rename, not atomic replacement support.
auto memory = std::make_shared<MemoryProvider>( "source" );
memory->putFile( URI( "source://t/file" ), "new content" );
vfs.registerProvider( memory );
EXPECT_TRUE(
waitFor( vfs.copy( URI( "source://t/file" ), URI( "mem://t/new" ) ).completion() ) );
EXPECT_STDSTREQ( "new content", *destination->fileContent( URI( "mem://t/new" ) ) );
}
UTEST( VFSSafety, concurrentDestinationRespectsFailAndSkip ) {
for ( auto policy : { OverwritePolicy::Fail, OverwritePolicy::Skip } ) {
VirtualFileSystem vfs;
auto destination = std::make_shared<MemoryProvider>();
auto source = std::make_shared<RacingReadProvider>( destination );
source->putFile( URI( "race://t/source" ), "x" );
vfs.registerProvider( destination );
vfs.registerProvider( source );
CopyOptions options;
options.overwrite = policy;
auto result =
waitFor( vfs.move( URI( "race://t/source" ), URI( "mem://t/destination" ), options )
.completion() );
if ( policy == OverwritePolicy::Fail ) {
ASSERT_FALSE( result.hasValue() );
EXPECT_EQ( VFSErrorCode::AlreadyExists, result.error().code );
} else {
ASSERT_TRUE( result.hasValue() );
EXPECT_EQ( 1u, result.value().filesSkipped );
EXPECT_EQ( 0u, result.value().sourcesRemoved );
}
EXPECT_STDSTREQ( "concurrently created data",
*destination->fileContent( URI( "mem://t/destination" ) ) );
EXPECT_TRUE( source->fileContent( URI( "race://t/source" ) ).has_value() );
}
}
UTEST( VFSSafety, failedOverwriteKeepsTheDestination ) {
VirtualFileSystem vfs;
auto mem = std::make_shared<MemoryProvider>();
auto fault = std::make_shared<FailingReadProvider>();
vfs.registerProvider( mem );
vfs.registerProvider( fault );
fault->putFile( URI( "fault://t/source" ), "new content" );
mem->putFile( URI( "mem://t/file" ), "valuable data" );
CopyOptions options;
options.overwrite = OverwritePolicy::Overwrite;
auto result = waitFor(
vfs.copy( URI( "fault://t/source" ), URI( "mem://t/file" ), options ).completion() );
EXPECT_FALSE( result.hasValue() );
EXPECT_STDSTREQ( "valuable data", readString( vfs, URI( "mem://t/file" ) ) );
// No temporary is left behind.
auto listed = waitFor( vfs.listAll( URI( "mem://t/" ) ) );
ASSERT_TRUE( listed.hasValue() );
EXPECT_EQ( 1u, listed.value().size() );
}
UTEST( VFSSafety, writeAllCreateParentsWithOneOrchestrationThread ) {
// writeAll() runs on the orchestration pool: creating the parents must not wait on another
// job of that pool.
VirtualFileSystem::Config config;
config.registerDefaultProviders = false;
config.orchestrationThreads = 1;
VirtualFileSystem vfs( config );
vfs.registerProvider( std::make_shared<MemoryProvider>() );
WriteOptions options;
options.createParents = true;
auto task = vfs.writeAll( URI( "mem://t/new/dir/file" ), std::string_view( "data" ), options );
ASSERT_EQ( TaskState::Ready, task.waitFor( std::chrono::seconds( 5 ) ) );
EXPECT_TRUE( std::move( task ).takeValue().hasValue() );
EXPECT_STDSTREQ( "data", readString( vfs, URI( "mem://t/new/dir/file" ) ) );
}
#if EE_PLATFORM != EE_PLATFORM_WIN
UTEST( VFSSafety, copyOntoAHardLinkKeepsTheSource ) {
TempDir tmp;
VirtualFileSystem vfs;
URI source = tmp.uri().withAppendedSegment( "file" );
URI alias = tmp.uri().withAppendedSegment( "hardlink" );
ASSERT_TRUE( waitFor( vfs.writeAll( source, std::string_view( "valuable data" ) ) ) );
ASSERT_EQ( 0, ::link( source.getFSPath().c_str(), alias.getFSPath().c_str() ) );
CopyOptions options;
options.overwrite = OverwritePolicy::Overwrite;
waitFor( vfs.copy( source, alias, options ).completion() );
EXPECT_STDSTREQ( "valuable data", readString( vfs, source ) );
EXPECT_STDSTREQ( "valuable data", readString( vfs, alias ) );
}
UTEST( VFSSafety, copyOntoASymlinkToTheSourceFails ) {
TempDir tmp;
VirtualFileSystem vfs;
URI source = tmp.uri().withAppendedSegment( "file" );
URI alias = tmp.uri().withAppendedSegment( "link" );
ASSERT_TRUE( waitFor( vfs.writeAll( source, std::string_view( "valuable data" ) ) ) );
ASSERT_EQ( 0, ::symlink( source.getFSPath().c_str(), alias.getFSPath().c_str() ) );
CopyOptions options;
options.overwrite = OverwritePolicy::Overwrite;
auto result = waitFor( vfs.copy( source, alias, options ).completion() );
EXPECT_EQ( VFSErrorCode::InvalidOperation, codeOf( result ) );
EXPECT_STDSTREQ( "valuable data", readString( vfs, source ) );
}
UTEST( VFSSafety, movingADirectorySymlinkAcrossProvidersKeepsItsTarget ) {
TempDir tmp;
VirtualFileSystem vfs;
vfs.registerProvider( std::make_shared<MemoryProvider>() );
URI real = tmp.uri().withAppendedSegment( "real" );
URI link = tmp.uri().withAppendedSegment( "link" );
ASSERT_TRUE( waitFor( vfs.makeDirs( real ) ) );
ASSERT_TRUE(
waitFor( vfs.writeAll( real.withAppendedSegment( "file" ), std::string_view( "data" ) ) ) );
ASSERT_EQ( 0, ::symlink( real.getFSPath().c_str(), link.getFSPath().c_str() ) );
auto result = waitFor( vfs.move( link, URI( "mem://t/moved" ) ).completion() );
EXPECT_EQ( VFSErrorCode::Unsupported, codeOf( result ) );
EXPECT_STDSTREQ( "data", readString( vfs, real.withAppendedSegment( "file" ) ) );
EXPECT_TRUE( waitFor( vfs.stat( link, StatOptions{ false } ) ).hasValue() );
}
UTEST( VFSSafety, atomicSaveKeepsTheExactMode ) {
TempDir tmp;
VirtualFileSystem vfs;
URI file = tmp.uri().withAppendedSegment( "file" );
ASSERT_TRUE( waitFor( vfs.writeAll( file, std::string_view( "v1" ) ) ) );
ASSERT_TRUE( waitFor( vfs.setPermissions( file, 0777 ) ) );
mode_t mask = ::umask( 0077 );
auto saved = waitFor( vfs.writeAll( file, std::string_view( "v2" ) ) );
::umask( mask );
ASSERT_TRUE( saved.hasValue() );
EXPECT_EQ( 0777u, *saved.value().permissions() );
EXPECT_STDSTREQ( "v2", readString( vfs, file ) );
}
#endif
UTEST( VFSWatch, localRegistrationRunsOnThePool ) {
// Registering touches the filesystem (slow mounts block): it must not run on the caller's
// (UI) thread. With the backend's pool busy, watch() returns a pending task.
TempDir tmp;
auto pool = ThreadPool::createShared( 1 );
std::mutex gate;
std::unique_lock<std::mutex> hold( gate );
pool->run( [&gate] { std::lock_guard<std::mutex> wait( gate ); } );
auto backend = std::make_shared<LocalWatchBackend>( pool );
WatchManager manager;
manager.registerBackend( "file", backend );
auto task = manager.watch( tmp.uri() );
EXPECT_EQ( TaskState::Pending, task.state() );
hold.unlock();
ASSERT_EQ( TaskState::Ready, task.waitFor( std::chrono::seconds( 5 ) ) );
auto handle = std::move( task ).takeValue();
ASSERT_TRUE( handle.hasValue() );
EXPECT_EQ( WatchStatus::Active, handle.value().status() );
handle.value().reset();
}
namespace {
/** open() blocks until released: a transfer reading from it keeps its thread busy. */
class GatedProvider : public MemoryProvider {
public:
GatedProvider() : MemoryProvider( "gated" ) {}
VFSTask<FileHandle> open( const URI& uri, const OpenOptions& options,
CancellationToken token ) override {
{
std::unique_lock<std::mutex> lock( mMutex );
mOpened = true;
mChanged.notify_all();
mChanged.wait( lock, [this] { return mReleased; } );
}
return MemoryProvider::open( uri, options, std::move( token ) );
}
void waitOpened() {
std::unique_lock<std::mutex> lock( mMutex );
mChanged.wait_for( lock, std::chrono::seconds( 5 ), [this] { return mOpened; } );
}
void release() {
std::lock_guard<std::mutex> lock( mMutex );
mReleased = true;
mChanged.notify_all();
}
private:
std::mutex mMutex;
std::condition_variable mChanged;
bool mOpened{ false };
bool mReleased{ false };
};
} // namespace
UTEST( VFSSafety, transfersDoNotDelayOtherOperations ) {
// Copies run on their own pool: with every transfer thread busy, saves still complete.
VirtualFileSystem::Config config;
config.registerDefaultProviders = false;
config.orchestrationThreads = 1;
config.transferThreads = 1;
VirtualFileSystem vfs( config );
auto mem = std::make_shared<MemoryProvider>();
auto gated = std::make_shared<GatedProvider>();
vfs.registerProvider( mem );
vfs.registerProvider( gated );
gated->putFile( URI( "gated://t/big" ), std::string( 1024, 'x' ) );
Transfer transfer = vfs.copy( URI( "gated://t/big" ), URI( "mem://t/copy" ) );
gated->waitOpened();
auto saved = vfs.writeAll( URI( "mem://t/other" ), std::string_view( "data" ) );
EXPECT_EQ( TaskState::Ready, saved.waitFor( std::chrono::seconds( 5 ) ) );
gated->release();
auto done = transfer.completion();
ASSERT_EQ( TaskState::Ready, done.waitFor( std::chrono::seconds( 5 ) ) );
EXPECT_TRUE( std::move( done ).takeValue().hasValue() );
}
+25 -6
View File
@@ -417,13 +417,32 @@ int App::run( int argc, char* argv[] ) {
if ( !init( locations.Get(), split.Get() ) )
return EXIT_FAILURE;
int result = mApp->run();
// Producers stop before the executor they deliver to goes away: continuations hold it by
// reference. Running transfers are cancelled (destroying the VFS waits for them).
for ( auto& transfer : mTransfers )
transfer.second.transfer.cancel();
mTransfers.clear();
for ( auto& pane : mPanes )
if ( pane )
pane->tabList.clear();
mExecutor.reset();
mPool.reset();
#ifdef EE_VFS_SFTP
if ( mSSHPrompts )
mSSHPrompts->shutdown();
#endif
// The scene is still alive: quit() stops the application loop without destroying it.
// Detach before waiting for producers, so late completions cannot schedule more UI work.
mExecutor->shutdown();
#ifdef EE_VFS_SFTP
if ( auto sftp = std::dynamic_pointer_cast<SFTPProvider>( mVFS->provider( "sftp" ) ) )
sftp->disconnectAll();
#endif
VirtualFileSystem::destroyInstance();
// The UI thread pool belongs to the scene too: work finishing while it is joined finds the
// executor detached (its functions are dropped), not destroyed.
mSystemIcons.reset();
mPool.reset();
mApp.reset();
mExecutor.reset();
return result;
}
@@ -625,8 +644,9 @@ bool App::init( const std::vector<std::string>& locations, bool forceSplit ) {
} );
mApp->getWindow()->setCloseRequestCallback( [this]( EE::Window::Window* ) {
saveState();
return true;
quit();
// Keep the scene and its dispatcher alive until producers and the executor stop.
return false;
} );
// The keyboard belongs to the files: a click on a pane's bars or empty areas focuses the
@@ -753,9 +773,8 @@ void App::saveState() {
}
void App::quit() {
// Closing the window programmatically skips the close request callback.
saveState();
mApp->getWindow()->close();
mApp->requestQuit();
}
void App::rememberViewState( FileTab* tab ) {
+17 -2
View File
@@ -65,12 +65,22 @@ void showError( UISceneNode* ui, const String& message ) {
#ifdef EE_VFS_SFTP
void SSHPrompts::shutdown() {
std::lock_guard<std::mutex> lock( mMutex );
mExecutor = nullptr;
}
Task<SSHHostKeyDecision> SSHPrompts::verify( const SSHHostKeyInfo& info ) {
auto source = std::make_shared<TaskSource<SSHHostKeyDecision>>();
auto task = source->task();
std::lock_guard<std::mutex> lock( mMutex );
if ( !mExecutor ) {
source->resolve( SSHHostKeyDecision::Reject );
return task;
}
UISceneNode* ui = mUI;
// If the executor drops this (shutdown) the source is abandoned and the key rejected.
mExecutor.execute( [ui, info, source] {
mExecutor->execute( [ui, info, source] {
String message = String::format(
ui->i18n( "eefm_unknown_host_key",
"The authenticity of host '%s' (port %d) can't be established.\n%s key "
@@ -92,8 +102,13 @@ SSHPrompts::request( const SSHAuthenticationRequest& request ) {
using Answers = std::optional<std::vector<std::string>>;
auto source = std::make_shared<TaskSource<Answers>>();
auto task = source->task();
std::lock_guard<std::mutex> lock( mMutex );
if ( !mExecutor ) {
source->resolve( std::nullopt );
return task;
}
UISceneNode* ui = mUI;
mExecutor.execute( [ui, request, source] {
mExecutor->execute( [ui, request, source] {
String title = ui->i18n( "eefm_authentication", "Authentication" ) + " - " +
String::fromUtf8( request.user + "@" + request.host );
// Prompts are asked one after another; any cancel aborts the whole request.
+7 -2
View File
@@ -9,6 +9,7 @@
#include <eepp/vfs/sftp/sftpprovider.hpp>
#endif
#include <functional>
#include <mutex>
#include <optional>
#include <string>
@@ -34,7 +35,10 @@ void showError( UISceneNode* ui, const String& message );
* I/O thread; dialogs are always created on the UI thread through the executor. */
class SSHPrompts : public SSHHostKeyVerifier, public SSHAuthenticationHandler {
public:
SSHPrompts( UISceneNode* ui, Executor& uiExecutor ) : mUI( ui ), mExecutor( uiExecutor ) {}
SSHPrompts( UISceneNode* ui, Executor& uiExecutor ) : mUI( ui ), mExecutor( &uiExecutor ) {}
/** Refuses late requests before the executor and scene are destroyed. */
void shutdown();
Task<SSHHostKeyDecision> verify( const SSHHostKeyInfo& info ) override;
@@ -43,7 +47,8 @@ class SSHPrompts : public SSHHostKeyVerifier, public SSHAuthenticationHandler {
private:
UISceneNode* mUI;
Executor& mExecutor;
std::mutex mMutex;
Executor* mExecutor;
};
#endif
+16 -4
View File
@@ -2,6 +2,7 @@
#include "apps.hpp"
#include "mime.hpp"
#include "trash.hpp"
#include <eepp/system/sha1.hpp>
namespace eefm {
@@ -86,6 +87,15 @@ void App::openRow( FileTab* tab, size_t row ) {
namespace {
/** Where remote files are downloaded to be opened: private to the user (the shared /tmp is
* not, on Linux and BSD). */
std::string remoteCacheDirectory() {
std::string cache = Sys::getCachePath( "eefm" );
FileSystem::dirAddSlashAtEnd( cache );
cache += "remote/";
return cache;
}
/** Calls done( localPaths ) with local copies of uris (remote files are downloaded to a cache
* folder first). */
void withLocalFiles( VirtualFileSystem* vfs, Executor& executor, UISceneNode* ui,
@@ -100,10 +110,12 @@ void withLocalFiles( VirtualFileSystem* vfs, Executor& executor, UISceneNode* ui
continue;
}
++*remaining;
std::string cache = Sys::getTempPath();
FileSystem::dirAddSlashAtEnd( cache );
cache += "eefm/" + uris[i].getScheme() + "/" + uris[i].getHost() + "/";
FileSystem::makeDir( cache, true );
// One private folder per remote file (its complete URI: user, port and path included),
// so files with the same name never overwrite each other; the file keeps its name for
// the application that opens it.
std::string cache =
remoteCacheDirectory() + SHA1::fromString( uris[i].toString() ).toHexString() + "/";
FileSystem::makeDir( cache, true, 0700 );
std::string local = cache + uris[i].getLastPathSegment();
CopyOptions options;
options.overwrite = OverwritePolicy::Overwrite;
+2
View File
@@ -243,6 +243,8 @@ class FileCell : public UITableCell {
Image preview = snapshot( dragLabel( getUISceneNode() ) );
if ( preview.getPixels() )
offer.preview.image = &preview;
// Drops on eefm's own window complete as eefm's drag (see endNativeDrag()).
offer.sourceHandlesDrops = true;
auto result = drag.startDrag( offer, []( DragResult result ) {
FileCell* cell = sNativeDragCell;
sNativeDragCell = nullptr;