This moves the loading of the keys from the database to the trait and
thus with types differing between public and secret keys. This
fetches the Config::ConfiguredAddr (configured_addr) directly from the
database in the SQL to simplify the API and consistency instead of
making this the responsiblity of all callers to get this right.
Since anyone invoking these methods also wants to be sure the keys
exist, move key generation here as well. This already simplifies some
code in contact.rs and will eventually replace all manual checks for
existing keys.
To make errors more manageable this gives EmailAddress it's own error
type and adds some conversions for it. Otherwise the general error
type leaks to far. The EmailAddress type also gets its ToSql trait impl
to be able to save it to the database directly.
CString::yolo was still used in the ffi, this was an unsafe
transitional thing. To remove it there were two choices: 1. make
errors in creating CStrings hard errors or 2. try and be as lenient as
possible. Given the to_string_lossy() convention adopted in the ffi
this choose the lenient option and simply skips over embedded null
bytes, leaving the rest of the strings intact.
Thus now CString::new_lossy(). It's only used for .strdup() however
so no longer a public trait.
This also cleans up the public visibility of things in the strings.rs
file:
- Rename StrExt/OptStrExt traits to what they actually do: provide
.strdup() -> Strdup/OptStrdup.
- dc_strdup() should be an implementation detail, replace all usages
with Strdup.strdup() method.
- Only allow visibility inside the crate for all things.
- Reduce visibility to only the module for things not used in lib.rs.
We already have a .strdup() method on AsRef<str>, this adds this
method also to an option of this. In case the option is None a NULL
pointer is returned.
This is done by using a new trait, as the type system otherwise
considers such an implementation conflicting with the existing one.
I had a look for two mintues at converting things to .try_inner() and
I don't think this is currently worth the effort or would increase
readability a lot. So let's leave this for now.
There is no need to have both a .log_warn() and .log_err(), even their
names are confusing by now. Let's just have the most liberal one and
reduce one more thing we need to know about.
Also, these traits don't need to be pub.
- Python should raise exceptions on error. Not return False.
- Negative durations are nonense.
- ChatID validity is already checked by the Rust API, let's not duplicate.
Configuring an online account generates a key, we would like this
code-path tested too. So add some functionality to the AccountManager
to not use the pre-generated keys.
Because this slows down interactively running the tests by hand add an
ignored marker which only runs if --ignored is used. This name was
chosen because this matches the naming used by rust/cargo #[ignored].
The difference however is that --ignored on cargo *only* runs ignored
tests while here it runs *all* tests.
To ensure the ignored/slow tests are run on CI we add it as an
argument to the tox configuration, which is used by the CI to run the
tests.
This changes the AccountMaker to use pre-generated keys when
available, speeding up test runs.
As a side-effect we no longer need to compile the integration tests in
release mode with debug symbols. Losing debug symbols (-g) means
cargo no longer wants to recompile everything all the time too.
Tested locally and seems to works.
The imex code was deleting the key if it was already used. Turns out
none of the callers would not want this behaviour, so let's just
handle this case generally.
I guess this is a little neater. Note that this does change all the
existing error logging to warning logging. I believe this is correct
as these error log messages are really programming errors rather than
errors which need to be show to the user.
Also chose to make entire with_inner unsafe, there is little to be
gained from hiding the unsafe as this is only used from unsafe
functions in the first place.
The user-visible change here is that it allows the FFI API to save
keys in the database for a context. This is primarily intended for
testing purposes as it allows you to get a key without having to
generate it.
Internally the most important change is to start using the
SignedPublicKey and SignedPrivateKey types from rpgp instead of
wrapping them into a single Key object. This allows APIs to be
specific about which they want instead of having to do runtime checks
like .is_public() or so. This means some of the functionality of the
Key impl now needs to be a trait.
A thid API change is to introduce the KeyPair struct, which binds
together the email address, public and private key for a keypair.
All these changes result in a bunch of cleanups, though more more
should be done to completely replace the Key type with the
SignedPublicKye/SignedPrivateKey + traits. But this change is large
enough already.
Testing-wise this adds two new keys which can be loaded from disk and
and avoids a few more key-generating tests. The encrypt/decrypt tests
are moved from the stress tests into the pgp tests and split up.
This return value was very complicated to understand. Some failure
returns were returned as Err and some as Ok with no consistency, but
resulting in the same behaviour.
This refactor makes the handle_securejoin_handshake the sole place
responsible for maintaining the state of the secure join
process (context.bob) and also in charge of terminating the ongoing
process. This is none of receive_imf's business.
The remaining returns are now cleanly classified in application-errors
and protocol errors:
Applications errors result in an Err and mean there is a bug or
something else serious went wrong, like database access suddenly
failed or so. In this case receive_imf is still responsible for
clearing the state and resetting ongoing-process. It may be possible
this should still be moved back to securejoin.rs so that recieve_imf
doesn't need to know anything about this either.
Protocol errors are not failures for receive_imf, it just means the
received message didn't follow the protocol. Receive_imf in this case
is told to ignore the message: that is hide it but not delete it.
Other Ok returns also only say what needs to happen to the message:
- It's fully processed and needs no further processing, instead should
be removed
- It should still be processed as a normal received message.
This changes some behaviour: if the chat creation/lookup for the
contact fails this is treated as an application error. Previously
this was silently ignored and send_msg() would be called with a 0
chat_id without checking the response. This resulted in the protocol
quitely being blocked.
This all shhould now make it easier to resultify more of the functions
called by this function, instead of having to deal with very
complicated application logic hidden in the return code.
The struct really represents a parsed MIME message and is not used as
a parser itself. After the from_bytes() call (which should arguably
use the FromStr trait instead) the struct is fully populated.
This is a new feature in Rust 1.40, it means users outside the crate
will not be able to create these structs, allowing us to add fields
without breaking the public API.
This is the first experiment towards using structs to define the API.
It adds it as a new method on the existing Chat struct.
The types in this struct could improve as well as many other things.
But this is a straight forward translation of the existing json! macro
into a struct.
This effectively reverts
https://github.com/deltachat/deltachat-core-rust/pull/964 for chat.rs,
which in that PR was thought to fix something. So maybe something is
still broken? But after improving tests the previous code seems to be
correct.
- Update Python bindings to not always use dc_prepare_msg path when
sending messages with attachements. When using dc_prepare_msg the
blobs need to be created in the blobdir since they will not get
copied and many tests where not doing this.
- Add a test that ensures that calling dc_prepare_msg with a
file **not** in the blobdir fails.
- Add a test that ensures that calling dc_send_msg directly with a
file **not** in the blobdir copies the file to the blobdir. This
test cheats a little by knowing what the filename in the blobdir
will be which is implementation-dependent and thus a bit brittle.
But for now it proves correct behaviour so let's go with this.
- Improve the test_forward_increation test to ensure that the
in-creation file only has it's final state before calling
dc_send_msg. This checks the correct file data is sent out and not
the preparing data, this fails with the chat.rs changes in
#964 (reverted here to make this work again). Also fix the test to
actually create the in-creation file in the blobdir.
- Fix test_send_file_twice_unicode_filename_mangling to not use
in-creation. It was not creating it's files in the blobdir and that
is an error when using in-creation and it didn't seem it was trying
to test something about the in-creation logic (which is tested in
test_increation.py already).
- Fix Message._msgtate code which presumably was not used before?
- Rename `BlobObject::create_from_path` to
`BlobObject::new_from_path`. All the `BlobObject::create*` calls
now always create new files which is much more consistent. APIs
should do what is obious.
This deletes a lot of code and complexity. Though comes at some cost:
- The type no longer fits in a register and will always be on the
stack.
- Constructing the errors is more verbose, no more auto Into casting.
The ongoing process of dc_join_securejoin() was stopped before the
corresponding chat was created. This resulted in a race-condition
between the sqlite threads executing the creation and query
statements, thus sometimes the query would not find the group and
mysteriously fail.
Tripple-programming with hpk & r10s.
Be more defensive: if somehow we got corrupt jobs in the database skip
over them rather than fail to do anything.
This only modifies the query_map() call, the rest is only split off
into it's own function to make it testable. Smaller functions are
good anyway.
When a new Selfavatar is set the file needs to be copied into the
blobdir and as a proper blob it should be stored in the database as
$BLOBDIR/$filename.
This should fix#868.
Double extensions are sometimes used to identify files correctly,
e.g. .tar.gz or .html.zip. Breaking those extensions is not very
nice.
This fixes#865.
Before we created an empty file and asked the OS to copy the file.
The OS is very good at this so this is a good idea generally. However
it seems that in some cases, possibly an Android Dowload folder, we
might be able to create a file but not overwrite it. Thus refactor
this a bit so we are copying the file ourselves.
There are no new tests here since the behaviour remains identical.
The good news is that the existing tests were good enough to catch
some bugs already.
This more strongly types the ubiquitous message id type by no longer
making it an integer. It keeps the actual ID opaque. Only for the
generic job API the number keeps being used. Some locations also need
to create it from an integer and call MsgId::new().
Turns out that anyone that uses these either justs wants a file or
wants a blob. Consolidate those patterns into one place and simplify
all the callers.
This creates a specific type for blobs, with well defined conversions
at the borders. It also introduces a strong type for the Param::File
value since that param is often used used by the public API to set
filenames using absolute paths, but then core changes the param to a
blob before it gets to the database.
This eliminates a few more functions with very mallable C-like
arguments behaviour which combine a number of operations in one.
Because blob filenames are stored so often in arbitrary strings this
does add more code when receiving those, until the storage is fixed.
File name sanitisation is now deletated to the sanitize-filename crate
which should do a slightly better job at this.
There is very little API guarantees about this, but clients seem to
expect *something* to work on a closed context. So split this up into
static info an more dynamic context-related info, but let's not
provide any guarantees about what keys are available when.
Fixes#599
This adopts the FFI API to use the new Context::new() and
Contest::drop() Rust-style APIs while preserving the semantics of the
existing FFI API. It introduces a wrapper around the context which
keeps an optional reference to the actual context, changing dc_open()
and dc_close() to create/destroy the actual reference.
This changes the events emitted on closing slightly: they only get
emitted when dc_close() was called while the context was open. While
dc_close() remains idempotent in that it can still be called safely
multiple times, the close events will only occur if the context is
actually closed.
A new context is now created by calling Context::new and therefore you
always have a valid context. This is much more in Rust style and will
allow a lot of furture simplifications on the context itself.
The FFI layer has not yet been adjusted in this commit and thus will
fail.
The Message struct had a reference to the context which made a few
APIs a little easier. However it has surprising consequences a long
way down the line as shown in #335: it means any object which has such
a reference needs to keep open a lock if we want to do this refactor
of no longer having a "closed" Context struct on the Rust API (which
has many benefits which will simply that Context struct and is more
the Rust way - RAII etc).
By refactoring away the context reference on the rust API as done in
here however, we push this behaviour of how these references are
handled back to the C-API pointer behaviour: that is unsafe but just
works in a C-like way. The resulting complexity in the FFI layer is
also notably less than in the #335 alternative.
As a consequence all APIs which require the context, now explicitly
need to get the context passed in as an argument. It looks like this
is certainly no downside and maybe even beneficial for further API
refactors.
For this strategy to work out the same should be done to
dc_chatlist_t, dc_chat_t and dc_contact_t. But this working for
dc_msg_t give a reasonable confidence that this is a good approach.
The function is made safe and now returns Result. Functionally it now
fails when it can not write the newly generated key to the database
whereas before it still returned the key but logged a warning. There
is no reason this shouldn't be able to store the key and silently not
storing the key may result in later operations assuming the key is
available, so failing seems like a better choice.
The function now also uses a proper mutex to guard against multiple
threads generating keys. And this mutex is Context-scoped rather than
fully global (static).
This removes the dc_context_unref function from the Rust API which was
just an alias for dc_close. It still exists on the C API where it
makes sure to free the memory.
It also implements Drop for the context which just calls dc_close to
make sure all the memory is freed. Since you can call dc_close as
many times as you like this ensures that at the Rust level you can't
Drop the struct without releasing the memory.
Finally since memory is now freed by dropping the struct this removes
the #[repr(C)] for the struct. This struct is fully opaque to the C
API.
This marks the function safe and returns Result, it also now returns
the ConfiguredAddr since it has to look this up anyway and it makes
testing more easy. Turns out it reduces some duplicate SQL query in
some callers too.
More test code has been moved from dc_imex to test_utils as it's
more genrally applicable.
Mostly safe string usage, but some other minor cleanups:
- This isn't used from any C code, so no extern C.
- Use Config enums rather than strings.
- Clean up old unused sql statements.
Add a trait for str.strdup() to replace to_cstring() which avoid the
signature ambiguity with .to_string().
Also instruduce CString::yolo() as a shortcut to
CString::new().unwrap() and use it whenever the variable does can be
deallocated by going out of scope. This is less error prone.
Use some Path.to_c_string() functions where possible.
This is writes pkg-config/deltachat.pc file in the target directory,
using the PREFIX environment variable at build time. If this is
undefined at build time /usr/local is used.
For a few of the locations where error handling is done correctly this
does the right thing. For most other places it gracefully ignores any
issues which is what the original code did as well. Errors are
already logged by the called functions in those cases.
- Pass extra_link_args when using an installed libdeltachat
- Allow setting the liveconfig by envvar
- Show lifeconfig path in the pytest summary line
- Pass required envvars through tox
- Fix broken liveconfig passing in run-integration-test.sh
This clears the way to start working on making the functions safe.
But small PRs are good PRs so let's get this rename out of the way and
have future PRs less noisy.
Also stop making this #[repr(C)] and start making fields that are not
used private. Lastly clean up some comments by moving them or
deleting them, so they make sense again after the translation.
This makes interactively running the tests a much more pleasant
experience rather than something one dreads. These tests will still
be run on the CI. To run these manually run:
cargo test [TESTNAME] -- --ignored
With the API changes already done, notably .is_open(), the
implementation of the sqlite can and should now be private.
This also doesn't need to be #[repr(C)] anymore since the C API does
not need to access the fields in the struct.
Experiment with refactoring the internal sql interface a bit. My
original goal was to modify the schema and thus refactor to a state
where it would be sane to write tests for dc_sqlite_open() (and/or
however it ends up refactored) to assert schame changes before/after.
The C API allows passing a NULL pointer is for the callback function.
However when calling the callback nothing checks for this null pointer
and thus things fail badly. Even worse since the C API is defined
using an "fn pointer" rather than a "*-ptr" or raw pointer to the
function rust does not realise this can be invalid and therefore the
typechecker does not catch this even though there are no unsafe casts.
Fix this by making the callback an Option in rust, this can be easily
checked when calling. Also add a Context.call_cb() function which
simplifies calling the callback, hides the weird syntax due to the
function pointer and makes the call a little easier. Finally it also
means the option checking is only needed in one place.
For the C API this needs to check if this is a NULL pointer or not,
this is implicitly done by rust using the "nullable pointer
optimisation":
https://doc.rust-lang.org/nomicon/ffi.html#the-nullable-pointer-optimization
This stops using the deprecated libc::uint32_t and libc::uint64_t
types in favour of the native u32 and u64 types.
It also uses a newer nightly compiler to get rid of the incorrect
"unused no_mangle" warning when compiling incrementally.
Finally the newer compiler prefers us to be explicit when implementing
traits using the new dyn keyword.
Floris Bruynooghe
