Commit Graph

461 Commits

Author SHA1 Message Date
Hocuri
e3e2adeea5 Fix all outgoing messages popping up in selfchat (#2456)
Fix https://github.com/deltachat/deltachat-android/issues/1940, fix https://github.com/deltachat/deltachat-core-rust/issues/2220 (I assume these are the same bug)

The problem was:
- Gmail adds a header `Bcc: <self-address>` to our bcc-self message
- `to_id` was just set to the first recipient, in this case _self_
- it was seen that `to_id` is _self_, so it's a self-sent message
2021-05-28 20:02:46 +02:00
link2xt
adac903818 Debloat the binary by using less AsRef arguments
Using `impl AsRef<str>` as the argument instead of `&str` makes it
possible to call the function with `&str`, `String` and other types
that implement `AsRef` trait.

The cost of it is that compiled binary contains mulitple versions of
the same function, one for each variant of types. If function contains
multiple generic `impl AsRef` arguments, the number of versions possibly
compiled into binary grows exponentially with the number of arguments.

Simple way to avoid it is to call `.as_ref()` on the caller side to
convert the argument to `&str`. In most cases even adding a `&` and
relying on `Deref` coercion is sufficient.

This patch changes many functions that accepted `impl AsRef<str>` and
`impl AsRef<Path>` to accept `&str` and `&Path` instead.

In some places `.clone()` calls are removed. Calling `.clone()` on
`String` and passing `String` to a function accepting `impl
AsRef<str>` is completely unnecessary as `&str` reference could be
passed instead. There is no clippy warning against it yet, but
changing argument type to `&str` allowed to find these cases.

The result of debloating is not impressive, several hundred kilobytes
are saved, which is about 3% of the `.so` binary, but the code is
cleaner too.
2021-05-09 16:25:11 +03:00
link2xt
30c463e0ba Add extension to avatars base64-encoded in headers 2021-05-03 01:32:37 +02:00
link2xt
bba437523a Process Chat-User-Avatar headers with embedded base64 images 2021-05-01 07:40:56 +03:00
link2xt
f76bc44cdc mimeparser: parse hidden headers 2021-05-01 07:40:56 +03:00
link2xt
8610b0c945 sql: switch from sqlx to rusqlite 2021-04-25 22:33:14 +03:00
Hocuri
179a2a50e6 Parse <blockquote> tags for better quote detection (#2313) 2021-04-07 18:45:00 +02:00
Friedel Ziegelmayer
6bb5721f29 feat: improve internal sql interface
Switches from rusqlite to sqlx to have a fully async based interface
to sqlite.

Co-authored-by: B. Petersen <r10s@b44t.com>
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
2021-04-06 16:06:11 +02:00
B. Petersen
6963fd877d allow one additional boolean
the function is not complex and is only called once.
refactoring that seems to be a bit too much effort
and at least out of scope of this pr.
2021-03-12 11:30:25 +01:00
B. Petersen
045fbab7cd remove subsequent images inside multipart/related 2021-03-12 11:30:25 +01:00
B. Petersen
f69bcc71ed mark child-parts of multipart/related as such 2021-03-12 11:30:25 +01:00
B. Petersen
4ae86b4e61 truncate long texts and make the whole text accessible via has_html()/get_html() 2021-03-04 23:26:02 +01:00
B. Petersen
e418d89c79 add failing test to test truncating 2021-03-04 23:26:02 +01:00
link2xt
a698a8dd84 Update to Rust 1.50
Also run rustfmt, fix new clippy warnings.
2021-03-01 16:48:33 +03:00
B. Petersen
5c2d6c22a0 remove additional text parts if we think they belong to a mailinglist footer 2021-02-28 21:03:39 +01:00
link2xt
f8755b505e Fix clippy::unnecessary_wraps warnings
This lint is enabled by default in 1.50 toolchain.
2021-02-25 12:35:45 +03:00
B. Petersen
0fa90a81e5 prefer X-Microsoft-Original-Message-ID, if set
outlooks SMTP-server change the Message-ID of messages
and put the original Message-ID to X-Microsoft-Original-Message-ID.

the changed Message-ID has some issues:

- outgoing messages with bcc_self enabled are shown twice
  as the self-copy got the changed Message-ID while the database uses
  the original one

- read receipts do not work as they refer to the changed message id

- in general, sender and recipient see different Message-IDs

the issues can be fixed by

(1) let all receivers use the original Message-ID

	this is what this pr is doing
    and this should fix all issues with delta-to-delta communication,
	including groups, group-images etc.
    there may be issues left in communication
	with other MUAs as they are using another Message-ID.

(2) ftr: updating the Message-ID in the database of the sender to the new one

	this requires bcc_self always enabled (which is not the case)
    and may also result easily in race conditions
    (Bob answers before Alice sees its self-sent message),
    however, has the advantage of better compatibility with other MUA.

if needed, the compatibility with other MUA could be improved by remembering
both Messages-IDs, maybe we could treat the modified as References or so,
however, i think, this could be part of another PR if we know better about
real, in the wild issues.
2021-02-19 00:25:46 +03:00
B. Petersen
e7c6667347 introduce MailinglistType for more exhaustive checking 2021-02-15 21:38:31 +01:00
B. Petersen
0b3fb9c0a3 prepend subject whenever we think sth. is a mailinglist, not only for a subset of mailing lists 2021-02-15 21:38:31 +01:00
link2xt
57841cdcc0 Remove &Context from Peerstate and MimeParser
Passing Context around explicitly removes the need for explicit lifetimes.
2021-02-14 14:19:55 +03:00
link2xt
a88893f262 clippy: fix needless_borrow 2021-02-14 00:55:23 +03:00
Floris Bruynooghe
46a3226e43 Bunch of feedback: renames, simple functions, no cow 2021-02-12 18:28:51 +01:00
Floris Bruynooghe
29f184b4c4 Strongly type stock strings
This changes the internal stock strings API to be more strongly typed,
ensuring that the caller can not construct the stock string in the
wrong way.

The old approach left it to the callers to figure out how a stock
string should be created, now each stock string has their specific
arguments and callers can not make mistakes.  In particular all the
subtleties and different ways of calling stock_system_msg() disappear.

This could not use a trait for stock strings, as this would not allow
having per-message typed arguments.  So we needed a type per message
with a custom method, only by convention this method is .stock_str().
The type is a enum without variants to avoid allowing someone to
create the type.

Sadly the fallback string and substitutions are still far away from
each other, but it is now only one place which needs to know how to
construct the string instead of many.
2021-02-12 18:28:51 +01:00
link2xt
0601b05cb7 Use footer as a contact status 2021-02-11 13:57:49 +03:00
B. Petersen
6aeda98c0a comment on the quoted-encded test 2021-02-08 22:06:31 +01:00
Hocuri
396ec131fc Implement receiving mailing lists (#1964)
* Copypaste-merge my old work

* Start implementing mailinglists the new way

* Create pseudo contact

* Fine-tune docs

* Remove some unnecessary changes

* style

* Make a stock str

* Fix a crash. Yes, this line caused a panic when reconfiguring on Android

(without a reasonable error log). Also, I could not receive any messages
anymore.

* rfmt

* Add tests and make them pass

* Even more tests

* rfmt

* Enhance test and fix bug

* Don't update the sender name when prefetching because maybe it's a mailing list

* Use an enum instead of numbers for the decision

* Don't remove anyone from mailing lists

* Fix bug in the regex

* Adjust error msg

* Compile error after rebase

* Correctly emit event

* Add dc_msg_is_mailing_list so that you can find out whether messages in the deaddrop belong to mailing lists.

* Add received headers to unit tests

* Comments, small tweaks

* Use dc_msg_get_override_sender_name instead of dc_msg_get_sender_name

* Add dc_msg_get_sender_first_name() because sometimes the first name was not correctly shown in mailing lists

* small fixes, don't let the user modify mailing list groups

* Hide contacts for addresses like noreply@github.com and reply+AEJ...@reply.github.com

When testing mailing lists, I noticed that sometimes a mailing list
contact got a name (like, hocuri <noreply@github.com>). It turned out
that ages ago, I had accidentally written an email to - in this example
- hocuri <noreply@github.com> and it had been added to the contacts
list.

This hides email addresses from the contacts list that are obviously not
meant to be written at and prevents updating the names.

* Comment, clippy

* Replace u32 with ChatId

* Resolve lost of small issues from the reviews

* remove dc_msg_get_sender_first_name

* add dc_msg_get_real_chat_id()

this allows to check if a contact request belongs to a mailing list
and to show name/avatar/whatever of the mailinglist.

another approach was to duplicate some chat-apis for messages
(eg. dc_msg_is_mailing_list()) however that would require far more new apis.

the idea to change the behavior of dc_msg_get_chat_id() would be
more clean, however, that easily breaks existing implementations
and can led to hard to find bugs.

* remove now unused Message.is_mailing_list()

* if a name for a mailing list is missing, use List-ID

* fix comment

* fix error message

* document how dc_get_chat_contacts() works for mailing lists

* refine decide api (#2185)

* add DC_DECIDE* constants to deltachat.h, tweak documentation

* use StartChat/Block/NotNow instead of Yes/Never/NotNow

* decide_on_contact_request works on ctx/msg-id

functions working on message-objects
usually do not read or write directly on the database.

therefore, decide_on_contact_request()
should not work with message-objects as well,
it is even a bit misleading, as eg. chat-id of the object is not modified.

instead, the function works on context,
similar to dc_send_msg(), dc_create_chat() and so on.
for now, i moved it to context, could maybe be part od MsgId.

* Update src/chatlist.rs

Co-authored-by: Hocuri <hocuri@gmx.de>

Co-authored-by: Hocuri <hocuri@gmx.de>

* refine documentation

* re-add accidentally deleted Param::MailingList

* remove pseudo-contact in domain @mailing.list

1. the pseudo-contact was added to the mailing list contacts,
   which is not needed.
   might be that we want to add the recent contacts there in a subsequent pr
   (we do not know all contacts of a mailing list)

2. the pseudo-contact was used to block
   mailing lists; this is done by setting the chat to Blocked::Manually now

3- the pseudo-contact was used for unblocking;
   as it is very neat not to require additional ui for mailing list unblocking,
   might be that we introduce a similar pseudo-contact for this limited purpose
   in a subsequent pr, however, the pseudo-contact needs to exist only
   during unblocking then, maybe also the special domain is not needed,
   we'll see :)

* Move dc_decide_on_contact_request() up to the dc_context section as it's a member of dc_context now

More specifically, to the "handle messages" section

* re-introduce Chattype::Mailinglist (#2195)

* re-introduce Chattype::Mailinglist

* exhaustive chattype-check in fetch_existing_msgs() and get_summary2()

* exhaustive chattype-check in ndn_maybe_add_info_msg()

* exhaustive chattype-check in message::fill()

* remove dc_chat_is_mailing_list() from ffi

Co-authored-by: B. Petersen <r10s@b44t.com>
2021-02-07 23:37:04 +01:00
bjoern
0ef6a3060a fix From:-header bug (#2193)
* add failing test for From:-header bug

the test fails on the last check for quote-encapsulated utf-8 strings.

* Update mailparse

Co-authored-by: link2xt <link2xt@testrun.org>
2021-02-05 20:57:38 +01:00
Floris Bruynooghe
355e0145c0 Depend on anyhow directly
This removes the proxy via crate::error to depend on anyhow directly.
There is no benefit to this indirection and this makes it simpler to
see which error types are used.
2021-01-24 17:29:52 +03:00
bjoern
e9c582c4e4 forward html-messages properly (#2151)
* add failing tests for forwarding html-mails

* let MsgId.get_html() return an option

* write html-part to local database on forwarding

* add html-part to forwarded non-dc messages

* read HTML-parts from encrypted messages

* avoid clone()

* Received:-header is no longer needed since #2152

* Update src/html.rs

Co-authored-by: Floris Bruynooghe <flub@devork.be>

* Update src/html.rs

Co-authored-by: Floris Bruynooghe <flub@devork.be>

* Update src/mimeparser.rs

Co-authored-by: Floris Bruynooghe <flub@devork.be>

* prefer 'orig' over 'org' as abbreviation for 'original'

* improve comment on tests

* prefer 'try_into()' over 'as u32' to avoid panics on bad data

* simplify ffi

Co-authored-by: Floris Bruynooghe <flub@devork.be>
2021-01-23 23:21:18 +01:00
Floris Bruynooghe
83dd1c6232 New version of clippy has a lot of new lints
Lots of new clippy lints due to toolchain upgrade.

Made the Message::error field pub(crate) again, it was the odd one out
and it seemed a reasonable way to shut up clippy.
2021-01-19 11:20:33 +01:00
Alexander Krotov
5e8e77dfb6 clippy: forbid wildcard_imports 2021-01-12 12:06:13 +03:00
bjoern
e2688f6355 add option to access original message (#2125)
* draft API to deal with uncut message texts

* add column mime_modified

* add mime_modified flag to MimeParser and save it in the database

* save mime_headers also when mime_modified is set

* cargo fmt

* set mime_modified on parsed html-texts and when there are multiple alternative-parts; add test for that

* prototype functions, add to repl and ffi

* use correct mime_modified flag

* basically parse Mime-Structure to HTML

* add basic tests for HTML-parsing

* convert text/plain to html for getting original

* respect charset for plain texts

* make test more specific

* fix handling non-utf-8 charsets for plain messages

* add test for plain_to_html()

* add failing test for plaintext linkify

* linkify urls in plain text

* fix regex

* plain text linkify: add failing test for encapsulated links as <https://domain.com>

* plain text linkify: make encapsulated links as <https://domain.com> work

* plain text linkify: require word boundary at beginning of link, add tests for that

* plain text linkify: linkify emails

* plain text: support format=flowed

* plain text: support quotes

* make clippy happy

* set mime-modified also when simplify() cuts non-html messages, add tests for that

* streamline mime recursion

* repl tool: write original html to file for further processing

* convert cid:- to data:-protocol

* add a test for cid: to data: conversion

* make clippy happy

* fix html-tests to work with windows-lineends

* clarify what the returned html-code may contain

* add some more detailed doc comments

* add mime_modified column only if not exist

this additional check is needed
as the column may added with another dbversion in
some shipped beta-versions.

* incorporate documentation suggestions from review

* rename get_original_mime_html() to more simple get_html()

* rename api is_mime_modified() to more simple has_html(); internally, mime_modified-flag stays as-is, however

* rename MimeS to MimeMultipartType

* do not set mime-modified flag for encrypted messages that need extra-handling for saved mime-structure

* fix typo

* move get_msg_html() to MsgId.get_html()

* incorporate more documentation suggestions from review

* remove unused return value from collect_texts_recursive()

* avoid mime_modified being mutable in write-parts-loop

* move 'use futures::future::FutureExt' atop of html.rs

* move attributes defining plain-text to a dedicated structure

* more PlainText to separate file

* escape cid when building regex

* let dc_get_msg_html() return NULL when calling with bad param
2021-01-11 17:40:35 +01:00
Alexander Krotov
f98aa0d906 Make maybe_remove_bad_parts() non-async 2020-12-22 20:51:57 +03:00
Alexander Krotov
179dd0f3a1 Fix "dehtlm_failed" typo 2020-12-22 20:51:42 +03:00
Hocuri
03232eb79c Adapt other tests to the fact that now the subject is prepended more often 2020-12-18 18:44:16 +01:00
Hocuri
9edc6702f1 Fix #2078 2020-12-18 18:44:16 +01:00
Hocuri
2b207e1375 rustfmt 2020-12-18 18:35:54 +01:00
Hocuri
4d2c2130e8 Replace all t.ctx with t
Not sure if that's the way to go because, you know, merge conflicts, but
we could do this now.
2020-12-18 18:35:54 +01:00
Hocuri
ec83fae314 Parse name="quote" divs (#2104)
fix #1560 Replies in html-only format are not converted nicely wrt Quoting
2020-12-13 18:02:20 +01:00
Alexander Krotov
518e87b0cf Compare Chat-Disposition-Notification-To: address case-insensitively
Otherwise users with uppercase letters in their addresses never receive
read receipts.
2020-12-06 23:29:11 +03:00
Alexander Krotov
fc1f44c6d6 Spellcheck 2020-12-06 04:26:20 +03:00
B. Petersen
f774665921 be more graceful when the apostrophed-encoding is used erroneous - either by other MUA or by Delta Chat up to core1.50 2020-12-06 04:24:39 +03:00
B. Petersen
8fcb8c3788 add failing test for erroneous apostrophed-encoding 2020-12-06 04:24:39 +03:00
B. Petersen
af045c245d warn about encoding errors; this requires Context 2020-12-04 12:25:45 +01:00
B. Petersen
0bdd1b7dc2 avoid unwrap() when falling back to latin1 charset 2020-12-04 11:42:14 +01:00
B. Petersen
e77706f7d0 add test apostrophed-encoding with cp1252 charset (aka ANSI aka Windows-1252) 2020-12-04 01:28:24 +01:00
B. Petersen
8de73c4566 handle apostrophed-encoding with non-utf-8 charsets 2020-12-04 01:19:03 +01:00
B. Petersen
56e6c2712b add failing test for apostroped windows-1251 encoding 2020-12-04 00:19:19 +01:00
B. Petersen
b510d74c4a add test for apostrophed-encoded filenames 2020-12-03 17:39:00 +01:00
B. Petersen
966712019f add test for bad encoded-words delimiter 2020-12-03 17:38:55 +01:00