Mirrors/chatmail-core
1
0
Fork 0
mirror of https://github.com/chatmail/core.git synced 2026-04-05 23:22:11 +03:00
Code Issues Packages Projects Releases Wiki Activity

Squashed commit of the following:

Browse Source 
commit 6bc5d1b90e
Author: holger krekel <holger@merlinux.eu>
Date:   Sun Jul 21 22:56:37 2019 +0200

    fix fmt

commit 197d94ad9d
Merge: 7ce337c 686678c
Author: holger krekel <holger@merlinux.eu>
Date:   Sun Jul 21 22:51:16 2019 +0200

    Merge remote-tracking branch 'origin/master' into eventlogging

commit 7ce337c6d0
Author: holger krekel <holger@merlinux.eu>
Date:   Sun Jul 21 22:44:27 2019 +0200

    left-over error logging

commit 10148d2e43
Author: holger krekel <holger@merlinux.eu>
Date:   Sun Jul 21 22:03:17 2019 +0200

    ignore non-utf8 parts of header fields (add comment why it shouldn't happen)
    don't throw error if no sql rows are returned

commit 69dc237ee3
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Sun Jul 21 12:56:04 2019 +0200

    fix(receive_imf): remove recursive sql call

commit df5464ea80
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Sat Jul 20 17:05:24 2019 +0200

    fix: blocked is an optional value

commit e4bf9956a5
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Sat Jul 20 16:50:56 2019 +0200

    fix(msg): handle optional in_reply_to

commit d353d9d9d8
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Sat Jul 20 16:17:25 2019 +0200

    fix(chat): remove recursive sql usage

commit 1ad45ed4d6
Author: holger krekel <holger@merlinux.eu>
Date:   Sat Jul 20 15:14:11 2019 +0200

    fix rust fmt

commit 496e980a17
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Sat Jul 20 14:34:20 2019 +0200

    use forked rusqlite

commit fa09e46ed9
Author: holger krekel <holger@merlinux.eu>
Date:   Sat Jul 20 12:37:51 2019 +0200

    another pace where we might (and in my case did) get invalid utf8

commit d6de420b9a
Author: holger krekel <holger@merlinux.eu>
Date:   Sat Jul 20 12:30:48 2019 +0200

    fix some string issues, introduce to_string_lossy such that to_string() continues to panic on non-utf8

commit 38eb708db8
Author: holger krekel <holger@merlinux.eu>
Date:   Sat Jul 20 01:17:53 2019 +0200

    for now make to_string() less strict as we often don't want to crash the whole app just because some non-proper utf8 came in (through a message we can't neccesarily congtrol)

commit 7a59da5f8f
Author: holger krekel <holger@merlinux.eu>
Date:   Fri Jul 19 22:48:39 2019 +0200

    fix linting

commit f13a1d4a2f
Author: holger krekel <holger@merlinux.eu>
Date:   Fri Jul 19 22:46:58 2019 +0200

    fix some test flakyness

commit 7b3a450918
Author: holger krekel <holger@merlinux.eu>
Date:   Fri Jul 19 22:35:07 2019 +0200

    - fix saved_mime test which broke to improper conversion of
      imf_raw_not_terminated
    - some cargo.toml updates no clue where they come from
    - log Message-ID for received messages

commit 169923b102
Author: holger krekel <holger@merlinux.eu>
Date:   Fri Jul 19 12:31:22 2019 +0200

    formatting

commit 42688a0622
Author: holger krekel <holger@merlinux.eu>
Date:   Fri Jul 19 12:24:56 2019 +0200

    remove some print statements

commit 35f3c0edd1
Merge: e7a2362 f58b1d6
Author: holger krekel <holger@merlinux.eu>
Date:   Fri Jul 19 10:25:21 2019 +0200

    Merge branch 'master' into eventlogging

commit e7a236264a
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Thu Jul 18 23:20:20 2019 +0200

    print invalid strings

commit aaa5b820d9
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Thu Jul 18 23:12:35 2019 +0200

    cleanup

commit e7f0745010
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Thu Jul 18 23:03:57 2019 +0200

    reduce direc usage of CString

commit c68e7ae14e
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Thu Jul 18 22:47:47 2019 +0200

    audit use of to_cstring and fix ub

commit 618087e5a7
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Thu Jul 18 21:38:52 2019 +0200

    fix(imap): body ptr lifetime

commit 245abb8384
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Thu Jul 18 19:44:10 2019 +0200

    remove debug

commit a3e1042001
Author: dignifiedquire <dignifiedquire@users.noreply.github.com>
Date:   Thu Jul 18 18:30:54 2019 +0200

    fix some things, add more debugging statements

commit 7b7ce9348f
Author: holger krekel <holger@merlinux.eu>
Date:   Thu Jul 18 15:11:57 2019 +0200

    fix python lint issues

commit 7a4808ba0d
Author: holger krekel <holger@merlinux.eu>
Date:   Thu Jul 18 14:35:54 2019 +0200

    cargofmt

commit 8f240f7153
Author: holger krekel <holger@merlinux.eu>
Date:   Thu Jul 18 14:03:57 2019 +0200

    (dig,hpk) pull out job collection from sql query/lock logic

commit 7d0b5d8abb
Author: holger krekel <holger@merlinux.eu>
Date:   Thu Jul 18 12:52:02 2019 +0200

    remove print statements and fix a crash

commit ee317cb1b5
Author: holger krekel <holger@merlinux.eu>
Date:   Thu Jul 18 11:38:10 2019 +0200

    fix some merge issues

commit 7b736fe635
Author: holger krekel <holger@merlinux.eu>
Date:   Thu Jul 18 11:16:38 2019 +0200

    (dig,hpk) add test and fix for wrong dbs

commit c7db15352a
Merge: 0b37167 0c5015d
Author: holger krekel <holger@merlinux.eu>
Date:   Thu Jul 18 09:59:44 2019 +0200

    Merge branch 'master' into eventlogging

commit 0b37167be8
Author: holger krekel <holger@merlinux.eu>
Date:   Thu Jul 18 00:06:05 2019 +0200

    address @dignifiedquire comments

commit 5cac4b5076
Author: holger krekel <holger@merlinux.eu>
Date:   Wed Jul 17 12:47:22 2019 +0200

    remove spurious print

commit 475a41beb3
Author: holger krekel <holger@merlinux.eu>
Date:   Wed Jul 17 12:31:12 2019 +0200

    address @dignifiedquire rustyness comment and fix changelog

commit ad4be80b4e
Author: holger krekel <holger@merlinux.eu>
Date:   Wed Jul 17 10:25:25 2019 +0200

    make smtp/imap connect() return bool instead of c-int

commit 8737c1d142
Author: holger krekel <holger@merlinux.eu>
Date:   Wed Jul 17 09:26:33 2019 +0200

    cleanup some parts, add comments

commit 964fe466cc
Author: holger krekel <holger@merlinux.eu>
Date:   Tue Jul 16 20:05:41 2019 +0200

    wip-commit which passes all tests with proper finalization

commit 43936e7db7
Author: holger krekel <holger@merlinux.eu>
Date:   Tue Jul 16 16:17:42 2019 +0200

    snapshot of my current debugging state

commit 0e80ce9c39
Author: holger krekel <holger@merlinux.eu>
Date:   Tue Jul 16 12:57:19 2019 +0200

    more aggressively skip perform API when threads are closing

commit c652bae68a
Author: holger krekel <holger@merlinux.eu>
Date:   Tue Jul 16 12:06:05 2019 +0200

    intermediate wip commit

commit bc904a495d
Author: holger krekel <holger@merlinux.eu>
Date:   Tue Jul 16 11:18:56 2019 +0200

    add some logging, and a more precise teardown for online python tests

commit 8d99444c6a
Author: holger krekel <holger@merlinux.eu>
Date:   Tue Jul 16 00:22:12 2019 +0200

    fix std

commit 9dab53e0af
Author: holger krekel <holger@merlinux.eu>
Date:   Tue Jul 16 00:20:54 2019 +0200

    rustfmt

commit 360089ac74
Author: holger krekel <holger@merlinux.eu>
Date:   Tue Jul 16 00:03:49 2019 +0200

    remove some debugging

commit e892c5cf4d
Author: holger krekel <holger@merlinux.eu>
Date:   Mon Jul 15 23:31:30 2019 +0200

    fix test for events

commit 9ad4c9a6fe
Author: holger krekel <holger@merlinux.eu>
Date:   Mon Jul 15 22:51:57 2019 +0200

    wip try test that we see INFO events from the core
This commit is contained in:
holger krekel
2019-07-21 23:31:14 +02:00
parent 686678c96c
commit 8089559958
45 changed files with 1165 additions and 836 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
python/CHANGELOG
View File

@@ -4,6 +4,12 @@
- introduce automatic versioning via setuptools_scm,
based on py-X.Y.Z tags
- integrate latest DCC core-rust with dc_close() fixes
- provide a account.shutdown() method and improve termination
logic also in tests. also fixes output-clubbering during
test runs.
0.600.0
---------

7
python/install_py_bindings.sh
View File

@@ -1,7 +0,0 @@
#!/usr/bin/env bash
set -ex
cargo build -p deltachat_ffi --release
rm -rf build/ src/deltachat/*.so
DCC_RS_DEV=`pwd`/.. pip install -e .

36
python/install_python_bindings.py Executable file
View File

@@ -0,0 +1,36 @@
#!/usr/bin/env python
"""
setup a python binding development in-place install with cargo debug symbols.
"""
import os
import subprocess
if __name__ == "__main__":
os.environ["DCC_RS_TARGET"] = target = "release"
toml = os.path.join(os.getcwd(), "..", "Cargo.toml")
assert os.path.exists(toml)
with open(toml) as f:
s = orig = f.read()
s += "\n"
s += "[profile.release]\n"
s += "debug = true\n"
with open(toml, "w") as f:
f.write(s)
print("temporarily modifying Cargo.toml to provide release build with debug symbols ")
try:
subprocess.check_call([
"cargo", "build", "-p", "deltachat_ffi", "--" + target
])
finally:
with open(toml, "w") as f:
f.write(orig)
print("\nreseted Cargo.toml to previous original state")
subprocess.check_call("rm -rf build/ src/deltachat/*.so" , shell=True)
subprocess.check_call([
"pip", "install", "-e", "."
])

8
python/src/deltachat/__init__.py
View File

@@ -43,6 +43,9 @@ def py_dc_callback(ctx, evt, data1, data2):
try:
ret = callback(ctx, evt_name, data1, data2)
if ret is None:
ret = 0
assert isinstance(ret, int), repr(ret)
if event_sig_types & 4:
return ffi.cast('uintptr_t', ret)
elif event_sig_types & 8:
@@ -58,7 +61,10 @@ def set_context_callback(dc_context, func):
def clear_context_callback(dc_context):
_DC_CALLBACK_MAP.pop(dc_context, None)
try:
_DC_CALLBACK_MAP.pop(dc_context, None)
except AttributeError:
pass
def get_dc_event_name(integer, _DC_EVENTNAME_MAP={}):

1
python/src/deltachat/_build.py
View File

@@ -25,6 +25,7 @@ def ffibuilder():
else:
raise NotImplementedError("Compilation not supported yet on Windows, can you help?")
objs = [os.path.join(projdir, 'target', target, 'libdeltachat.a')]
assert os.path.exists(objs[0]), objs
incs = [os.path.join(projdir, 'deltachat-ffi')]
else:
libs = ['deltachat']

43
python/src/deltachat/account.py
View File

@@ -23,28 +23,36 @@ class Account(object):
by the underlying deltachat c-library. All public Account methods are
meant to be memory-safe and return memory-safe objects.
"""
def __init__(self, db_path, logid=None):
def __init__(self, db_path, logid=None, eventlogging=True):
""" initialize account object.
:param db_path: a path to the account database. The database
will be created if it doesn't exist.
:param logid: an optional logging prefix that should be used with
the default internal logging.
:param eventlogging: if False no eventlogging and no context callback will be configured
"""
self._dc_context = ffi.gc(
lib.dc_context_new(lib.py_dc_callback, ffi.NULL, ffi.NULL),
_destroy_dc_context,
)
if eventlogging:
self._evlogger = EventLogger(self._dc_context, logid)
deltachat.set_context_callback(self._dc_context, self._process_event)
self._threads = IOThreads(self._dc_context, self._evlogger._log_event)
else:
self._threads = IOThreads(self._dc_context)
if hasattr(db_path, "encode"):
db_path = db_path.encode("utf8")
if not lib.dc_open(self._dc_context, db_path, ffi.NULL):
raise ValueError("Could not dc_open: {}".format(db_path))
self._evlogger = EventLogger(self._dc_context, logid)
deltachat.set_context_callback(self._dc_context, self._process_event)
self._threads = IOThreads(self._dc_context)
self._configkeys = self.get_config("sys.config_keys").split()
self._imex_completed = threading.Event()
def __del__(self):
self.shutdown()
def _check_config_key(self, name):
if name not in self._configkeys:
raise KeyError("{!r} not a valid config key, existing keys: {!r}".format(
@@ -333,12 +341,22 @@ class Account(object):
lib.dc_stop_ongoing_process(self._dc_context)
self._threads.stop(wait=wait)
def shutdown(self, wait=True):
""" stop threads and close and remove underlying dc_context and callbacks. """
if hasattr(self, "_dc_context") and hasattr(self, "_threads"):
self.stop_threads(wait=False) # to interrupt idle and tell python threads to stop
lib.dc_close(self._dc_context)
self.stop_threads(wait=wait) # to wait for threads
deltachat.clear_context_callback(self._dc_context)
del self._dc_context
def _process_event(self, ctx, evt_name, data1, data2):
assert ctx == self._dc_context
self._evlogger(evt_name, data1, data2)
method = getattr(self, "on_" + evt_name.lower(), None)
if method is not None:
method(data1, data2)
if hasattr(self, "_evlogger"):
self._evlogger(evt_name, data1, data2)
method = getattr(self, "on_" + evt_name.lower(), None)
if method is not None:
method(data1, data2)
return 0
def on_dc_event_imex_progress(self, data1, data2):
@@ -347,10 +365,11 @@ class Account(object):
class IOThreads:
def __init__(self, dc_context):
def __init__(self, dc_context, log_event=lambda *args: None):
self._dc_context = dc_context
self._thread_quitflag = False
self._name2thread = {}
self._log_event = log_event
def is_started(self):
return len(self._name2thread) > 0
@@ -376,15 +395,19 @@ class IOThreads:
thread.join()
def imap_thread_run(self):
self._log_event("py-bindings-info", 0, "IMAP THREAD START")
while not self._thread_quitflag:
lib.dc_perform_imap_jobs(self._dc_context)
lib.dc_perform_imap_fetch(self._dc_context)
lib.dc_perform_imap_idle(self._dc_context)
self._log_event("py-bindings-info", 0, "IMAP THREAD FINISHED")
def smtp_thread_run(self):
self._log_event("py-bindings-info", 0, "SMTP THREAD START")
while not self._thread_quitflag:
lib.dc_perform_smtp_jobs(self._dc_context)
lib.dc_perform_smtp_idle(self._dc_context)
self._log_event("py-bindings-info", 0, "SMTP THREAD FINISHED")
class EventLogger:
@@ -414,7 +437,7 @@ class EventLogger:
raise ValueError("{}({!r},{!r})".format(*ev))
return ev
def get_matching(self, event_name_regex):
def get_matching(self, event_name_regex, check_error=True):
self._log("-- waiting for event with regex: {} --".format(event_name_regex))
rex = re.compile("(?:{}).*".format(event_name_regex))
while 1:

21
python/tests/conftest.py
View File

@@ -16,12 +16,23 @@ def pytest_addoption(parser):
)
@pytest.hookimpl(trylast=True)
def pytest_runtest_call(item):
# perform early finalization because we otherwise get cloberred
# output from concurrent threads printing between execution
# of the test function and the teardown phase of that test function
if "acfactory" in item.funcargs:
print("*"*30, "finalizing", "*"*30)
acfactory = item.funcargs["acfactory"]
acfactory.finalize()
def pytest_report_header(config, startdir):
t = tempfile.mktemp()
try:
ac = Account(t)
ac = Account(t, eventlogging=False)
info = ac.get_info()
del ac
ac.shutdown()
finally:
os.remove(t)
return "Deltachat core={} sqlite={}".format(
@@ -52,7 +63,6 @@ def acfactory(pytestconfig, tmpdir, request):
self.live_count = 0
self.offline_count = 0
self._finalizers = []
request.addfinalizer(self.finalize)
self.init_time = time.time()
def finalize(self):
@@ -78,6 +88,7 @@ def acfactory(pytestconfig, tmpdir, request):
ac = Account(tmpdb.strpath, logid="ac{}".format(self.offline_count))
ac._evlogger.init_time = self.init_time
ac._evlogger.set_timeout(2)
self._finalizers.append(ac.shutdown)
return ac
def get_configured_offline_account(self):
@@ -103,7 +114,7 @@ def acfactory(pytestconfig, tmpdir, request):
ac._evlogger.set_timeout(30)
ac.configure(**configdict)
ac.start_threads()
self._finalizers.append(lambda: ac.stop_threads(wait=False))
self._finalizers.append(ac.shutdown)
return ac
def clone_online_account(self, account):
@@ -114,7 +125,7 @@ def acfactory(pytestconfig, tmpdir, request):
ac._evlogger.set_timeout(30)
ac.configure(addr=account.get_config("addr"), mail_pw=account.get_config("mail_pw"))
ac.start_threads()
self._finalizers.append(lambda: ac.stop_threads(wait=False))
self._finalizers.append(ac.shutdown)
return ac
return AccountMaker()

8
python/tests/test_account.py
View File

@@ -1,12 +1,18 @@
from __future__ import print_function
import pytest
import os
from deltachat import const
from deltachat import const, Account
from datetime import datetime, timedelta
from conftest import wait_configuration_progress, wait_successful_IMAP_SMTP_connection
class TestOfflineAccount:
def test_wrong_db(self, tmpdir):
p = tmpdir.join("hello.db")
p.write("123")
with pytest.raises(ValueError):
Account(p.strpath)
def test_getinfo(self, acfactory):
ac1 = acfactory.get_unconfigured_account()
d = ac1.get_info()

4
python/tests/test_increation.py
View File

@@ -32,7 +32,11 @@ class TestInCreation:
chat2.add_contact(c2)
wait_msgs_changed(ac1, 0, 0) # why not chat id?
ac1.forward_messages([prepared_original], chat2)
# XXX there might be two EVENT_MSGS_CHANGED and only one of them
# is the one caused by forwarding
forwarded_id = wait_msgs_changed(ac1, chat2.id)
if forwarded_id == 0:
forwarded_id = wait_msgs_changed(ac1, chat2.id)
forwarded_msg = ac1.get_message_by_id(forwarded_id)
assert forwarded_msg.get_state().is_out_preparing()

47
python/tests/test_lowlevel.py
View File

@@ -1,6 +1,8 @@
from __future__ import print_function
import pytest
from deltachat import capi, Account, const
from deltachat import capi, const, set_context_callback, clear_context_callback
from deltachat.capi import ffi
from deltachat.capi import lib
from deltachat.account import EventLogger
def test_empty_context():
@@ -8,10 +10,45 @@ def test_empty_context():
capi.lib.dc_close(ctx)
def test_callback_None2int():
ctx = capi.lib.dc_context_new(capi.lib.py_dc_callback, ffi.NULL, ffi.NULL)
set_context_callback(ctx, lambda *args: None)
capi.lib.dc_close(ctx)
clear_context_callback(ctx)
def test_dc_close_events():
ctx = capi.lib.dc_context_new(capi.lib.py_dc_callback, ffi.NULL, ffi.NULL)
evlog = EventLogger(ctx)
evlog.set_timeout(5)
set_context_callback(ctx, lambda ctx, evt_name, data1, data2: evlog(evt_name, data1, data2))
capi.lib.dc_close(ctx)
def find(info_string):
while 1:
ev = evlog.get_matching("DC_EVENT_INFO", check_error=False)
data2 = ev[2]
if info_string in data2:
return
else:
print("skipping event", *ev)
find("disconnecting INBOX-watch")
find("disconnecting sentbox-thread")
find("disconnecting mvbox-thread")
find("disconnecting SMTP")
find("Database closed")
def test_wrong_db(tmpdir):
tmpdir.join("hello.db").write("123")
with pytest.raises(ValueError):
Account(db_path=tmpdir.strpath)
dc_context = ffi.gc(
lib.dc_context_new(lib.py_dc_callback, ffi.NULL, ffi.NULL),
lib.dc_context_unref,
)
p = tmpdir.join("hello.db")
# write an invalid database file
p.write("x123" * 10)
assert not lib.dc_open(dc_context, p.strpath.encode("ascii"), ffi.NULL)
def test_event_defines():

1
python/tox.ini
View File

@@ -52,6 +52,7 @@ commands =
python_files = tests/test_*.py
norecursedirs = .tox
xfail_strict=true
timeout = 60
[flake8]
max-line-length = 120