Slavasil/shortener-bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial

Browse Source
This commit is contained in:
Slavasil
2024-10-10 19:05:48 +00:00
commit cffdcba6af
1880 changed files with 813614 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
td/example/web/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
/OpenSSL_*.tar.gz
/openssl-OpenSSL_*/
/tdweb/dist/
/tdweb/node_modules/
/tdweb/src/prebuilt/

29
td/example/web/README.md Normal file
View File

@@ -0,0 +1,29 @@
# TDLib Web example
This is an example of building `TDLib` for browsers using [Emscripten](https://github.com/kripken/emscripten).
These scripts build `TDLib` and create an [NPM](https://www.npmjs.com/) package [tdweb](https://www.npmjs.com/package/tdweb).
You need a Unix shell with `sed`, `tar` and `wget` utilities to run the provided scripts.
## Building tdweb NPM package
* Install the 3.1.1 [emsdk](https://kripken.github.io/emscripten-site/docs/getting_started/downloads.html), which is known to work. Do not use the system-provided `emscripten` package, because it contains a too old emsdk version.
* Install all `TDLib` build dependencies described in [Building](https://github.com/tdlib/td#building) and `sed`, `tar` and `wget` utilities.
* Run `source ./emsdk_env.sh` from `emsdk` directory to set up the correct build environment.
* On `macOS`, install the `coreutils` [Homebrew](https://brew.sh) package and replace `realpath` in scripts with `grealpath`:
```
brew install coreutils
sed -i.bak 's/[(]realpath/(grealpath/g' build-tdlib.sh
```
* Run build scripts in the following order:
```
cd <path to TDLib sources>/example/web
./build-openssl.sh
./build-tdlib.sh
./copy-tdlib.sh
./build-tdweb.sh
```
* The built package is now located in the `tdweb` directory.
## Using tdweb NPM package
See [tdweb](https://www.npmjs.com/package/tdweb) or [README.md](https://github.com/tdlib/td/tree/master/example/web/tdweb/README.md) for package documentation.

28
td/example/web/build-openssl.sh Executable file
View File

@@ -0,0 +1,28 @@
#!/bin/sh
cd $(dirname $0)
emconfigure true 2> /dev/null || { echo 'emconfigure not found. Install emsdk and add emconfigure and emmake to PATH environment variable. See instruction at https://kripken.github.io/emscripten-site/docs/getting_started/downloads.html. Do not forget to add `emconfigure` and `emmake` to the PATH environment variable via `emsdk/emsdk_env.sh` script.'; exit 1; }
OPENSSL=OpenSSL_1_1_0l
if [ ! -f $OPENSSL.tar.gz ]; then
echo "Downloading OpenSSL sources..."
curl -sfLO https://github.com/openssl/openssl/archive/$OPENSSL.tar.gz
fi
rm -rf ./openssl-$OPENSSL
echo "Unpacking OpenSSL sources..."
tar xzf $OPENSSL.tar.gz || exit 1
cd openssl-$OPENSSL
emconfigure ./Configure linux-generic32 no-shared no-threads no-dso no-engine no-unit-test no-ui || exit 1
sed -i.bak 's/CROSS_COMPILE=.*/CROSS_COMPILE=/g' Makefile || exit 1
sed -i.bak 's/-ldl //g' Makefile || exit 1
sed -i.bak 's/-O3/-Os/g' Makefile || exit 1
echo "Building OpenSSL..."
emmake make depend || exit 1
emmake make -j 4 || exit 1
rm -rf ../build/crypto || exit 1
mkdir -p ../build/crypto/lib || exit 1
cp libcrypto.a libssl.a ../build/crypto/lib/ || exit 1
cp -r include ../build/crypto/ || exit 1
cd ..

44
td/example/web/build-tdlib.sh Executable file
View File

@@ -0,0 +1,44 @@
#!/bin/sh
cd $(dirname $0)
emcmake true 2> /dev/null || { echo 'emcmake not found. Install emsdk and add emcmake and emmake to PATH environment variable. See instruction at https://kripken.github.io/emscripten-site/docs/getting_started/downloads.html. Do not forget to add `emcmake` and `emmake` to the PATH environment variable via `emsdk/emsdk_env.sh` script.'; exit 1; }
rm -rf build/generate
rm -rf build/asmjs
rm -rf build/wasm
mkdir -p build/generate
mkdir -p build/asmjs
mkdir -p build/wasm
TD_ROOT=$(realpath ../../)
OPENSSL_ROOT=$(realpath ./build/crypto/)
OPENSSL_CRYPTO_LIBRARY=$OPENSSL_ROOT/lib/libcrypto.a
OPENSSL_SSL_LIBRARY=$OPENSSL_ROOT/lib/libssl.a
OPENSSL_OPTIONS="-DOPENSSL_FOUND=1 \
-DOPENSSL_ROOT_DIR=\"$OPENSSL_ROOT\" \
-DOPENSSL_INCLUDE_DIR=\"$OPENSSL_ROOT/include\" \
-DOPENSSL_CRYPTO_LIBRARY=\"$OPENSSL_CRYPTO_LIBRARY\" \
-DOPENSSL_SSL_LIBRARY=\"$OPENSSL_SSL_LIBRARY\" \
-DOPENSSL_LIBRARIES=\"$OPENSSL_SSL_LIBRARY;$OPENSSL_CRYPTO_LIBRARY\" \
-DOPENSSL_VERSION=\"1.1.0l\""
cd build/generate
cmake -DTD_GENERATE_SOURCE_FILES=ON $TD_ROOT || exit 1
cd ../..
cd build/wasm
eval emcmake cmake -DCMAKE_BUILD_TYPE=MinSizeRel $OPENSSL_OPTIONS $TD_ROOT || exit 1
cd ../..
cd build/asmjs
eval emcmake cmake -DCMAKE_BUILD_TYPE=MinSizeRel $OPENSSL_OPTIONS -DASMJS=1 $TD_ROOT || exit 1
cd ../..
echo "Generating TDLib autogenerated source files..."
cmake --build build/generate || exit 1
echo "Building TDLib to WebAssembly..."
cmake --build build/wasm --target td_wasm || exit 1
echo "Building TDLib to asm.js..."
cmake --build build/asmjs --target td_asmjs || exit 1

7
td/example/web/build-tdweb.sh Executable file
View File

@@ -0,0 +1,7 @@
#!/bin/sh
cd $(dirname $0)
cd tdweb || exit 1
npm install --no-save || exit 1
npm run build || exit 1
cd ..

7
td/example/web/copy-tdlib.sh Executable file
View File

@@ -0,0 +1,7 @@
#!/bin/sh
cd $(dirname $0)
DEST=tdweb/src/prebuilt/release/
mkdir -p $DEST || exit 1
cp build/wasm/td_wasm.js build/wasm/td_wasm.wasm $DEST || exit 1
cp build/asmjs/td_asmjs.js build/asmjs/td_asmjs.js.mem $DEST || exit 1

29
td/example/web/tdweb/README.md Normal file
View File

@@ -0,0 +1,29 @@
## tdweb - TDLib in a browser
[TDLib](https://github.com/tdlib/td) is a library for building Telegram clients. tdweb is a convenient wrapper for TDLib in a browser which controls TDLib instance creation,
handles interaction with TDLib and manages a filesystem for persistent TDLib data.
For interaction with TDLib, you need to create an instance of the class `TdClient`, providing a handler for incoming updates and other options if needed.
Once this is done, you can send queries to the TDLib instance using the method `TdClient.send` which returns a Promise object representing the result of the query.
See [Getting Started](https://core.telegram.org/tdlib/getting-started) for a description of basic TDLib concepts and a short introduction to TDLib usage.
See the [td_api.tl](https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl) scheme or
the automatically generated [HTML documentation](https://core.telegram.org/tdlib/docs/td__api_8h.html) for a list of all available
TDLib [methods](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_function.html) and [classes](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_object.html).
The JSON representation of TDLib API objects is straightforward: all API objects are represented as JSON objects with the same keys as the API object field names in the
[td_api.tl](https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl) scheme. Note that in the automatically generated C++ documentation all fields have an additional terminating underscore
which shouldn't be used in the JSON interface. The object type name is stored in the special field '@type' which is optional in places where type is uniquely determined by the context.
Fields of Bool type are stored as Boolean, fields of int32, int53, and double types are stored as Number, fields of int64 and string types are stored as String,
fields of bytes type are base64 encoded and then stored as String, fields of array type are stored as Array.
You can also add the field '@extra' to any query to TDLib and the response will contain the field '@extra' with exactly the same value.
## Installation
As usual, add npm tdweb package into your project:
```
npm install tdweb
```
All files will be installed into `node_modules/tdweb/dist/` folder. For now, it is your responsibility to make
those files loadable from your server. For example, [telegram-react](https://github.com/evgeny-nadymov/telegram-react)
manually copies these files into the `public` folder. If you know how to avoid this problem, please tell us.

7681
td/example/web/tdweb/package-lock.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

102
td/example/web/tdweb/package.json Normal file
View File

@@ -0,0 +1,102 @@
{
"name": "tdweb",
"version": "1.8.1",
"description": "JavaScript interface for TDLib (Telegram library)",
"main": "dist/tdweb.js",
"repository": {
"type": "git",
"url": "https://github.com/tdlib/td.git",
"directory": "example/web/tdweb"
},
"files": [
"dist"
],
"scripts": {
"build": "webpack --mode production",
"start": "webpack-dev-server --open"
},
"keywords": [
"telegram"
],
"author": "Arseny Smirnov",
"license": "MIT",
"devDependencies": {
"@babel/core": "^7.4.3",
"@babel/plugin-syntax-dynamic-import": "^7.2.0",
"@babel/plugin-transform-runtime": "^7.4.3",
"@babel/preset-env": "^7.4.3",
"@typescript-eslint/eslint-plugin": "^1.7.0",
"acorn": "^6.4.1",
"babel-eslint": "^10.0.1",
"babel-loader": "^8.0.5",
"clean-webpack-plugin": "^2.0.1",
"eslint": "^5.16.0",
"eslint-config-react-app": "^4.0.0",
"eslint-loader": "^2.1.2",
"eslint-plugin-flowtype": "^2.0.0",
"eslint-plugin-import": "^2.17.2",
"eslint-plugin-jsx-a11y": "^6.2.1",
"eslint-plugin-react": "^7.12.4",
"eslint-plugin-react-hooks": "^1.6.0",
"file-loader": "^3.0.1",
"husky": "^1.3.1",
"lint-staged": "^8.1.5",
"prettier": "^1.17.0",
"typescript": "^3.4.5",
"webpack": "^4.44.1",
"webpack-cli": "^3.3.12",
"worker-loader": "^2.0.0"
},
"husky": {
"hooks": {
"pre-commit": "lint-staged"
}
},
"lint-staged": {
"linters": {
"webpack.config.json": [
"prettier --single-quote --write",
"git add"
],
"package.json": [
"prettier --single-quote --write",
"git add"
],
"src/*.{js,jsx,json,css}": [
"prettier --single-quote --write",
"git add"
]
}
},
"dependencies": {
"@babel/runtime": "^7.4.3",
"broadcast-channel": "^2.1.12",
"localforage": "^1.7.3",
"uuid": "^3.3.2"
},
"babel": {
"presets": [
"@babel/env"
],
"plugins": [
"@babel/syntax-dynamic-import",
"@babel/transform-runtime"
]
},
"eslintConfig": {
"extends": "eslint-config-react-app",
"env": {
"worker": true,
"node": true,
"browser": true
},
"globals": {
"WebAssembly": true
},
"settings": {
"react": {
"version": "999.999.999"
}
}
}
}

680
td/example/web/tdweb/src/index.js Normal file
View File

@@ -0,0 +1,680 @@
import MyWorker from './worker.js';
//import localforage from 'localforage';
import BroadcastChannel from 'broadcast-channel';
import uuid4 from 'uuid/v4';
import log from './logger.js';
const sleep = ms => new Promise(res => setTimeout(res, ms));
/**
* TDLib in a browser
*
* TDLib can be compiled to WebAssembly or asm.js using Emscripten compiler and used in a browser from JavaScript.
* This is a convenient wrapper for TDLib in a browser which controls TDLib instance creation, handles interaction
* with TDLib and manages a filesystem for persistent TDLib data.
* TDLib instance is created in a Web Worker to run it in a separate thread.
* TdClient just sends queries to the Web Worker and receives updates and results from it.
* <br>
* <br>
* Differences from the TDLib JSON API:<br>
* 1. Added the update <code>updateFatalError error:string = Update;</code> which is sent whenever TDLib encounters a fatal error.<br>
* 2. Added the method <code>setJsLogVerbosityLevel new_verbosity_level:string = Ok;</code>, which allows to change the verbosity level of tdweb logging.<br>
* 3. Added the possibility to use blobs as input files via the constructor <code>inputFileBlob data:<JavaScript blob> = InputFile;</code>.<br>
* 4. The class <code>filePart</code> contains data as a JavaScript blob instead of a base64-encoded string.<br>
* 5. The methods <code>getStorageStatistics</code>, <code>getStorageStatisticsFast</code>, <code>optimizeStorage</code>, and <code>addProxy</code> are not supported.<br>
* <br>
*/
class TdClient {
/**
* @callback TdClient~updateCallback
* @param {Object} update The update.
*/
/**
* Create TdClient.
* @param {Object} options - Options for TDLib instance creation.
* @param {TdClient~updateCallback} options.onUpdate - Callback for all incoming updates.
* @param {string} [options.instanceName=tdlib] - Name of the TDLib instance. Currently, only one instance of TdClient with a given name is allowed. All but one instances with the same name will be automatically closed. Usually, the newest non-background instance is kept alive. Files will be stored in an IndexedDb table with the same name.
* @param {boolean} [options.isBackground=false] - Pass true if the instance is opened from the background.
* @param {string} [options.jsLogVerbosityLevel=info] - The initial verbosity level of the JavaScript part of the code (one of 'error', 'warning', 'info', 'log', 'debug').
* @param {number} [options.logVerbosityLevel=2] - The initial verbosity level for the TDLib internal logging (0-1023).
* @param {boolean} [options.useDatabase=true] - Pass false to use TDLib without database and secret chats. It significantly improves loading time, but some functionality is unavailable without the database.
* @param {boolean} [options.readOnly=false] - For debug only. Pass true to open TDLib database in read-only mode
* @param {string} [options.mode=auto] - For debug only. The type of the TDLib build to use. 'asmjs' for asm.js and 'wasm' for WebAssembly. If mode == 'auto' WebAbassembly is used if supported by browser; otherwise, asm.js is used.
*/
constructor(options) {
log.setVerbosity(options.jsLogVerbosityLevel);
this.worker = new MyWorker();
this.worker.onmessage = e => {
this.onResponse(e.data);
};
this.query_id = 0;
this.query_callbacks = new Map();
if ('onUpdate' in options) {
this.onUpdate = options.onUpdate;
delete options.onUpdate;
}
options.instanceName = options.instanceName || 'tdlib';
this.fileManager = new FileManager(options.instanceName, this);
this.worker.postMessage({ '@type': 'init', options: options });
this.closeOtherClients(options);
}
/**
* Send a query to TDLib.
*
* If the query contains the field '@extra', the same field will be added into the result.
*
* @param {Object} query - The query for TDLib. See the [td_api.tl]{@link https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl} scheme or
* the automatically generated [HTML documentation]{@link https://core.telegram.org/tdlib/docs/td__api_8h.html}
* for a list of all available TDLib [methods]{@link https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_function.html} and
* [classes]{@link https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_object.html}.
* @returns {Promise} Promise object represents the result of the query.
*/
send(query) {
return this.doSend(query, true);
}
/** @private */
sendInternal(query) {
return this.doSend(query, false);
}
/** @private */
doSend(query, isExternal) {
this.query_id++;
if (query['@extra']) {
query['@extra'] = {
'@old_extra': JSON.parse(JSON.stringify(query['@extra'])),
query_id: this.query_id
};
} else {
query['@extra'] = {
query_id: this.query_id
};
}
if (query['@type'] === 'setJsLogVerbosityLevel') {
log.setVerbosity(query.new_verbosity_level);
}
log.debug('send to worker: ', query);
const res = new Promise((resolve, reject) => {
this.query_callbacks.set(this.query_id, [resolve, reject]);
});
if (isExternal) {
this.externalPostMessage(query);
} else {
this.worker.postMessage(query);
}
return res;
}
/** @private */
externalPostMessage(query) {
const unsupportedMethods = [
'getStorageStatistics',
'getStorageStatisticsFast',
'optimizeStorage',
'addProxy',
'init',
'start'
];
if (unsupportedMethods.includes(query['@type'])) {
this.onResponse({
'@type': 'error',
'@extra': query['@extra'],
code: 400,
message: "Method '" + query['@type'] + "' is not supported"
});
return;
}
if (query['@type'] === 'readFile' || query['@type'] === 'readFilePart') {
this.readFile(query);
return;
}
if (query['@type'] === 'deleteFile') {
this.deleteFile(query);
return;
}
this.worker.postMessage(query);
}
/** @private */
async readFile(query) {
const response = await this.fileManager.readFile(query);
this.onResponse(response);
}
/** @private */
async deleteFile(query) {
const response = this.fileManager.deleteFile(query);
try {
if (response.idb_key) {
await this.sendInternal({
'@type': 'deleteIdbKey',
idb_key: response.idb_key
});
delete response.idb_key;
}
await this.sendInternal({
'@type': 'deleteFile',
file_id: query.file_id
});
} catch (e) {}
this.onResponse(response);
}
/** @private */
onResponse(response) {
log.debug(
'receive from worker: ',
JSON.parse(
JSON.stringify(response, (key, value) => {
if (key === 'arr' || key === 'data') {
return undefined;
}
return value;
})
)
);
// for FileManager
response = this.prepareResponse(response);
if ('@extra' in response) {
const query_id = response['@extra'].query_id;
const [resolve, reject] = this.query_callbacks.get(query_id);
this.query_callbacks.delete(query_id);
if ('@old_extra' in response['@extra']) {
response['@extra'] = response['@extra']['@old_extra'];
}
if (resolve) {
if (response['@type'] === 'error') {
reject(response);
} else {
resolve(response);
}
}
} else {
if (response['@type'] === 'inited') {
this.onInited();
return;
}
if (response['@type'] === 'fsInited') {
this.onFsInited();
return;
}
if (
response['@type'] === 'updateAuthorizationState' &&
response.authorization_state['@type'] === 'authorizationStateClosed'
) {
this.onClosed();
}
this.onUpdate(response);
}
}
/** @private */
prepareFile(file) {
return this.fileManager.registerFile(file);
}
/** @private */
prepareResponse(response) {
if (response['@type'] === 'file') {
if (false && Math.random() < 0.1) {
(async () => {
log.warn('DELETE FILE', response.id);
try {
await this.send({ '@type': 'deleteFile', file_id: response.id });
} catch (e) {}
})();
}
return this.prepareFile(response);
}
for (const key in response) {
const field = response[key];
if (
field &&
typeof field === 'object' &&
key !== 'data' &&
key !== 'arr'
) {
response[key] = this.prepareResponse(field);
}
}
return response;
}
/** @private */
onBroadcastMessage(e) {
//const message = e.data;
const message = e;
if (message.uid === this.uid) {
log.info('ignore self broadcast message: ', message);
return;
}
log.info('receive broadcast message: ', message);
if (message.isBackground && !this.isBackground) {
// continue
} else if (
(!message.isBackground && this.isBackground) ||
message.timestamp > this.timestamp
) {
this.close();
return;
}
if (message.state === 'closed') {
this.waitSet.delete(message.uid);
if (this.waitSet.size === 0) {
log.info('onWaitSetEmpty');
this.onWaitSetEmpty();
this.onWaitSetEmpty = () => {};
}
} else {
this.waitSet.add(message.uid);
if (message.state !== 'closing') {
this.postState();
}
}
}
/** @private */
postState() {
const state = {
uid: this.uid,
state: this.state,
timestamp: this.timestamp,
isBackground: this.isBackground
};
log.info('Post state: ', state);
this.channel.postMessage(state);
}
/** @private */
onWaitSetEmpty() {
// nop
}
/** @private */
onFsInited() {
this.fileManager.init();
}
/** @private */
onInited() {
this.isInited = true;
this.doSendStart();
}
/** @private */
sendStart() {
this.wantSendStart = true;
this.doSendStart();
}
/** @private */
doSendStart() {
if (!this.isInited || !this.wantSendStart || this.state !== 'start') {
return;
}
this.wantSendStart = false;
this.state = 'active';
const query = { '@type': 'start' };
log.info('send to worker: ', query);
this.worker.postMessage(query);
}
/** @private */
onClosed() {
this.isClosing = true;
this.worker.terminate();
log.info('worker is terminated');
this.state = 'closed';
this.postState();
}
/** @private */
close() {
if (this.isClosing) {
return;
}
this.isClosing = true;
log.info('close state: ', this.state);
if (this.state === 'start') {
this.onClosed();
this.onUpdate({
'@type': 'updateAuthorizationState',
authorization_state: {
'@type': 'authorizationStateClosed'
}
});
return;
}
const query = { '@type': 'close' };
log.info('send to worker: ', query);
this.worker.postMessage(query);
this.state = 'closing';
this.postState();
}
/** @private */
async closeOtherClients(options) {
this.uid = uuid4();
this.state = 'start';
this.isBackground = !!options.isBackground;
this.timestamp = Date.now();
this.waitSet = new Set();
log.info('close other clients');
this.channel = new BroadcastChannel(options.instanceName, {
webWorkerSupport: false
});
this.postState();
this.channel.onmessage = message => {
this.onBroadcastMessage(message);
};
await sleep(300);
if (this.waitSet.size !== 0) {
await new Promise(resolve => {
this.onWaitSetEmpty = resolve;
});
}
this.sendStart();
}
/** @private */
onUpdate(update) {
log.info('ignore onUpdate');
//nop
}
}
/** @private */
class ListNode {
constructor(value) {
this.value = value;
this.clear();
}
erase() {
this.prev.connect(this.next);
this.clear();
}
clear() {
this.prev = this;
this.next = this;
}
connect(other) {
this.next = other;
other.prev = this;
}
onUsed(other) {
other.usedAt = Date.now();
other.erase();
other.connect(this.next);
log.debug('LRU: used file_id: ', other.value);
this.connect(other);
}
getLru() {
if (this === this.next) {
throw new Error('popLru from empty list');
}
return this.prev;
}
}
/** @private */
class FileManager {
constructor(instanceName, client) {
this.instanceName = instanceName;
this.cache = new Map();
this.pending = [];
this.transaction_id = 0;
this.totalSize = 0;
this.lru = new ListNode(-1);
this.client = client;
}
init() {
this.idb = new Promise((resolve, reject) => {
const request = indexedDB.open(this.instanceName);
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error);
});
//this.store = localforage.createInstance({
//name: instanceName
//});
this.isInited = true;
}
unload(info) {
if (info.arr) {
log.debug(
'LRU: delete file_id: ',
info.node.value,
' with arr.length: ',
info.arr.length
);
this.totalSize -= info.arr.length;
delete info.arr;
}
if (info.node) {
info.node.erase();
delete info.node;
}
}
registerFile(file) {
if (file.idb_key || file.arr) {
file.local.is_downloading_completed = true;
} else {
file.local.is_downloading_completed = false;
}
let info = {};
const cached_info = this.cache.get(file.id);
if (cached_info) {
info = cached_info;
} else {
this.cache.set(file.id, info);
}
if (file.idb_key) {
info.idb_key = file.idb_key;
delete file.idb_key;
} else {
delete info.idb_key;
}
if (file.arr) {
const now = Date.now();
while (this.totalSize > 100000000) {
const node = this.lru.getLru();
// immunity for 60 seconds
if (node.usedAt + 60 * 1000 > now) {
break;
}
const lru_info = this.cache.get(node.value);
this.unload(lru_info);
}
if (info.arr) {
log.warn('Receive file.arr at least twice for the same file');
this.totalSize -= info.arr.length;
}
info.arr = file.arr;
delete file.arr;
this.totalSize += info.arr.length;
if (!info.node) {
log.debug(
'LRU: create file_id: ',
file.id,
' with arr.length: ',
info.arr.length
);
info.node = new ListNode(file.id);
}
this.lru.onUsed(info.node);
log.info('Total file.arr size: ', this.totalSize);
}
info.file = file;
return file;
}
async flushLoad() {
const pending = this.pending;
this.pending = [];
const idb = await this.idb;
const transaction_id = this.transaction_id++;
const read = idb
.transaction(['keyvaluepairs'], 'readonly')
.objectStore('keyvaluepairs');
log.debug('Load group of files from idb', pending.length);
for (const query of pending) {
const request = read.get(query.key);
request.onsuccess = event => {
const blob = event.target.result;
if (blob) {
if (blob.size === 0) {
log.error('Receive empty blob from db ', query.key);
}
query.resolve({ data: blob, transaction_id: transaction_id });
} else {
query.reject();
}
};
request.onerror = () => query.reject(request.error);
}
}
load(key, resolve, reject) {
if (this.pending.length === 0) {
setTimeout(() => {
this.flushLoad();
}, 1);
}
this.pending.push({ key: key, resolve: resolve, reject: reject });
}
async doLoadFull(info) {
if (info.arr) {
return { data: new Blob([info.arr]), transaction_id: -1 };
}
if (info.idb_key) {
const idb_key = info.idb_key;
//return this.store.getItem(idb_key);
return await new Promise((resolve, reject) => {
this.load(idb_key, resolve, reject);
});
}
throw new Error('File is not loaded');
}
async doLoad(info, offset, size) {
if (!info.arr && !info.idb_key && info.file.local.path) {
try {
const count = await this.client.sendInternal({
'@type': 'getFileDownloadedPrefixSize',
file_id: info.file.id,
offset: offset
});
//log.error(count, size);
if (!size) {
size = count.count;
} else if (size > count.count) {
throw new Error('File not loaded yet');
}
const res = await this.client.sendInternal({
'@type': 'readFilePart',
path: info.file.local.path,
offset: offset,
count: size
});
res.data = new Blob([res.data]);
res.transaction_id = -2;
//log.error(res);
return res;
} catch (e) {
log.info('readFilePart failed', info, offset, size, e);
}
}
const res = await this.doLoadFull(info);
// return slice(size, offset + size)
const data_size = res.data.size;
if (!size) {
size = data_size;
}
if (offset > data_size) {
offset = data_size;
}
res.data = res.data.slice(offset, offset + size);
return res;
}
doDelete(info) {
this.unload(info);
return info.idb_key;
}
async readFile(query) {
try {
if (!this.isInited) {
throw new Error('FileManager is not inited');
}
const info = this.cache.get(query.file_id);
if (!info) {
throw new Error('File is not loaded');
}
if (info.node) {
this.lru.onUsed(info.node);
}
query.offset = query.offset || 0;
query.size = query.count || query.size || 0;
const response = await this.doLoad(info, query.offset, query.size);
return {
'@type': 'filePart',
'@extra': query['@extra'],
data: response.data,
transaction_id: response.transaction_id
};
} catch (e) {
return {
'@type': 'error',
'@extra': query['@extra'],
code: 400,
message: e
};
}
}
deleteFile(query) {
const res = {
'@type': 'ok',
'@extra': query['@extra']
};
try {
if (!this.isInited) {
throw new Error('FileManager is not inited');
}
const info = this.cache.get(query.file_id);
if (!info) {
throw new Error('File is not loaded');
}
const idb_key = this.doDelete(info);
if (idb_key) {
res.idb_key = idb_key;
}
} catch (e) {}
return res;
}
}
export default TdClient;

47
td/example/web/tdweb/src/logger.js Normal file
View File

@@ -0,0 +1,47 @@
class Logger {
constructor() {
this.setVerbosity('WARNING');
}
debug(...str) {
if (this.checkVerbosity(4)) {
console.log(...str);
}
}
log(...str) {
if (this.checkVerbosity(4)) {
console.log(...str);
}
}
info(...str) {
if (this.checkVerbosity(3)) {
console.info(...str);
}
}
warn(...str) {
if (this.checkVerbosity(2)) {
console.warn(...str);
}
}
error(...str) {
if (this.checkVerbosity(1)) {
console.error(...str);
}
}
setVerbosity(level, default_level = 'info') {
if (level === undefined) {
level = default_level;
}
if (typeof level === 'string') {
level =
{ ERROR: 1, WARNING: 2, INFO: 3, LOG: 4, DEBUG: 4 }[
level.toUpperCase()
] || 2;
}
this.level = level;
}
checkVerbosity(level) {
return this.level >= level;
}
}
let log = new Logger();
export default log;

136
td/example/web/tdweb/src/wasm-utils.js Normal file
View File

@@ -0,0 +1,136 @@
// 1. +++ fetchAndInstantiate() +++ //
// This library function fetches the wasm module at 'url', instantiates it with
// the given 'importObject', and returns the instantiated object instance
export async function instantiateStreaming(url, importObject) {
let result = await WebAssembly.instantiateStreaming(fetch(url), importObject);
return result.instance;
}
export function fetchAndInstantiate(url, importObject) {
return fetch(url)
.then(response => response.arrayBuffer())
.then(bytes => WebAssembly.instantiate(bytes, importObject))
.then(results => results.instance);
}
// 2. +++ instantiateCachedURL() +++ //
// This library function fetches the wasm Module at 'url', instantiates it with
// the given 'importObject', and returns a Promise resolving to the finished
// wasm Instance. Additionally, the function attempts to cache the compiled wasm
// Module in IndexedDB using 'url' as the key. The entire site's wasm cache (not
// just the given URL) is versioned by dbVersion and any change in dbVersion on
// any call to instantiateCachedURL() will conservatively clear out the entire
// cache to avoid stale modules.
export function instantiateCachedURL(dbVersion, url, importObject) {
const dbName = 'wasm-cache';
const storeName = 'wasm-cache';
// This helper function Promise-ifies the operation of opening an IndexedDB
// database and clearing out the cache when the version changes.
function openDatabase() {
return new Promise((resolve, reject) => {
var request = indexedDB.open(dbName, dbVersion);
request.onerror = reject.bind(null, 'Error opening wasm cache database');
request.onsuccess = () => {
resolve(request.result);
};
request.onupgradeneeded = event => {
var db = request.result;
if (db.objectStoreNames.contains(storeName)) {
console.log(`Clearing out version ${event.oldVersion} wasm cache`);
db.deleteObjectStore(storeName);
}
console.log(`Creating version ${event.newVersion} wasm cache`);
db.createObjectStore(storeName);
};
});
}
// This helper function Promise-ifies the operation of looking up 'url' in the
// given IDBDatabase.
function lookupInDatabase(db) {
return new Promise((resolve, reject) => {
var store = db.transaction([storeName]).objectStore(storeName);
var request = store.get(url);
request.onerror = reject.bind(null, `Error getting wasm module ${url}`);
request.onsuccess = event => {
if (request.result) resolve(request.result);
else reject(`Module ${url} was not found in wasm cache`);
};
});
}
// This helper function fires off an async operation to store the given wasm
// Module in the given IDBDatabase.
function storeInDatabase(db, module) {
var store = db.transaction([storeName], 'readwrite').objectStore(storeName);
var request = store.put(module, url);
request.onerror = err => {
console.log(`Failed to store in wasm cache: ${err}`);
};
request.onsuccess = err => {
console.log(`Successfully stored ${url} in wasm cache`);
};
}
// This helper function fetches 'url', compiles it into a Module,
// instantiates the Module with the given import object.
function fetchAndInstantiate() {
return fetch(url)
.then(response => response.arrayBuffer())
.then(buffer => WebAssembly.instantiate(buffer, importObject));
}
// With all the Promise helper functions defined, we can now express the core
// logic of an IndexedDB cache lookup. We start by trying to open a database.
return openDatabase().then(
db => {
// Now see if we already have a compiled Module with key 'url' in 'db':
return lookupInDatabase(db).then(
module => {
// We do! Instantiate it with the given import object.
console.log(`Found ${url} in wasm cache`);
return WebAssembly.instantiate(module, importObject);
},
errMsg => {
// Nope! Compile from scratch and then store the compiled Module in 'db'
// with key 'url' for next time.
console.log(errMsg);
return fetchAndInstantiate().then(results => {
try {
storeInDatabase(db, results.module);
} catch (e) {
console.log('Failed to store module into db');
}
return results.instance;
});
}
);
},
errMsg => {
// If opening the database failed (due to permissions or quota), fall back
// to simply fetching and compiling the module and don't try to store the
// results.
console.log(errMsg);
return fetchAndInstantiate().then(results => results.instance);
}
);
}
export async function instantiateAny(version, url, importObject) {
console.log("instantiate");
try {
return await instantiateStreaming(url, importObject);
} catch (e) {
console.log("instantiateStreaming failed", e);
}
try {
return await instantiateCachedURL(version, url, importObject);
} catch (e) {
console.log("instantiateCachedURL failed", e);
}
throw new Error("can't instantiate wasm");
}

1032
td/example/web/tdweb/src/worker.js Normal file
View File

File diff suppressed because it is too large Load Diff

86
td/example/web/tdweb/webpack.config.js Normal file
View File

@@ -0,0 +1,86 @@
const path = require('path');
const CleanWebpackPlugin = require('clean-webpack-plugin');
module.exports = {
entry: ['./src/index.js'],
output: {
filename: 'tdweb.js',
path: path.resolve(__dirname, 'dist'),
library: 'tdweb',
libraryTarget: 'umd',
umdNamedDefine: true,
globalObject: 'this'
},
devServer: {
contentBase: './dist'
},
plugins: [
// new HtmlWebpackPlugin(),
new CleanWebpackPlugin({})
//, new UglifyJSPlugin()
],
optimization:{
minimize: false, // <---- disables uglify.
},
module: {
noParse: /td_asmjs\.js$/,
rules: [
{
test: /\.(js|jsx)$/,
exclude: /prebuilt/,
enforce: 'pre',
include: [path.resolve(__dirname, 'src')],
use: [
{
loader: require.resolve('eslint-loader')
}
]
},
{
test: /worker\.(js|jsx)$/,
include: [path.resolve(__dirname, 'src')],
use: [
{
loader: require.resolve('worker-loader')
}
]
},
{
test: /\.(js|jsx)$/,
exclude: /prebuilt/,
include: [path.resolve(__dirname, 'src')],
use: [
{
loader: require.resolve('babel-loader')
}
]
},
{
test: /\.(wasm|mem)$/,
include: [path.resolve(__dirname, 'src')],
type: "javascript/auto",
use: [
{
loader: require.resolve('file-loader')
}
]
}
]
},
node: {
dgram: 'empty',
fs: 'empty',
net: 'empty',
tls: 'empty',
crypto: 'empty',
child_process: 'empty'
},
performance: {
maxAssetSize: 30000000
},
resolve: {
alias: {
ws$: 'fs'
}
}
};