Site blocked
+This site is on your blocklist:
+
+
+
+Too much image
+ on 2015-01-25 12:32:11
+ 
](https://play.google.com/store/apps/details?id=in.canews.zeronetmobile)
-- Скачать APK: https://github.com/canewsin/zeronet_mobile/releases
+Смотрите [ArchWiki](https://wiki.archlinux.org)'s [ZeroNet
+article](https://wiki.archlinux.org/index.php/ZeroNet) для дальнейшей помощи.
-### Android (arm, arm64, x86) Облегчённый клиент только для просмотра (1МБ)
+### [Gentoo Linux](https://www.gentoo.org)
-- Для работы требуется Android как минимум версии 4.1 Jelly Bean
-- [](https://play.google.com/store/apps/details?id=dev.zeronetx.app.lite)
+* [`layman -a raiagent`](https://github.com/leycec/raiagent)
+* `echo '>=net-vpn/zeronet-0.5.4' >> /etc/portage/package.accept_keywords`
+* *(Опционально)* Включить поддержку Tor: `echo 'net-vpn/zeronet tor' >>
+ /etc/portage/package.use`
+* `emerge zeronet`
+* `rc-service zeronet start`
+* Откройте http://127.0.0.1:43110/ в вашем браузере.
-### Установка из исходного кода
+Смотрите `/usr/share/doc/zeronet-*/README.gentoo.bz2` для дальнейшей помощи.
-```sh
-wget https://github.com/ZeroNetX/ZeroNet/releases/latest/download/ZeroNet-src.zip
-unzip ZeroNet-src.zip
-cd ZeroNet
-sudo apt-get update
-sudo apt-get install python3-pip
-sudo python3 -m pip install -r requirements.txt
+### [FreeBSD](https://www.freebsd.org/)
+
+* `pkg install zeronet` or `cd /usr/ports/security/zeronet/ && make install clean`
+* `sysrc zeronet_enable="YES"`
+* `service zeronet start`
+* Откройте http://127.0.0.1:43110/ в вашем браузере.
+
+### [Vagrant](https://www.vagrantup.com/)
+
+* `vagrant up`
+* Подключитесь к VM с помощью `vagrant ssh`
+* `cd /vagrant`
+* Запустите `python2 zeronet.py --ui_ip 0.0.0.0`
+* Откройте http://127.0.0.1:43110/ в вашем браузере.
+
+### [Docker](https://www.docker.com/)
+* `docker run -d -v ](https://play.google.com/store/apps/details?id=in.canews.zeronetmobile)
- - APK download: https://github.com/canewsin/zeronet_mobile/releases
+### [FreeBSD](https://www.freebsd.org/)
-### Android (arm, arm64, x86) Thin Client for Preview Only (Size 1MB)
- - minimum Android version supported 16 (JellyBean)
- - [](https://play.google.com/store/apps/details?id=dev.zeronetx.app.lite)
+* `pkg install zeronet` 或者 `cd /usr/ports/security/zeronet/ && make install clean`
+* `sysrc zeronet_enable="YES"`
+* `service zeronet start`
+* 在你的浏览器中打开 http://127.0.0.1:43110/
+
+### [Vagrant](https://www.vagrantup.com/)
+
+* `vagrant up`
+* 通过 `vagrant ssh` 连接到 VM
+* `cd /vagrant`
+* 运行 `python2 zeronet.py --ui_ip 0.0.0.0`
+* 在你的浏览器中打开 http://127.0.0.1:43110/
+
+### [Docker](https://www.docker.com/)
+* `docker run -d -v ](https://play.google.com/store/apps/details?id=in.canews.zeronetmobile)
- - APK download: https://github.com/canewsin/zeronet_mobile/releases
-
-### Android (arm, arm64, x86) Thin Client for Preview Only (Size 1MB)
- - minimum Android version supported 16 (JellyBean)
- - [](https://play.google.com/store/apps/details?id=dev.zeronetx.app.lite)
-
-
-#### Docker
-There is an official image, built from source at: https://hub.docker.com/r/canewsin/zeronet/
-
-### Online Proxies
-Proxies are like seed boxes for sites(i.e ZNX runs on a cloud vps), you can try zeronet experience from proxies. Add your proxy below if you have one.
-
-#### Official ZNX Proxy :
-
-https://proxy.zeronet.dev/
-
-https://zeronet.dev/
-
-#### From Community
-
-https://0net-preview.com/
-
-https://portal.ngnoid.tv/
-
-https://zeronet.ipfsscan.io/
+### Install from package for your distribution
+* Arch Linux: [zeronet](https://aur.archlinux.org/zeronet.git), [zeronet-git](https://aur.archlinux.org/zeronet-git.git)
+* Gentoo: [emerge repository](https://github.com/leycec/raiagent)
+* FreeBSD: zeronet
+* Whonix: [instructions](https://www.whonix.org/wiki/ZeroNet)
### Install from source
- - `wget https://github.com/ZeroNetX/ZeroNet/releases/latest/download/ZeroNet-src.zip`
- - `unzip ZeroNet-src.zip`
- - `cd ZeroNet`
- - `sudo apt-get update`
- - `sudo apt-get install python3-pip`
- - `sudo python3 -m pip install -r requirements.txt`
- - Start with: `python3 zeronet.py`
- - Open the ZeroHello landing page in your browser by navigating to: http://127.0.0.1:43110/
+Fetch and extract the source:
+
+ wget https://github.com/HelloZeroNet/ZeroNet/archive/py3.tar.gz
+ tar xvpfz py3.tar.gz
+ cd ZeroNet-py3
+
+Install Python module dependencies either:
+
+* (Option A) into a [virtual env](https://virtualenv.readthedocs.org/en/latest/)
+
+ ```
+ virtualenv zeronet
+ source zeronet/bin/activate
+ python -m pip install -r requirements.txt
+ ```
+
+* (Option B) into the system (requires root), for example, on Debian/Ubuntu:
+
+ ```
+ sudo apt-get update
+ sudo apt-get install python3-pip
+ sudo python3 -m pip install -r requirements.txt
+ ```
+
+Start Zeronet:
+
+ python3 zeronet.py
+
+Open the ZeroHello landing page in your browser by navigating to:
+
+ http://127.0.0.1:43110/
## Current limitations
-* File transactions are not compressed
+* ~~No torrent-like file splitting for big file support~~ (big file support added)
+* ~~No more anonymous than Bittorrent~~ (built-in full Tor support added)
+* File transactions are not compressed ~~or encrypted yet~~ (TLS encryption added)
* No private sites
## How can I create a ZeroNet site?
- * Click on **⋮** > **"Create new, empty site"** menu item on the site [ZeroHello](http://127.0.0.1:43110/1HELLoE3sFD9569CLCbHEAVqvqV7U2Ri9d).
- * You will be **redirected** to a completely new site that is only modifiable by you!
- * You can find and modify your site's content in **data/[yoursiteaddress]** directory
- * After the modifications open your site, drag the topright "0" button to left, then press **sign** and **publish** buttons on the bottom
+Shut down zeronet if you are running it already
+
+```bash
+$ zeronet.py siteCreate
+...
+- Site private key: 23DKQpzxhbVBrAtvLEc2uvk7DZweh4qL3fn3jpM3LgHDczMK2TtYUq
+- Site address: 13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2
+...
+- Site created!
+$ zeronet.py
+...
+```
+
+Congratulations, you're finished! Now anyone can access your site using
+`http://localhost:43110/13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2`
+
+Next steps: [ZeroNet Developer Documentation](https://zeronet.io/docs/site_development/getting_started/)
+
+
+## How can I modify a ZeroNet site?
+
+* Modify files located in data/13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2 directory.
+ After you're finished:
+
+```bash
+$ zeronet.py siteSign 13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2
+- Signing site: 13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2...
+Private key (input hidden):
+```
+
+* Enter the private key you got when you created the site, then:
+
+```bash
+$ zeronet.py sitePublish 13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2
+...
+Site:13DNDk..bhC2 Publishing to 3/10 peers...
+Site:13DNDk..bhC2 Successfuly published to 3 peers
+- Serving files....
+```
+
+* That's it! You've successfully signed and published your modifications.
-Next steps: [ZeroNet Developer Documentation](https://docs.zeronet.dev/1DeveLopDZL1cHfKi8UXHh2UBEhzH6HhMp/site_development/getting_started/)
## Help keep this project alive
-- Bitcoin: 1ZeroNetyV5mKY9JF1gsm82TuBXHpfdLX (Preferred)
-- LiberaPay: https://liberapay.com/PramUkesh
-- Paypal: https://paypal.me/PramUkesh
-- Others: [Donate](!https://docs.zeronet.dev/1DeveLopDZL1cHfKi8UXHh2UBEhzH6HhMp/help_zeronet/donate/#help-to-keep-zeronet-development-alive)
+
+- Bitcoin: 1QDhxQ6PraUZa21ET5fYUCPgdrwBomnFgX
+- Paypal: https://zeronet.io/docs/help_zeronet/donate/
+
+### Sponsors
+
+* Better macOS/Safari compatibility made possible by [BrowserStack.com](https://www.browserstack.com)
#### Thank you!
-* More info, help, changelog, zeronet sites: https://www.reddit.com/r/zeronetx/
-* Come, chat with us: [#zeronet @ FreeNode](https://kiwiirc.com/client/irc.freenode.net/zeronet) or on [gitter](https://gitter.im/canewsin/ZeroNet)
-* Email: canews.in@gmail.com
+* More info, help, changelog, zeronet sites: https://www.reddit.com/r/zeronet/
+* Come, chat with us: [#zeronet @ FreeNode](https://kiwiirc.com/client/irc.freenode.net/zeronet) or on [gitter](https://gitter.im/HelloZeroNet/ZeroNet)
+* Email: hello@zeronet.io (PGP: CB9613AE)
diff --git a/plugins b/plugins
deleted file mode 160000
index 689d9309..00000000
--- a/plugins
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit 689d9309f73371f4681191b125ec3f2e14075eeb
diff --git a/plugins/AnnounceLocal/AnnounceLocalPlugin.py b/plugins/AnnounceLocal/AnnounceLocalPlugin.py
new file mode 100644
index 00000000..0919762a
--- /dev/null
+++ b/plugins/AnnounceLocal/AnnounceLocalPlugin.py
@@ -0,0 +1,148 @@
+import time
+
+import gevent
+
+from Plugin import PluginManager
+from Config import config
+from . import BroadcastServer
+
+
+@PluginManager.registerTo("SiteAnnouncer")
+class SiteAnnouncerPlugin(object):
+ def announce(self, force=False, *args, **kwargs):
+ local_announcer = self.site.connection_server.local_announcer
+
+ thread = None
+ if local_announcer and (force or time.time() - local_announcer.last_discover > 5 * 60):
+ thread = gevent.spawn(local_announcer.discover, force=force)
+ back = super(SiteAnnouncerPlugin, self).announce(force=force, *args, **kwargs)
+
+ if thread:
+ thread.join()
+
+ return back
+
+
+class LocalAnnouncer(BroadcastServer.BroadcastServer):
+ def __init__(self, server, listen_port):
+ super(LocalAnnouncer, self).__init__("zeronet", listen_port=listen_port)
+ self.server = server
+
+ self.sender_info["peer_id"] = self.server.peer_id
+ self.sender_info["port"] = self.server.port
+ self.sender_info["broadcast_port"] = listen_port
+ self.sender_info["rev"] = config.rev
+
+ self.known_peers = {}
+ self.last_discover = 0
+
+ def discover(self, force=False):
+ self.log.debug("Sending discover request (force: %s)" % force)
+ self.last_discover = time.time()
+ if force: # Probably new site added, clean cache
+ self.known_peers = {}
+
+ for peer_id, known_peer in list(self.known_peers.items()):
+ if time.time() - known_peer["found"] > 20 * 60:
+ del(self.known_peers[peer_id])
+ self.log.debug("Timeout, removing from known_peers: %s" % peer_id)
+ self.broadcast({"cmd": "discoverRequest", "params": {}}, port=self.listen_port)
+
+ def actionDiscoverRequest(self, sender, params):
+ back = {
+ "cmd": "discoverResponse",
+ "params": {
+ "sites_changed": self.server.site_manager.sites_changed
+ }
+ }
+
+ if sender["peer_id"] not in self.known_peers:
+ self.known_peers[sender["peer_id"]] = {"added": time.time(), "sites_changed": 0, "updated": 0, "found": time.time()}
+ self.log.debug("Got discover request from unknown peer %s (%s), time to refresh known peers" % (sender["ip"], sender["peer_id"]))
+ gevent.spawn_later(1.0, self.discover) # Let the response arrive first to the requester
+
+ return back
+
+ def actionDiscoverResponse(self, sender, params):
+ if sender["peer_id"] in self.known_peers:
+ self.known_peers[sender["peer_id"]]["found"] = time.time()
+ if params["sites_changed"] != self.known_peers.get(sender["peer_id"], {}).get("sites_changed"):
+ # Peer's site list changed, request the list of new sites
+ return {"cmd": "siteListRequest"}
+ else:
+ # Peer's site list is the same
+ for site in self.server.sites.values():
+ peer = site.peers.get("%s:%s" % (sender["ip"], sender["port"]))
+ if peer:
+ peer.found("local")
+
+ def actionSiteListRequest(self, sender, params):
+ back = []
+ sites = list(self.server.sites.values())
+
+ # Split adresses to group of 100 to avoid UDP size limit
+ site_groups = [sites[i:i + 100] for i in range(0, len(sites), 100)]
+ for site_group in site_groups:
+ res = {}
+ res["sites_changed"] = self.server.site_manager.sites_changed
+ res["sites"] = [site.address_hash for site in site_group]
+ back.append({"cmd": "siteListResponse", "params": res})
+ return back
+
+ def actionSiteListResponse(self, sender, params):
+ s = time.time()
+ peer_sites = set(params["sites"])
+ num_found = 0
+ added_sites = []
+ for site in self.server.sites.values():
+ if site.address_hash in peer_sites:
+ added = site.addPeer(sender["ip"], sender["port"], source="local")
+ num_found += 1
+ if added:
+ site.worker_manager.onPeers()
+ site.updateWebsocket(peers_added=1)
+ added_sites.append(site)
+
+ # Save sites changed value to avoid unnecessary site list download
+ if sender["peer_id"] not in self.known_peers:
+ self.known_peers[sender["peer_id"]] = {"added": time.time()}
+
+ self.known_peers[sender["peer_id"]]["sites_changed"] = params["sites_changed"]
+ self.known_peers[sender["peer_id"]]["updated"] = time.time()
+ self.known_peers[sender["peer_id"]]["found"] = time.time()
+
+ self.log.debug(
+ "Tracker result: Discover from %s response parsed in %.3fs, found: %s added: %s of %s" %
+ (sender["ip"], time.time() - s, num_found, added_sites, len(peer_sites))
+ )
+
+
+@PluginManager.registerTo("FileServer")
+class FileServerPlugin(object):
+ def __init__(self, *args, **kwargs):
+ res = super(FileServerPlugin, self).__init__(*args, **kwargs)
+ if config.broadcast_port and config.tor != "always" and not config.disable_udp:
+ self.local_announcer = LocalAnnouncer(self, config.broadcast_port)
+ else:
+ self.local_announcer = None
+ return res
+
+ def start(self, *args, **kwargs):
+ if self.local_announcer:
+ gevent.spawn(self.local_announcer.start)
+ return super(FileServerPlugin, self).start(*args, **kwargs)
+
+ def stop(self):
+ if self.local_announcer:
+ self.local_announcer.stop()
+ res = super(FileServerPlugin, self).stop()
+ return res
+
+
+@PluginManager.registerTo("ConfigPlugin")
+class ConfigPlugin(object):
+ def createArguments(self):
+ group = self.parser.add_argument_group("AnnounceLocal plugin")
+ group.add_argument('--broadcast_port', help='UDP broadcasting port for local peer discovery', default=1544, type=int, metavar='port')
+
+ return super(ConfigPlugin, self).createArguments()
diff --git a/plugins/AnnounceLocal/BroadcastServer.py b/plugins/AnnounceLocal/BroadcastServer.py
new file mode 100644
index 00000000..74678896
--- /dev/null
+++ b/plugins/AnnounceLocal/BroadcastServer.py
@@ -0,0 +1,139 @@
+import socket
+import logging
+import time
+from contextlib import closing
+
+from Debug import Debug
+from util import UpnpPunch
+from util import Msgpack
+
+
+class BroadcastServer(object):
+ def __init__(self, service_name, listen_port=1544, listen_ip=''):
+ self.log = logging.getLogger("BroadcastServer")
+ self.listen_port = listen_port
+ self.listen_ip = listen_ip
+
+ self.running = False
+ self.sock = None
+ self.sender_info = {"service": service_name}
+
+ def createBroadcastSocket(self):
+ sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
+ sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+ if hasattr(socket, 'SO_REUSEPORT'):
+ try:
+ sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
+ except Exception as err:
+ self.log.warning("Error setting SO_REUSEPORT: %s" % err)
+
+ binded = False
+ for retry in range(3):
+ try:
+ sock.bind((self.listen_ip, self.listen_port))
+ binded = True
+ break
+ except Exception as err:
+ self.log.error(
+ "Socket bind to %s:%s error: %s, retry #%s" %
+ (self.listen_ip, self.listen_port, Debug.formatException(err), retry)
+ )
+ time.sleep(retry)
+
+ if binded:
+ return sock
+ else:
+ return False
+
+ def start(self): # Listens for discover requests
+ self.sock = self.createBroadcastSocket()
+ if not self.sock:
+ self.log.error("Unable to listen on port %s" % self.listen_port)
+ return
+
+ self.log.debug("Started on port %s" % self.listen_port)
+
+ self.running = True
+
+ while self.running:
+ try:
+ data, addr = self.sock.recvfrom(8192)
+ except Exception as err:
+ if self.running:
+ self.log.error("Listener receive error: %s" % err)
+ continue
+
+ if not self.running:
+ break
+
+ try:
+ message = Msgpack.unpack(data)
+ response_addr, message = self.handleMessage(addr, message)
+ if message:
+ self.send(response_addr, message)
+ except Exception as err:
+ self.log.error("Handlemessage error: %s" % Debug.formatException(err))
+ self.log.debug("Stopped listening on port %s" % self.listen_port)
+
+ def stop(self):
+ self.log.debug("Stopping, socket: %s" % self.sock)
+ self.running = False
+ if self.sock:
+ self.sock.close()
+
+ def send(self, addr, message):
+ if type(message) is not list:
+ message = [message]
+
+ for message_part in message:
+ message_part["sender"] = self.sender_info
+
+ self.log.debug("Send to %s: %s" % (addr, message_part["cmd"]))
+ with closing(socket.socket(socket.AF_INET, socket.SOCK_DGRAM)) as sock:
+ sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ sock.sendto(Msgpack.pack(message_part), addr)
+
+ def getMyIps(self):
+ return UpnpPunch._get_local_ips()
+
+ def broadcast(self, message, port=None):
+ if not port:
+ port = self.listen_port
+
+ my_ips = self.getMyIps()
+ addr = ("255.255.255.255", port)
+
+ message["sender"] = self.sender_info
+ self.log.debug("Broadcast using ips %s on port %s: %s" % (my_ips, port, message["cmd"]))
+
+ for my_ip in my_ips:
+ try:
+ with closing(socket.socket(socket.AF_INET, socket.SOCK_DGRAM)) as sock:
+ sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
+ sock.bind((my_ip, 0))
+ sock.sendto(Msgpack.pack(message), addr)
+ except Exception as err:
+ self.log.warning("Error sending broadcast using ip %s: %s" % (my_ip, err))
+
+ def handleMessage(self, addr, message):
+ self.log.debug("Got from %s: %s" % (addr, message["cmd"]))
+ cmd = message["cmd"]
+ params = message.get("params", {})
+ sender = message["sender"]
+ sender["ip"] = addr[0]
+
+ func_name = "action" + cmd[0].upper() + cmd[1:]
+ func = getattr(self, func_name, None)
+
+ if sender["service"] != "zeronet" or sender["peer_id"] == self.sender_info["peer_id"]:
+ # Skip messages not for us or sent by us
+ message = None
+ elif func:
+ message = func(sender, params)
+ else:
+ self.log.debug("Unknown cmd: %s" % cmd)
+ message = None
+
+ return (sender["ip"], sender["broadcast_port"]), message
diff --git a/plugins/AnnounceLocal/Test/TestAnnounce.py b/plugins/AnnounceLocal/Test/TestAnnounce.py
new file mode 100644
index 00000000..4def02ed
--- /dev/null
+++ b/plugins/AnnounceLocal/Test/TestAnnounce.py
@@ -0,0 +1,113 @@
+import time
+import copy
+
+import gevent
+import pytest
+import mock
+
+from AnnounceLocal import AnnounceLocalPlugin
+from File import FileServer
+from Test import Spy
+
+@pytest.fixture
+def announcer(file_server, site):
+ file_server.sites[site.address] = site
+ announcer = AnnounceLocalPlugin.LocalAnnouncer(file_server, listen_port=1100)
+ file_server.local_announcer = announcer
+ announcer.listen_port = 1100
+ announcer.sender_info["broadcast_port"] = 1100
+ announcer.getMyIps = mock.MagicMock(return_value=["127.0.0.1"])
+ announcer.discover = mock.MagicMock(return_value=False) # Don't send discover requests automatically
+ gevent.spawn(announcer.start)
+ time.sleep(0.5)
+
+ assert file_server.local_announcer.running
+ return file_server.local_announcer
+
+@pytest.fixture
+def announcer_remote(request, site_temp):
+ file_server_remote = FileServer("127.0.0.1", 1545)
+ file_server_remote.sites[site_temp.address] = site_temp
+ announcer = AnnounceLocalPlugin.LocalAnnouncer(file_server_remote, listen_port=1101)
+ file_server_remote.local_announcer = announcer
+ announcer.listen_port = 1101
+ announcer.sender_info["broadcast_port"] = 1101
+ announcer.getMyIps = mock.MagicMock(return_value=["127.0.0.1"])
+ announcer.discover = mock.MagicMock(return_value=False) # Don't send discover requests automatically
+ gevent.spawn(announcer.start)
+ time.sleep(0.5)
+
+ assert file_server_remote.local_announcer.running
+
+ def cleanup():
+ file_server_remote.stop()
+ request.addfinalizer(cleanup)
+
+
+ return file_server_remote.local_announcer
+
+@pytest.mark.usefixtures("resetSettings")
+@pytest.mark.usefixtures("resetTempSettings")
+class TestAnnounce:
+ def testSenderInfo(self, announcer):
+ sender_info = announcer.sender_info
+ assert sender_info["port"] > 0
+ assert len(sender_info["peer_id"]) == 20
+ assert sender_info["rev"] > 0
+
+ def testIgnoreSelfMessages(self, announcer):
+ # No response to messages that has same peer_id as server
+ assert not announcer.handleMessage(("0.0.0.0", 123), {"cmd": "discoverRequest", "sender": announcer.sender_info, "params": {}})[1]
+
+ # Response to messages with different peer id
+ sender_info = copy.copy(announcer.sender_info)
+ sender_info["peer_id"] += "-"
+ addr, res = announcer.handleMessage(("0.0.0.0", 123), {"cmd": "discoverRequest", "sender": sender_info, "params": {}})
+ assert res["params"]["sites_changed"] > 0
+
+ def testDiscoverRequest(self, announcer, announcer_remote):
+ assert len(announcer_remote.known_peers) == 0
+ with Spy.Spy(announcer_remote, "handleMessage") as responses:
+ announcer_remote.broadcast({"cmd": "discoverRequest", "params": {}}, port=announcer.listen_port)
+ time.sleep(0.1)
+
+ response_cmds = [response[1]["cmd"] for response in responses]
+ assert response_cmds == ["discoverResponse", "siteListResponse"]
+ assert len(responses[-1][1]["params"]["sites"]) == 1
+
+ # It should only request siteList if sites_changed value is different from last response
+ with Spy.Spy(announcer_remote, "handleMessage") as responses:
+ announcer_remote.broadcast({"cmd": "discoverRequest", "params": {}}, port=announcer.listen_port)
+ time.sleep(0.1)
+
+ response_cmds = [response[1]["cmd"] for response in responses]
+ assert response_cmds == ["discoverResponse"]
+
+ def testPeerDiscover(self, announcer, announcer_remote, site):
+ assert announcer.server.peer_id != announcer_remote.server.peer_id
+ assert len(list(announcer.server.sites.values())[0].peers) == 0
+ announcer.broadcast({"cmd": "discoverRequest"}, port=announcer_remote.listen_port)
+ time.sleep(0.1)
+ assert len(list(announcer.server.sites.values())[0].peers) == 1
+
+ def testRecentPeerList(self, announcer, announcer_remote, site):
+ assert len(site.peers_recent) == 0
+ assert len(site.peers) == 0
+ with Spy.Spy(announcer, "handleMessage") as responses:
+ announcer.broadcast({"cmd": "discoverRequest", "params": {}}, port=announcer_remote.listen_port)
+ time.sleep(0.1)
+ assert [response[1]["cmd"] for response in responses] == ["discoverResponse", "siteListResponse"]
+ assert len(site.peers_recent) == 1
+ assert len(site.peers) == 1
+
+ # It should update peer without siteListResponse
+ last_time_found = list(site.peers.values())[0].time_found
+ site.peers_recent.clear()
+ with Spy.Spy(announcer, "handleMessage") as responses:
+ announcer.broadcast({"cmd": "discoverRequest", "params": {}}, port=announcer_remote.listen_port)
+ time.sleep(0.1)
+ assert [response[1]["cmd"] for response in responses] == ["discoverResponse"]
+ assert len(site.peers_recent) == 1
+ assert list(site.peers.values())[0].time_found > last_time_found
+
+
diff --git a/plugins/AnnounceLocal/Test/conftest.py b/plugins/AnnounceLocal/Test/conftest.py
new file mode 100644
index 00000000..a88c642c
--- /dev/null
+++ b/plugins/AnnounceLocal/Test/conftest.py
@@ -0,0 +1,4 @@
+from src.Test.conftest import *
+
+from Config import config
+config.broadcast_port = 0
diff --git a/plugins/AnnounceLocal/Test/pytest.ini b/plugins/AnnounceLocal/Test/pytest.ini
new file mode 100644
index 00000000..d09210d1
--- /dev/null
+++ b/plugins/AnnounceLocal/Test/pytest.ini
@@ -0,0 +1,5 @@
+[pytest]
+python_files = Test*.py
+addopts = -rsxX -v --durations=6
+markers =
+ webtest: mark a test as a webtest.
\ No newline at end of file
diff --git a/plugins/AnnounceLocal/__init__.py b/plugins/AnnounceLocal/__init__.py
new file mode 100644
index 00000000..5b80abd2
--- /dev/null
+++ b/plugins/AnnounceLocal/__init__.py
@@ -0,0 +1 @@
+from . import AnnounceLocalPlugin
\ No newline at end of file
diff --git a/plugins/AnnounceShare/AnnounceSharePlugin.py b/plugins/AnnounceShare/AnnounceSharePlugin.py
new file mode 100644
index 00000000..057ce55a
--- /dev/null
+++ b/plugins/AnnounceShare/AnnounceSharePlugin.py
@@ -0,0 +1,190 @@
+import time
+import os
+import logging
+import json
+import atexit
+
+import gevent
+
+from Config import config
+from Plugin import PluginManager
+from util import helper
+
+
+class TrackerStorage(object):
+ def __init__(self):
+ self.log = logging.getLogger("TrackerStorage")
+ self.file_path = "%s/trackers.json" % config.data_dir
+ self.load()
+ self.time_discover = 0.0
+ atexit.register(self.save)
+
+ def getDefaultFile(self):
+ return {"shared": {}}
+
+ def onTrackerFound(self, tracker_address, type="shared", my=False):
+ if not tracker_address.startswith("zero://"):
+ return False
+
+ trackers = self.getTrackers()
+ added = False
+ if tracker_address not in trackers:
+ trackers[tracker_address] = {
+ "time_added": time.time(),
+ "time_success": 0,
+ "latency": 99.0,
+ "num_error": 0,
+ "my": False
+ }
+ self.log.debug("New tracker found: %s" % tracker_address)
+ added = True
+
+ trackers[tracker_address]["time_found"] = time.time()
+ trackers[tracker_address]["my"] = my
+ return added
+
+ def onTrackerSuccess(self, tracker_address, latency):
+ trackers = self.getTrackers()
+ if tracker_address not in trackers:
+ return False
+
+ trackers[tracker_address]["latency"] = latency
+ trackers[tracker_address]["time_success"] = time.time()
+ trackers[tracker_address]["num_error"] = 0
+
+ def onTrackerError(self, tracker_address):
+ trackers = self.getTrackers()
+ if tracker_address not in trackers:
+ return False
+
+ trackers[tracker_address]["time_error"] = time.time()
+ trackers[tracker_address]["num_error"] += 1
+
+ if len(self.getWorkingTrackers()) >= config.working_shared_trackers_limit:
+ error_limit = 5
+ else:
+ error_limit = 30
+ error_limit
+
+ if trackers[tracker_address]["num_error"] > error_limit and trackers[tracker_address]["time_success"] < time.time() - 60 * 60:
+ self.log.debug("Tracker %s looks down, removing." % tracker_address)
+ del trackers[tracker_address]
+
+ def getTrackers(self, type="shared"):
+ return self.file_content.setdefault(type, {})
+
+ def getWorkingTrackers(self, type="shared"):
+ trackers = {
+ key: tracker for key, tracker in self.getTrackers(type).items()
+ if tracker["time_success"] > time.time() - 60 * 60
+ }
+ return trackers
+
+ def getFileContent(self):
+ if not os.path.isfile(self.file_path):
+ open(self.file_path, "w").write("{}")
+ return self.getDefaultFile()
+ try:
+ return json.load(open(self.file_path))
+ except Exception as err:
+ self.log.error("Error loading trackers list: %s" % err)
+ return self.getDefaultFile()
+
+ def load(self):
+ self.file_content = self.getFileContent()
+
+ trackers = self.getTrackers()
+ self.log.debug("Loaded %s shared trackers" % len(trackers))
+ for address, tracker in list(trackers.items()):
+ tracker["num_error"] = 0
+ if not address.startswith("zero://"):
+ del trackers[address]
+
+ def save(self):
+ s = time.time()
+ helper.atomicWrite(self.file_path, json.dumps(self.file_content, indent=2, sort_keys=True).encode("utf8"))
+ self.log.debug("Saved in %.3fs" % (time.time() - s))
+
+ def discoverTrackers(self, peers):
+ if len(self.getWorkingTrackers()) > config.working_shared_trackers_limit:
+ return False
+ s = time.time()
+ num_success = 0
+ for peer in peers:
+ if peer.connection and peer.connection.handshake.get("rev", 0) < 3560:
+ continue # Not supported
+
+ res = peer.request("getTrackers")
+ if not res or "error" in res:
+ continue
+
+ num_success += 1
+ for tracker_address in res["trackers"]:
+ if type(tracker_address) is bytes: # Backward compatibilitys
+ tracker_address = tracker_address.decode("utf8")
+ added = self.onTrackerFound(tracker_address)
+ if added: # Only add one tracker from one source
+ break
+
+ if not num_success and len(peers) < 20:
+ self.time_discover = 0.0
+
+ if num_success:
+ self.save()
+
+ self.log.debug("Trackers discovered from %s/%s peers in %.3fs" % (num_success, len(peers), time.time() - s))
+
+
+if "tracker_storage" not in locals():
+ tracker_storage = TrackerStorage()
+
+
+@PluginManager.registerTo("SiteAnnouncer")
+class SiteAnnouncerPlugin(object):
+ def getTrackers(self):
+ if tracker_storage.time_discover < time.time() - 5 * 60:
+ tracker_storage.time_discover = time.time()
+ gevent.spawn(tracker_storage.discoverTrackers, self.site.getConnectedPeers())
+ trackers = super(SiteAnnouncerPlugin, self).getTrackers()
+ shared_trackers = list(tracker_storage.getTrackers("shared").keys())
+ if shared_trackers:
+ return trackers + shared_trackers
+ else:
+ return trackers
+
+ def announceTracker(self, tracker, *args, **kwargs):
+ res = super(SiteAnnouncerPlugin, self).announceTracker(tracker, *args, **kwargs)
+ if res:
+ latency = res
+ tracker_storage.onTrackerSuccess(tracker, latency)
+ elif res is False:
+ tracker_storage.onTrackerError(tracker)
+
+ return res
+
+
+@PluginManager.registerTo("FileRequest")
+class FileRequestPlugin(object):
+ def actionGetTrackers(self, params):
+ shared_trackers = list(tracker_storage.getWorkingTrackers("shared").keys())
+ self.response({"trackers": shared_trackers})
+
+
+@PluginManager.registerTo("FileServer")
+class FileServerPlugin(object):
+ def portCheck(self, *args, **kwargs):
+ res = super(FileServerPlugin, self).portCheck(*args, **kwargs)
+ if res and not config.tor == "always" and "Bootstrapper" in PluginManager.plugin_manager.plugin_names:
+ for ip in self.ip_external_list:
+ my_tracker_address = "zero://%s:%s" % (ip, config.fileserver_port)
+ tracker_storage.onTrackerFound(my_tracker_address, my=True)
+ return res
+
+
+@PluginManager.registerTo("ConfigPlugin")
+class ConfigPlugin(object):
+ def createArguments(self):
+ group = self.parser.add_argument_group("AnnounceShare plugin")
+ group.add_argument('--working_shared_trackers_limit', help='Stop discovering new shared trackers after this number of shared trackers reached', default=5, type=int, metavar='limit')
+
+ return super(ConfigPlugin, self).createArguments()
diff --git a/plugins/AnnounceShare/Test/TestAnnounceShare.py b/plugins/AnnounceShare/Test/TestAnnounceShare.py
new file mode 100644
index 00000000..7178eac8
--- /dev/null
+++ b/plugins/AnnounceShare/Test/TestAnnounceShare.py
@@ -0,0 +1,24 @@
+import pytest
+
+from AnnounceShare import AnnounceSharePlugin
+from Peer import Peer
+from Config import config
+
+
+@pytest.mark.usefixtures("resetSettings")
+@pytest.mark.usefixtures("resetTempSettings")
+class TestAnnounceShare:
+ def testAnnounceList(self, file_server):
+ open("%s/trackers.json" % config.data_dir, "w").write("{}")
+ tracker_storage = AnnounceSharePlugin.tracker_storage
+ tracker_storage.load()
+ peer = Peer(file_server.ip, 1544, connection_server=file_server)
+ assert peer.request("getTrackers")["trackers"] == []
+
+ tracker_storage.onTrackerFound("zero://%s:15441" % file_server.ip)
+ assert peer.request("getTrackers")["trackers"] == []
+
+ # It needs to have at least one successfull announce to be shared to other peers
+ tracker_storage.onTrackerSuccess("zero://%s:15441" % file_server.ip, 1.0)
+ assert peer.request("getTrackers")["trackers"] == ["zero://%s:15441" % file_server.ip]
+
diff --git a/plugins/AnnounceShare/Test/conftest.py b/plugins/AnnounceShare/Test/conftest.py
new file mode 100644
index 00000000..5abd4dd6
--- /dev/null
+++ b/plugins/AnnounceShare/Test/conftest.py
@@ -0,0 +1,3 @@
+from src.Test.conftest import *
+
+from Config import config
diff --git a/plugins/AnnounceShare/Test/pytest.ini b/plugins/AnnounceShare/Test/pytest.ini
new file mode 100644
index 00000000..d09210d1
--- /dev/null
+++ b/plugins/AnnounceShare/Test/pytest.ini
@@ -0,0 +1,5 @@
+[pytest]
+python_files = Test*.py
+addopts = -rsxX -v --durations=6
+markers =
+ webtest: mark a test as a webtest.
\ No newline at end of file
diff --git a/plugins/AnnounceShare/__init__.py b/plugins/AnnounceShare/__init__.py
new file mode 100644
index 00000000..dc1e40bd
--- /dev/null
+++ b/plugins/AnnounceShare/__init__.py
@@ -0,0 +1 @@
+from . import AnnounceSharePlugin
diff --git a/plugins/AnnounceZero/AnnounceZeroPlugin.py b/plugins/AnnounceZero/AnnounceZeroPlugin.py
new file mode 100644
index 00000000..dcaa04f0
--- /dev/null
+++ b/plugins/AnnounceZero/AnnounceZeroPlugin.py
@@ -0,0 +1,138 @@
+import time
+import itertools
+
+from Plugin import PluginManager
+from util import helper
+from Crypt import CryptRsa
+
+allow_reload = False # No source reload supported in this plugin
+time_full_announced = {} # Tracker address: Last announced all site to tracker
+connection_pool = {} # Tracker address: Peer object
+
+
+# We can only import plugin host clases after the plugins are loaded
+@PluginManager.afterLoad
+def importHostClasses():
+ global Peer, AnnounceError
+ from Peer import Peer
+ from Site.SiteAnnouncer import AnnounceError
+
+
+# Process result got back from tracker
+def processPeerRes(tracker_address, site, peers):
+ added = 0
+ # Ip4
+ found_ipv4 = 0
+ peers_normal = itertools.chain(peers.get("ip4", []), peers.get("ipv4", []), peers.get("ipv6", []))
+ for packed_address in peers_normal:
+ found_ipv4 += 1
+ peer_ip, peer_port = helper.unpackAddress(packed_address)
+ if site.addPeer(peer_ip, peer_port, source="tracker"):
+ added += 1
+ # Onion
+ found_onion = 0
+ for packed_address in peers["onion"]:
+ found_onion += 1
+ peer_onion, peer_port = helper.unpackOnionAddress(packed_address)
+ if site.addPeer(peer_onion, peer_port, source="tracker"):
+ added += 1
+
+ if added:
+ site.worker_manager.onPeers()
+ site.updateWebsocket(peers_added=added)
+ return added
+
+
+@PluginManager.registerTo("SiteAnnouncer")
+class SiteAnnouncerPlugin(object):
+ def getTrackerHandler(self, protocol):
+ if protocol == "zero":
+ return self.announceTrackerZero
+ else:
+ return super(SiteAnnouncerPlugin, self).getTrackerHandler(protocol)
+
+ def announceTrackerZero(self, tracker_address, mode="start", num_want=10):
+ global time_full_announced
+ s = time.time()
+
+ need_types = ["ip4"] # ip4 for backward compatibility reasons
+ need_types += self.site.connection_server.supported_ip_types
+ if self.site.connection_server.tor_manager.enabled:
+ need_types.append("onion")
+
+ if mode == "start" or mode == "more": # Single: Announce only this site
+ sites = [self.site]
+ full_announce = False
+ else: # Multi: Announce all currently serving site
+ full_announce = True
+ if time.time() - time_full_announced.get(tracker_address, 0) < 60 * 15: # No reannounce all sites within short time
+ return None
+ time_full_announced[tracker_address] = time.time()
+ from Site import SiteManager
+ sites = [site for site in SiteManager.site_manager.sites.values() if site.isServing()]
+
+ # Create request
+ add_types = self.getOpenedServiceTypes()
+ request = {
+ "hashes": [], "onions": [], "port": self.fileserver_port, "need_types": need_types, "need_num": 20, "add": add_types
+ }
+ for site in sites:
+ if "onion" in add_types:
+ onion = self.site.connection_server.tor_manager.getOnion(site.address)
+ request["onions"].append(onion)
+ request["hashes"].append(site.address_hash)
+
+ # Tracker can remove sites that we don't announce
+ if full_announce:
+ request["delete"] = True
+
+ # Sent request to tracker
+ tracker_peer = connection_pool.get(tracker_address) # Re-use tracker connection if possible
+ if not tracker_peer:
+ tracker_ip, tracker_port = tracker_address.rsplit(":", 1)
+ tracker_peer = Peer(str(tracker_ip), int(tracker_port), connection_server=self.site.connection_server)
+ tracker_peer.is_tracker_connection = True
+ connection_pool[tracker_address] = tracker_peer
+
+ res = tracker_peer.request("announce", request)
+
+ if not res or "peers" not in res:
+ if full_announce:
+ time_full_announced[tracker_address] = 0
+ raise AnnounceError("Invalid response: %s" % res)
+
+ # Add peers from response to site
+ site_index = 0
+ peers_added = 0
+ for site_res in res["peers"]:
+ site = sites[site_index]
+ peers_added += processPeerRes(tracker_address, site, site_res)
+ site_index += 1
+
+ # Check if we need to sign prove the onion addresses
+ if "onion_sign_this" in res:
+ self.site.log.debug("Signing %s for %s to add %s onions" % (res["onion_sign_this"], tracker_address, len(sites)))
+ request["onion_signs"] = {}
+ request["onion_sign_this"] = res["onion_sign_this"]
+ request["need_num"] = 0
+ for site in sites:
+ onion = self.site.connection_server.tor_manager.getOnion(site.address)
+ publickey = self.site.connection_server.tor_manager.getPublickey(onion)
+ if publickey not in request["onion_signs"]:
+ sign = CryptRsa.sign(res["onion_sign_this"].encode("utf8"), self.site.connection_server.tor_manager.getPrivatekey(onion))
+ request["onion_signs"][publickey] = sign
+ res = tracker_peer.request("announce", request)
+ if not res or "onion_sign_this" in res:
+ if full_announce:
+ time_full_announced[tracker_address] = 0
+ raise AnnounceError("Announce onion address to failed: %s" % res)
+
+ if full_announce:
+ tracker_peer.remove() # Close connection, we don't need it in next 5 minute
+
+ self.site.log.debug(
+ "Tracker announce result: zero://%s (sites: %s, new peers: %s, add: %s) in %.3fs" %
+ (tracker_address, site_index, peers_added, add_types, time.time() - s)
+ )
+
+ return True
diff --git a/plugins/AnnounceZero/__init__.py b/plugins/AnnounceZero/__init__.py
new file mode 100644
index 00000000..8aec5ddb
--- /dev/null
+++ b/plugins/AnnounceZero/__init__.py
@@ -0,0 +1 @@
+from . import AnnounceZeroPlugin
\ No newline at end of file
diff --git a/plugins/Bigfile/BigfilePiecefield.py b/plugins/Bigfile/BigfilePiecefield.py
new file mode 100644
index 00000000..ee770573
--- /dev/null
+++ b/plugins/Bigfile/BigfilePiecefield.py
@@ -0,0 +1,164 @@
+import array
+
+
+def packPiecefield(data):
+ assert isinstance(data, bytes) or isinstance(data, bytearray)
+ res = []
+ if not data:
+ return array.array("H", b"")
+
+ if data[0] == b"\x00":
+ res.append(0)
+ find = b"\x01"
+ else:
+ find = b"\x00"
+ last_pos = 0
+ pos = 0
+ while 1:
+ pos = data.find(find, pos)
+ if find == b"\x00":
+ find = b"\x01"
+ else:
+ find = b"\x00"
+ if pos == -1:
+ res.append(len(data) - last_pos)
+ break
+ res.append(pos - last_pos)
+ last_pos = pos
+ return array.array("H", res)
+
+
+def unpackPiecefield(data):
+ if not data:
+ return b""
+
+ res = []
+ char = b"\x01"
+ for times in data:
+ if times > 10000:
+ return b""
+ res.append(char * times)
+ if char == b"\x01":
+ char = b"\x00"
+ else:
+ char = b"\x01"
+ return b"".join(res)
+
+
+def spliceBit(data, idx, bit):
+ assert bit == b"\x00" or bit == b"\x01"
+ if len(data) < idx:
+ data = data.ljust(idx + 1, b"\x00")
+ return data[:idx] + bit + data[idx+ 1:]
+
+class Piecefield(object):
+ def tostring(self):
+ return "".join(["1" if b else "0" for b in self.tobytes()])
+
+
+class BigfilePiecefield(Piecefield):
+ __slots__ = ["data"]
+
+ def __init__(self):
+ self.data = b""
+
+ def frombytes(self, s):
+ assert isinstance(s, bytes) or isinstance(s, bytearray)
+ self.data = s
+
+ def tobytes(self):
+ return self.data
+
+ def pack(self):
+ return packPiecefield(self.data).tobytes()
+
+ def unpack(self, s):
+ self.data = unpackPiecefield(array.array("H", s))
+
+ def __getitem__(self, key):
+ try:
+ return self.data[key]
+ except IndexError:
+ return False
+
+ def __setitem__(self, key, value):
+ self.data = spliceBit(self.data, key, value)
+
+class BigfilePiecefieldPacked(Piecefield):
+ __slots__ = ["data"]
+
+ def __init__(self):
+ self.data = b""
+
+ def frombytes(self, data):
+ assert isinstance(data, bytes) or isinstance(data, bytearray)
+ self.data = packPiecefield(data).tobytes()
+
+ def tobytes(self):
+ return unpackPiecefield(array.array("H", self.data))
+
+ def pack(self):
+ return array.array("H", self.data).tobytes()
+
+ def unpack(self, data):
+ self.data = data
+
+ def __getitem__(self, key):
+ try:
+ return self.tobytes()[key]
+ except IndexError:
+ return False
+
+ def __setitem__(self, key, value):
+ data = spliceBit(self.tobytes(), key, value)
+ self.frombytes(data)
+
+
+if __name__ == "__main__":
+ import os
+ import psutil
+ import time
+ testdata = b"\x01" * 100 + b"\x00" * 900 + b"\x01" * 4000 + b"\x00" * 4999 + b"\x01"
+ meminfo = psutil.Process(os.getpid()).memory_info
+
+ for storage in [BigfilePiecefieldPacked, BigfilePiecefield]:
+ print("-- Testing storage: %s --" % storage)
+ m = meminfo()[0]
+ s = time.time()
+ piecefields = {}
+ for i in range(10000):
+ piecefield = storage()
+ piecefield.frombytes(testdata[:i] + b"\x00" + testdata[i + 1:])
+ piecefields[i] = piecefield
+
+ print("Create x10000: +%sKB in %.3fs (len: %s)" % ((meminfo()[0] - m) / 1024, time.time() - s, len(piecefields[0].data)))
+
+ m = meminfo()[0]
+ s = time.time()
+ for piecefield in list(piecefields.values()):
+ val = piecefield[1000]
+
+ print("Query one x10000: +%sKB in %.3fs" % ((meminfo()[0] - m) / 1024, time.time() - s))
+
+ m = meminfo()[0]
+ s = time.time()
+ for piecefield in list(piecefields.values()):
+ piecefield[1000] = b"\x01"
+
+ print("Change one x10000: +%sKB in %.3fs" % ((meminfo()[0] - m) / 1024, time.time() - s))
+
+ m = meminfo()[0]
+ s = time.time()
+ for piecefield in list(piecefields.values()):
+ packed = piecefield.pack()
+
+ print("Pack x10000: +%sKB in %.3fs (len: %s)" % ((meminfo()[0] - m) / 1024, time.time() - s, len(packed)))
+
+ m = meminfo()[0]
+ s = time.time()
+ for piecefield in list(piecefields.values()):
+ piecefield.unpack(packed)
+
+ print("Unpack x10000: +%sKB in %.3fs (len: %s)" % ((meminfo()[0] - m) / 1024, time.time() - s, len(piecefields[0].data)))
+
+ piecefields = {}
diff --git a/plugins/Bigfile/BigfilePlugin.py b/plugins/Bigfile/BigfilePlugin.py
new file mode 100644
index 00000000..03a0f44f
--- /dev/null
+++ b/plugins/Bigfile/BigfilePlugin.py
@@ -0,0 +1,784 @@
+import time
+import os
+import subprocess
+import shutil
+import collections
+import math
+import warnings
+import base64
+import binascii
+import json
+
+import gevent
+import gevent.lock
+
+from Plugin import PluginManager
+from Debug import Debug
+from Crypt import CryptHash
+with warnings.catch_warnings():
+ warnings.filterwarnings("ignore") # Ignore missing sha3 warning
+ import merkletools
+
+from util import helper
+from util import Msgpack
+import util
+from .BigfilePiecefield import BigfilePiecefield, BigfilePiecefieldPacked
+
+
+# We can only import plugin host clases after the plugins are loaded
+@PluginManager.afterLoad
+def importPluginnedClasses():
+ global VerifyError, config
+ from Content.ContentManager import VerifyError
+ from Config import config
+
+if "upload_nonces" not in locals():
+ upload_nonces = {}
+
+
+@PluginManager.registerTo("UiRequest")
+class UiRequestPlugin(object):
+ def isCorsAllowed(self, path):
+ if path == "/ZeroNet-Internal/BigfileUpload":
+ return True
+ else:
+ return super(UiRequestPlugin, self).isCorsAllowed(path)
+
+ @helper.encodeResponse
+ def actionBigfileUpload(self):
+ nonce = self.get.get("upload_nonce")
+ if nonce not in upload_nonces:
+ return self.error403("Upload nonce error.")
+
+ upload_info = upload_nonces[nonce]
+ del upload_nonces[nonce]
+
+ self.sendHeader(200, "text/html", noscript=True, extra_headers={
+ "Access-Control-Allow-Origin": "null",
+ "Access-Control-Allow-Credentials": "true"
+ })
+
+ self.readMultipartHeaders(self.env['wsgi.input']) # Skip http headers
+
+ site = upload_info["site"]
+ inner_path = upload_info["inner_path"]
+
+ with site.storage.open(inner_path, "wb", create_dirs=True) as out_file:
+ merkle_root, piece_size, piecemap_info = site.content_manager.hashBigfile(
+ self.env['wsgi.input'], upload_info["size"], upload_info["piece_size"], out_file
+ )
+
+ if len(piecemap_info["sha512_pieces"]) == 1: # Small file, don't split
+ hash = binascii.hexlify(piecemap_info["sha512_pieces"][0])
+ hash_id = site.content_manager.hashfield.getHashId(hash)
+ site.content_manager.optionalDownloaded(inner_path, hash_id, upload_info["size"], own=True)
+
+ else: # Big file
+ file_name = helper.getFilename(inner_path)
+ site.storage.open(upload_info["piecemap"], "wb").write(Msgpack.pack({file_name: piecemap_info}))
+
+ # Find piecemap and file relative path to content.json
+ file_info = site.content_manager.getFileInfo(inner_path, new_file=True)
+ content_inner_path_dir = helper.getDirname(file_info["content_inner_path"])
+ piecemap_relative_path = upload_info["piecemap"][len(content_inner_path_dir):]
+ file_relative_path = inner_path[len(content_inner_path_dir):]
+
+ # Add file to content.json
+ if site.storage.isFile(file_info["content_inner_path"]):
+ content = site.storage.loadJson(file_info["content_inner_path"])
+ else:
+ content = {}
+ if "files_optional" not in content:
+ content["files_optional"] = {}
+
+ content["files_optional"][file_relative_path] = {
+ "sha512": merkle_root,
+ "size": upload_info["size"],
+ "piecemap": piecemap_relative_path,
+ "piece_size": piece_size
+ }
+
+ merkle_root_hash_id = site.content_manager.hashfield.getHashId(merkle_root)
+ site.content_manager.optionalDownloaded(inner_path, merkle_root_hash_id, upload_info["size"], own=True)
+ site.storage.writeJson(file_info["content_inner_path"], content)
+
+ site.content_manager.contents.loadItem(file_info["content_inner_path"]) # reload cache
+
+ return json.dumps({
+ "merkle_root": merkle_root,
+ "piece_num": len(piecemap_info["sha512_pieces"]),
+ "piece_size": piece_size,
+ "inner_path": inner_path
+ })
+
+ def readMultipartHeaders(self, wsgi_input):
+ found = False
+ for i in range(100):
+ line = wsgi_input.readline()
+ if line == b"\r\n":
+ found = True
+ break
+ if not found:
+ raise Exception("No multipart header found")
+ return i
+
+ def actionFile(self, file_path, *args, **kwargs):
+ if kwargs.get("file_size", 0) > 1024 * 1024 and kwargs.get("path_parts"): # Only check files larger than 1MB
+ path_parts = kwargs["path_parts"]
+ site = self.server.site_manager.get(path_parts["address"])
+ big_file = site.storage.openBigfile(path_parts["inner_path"], prebuffer=2 * 1024 * 1024)
+ if big_file:
+ kwargs["file_obj"] = big_file
+ kwargs["file_size"] = big_file.size
+
+ return super(UiRequestPlugin, self).actionFile(file_path, *args, **kwargs)
+
+
+@PluginManager.registerTo("UiWebsocket")
+class UiWebsocketPlugin(object):
+ def actionBigfileUploadInit(self, to, inner_path, size):
+ valid_signers = self.site.content_manager.getValidSigners(inner_path)
+ auth_address = self.user.getAuthAddress(self.site.address)
+ if not self.site.settings["own"] and auth_address not in valid_signers:
+ self.log.error("FileWrite forbidden %s not in valid_signers %s" % (auth_address, valid_signers))
+ return self.response(to, {"error": "Forbidden, you can only modify your own files"})
+
+ nonce = CryptHash.random()
+ piece_size = 1024 * 1024
+ inner_path = self.site.content_manager.sanitizePath(inner_path)
+ file_info = self.site.content_manager.getFileInfo(inner_path, new_file=True)
+
+ content_inner_path_dir = helper.getDirname(file_info["content_inner_path"])
+ file_relative_path = inner_path[len(content_inner_path_dir):]
+
+ upload_nonces[nonce] = {
+ "added": time.time(),
+ "site": self.site,
+ "inner_path": inner_path,
+ "websocket_client": self,
+ "size": size,
+ "piece_size": piece_size,
+ "piecemap": inner_path + ".piecemap.msgpack"
+ }
+ return {
+ "url": "/ZeroNet-Internal/BigfileUpload?upload_nonce=" + nonce,
+ "piece_size": piece_size,
+ "inner_path": inner_path,
+ "file_relative_path": file_relative_path
+ }
+
+ def actionSiteSetAutodownloadBigfileLimit(self, to, limit):
+ permissions = self.getPermissions(to)
+ if "ADMIN" not in permissions:
+ return self.response(to, "You don't have permission to run this command")
+
+ self.site.settings["autodownload_bigfile_size_limit"] = int(limit)
+ self.response(to, "ok")
+
+ def actionFileDelete(self, to, inner_path):
+ piecemap_inner_path = inner_path + ".piecemap.msgpack"
+ if self.hasFilePermission(inner_path) and self.site.storage.isFile(piecemap_inner_path):
+ # Also delete .piecemap.msgpack file if exists
+ self.log.debug("Deleting piecemap: %s" % piecemap_inner_path)
+ file_info = self.site.content_manager.getFileInfo(piecemap_inner_path)
+ if file_info:
+ content_json = self.site.storage.loadJson(file_info["content_inner_path"])
+ relative_path = file_info["relative_path"]
+ if relative_path in content_json.get("files_optional", {}):
+ del content_json["files_optional"][relative_path]
+ self.site.storage.writeJson(file_info["content_inner_path"], content_json)
+ self.site.content_manager.loadContent(file_info["content_inner_path"], add_bad_files=False, force=True)
+ try:
+ self.site.storage.delete(piecemap_inner_path)
+ except Exception as err:
+ self.log.error("File %s delete error: %s" % (piecemap_inner_path, err))
+
+ return super(UiWebsocketPlugin, self).actionFileDelete(to, inner_path)
+
+
+@PluginManager.registerTo("ContentManager")
+class ContentManagerPlugin(object):
+ def getFileInfo(self, inner_path, *args, **kwargs):
+ if "|" not in inner_path:
+ return super(ContentManagerPlugin, self).getFileInfo(inner_path, *args, **kwargs)
+
+ inner_path, file_range = inner_path.split("|")
+ pos_from, pos_to = map(int, file_range.split("-"))
+ file_info = super(ContentManagerPlugin, self).getFileInfo(inner_path, *args, **kwargs)
+ return file_info
+
+ def readFile(self, file_in, size, buff_size=1024 * 64):
+ part_num = 0
+ recv_left = size
+
+ while 1:
+ part_num += 1
+ read_size = min(buff_size, recv_left)
+ part = file_in.read(read_size)
+
+ if not part:
+ break
+ yield part
+
+ if part_num % 100 == 0: # Avoid blocking ZeroNet execution during upload
+ time.sleep(0.001)
+
+ recv_left -= read_size
+ if recv_left <= 0:
+ break
+
+ def hashBigfile(self, file_in, size, piece_size=1024 * 1024, file_out=None):
+ self.site.settings["has_bigfile"] = True
+
+ recv = 0
+ try:
+ piece_hash = CryptHash.sha512t()
+ piece_hashes = []
+ piece_recv = 0
+
+ mt = merkletools.MerkleTools()
+ mt.hash_function = CryptHash.sha512t
+
+ part = ""
+ for part in self.readFile(file_in, size):
+ if file_out:
+ file_out.write(part)
+
+ recv += len(part)
+ piece_recv += len(part)
+ piece_hash.update(part)
+ if piece_recv >= piece_size:
+ piece_digest = piece_hash.digest()
+ piece_hashes.append(piece_digest)
+ mt.leaves.append(piece_digest)
+ piece_hash = CryptHash.sha512t()
+ piece_recv = 0
+
+ if len(piece_hashes) % 100 == 0 or recv == size:
+ self.log.info("- [HASHING:%.0f%%] Pieces: %s, %.1fMB/%.1fMB" % (
+ float(recv) / size * 100, len(piece_hashes), recv / 1024 / 1024, size / 1024 / 1024
+ ))
+ part = ""
+ if len(part) > 0:
+ piece_digest = piece_hash.digest()
+ piece_hashes.append(piece_digest)
+ mt.leaves.append(piece_digest)
+ except Exception as err:
+ raise err
+ finally:
+ if file_out:
+ file_out.close()
+
+ mt.make_tree()
+ merkle_root = mt.get_merkle_root()
+ if type(merkle_root) is bytes: # Python <3.5
+ merkle_root = merkle_root.decode()
+ return merkle_root, piece_size, {
+ "sha512_pieces": piece_hashes
+ }
+
+ def hashFile(self, dir_inner_path, file_relative_path, optional=False):
+ inner_path = dir_inner_path + file_relative_path
+
+ file_size = self.site.storage.getSize(inner_path)
+ # Only care about optional files >1MB
+ if not optional or file_size < 1 * 1024 * 1024:
+ return super(ContentManagerPlugin, self).hashFile(dir_inner_path, file_relative_path, optional)
+
+ back = {}
+ content = self.contents.get(dir_inner_path + "content.json")
+
+ hash = None
+ piecemap_relative_path = None
+ piece_size = None
+
+ # Don't re-hash if it's already in content.json
+ if content and file_relative_path in content.get("files_optional", {}):
+ file_node = content["files_optional"][file_relative_path]
+ if file_node["size"] == file_size:
+ self.log.info("- [SAME SIZE] %s" % file_relative_path)
+ hash = file_node.get("sha512")
+ piecemap_relative_path = file_node.get("piecemap")
+ piece_size = file_node.get("piece_size")
+
+ if not hash or not piecemap_relative_path: # Not in content.json yet
+ if file_size < 5 * 1024 * 1024: # Don't create piecemap automatically for files smaller than 5MB
+ return super(ContentManagerPlugin, self).hashFile(dir_inner_path, file_relative_path, optional)
+
+ self.log.info("- [HASHING] %s" % file_relative_path)
+ merkle_root, piece_size, piecemap_info = self.hashBigfile(self.site.storage.open(inner_path, "rb"), file_size)
+ if not hash:
+ hash = merkle_root
+
+ if not piecemap_relative_path:
+ file_name = helper.getFilename(file_relative_path)
+ piecemap_relative_path = file_relative_path + ".piecemap.msgpack"
+ piecemap_inner_path = inner_path + ".piecemap.msgpack"
+
+ self.site.storage.open(piecemap_inner_path, "wb").write(Msgpack.pack({file_name: piecemap_info}))
+
+ back.update(super(ContentManagerPlugin, self).hashFile(dir_inner_path, piecemap_relative_path, optional=True))
+
+ piece_num = int(math.ceil(float(file_size) / piece_size))
+
+ # Add the merkle root to hashfield
+ hash_id = self.site.content_manager.hashfield.getHashId(hash)
+ self.optionalDownloaded(inner_path, hash_id, file_size, own=True)
+ self.site.storage.piecefields[hash].frombytes(b"\x01" * piece_num)
+
+ back[file_relative_path] = {"sha512": hash, "size": file_size, "piecemap": piecemap_relative_path, "piece_size": piece_size}
+ return back
+
+ def getPiecemap(self, inner_path):
+ file_info = self.site.content_manager.getFileInfo(inner_path)
+ piecemap_inner_path = helper.getDirname(file_info["content_inner_path"]) + file_info["piecemap"]
+ self.site.needFile(piecemap_inner_path, priority=20)
+ piecemap = Msgpack.unpack(self.site.storage.open(piecemap_inner_path, "rb").read())[helper.getFilename(inner_path)]
+ piecemap["piece_size"] = file_info["piece_size"]
+ return piecemap
+
+ def verifyPiece(self, inner_path, pos, piece):
+ piecemap = self.getPiecemap(inner_path)
+ piece_i = int(pos / piecemap["piece_size"])
+ if CryptHash.sha512sum(piece, format="digest") != piecemap["sha512_pieces"][piece_i]:
+ raise VerifyError("Invalid hash")
+ return True
+
+ def verifyFile(self, inner_path, file, ignore_same=True):
+ if "|" not in inner_path:
+ return super(ContentManagerPlugin, self).verifyFile(inner_path, file, ignore_same)
+
+ inner_path, file_range = inner_path.split("|")
+ pos_from, pos_to = map(int, file_range.split("-"))
+
+ return self.verifyPiece(inner_path, pos_from, file)
+
+ def optionalDownloaded(self, inner_path, hash_id, size=None, own=False):
+ if "|" in inner_path:
+ inner_path, file_range = inner_path.split("|")
+ pos_from, pos_to = map(int, file_range.split("-"))
+ file_info = self.getFileInfo(inner_path)
+
+ # Mark piece downloaded
+ piece_i = int(pos_from / file_info["piece_size"])
+ self.site.storage.piecefields[file_info["sha512"]][piece_i] = b"\x01"
+
+ # Only add to site size on first request
+ if hash_id in self.hashfield:
+ size = 0
+ elif size > 1024 * 1024:
+ file_info = self.getFileInfo(inner_path)
+ if file_info and "sha512" in file_info: # We already have the file, but not in piecefield
+ sha512 = file_info["sha512"]
+ if sha512 not in self.site.storage.piecefields:
+ self.site.storage.checkBigfile(inner_path)
+
+ return super(ContentManagerPlugin, self).optionalDownloaded(inner_path, hash_id, size, own)
+
+ def optionalRemoved(self, inner_path, hash_id, size=None):
+ if size and size > 1024 * 1024:
+ file_info = self.getFileInfo(inner_path)
+ sha512 = file_info["sha512"]
+ if sha512 in self.site.storage.piecefields:
+ del self.site.storage.piecefields[sha512]
+
+ # Also remove other pieces of the file from download queue
+ for key in list(self.site.bad_files.keys()):
+ if key.startswith(inner_path + "|"):
+ del self.site.bad_files[key]
+ self.site.worker_manager.removeSolvedFileTasks()
+ return super(ContentManagerPlugin, self).optionalRemoved(inner_path, hash_id, size)
+
+
+@PluginManager.registerTo("SiteStorage")
+class SiteStoragePlugin(object):
+ def __init__(self, *args, **kwargs):
+ super(SiteStoragePlugin, self).__init__(*args, **kwargs)
+ self.piecefields = collections.defaultdict(BigfilePiecefield)
+ if "piecefields" in self.site.settings.get("cache", {}):
+ for sha512, piecefield_packed in self.site.settings["cache"].get("piecefields").items():
+ if piecefield_packed:
+ self.piecefields[sha512].unpack(base64.b64decode(piecefield_packed))
+ self.site.settings["cache"]["piecefields"] = {}
+
+ def createSparseFile(self, inner_path, size, sha512=None):
+ file_path = self.getPath(inner_path)
+
+ file_dir = os.path.dirname(file_path)
+ if not os.path.isdir(file_dir):
+ os.makedirs(file_dir)
+
+ f = open(file_path, 'wb')
+ f.truncate(min(1024 * 1024 * 5, size)) # Only pre-allocate up to 5MB
+ f.close()
+ if os.name == "nt":
+ startupinfo = subprocess.STARTUPINFO()
+ startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
+ subprocess.call(["fsutil", "sparse", "setflag", file_path], close_fds=True, startupinfo=startupinfo)
+
+ if sha512 and sha512 in self.piecefields:
+ self.log.debug("%s: File not exists, but has piecefield. Deleting piecefield." % inner_path)
+ del self.piecefields[sha512]
+
+ def write(self, inner_path, content):
+ if "|" not in inner_path:
+ return super(SiteStoragePlugin, self).write(inner_path, content)
+
+ # Write to specific position by passing |{pos} after the filename
+ inner_path, file_range = inner_path.split("|")
+ pos_from, pos_to = map(int, file_range.split("-"))
+ file_path = self.getPath(inner_path)
+
+ # Create dir if not exist
+ file_dir = os.path.dirname(file_path)
+ if not os.path.isdir(file_dir):
+ os.makedirs(file_dir)
+
+ if not os.path.isfile(file_path):
+ file_info = self.site.content_manager.getFileInfo(inner_path)
+ self.createSparseFile(inner_path, file_info["size"])
+
+ # Write file
+ with open(file_path, "rb+") as file:
+ file.seek(pos_from)
+ if hasattr(content, 'read'): # File-like object
+ shutil.copyfileobj(content, file) # Write buff to disk
+ else: # Simple string
+ file.write(content)
+ del content
+ self.onUpdated(inner_path)
+
+ def checkBigfile(self, inner_path):
+ file_info = self.site.content_manager.getFileInfo(inner_path)
+ if not file_info or (file_info and "piecemap" not in file_info): # It's not a big file
+ return False
+
+ self.site.settings["has_bigfile"] = True
+ file_path = self.getPath(inner_path)
+ sha512 = file_info["sha512"]
+ piece_num = int(math.ceil(float(file_info["size"]) / file_info["piece_size"]))
+ if os.path.isfile(file_path):
+ if sha512 not in self.piecefields:
+ if open(file_path, "rb").read(128) == b"\0" * 128:
+ piece_data = b"\x00"
+ else:
+ piece_data = b"\x01"
+ self.log.debug("%s: File exists, but not in piecefield. Filling piecefiled with %s * %s." % (inner_path, piece_num, piece_data))
+ self.piecefields[sha512].frombytes(piece_data * piece_num)
+ else:
+ self.log.debug("Creating bigfile: %s" % inner_path)
+ self.createSparseFile(inner_path, file_info["size"], sha512)
+ self.piecefields[sha512].frombytes(b"\x00" * piece_num)
+ self.log.debug("Created bigfile: %s" % inner_path)
+ return True
+
+ def openBigfile(self, inner_path, prebuffer=0):
+ if not self.checkBigfile(inner_path):
+ return False
+ self.site.needFile(inner_path, blocking=False) # Download piecemap
+ return BigFile(self.site, inner_path, prebuffer=prebuffer)
+
+
+class BigFile(object):
+ def __init__(self, site, inner_path, prebuffer=0):
+ self.site = site
+ self.inner_path = inner_path
+ file_path = site.storage.getPath(inner_path)
+ file_info = self.site.content_manager.getFileInfo(inner_path)
+ self.piece_size = file_info["piece_size"]
+ self.sha512 = file_info["sha512"]
+ self.size = file_info["size"]
+ self.prebuffer = prebuffer
+ self.read_bytes = 0
+
+ self.piecefield = self.site.storage.piecefields[self.sha512]
+ self.f = open(file_path, "rb+")
+ self.read_lock = gevent.lock.Semaphore()
+
+ def read(self, buff=64 * 1024):
+ with self.read_lock:
+ pos = self.f.tell()
+ read_until = min(self.size, pos + buff)
+ requests = []
+ # Request all required blocks
+ while 1:
+ piece_i = int(pos / self.piece_size)
+ if piece_i * self.piece_size >= read_until:
+ break
+ pos_from = piece_i * self.piece_size
+ pos_to = pos_from + self.piece_size
+ if not self.piecefield[piece_i]:
+ requests.append(self.site.needFile("%s|%s-%s" % (self.inner_path, pos_from, pos_to), blocking=False, update=True, priority=10))
+ pos += self.piece_size
+
+ if not all(requests):
+ return None
+
+ # Request prebuffer
+ if self.prebuffer:
+ prebuffer_until = min(self.size, read_until + self.prebuffer)
+ priority = 3
+ while 1:
+ piece_i = int(pos / self.piece_size)
+ if piece_i * self.piece_size >= prebuffer_until:
+ break
+ pos_from = piece_i * self.piece_size
+ pos_to = pos_from + self.piece_size
+ if not self.piecefield[piece_i]:
+ self.site.needFile("%s|%s-%s" % (self.inner_path, pos_from, pos_to), blocking=False, update=True, priority=max(0, priority))
+ priority -= 1
+ pos += self.piece_size
+
+ gevent.joinall(requests)
+ self.read_bytes += buff
+
+ # Increase buffer for long reads
+ if self.read_bytes > 7 * 1024 * 1024 and self.prebuffer < 5 * 1024 * 1024:
+ self.site.log.debug("%s: Increasing bigfile buffer size to 5MB..." % self.inner_path)
+ self.prebuffer = 5 * 1024 * 1024
+
+ return self.f.read(buff)
+
+ def seek(self, pos, whence=0):
+ with self.read_lock:
+ if whence == 2: # Relative from file end
+ pos = self.size + pos # Use the real size instead of size on the disk
+ whence = 0
+ return self.f.seek(pos, whence)
+
+ def tell(self):
+ return self.f.tell()
+
+ def close(self):
+ self.f.close()
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ self.close()
+
+
+@PluginManager.registerTo("WorkerManager")
+class WorkerManagerPlugin(object):
+ def addTask(self, inner_path, *args, **kwargs):
+ file_info = kwargs.get("file_info")
+ if file_info and "piecemap" in file_info: # Bigfile
+ self.site.settings["has_bigfile"] = True
+
+ piecemap_inner_path = helper.getDirname(file_info["content_inner_path"]) + file_info["piecemap"]
+ piecemap_task = None
+ if not self.site.storage.isFile(piecemap_inner_path):
+ # Start download piecemap
+ piecemap_task = super(WorkerManagerPlugin, self).addTask(piecemap_inner_path, priority=30)
+ autodownload_bigfile_size_limit = self.site.settings.get("autodownload_bigfile_size_limit", config.autodownload_bigfile_size_limit)
+ if "|" not in inner_path and self.site.isDownloadable(inner_path) and file_info["size"] / 1024 / 1024 <= autodownload_bigfile_size_limit:
+ gevent.spawn_later(0.1, self.site.needFile, inner_path + "|all") # Download all pieces
+
+ if "|" in inner_path:
+ # Start download piece
+ task = super(WorkerManagerPlugin, self).addTask(inner_path, *args, **kwargs)
+
+ inner_path, file_range = inner_path.split("|")
+ pos_from, pos_to = map(int, file_range.split("-"))
+ task["piece_i"] = int(pos_from / file_info["piece_size"])
+ task["sha512"] = file_info["sha512"]
+ else:
+ if inner_path in self.site.bad_files:
+ del self.site.bad_files[inner_path]
+ if piecemap_task:
+ task = piecemap_task
+ else:
+ fake_evt = gevent.event.AsyncResult() # Don't download anything if no range specified
+ fake_evt.set(True)
+ task = {"evt": fake_evt}
+
+ if not self.site.storage.isFile(inner_path):
+ self.site.storage.createSparseFile(inner_path, file_info["size"], file_info["sha512"])
+ piece_num = int(math.ceil(float(file_info["size"]) / file_info["piece_size"]))
+ self.site.storage.piecefields[file_info["sha512"]].frombytes(b"\x00" * piece_num)
+ else:
+ task = super(WorkerManagerPlugin, self).addTask(inner_path, *args, **kwargs)
+ return task
+
+ def taskAddPeer(self, task, peer):
+ if "piece_i" in task:
+ if not peer.piecefields[task["sha512"]][task["piece_i"]]:
+ if task["sha512"] not in peer.piecefields:
+ gevent.spawn(peer.updatePiecefields, force=True)
+ elif not task["peers"]:
+ gevent.spawn(peer.updatePiecefields)
+
+ return False # Deny to add peers to task if file not in piecefield
+ return super(WorkerManagerPlugin, self).taskAddPeer(task, peer)
+
+
+@PluginManager.registerTo("FileRequest")
+class FileRequestPlugin(object):
+ def isReadable(self, site, inner_path, file, pos):
+ # Peek into file
+ if file.read(10) == b"\0" * 10:
+ # Looks empty, but makes sures we don't have that piece
+ file_info = site.content_manager.getFileInfo(inner_path)
+ if "piece_size" in file_info:
+ piece_i = int(pos / file_info["piece_size"])
+ if not site.storage.piecefields[file_info["sha512"]][piece_i]:
+ return False
+ # Seek back to position we want to read
+ file.seek(pos)
+ return super(FileRequestPlugin, self).isReadable(site, inner_path, file, pos)
+
+ def actionGetPiecefields(self, params):
+ site = self.sites.get(params["site"])
+ if not site or not site.isServing(): # Site unknown or not serving
+ self.response({"error": "Unknown site"})
+ return False
+
+ # Add peer to site if not added before
+ peer = site.addPeer(self.connection.ip, self.connection.port, return_peer=True)
+ if not peer.connection: # Just added
+ peer.connect(self.connection) # Assign current connection to peer
+
+ piecefields_packed = {sha512: piecefield.pack() for sha512, piecefield in site.storage.piecefields.items()}
+ self.response({"piecefields_packed": piecefields_packed})
+
+ def actionSetPiecefields(self, params):
+ site = self.sites.get(params["site"])
+ if not site or not site.isServing(): # Site unknown or not serving
+ self.response({"error": "Unknown site"})
+ self.connection.badAction(5)
+ return False
+
+ # Add or get peer
+ peer = site.addPeer(self.connection.ip, self.connection.port, return_peer=True, connection=self.connection)
+ if not peer.connection:
+ peer.connect(self.connection)
+
+ peer.piecefields = collections.defaultdict(BigfilePiecefieldPacked)
+ for sha512, piecefield_packed in params["piecefields_packed"].items():
+ peer.piecefields[sha512].unpack(piecefield_packed)
+ site.settings["has_bigfile"] = True
+
+ self.response({"ok": "Updated"})
+
+
+@PluginManager.registerTo("Peer")
+class PeerPlugin(object):
+ def __getattr__(self, key):
+ if key == "piecefields":
+ self.piecefields = collections.defaultdict(BigfilePiecefieldPacked)
+ return self.piecefields
+ elif key == "time_piecefields_updated":
+ self.time_piecefields_updated = None
+ return self.time_piecefields_updated
+ else:
+ return super(PeerPlugin, self).__getattr__(key)
+
+ @util.Noparallel(ignore_args=True)
+ def updatePiecefields(self, force=False):
+ if self.connection and self.connection.handshake.get("rev", 0) < 2190:
+ return False # Not supported
+
+ # Don't update piecefield again in 1 min
+ if self.time_piecefields_updated and time.time() - self.time_piecefields_updated < 60 and not force:
+ return False
+
+ self.time_piecefields_updated = time.time()
+ res = self.request("getPiecefields", {"site": self.site.address})
+ if not res or "error" in res:
+ return False
+
+ self.piecefields = collections.defaultdict(BigfilePiecefieldPacked)
+ try:
+ for sha512, piecefield_packed in res["piecefields_packed"].items():
+ self.piecefields[sha512].unpack(piecefield_packed)
+ except Exception as err:
+ self.log("Invalid updatePiecefields response: %s" % Debug.formatException(err))
+
+ return self.piecefields
+
+ def sendMyHashfield(self, *args, **kwargs):
+ return super(PeerPlugin, self).sendMyHashfield(*args, **kwargs)
+
+ def updateHashfield(self, *args, **kwargs):
+ if self.site.settings.get("has_bigfile"):
+ thread = gevent.spawn(self.updatePiecefields, *args, **kwargs)
+ back = super(PeerPlugin, self).updateHashfield(*args, **kwargs)
+ thread.join()
+ return back
+ else:
+ return super(PeerPlugin, self).updateHashfield(*args, **kwargs)
+
+ def getFile(self, site, inner_path, *args, **kwargs):
+ if "|" in inner_path:
+ inner_path, file_range = inner_path.split("|")
+ pos_from, pos_to = map(int, file_range.split("-"))
+ kwargs["pos_from"] = pos_from
+ kwargs["pos_to"] = pos_to
+ return super(PeerPlugin, self).getFile(site, inner_path, *args, **kwargs)
+
+
+@PluginManager.registerTo("Site")
+class SitePlugin(object):
+ def isFileDownloadAllowed(self, inner_path, file_info):
+ if "piecemap" in file_info:
+ file_size_mb = file_info["size"] / 1024 / 1024
+ if config.bigfile_size_limit and file_size_mb > config.bigfile_size_limit:
+ self.log.debug(
+ "Bigfile size %s too large: %sMB > %sMB, skipping..." %
+ (inner_path, file_size_mb, config.bigfile_size_limit)
+ )
+ return False
+
+ file_info = file_info.copy()
+ file_info["size"] = file_info["piece_size"]
+ return super(SitePlugin, self).isFileDownloadAllowed(inner_path, file_info)
+
+ def getSettingsCache(self):
+ back = super(SitePlugin, self).getSettingsCache()
+ if self.storage.piecefields:
+ back["piecefields"] = {sha512: base64.b64encode(piecefield.pack()).decode("utf8") for sha512, piecefield in self.storage.piecefields.items()}
+ return back
+
+ def needFile(self, inner_path, *args, **kwargs):
+ if inner_path.endswith("|all"):
+ @util.Pooled(20)
+ def pooledNeedBigfile(inner_path, *args, **kwargs):
+ if inner_path not in self.bad_files:
+ self.log.debug("Cancelled piece, skipping %s" % inner_path)
+ return False
+ return self.needFile(inner_path, *args, **kwargs)
+
+ inner_path = inner_path.replace("|all", "")
+ file_info = self.needFileInfo(inner_path)
+ file_size = file_info["size"]
+ piece_size = file_info["piece_size"]
+
+ piece_num = int(math.ceil(float(file_size) / piece_size))
+
+ file_threads = []
+
+ piecefield = self.storage.piecefields.get(file_info["sha512"])
+
+ for piece_i in range(piece_num):
+ piece_from = piece_i * piece_size
+ piece_to = min(file_size, piece_from + piece_size)
+ if not piecefield or not piecefield[piece_i]:
+ inner_path_piece = "%s|%s-%s" % (inner_path, piece_from, piece_to)
+ self.bad_files[inner_path_piece] = self.bad_files.get(inner_path_piece, 1)
+ res = pooledNeedBigfile(inner_path_piece, blocking=False)
+ if res is not True and res is not False:
+ file_threads.append(res)
+ gevent.joinall(file_threads)
+ else:
+ return super(SitePlugin, self).needFile(inner_path, *args, **kwargs)
+
+
+@PluginManager.registerTo("ConfigPlugin")
+class ConfigPlugin(object):
+ def createArguments(self):
+ group = self.parser.add_argument_group("Bigfile plugin")
+ group.add_argument('--autodownload_bigfile_size_limit', help='Also download bigfiles smaller than this limit if help distribute option is checked', default=1, metavar="MB", type=int)
+ group.add_argument('--bigfile_size_limit', help='Maximum size of downloaded big files', default=False, metavar="MB", type=int)
+
+ return super(ConfigPlugin, self).createArguments()
diff --git a/plugins/Bigfile/Test/TestBigfile.py b/plugins/Bigfile/Test/TestBigfile.py
new file mode 100644
index 00000000..7d112860
--- /dev/null
+++ b/plugins/Bigfile/Test/TestBigfile.py
@@ -0,0 +1,574 @@
+import time
+import io
+import binascii
+
+import pytest
+import mock
+
+from Connection import ConnectionServer
+from Content.ContentManager import VerifyError
+from File import FileServer
+from File import FileRequest
+from Worker import WorkerManager
+from Peer import Peer
+from Bigfile import BigfilePiecefield, BigfilePiecefieldPacked
+from Test import Spy
+from util import Msgpack
+
+
+@pytest.mark.usefixtures("resetSettings")
+@pytest.mark.usefixtures("resetTempSettings")
+class TestBigfile:
+ privatekey = "5KUh3PvNm5HUWoCfSUfcYvfQ2g3PrRNJWr6Q9eqdBGu23mtMntv"
+ piece_size = 1024 * 1024
+
+ def createBigfile(self, site, inner_path="data/optional.any.iso", pieces=10):
+ f = site.storage.open(inner_path, "w")
+ for i in range(pieces * 100):
+ f.write(("Test%s" % i).ljust(10, "-") * 1000)
+ f.close()
+ assert site.content_manager.sign("content.json", self.privatekey)
+ return inner_path
+
+ def testPiecemapCreate(self, site):
+ inner_path = self.createBigfile(site)
+ content = site.storage.loadJson("content.json")
+ assert "data/optional.any.iso" in content["files_optional"]
+ file_node = content["files_optional"][inner_path]
+ assert file_node["size"] == 10 * 1000 * 1000
+ assert file_node["sha512"] == "47a72cde3be80b4a829e7674f72b7c6878cf6a70b0c58c6aa6c17d7e9948daf6"
+ assert file_node["piecemap"] == inner_path + ".piecemap.msgpack"
+
+ piecemap = Msgpack.unpack(site.storage.open(file_node["piecemap"], "rb").read())["optional.any.iso"]
+ assert len(piecemap["sha512_pieces"]) == 10
+ assert piecemap["sha512_pieces"][0] != piecemap["sha512_pieces"][1]
+ assert binascii.hexlify(piecemap["sha512_pieces"][0]) == b"a73abad9992b3d0b672d0c2a292046695d31bebdcb1e150c8410bbe7c972eff3"
+
+ def testVerifyPiece(self, site):
+ inner_path = self.createBigfile(site)
+
+ # Verify all 10 piece
+ f = site.storage.open(inner_path, "rb")
+ for i in range(10):
+ piece = io.BytesIO(f.read(1024 * 1024))
+ piece.seek(0)
+ site.content_manager.verifyPiece(inner_path, i * 1024 * 1024, piece)
+ f.close()
+
+ # Try to verify piece 0 with piece 1 hash
+ with pytest.raises(VerifyError) as err:
+ i = 1
+ f = site.storage.open(inner_path, "rb")
+ piece = io.BytesIO(f.read(1024 * 1024))
+ f.close()
+ site.content_manager.verifyPiece(inner_path, i * 1024 * 1024, piece)
+ assert "Invalid hash" in str(err)
+
+ def testSparseFile(self, site):
+ inner_path = "sparsefile"
+
+ # Create a 100MB sparse file
+ site.storage.createSparseFile(inner_path, 100 * 1024 * 1024)
+
+ # Write to file beginning
+ s = time.time()
+ f = site.storage.write("%s|%s-%s" % (inner_path, 0, 1024 * 1024), b"hellostart" * 1024)
+ time_write_start = time.time() - s
+
+ # Write to file end
+ s = time.time()
+ f = site.storage.write("%s|%s-%s" % (inner_path, 99 * 1024 * 1024, 99 * 1024 * 1024 + 1024 * 1024), b"helloend" * 1024)
+ time_write_end = time.time() - s
+
+ # Verify writes
+ f = site.storage.open(inner_path)
+ assert f.read(10) == b"hellostart"
+ f.seek(99 * 1024 * 1024)
+ assert f.read(8) == b"helloend"
+ f.close()
+
+ site.storage.delete(inner_path)
+
+ # Writing to end shold not take much longer, than writing to start
+ assert time_write_end <= max(0.1, time_write_start * 1.1)
+
+ def testRangedFileRequest(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ file_server.sites[site.address] = site
+ client = FileServer(file_server.ip, 1545)
+ client.sites[site_temp.address] = site_temp
+ site_temp.connection_server = client
+ connection = client.getConnection(file_server.ip, 1544)
+
+ # Add file_server as peer to client
+ peer_file_server = site_temp.addPeer(file_server.ip, 1544)
+
+ buff = peer_file_server.getFile(site_temp.address, "%s|%s-%s" % (inner_path, 5 * 1024 * 1024, 6 * 1024 * 1024))
+
+ assert len(buff.getvalue()) == 1 * 1024 * 1024 # Correct block size
+ assert buff.getvalue().startswith(b"Test524") # Correct data
+ buff.seek(0)
+ assert site.content_manager.verifyPiece(inner_path, 5 * 1024 * 1024, buff) # Correct hash
+
+ connection.close()
+ client.stop()
+
+ def testRangedFileDownload(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Make sure the file and the piecemap in the optional hashfield
+ file_info = site.content_manager.getFileInfo(inner_path)
+ assert site.content_manager.hashfield.hasHash(file_info["sha512"])
+
+ piecemap_hash = site.content_manager.getFileInfo(file_info["piecemap"])["sha512"]
+ assert site.content_manager.hashfield.hasHash(piecemap_hash)
+
+ # Init client server
+ client = ConnectionServer(file_server.ip, 1545)
+ site_temp.connection_server = client
+ peer_client = site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ bad_files = site_temp.storage.verifyFiles(quick_check=True)["bad_files"]
+ assert not bad_files
+
+ # client_piecefield = peer_client.piecefields[file_info["sha512"]].tostring()
+ # assert client_piecefield == "1" * 10
+
+ # Download 5. and 10. block
+
+ site_temp.needFile("%s|%s-%s" % (inner_path, 5 * 1024 * 1024, 6 * 1024 * 1024))
+ site_temp.needFile("%s|%s-%s" % (inner_path, 9 * 1024 * 1024, 10 * 1024 * 1024))
+
+ # Verify 0. block not downloaded
+ f = site_temp.storage.open(inner_path)
+ assert f.read(10) == b"\0" * 10
+ # Verify 5. and 10. block downloaded
+ f.seek(5 * 1024 * 1024)
+ assert f.read(7) == b"Test524"
+ f.seek(9 * 1024 * 1024)
+ assert f.read(7) == b"943---T"
+
+ # Verify hashfield
+ assert set(site_temp.content_manager.hashfield) == set([18343, 43727]) # 18343: data/optional.any.iso, 43727: data/optional.any.iso.hashmap.msgpack
+
+ def testOpenBigfile(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Init client server
+ client = ConnectionServer(file_server.ip, 1545)
+ site_temp.connection_server = client
+ site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ # Open virtual file
+ assert not site_temp.storage.isFile(inner_path)
+
+ with site_temp.storage.openBigfile(inner_path) as f:
+ with Spy.Spy(FileRequest, "route") as requests:
+ f.seek(5 * 1024 * 1024)
+ assert f.read(7) == b"Test524"
+
+ f.seek(9 * 1024 * 1024)
+ assert f.read(7) == b"943---T"
+
+ assert len(requests) == 4 # 1x peicemap + 1x getpiecefield + 2x for pieces
+
+ assert set(site_temp.content_manager.hashfield) == set([18343, 43727])
+
+ assert site_temp.storage.piecefields[f.sha512].tostring() == "0000010001"
+ assert f.sha512 in site_temp.getSettingsCache()["piecefields"]
+
+ # Test requesting already downloaded
+ with Spy.Spy(FileRequest, "route") as requests:
+ f.seek(5 * 1024 * 1024)
+ assert f.read(7) == b"Test524"
+
+ assert len(requests) == 0
+
+ # Test requesting multi-block overflow reads
+ with Spy.Spy(FileRequest, "route") as requests:
+ f.seek(5 * 1024 * 1024) # We already have this block
+ data = f.read(1024 * 1024 * 3) # Our read overflow to 6. and 7. block
+ assert data.startswith(b"Test524")
+ assert data.endswith(b"Test838-")
+ assert b"\0" not in data # No null bytes allowed
+
+ assert len(requests) == 2 # Two block download
+
+ # Test out of range request
+ f.seek(5 * 1024 * 1024)
+ data = f.read(1024 * 1024 * 30)
+ assert len(data) == 10 * 1000 * 1000 - (5 * 1024 * 1024)
+
+ f.seek(30 * 1024 * 1024)
+ data = f.read(1024 * 1024 * 30)
+ assert len(data) == 0
+
+ @pytest.mark.parametrize("piecefield_obj", [BigfilePiecefield, BigfilePiecefieldPacked])
+ def testPiecefield(self, piecefield_obj, site):
+ testdatas = [
+ b"\x01" * 100 + b"\x00" * 900 + b"\x01" * 4000 + b"\x00" * 4999 + b"\x01",
+ b"\x00\x01\x00\x01\x00\x01" * 10 + b"\x00\x01" * 90 + b"\x01\x00" * 400 + b"\x00" * 4999,
+ b"\x01" * 10000,
+ b"\x00" * 10000
+ ]
+ for testdata in testdatas:
+ piecefield = piecefield_obj()
+
+ piecefield.frombytes(testdata)
+ assert piecefield.tobytes() == testdata
+ assert piecefield[0] == testdata[0]
+ assert piecefield[100] == testdata[100]
+ assert piecefield[1000] == testdata[1000]
+ assert piecefield[len(testdata) - 1] == testdata[len(testdata) - 1]
+
+ packed = piecefield.pack()
+ piecefield_new = piecefield_obj()
+ piecefield_new.unpack(packed)
+ assert piecefield.tobytes() == piecefield_new.tobytes()
+ assert piecefield_new.tobytes() == testdata
+
+ def testFileGet(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Init client server
+ site_temp.connection_server = FileServer(file_server.ip, 1545)
+ site_temp.connection_server.sites[site_temp.address] = site_temp
+ site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ # Download second block
+ with site_temp.storage.openBigfile(inner_path) as f:
+ f.seek(1024 * 1024)
+ assert f.read(1024)[0:1] != b"\0"
+
+ # Make sure first block not download
+ with site_temp.storage.open(inner_path) as f:
+ assert f.read(1024)[0:1] == b"\0"
+
+ peer2 = site.addPeer(file_server.ip, 1545, return_peer=True)
+
+ # Should drop error on first block request
+ assert not peer2.getFile(site.address, "%s|0-%s" % (inner_path, 1024 * 1024 * 1))
+
+ # Should not drop error for second block request
+ assert peer2.getFile(site.address, "%s|%s-%s" % (inner_path, 1024 * 1024 * 1, 1024 * 1024 * 2))
+
+ def benchmarkPeerMemory(self, site, file_server):
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ import psutil, os
+ meminfo = psutil.Process(os.getpid()).memory_info
+
+ mem_s = meminfo()[0]
+ s = time.time()
+ for i in range(25000):
+ site.addPeer(file_server.ip, i)
+ print("%.3fs MEM: + %sKB" % (time.time() - s, (meminfo()[0] - mem_s) / 1024)) # 0.082s MEM: + 6800KB
+ print(list(site.peers.values())[0].piecefields)
+
+ def testUpdatePiecefield(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ server1 = file_server
+ server1.sites[site.address] = site
+ server2 = FileServer(file_server.ip, 1545)
+ server2.sites[site_temp.address] = site_temp
+ site_temp.connection_server = server2
+
+ # Add file_server as peer to client
+ server2_peer1 = site_temp.addPeer(file_server.ip, 1544)
+
+ # Testing piecefield sync
+ assert len(server2_peer1.piecefields) == 0
+ assert server2_peer1.updatePiecefields() # Query piecefields from peer
+ assert len(server2_peer1.piecefields) > 0
+
+ def testWorkerManagerPiecefieldDeny(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ server1 = file_server
+ server1.sites[site.address] = site
+ server2 = FileServer(file_server.ip, 1545)
+ server2.sites[site_temp.address] = site_temp
+ site_temp.connection_server = server2
+
+ # Add file_server as peer to client
+ server2_peer1 = site_temp.addPeer(file_server.ip, 1544) # Working
+
+ site_temp.downloadContent("content.json", download_files=False)
+ site_temp.needFile("data/optional.any.iso.piecemap.msgpack")
+
+ # Add fake peers with optional files downloaded
+ for i in range(5):
+ fake_peer = site_temp.addPeer("127.0.1.%s" % i, 1544)
+ fake_peer.hashfield = site.content_manager.hashfield
+ fake_peer.has_hashfield = True
+
+ with Spy.Spy(WorkerManager, "addWorker") as requests:
+ site_temp.needFile("%s|%s-%s" % (inner_path, 5 * 1024 * 1024, 6 * 1024 * 1024))
+ site_temp.needFile("%s|%s-%s" % (inner_path, 6 * 1024 * 1024, 7 * 1024 * 1024))
+
+ # It should only request parts from peer1 as the other peers does not have the requested parts in piecefields
+ assert len([request[1] for request in requests if request[1] != server2_peer1]) == 0
+
+ def testWorkerManagerPiecefieldDownload(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ server1 = file_server
+ server1.sites[site.address] = site
+ server2 = FileServer(file_server.ip, 1545)
+ server2.sites[site_temp.address] = site_temp
+ site_temp.connection_server = server2
+ sha512 = site.content_manager.getFileInfo(inner_path)["sha512"]
+
+ # Create 10 fake peer for each piece
+ for i in range(10):
+ peer = Peer(file_server.ip, 1544, site_temp, server2)
+ peer.piecefields[sha512][i] = b"\x01"
+ peer.updateHashfield = mock.MagicMock(return_value=False)
+ peer.updatePiecefields = mock.MagicMock(return_value=False)
+ peer.findHashIds = mock.MagicMock(return_value={"nope": []})
+ peer.hashfield = site.content_manager.hashfield
+ peer.has_hashfield = True
+ peer.key = "Peer:%s" % i
+ site_temp.peers["Peer:%s" % i] = peer
+
+ site_temp.downloadContent("content.json", download_files=False)
+ site_temp.needFile("data/optional.any.iso.piecemap.msgpack")
+
+ with Spy.Spy(Peer, "getFile") as requests:
+ for i in range(10):
+ site_temp.needFile("%s|%s-%s" % (inner_path, i * 1024 * 1024, (i + 1) * 1024 * 1024))
+
+ assert len(requests) == 10
+ for i in range(10):
+ assert requests[i][0] == site_temp.peers["Peer:%s" % i] # Every part should be requested from piece owner peer
+
+ def testDownloadStats(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Init client server
+ client = ConnectionServer(file_server.ip, 1545)
+ site_temp.connection_server = client
+ site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ # Open virtual file
+ assert not site_temp.storage.isFile(inner_path)
+
+ # Check size before downloads
+ assert site_temp.settings["size"] < 10 * 1024 * 1024
+ assert site_temp.settings["optional_downloaded"] == 0
+ size_piecemap = site_temp.content_manager.getFileInfo(inner_path + ".piecemap.msgpack")["size"]
+ size_bigfile = site_temp.content_manager.getFileInfo(inner_path)["size"]
+
+ with site_temp.storage.openBigfile(inner_path) as f:
+ assert b"\0" not in f.read(1024)
+ assert site_temp.settings["optional_downloaded"] == size_piecemap + size_bigfile
+
+ with site_temp.storage.openBigfile(inner_path) as f:
+ # Don't count twice
+ assert b"\0" not in f.read(1024)
+ assert site_temp.settings["optional_downloaded"] == size_piecemap + size_bigfile
+
+ # Add second block
+ assert b"\0" not in f.read(1024 * 1024)
+ assert site_temp.settings["optional_downloaded"] == size_piecemap + size_bigfile
+
+ def testPrebuffer(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Init client server
+ client = ConnectionServer(file_server.ip, 1545)
+ site_temp.connection_server = client
+ site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ # Open virtual file
+ assert not site_temp.storage.isFile(inner_path)
+
+ with site_temp.storage.openBigfile(inner_path, prebuffer=1024 * 1024 * 2) as f:
+ with Spy.Spy(FileRequest, "route") as requests:
+ f.seek(5 * 1024 * 1024)
+ assert f.read(7) == b"Test524"
+ # assert len(requests) == 3 # 1x piecemap + 1x getpiecefield + 1x for pieces
+ assert len([task for task in site_temp.worker_manager.tasks if task["inner_path"].startswith(inner_path)]) == 2
+
+ time.sleep(0.5) # Wait prebuffer download
+
+ sha512 = site.content_manager.getFileInfo(inner_path)["sha512"]
+ assert site_temp.storage.piecefields[sha512].tostring() == "0000011100"
+
+ # No prebuffer beyond end of the file
+ f.seek(9 * 1024 * 1024)
+ assert b"\0" not in f.read(7)
+
+ assert len([task for task in site_temp.worker_manager.tasks if task["inner_path"].startswith(inner_path)]) == 0
+
+ def testDownloadAllPieces(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Init client server
+ client = ConnectionServer(file_server.ip, 1545)
+ site_temp.connection_server = client
+ site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ # Open virtual file
+ assert not site_temp.storage.isFile(inner_path)
+
+ with Spy.Spy(FileRequest, "route") as requests:
+ site_temp.needFile("%s|all" % inner_path)
+
+ assert len(requests) == 12 # piecemap.msgpack, getPiecefields, 10 x piece
+
+ # Don't re-download already got pieces
+ with Spy.Spy(FileRequest, "route") as requests:
+ site_temp.needFile("%s|all" % inner_path)
+
+ assert len(requests) == 0
+
+ def testFileSize(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Init client server
+ client = ConnectionServer(file_server.ip, 1545)
+ site_temp.connection_server = client
+ site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ # Open virtual file
+ assert not site_temp.storage.isFile(inner_path)
+
+ # Download first block
+ site_temp.needFile("%s|%s-%s" % (inner_path, 0 * 1024 * 1024, 1 * 1024 * 1024))
+ assert site_temp.storage.getSize(inner_path) < 1000 * 1000 * 10 # Size on the disk should be smaller than the real size
+
+ site_temp.needFile("%s|%s-%s" % (inner_path, 9 * 1024 * 1024, 10 * 1024 * 1024))
+ assert site_temp.storage.getSize(inner_path) == site.storage.getSize(inner_path)
+
+ def testFileRename(self, file_server, site, site_temp):
+ inner_path = self.createBigfile(site)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Init client server
+ site_temp.connection_server = FileServer(file_server.ip, 1545)
+ site_temp.connection_server.sites[site_temp.address] = site_temp
+ site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ with Spy.Spy(FileRequest, "route") as requests:
+ site_temp.needFile("%s|%s-%s" % (inner_path, 0, 1 * self.piece_size))
+
+ assert len([req for req in requests if req[1] == "streamFile"]) == 2 # 1 piece + piecemap
+
+ # Rename the file
+ inner_path_new = inner_path.replace(".iso", "-new.iso")
+ site.storage.rename(inner_path, inner_path_new)
+ site.storage.delete("data/optional.any.iso.piecemap.msgpack")
+ assert site.content_manager.sign("content.json", self.privatekey, remove_missing_optional=True)
+
+ files_optional = site.content_manager.contents["content.json"]["files_optional"].keys()
+
+ assert "data/optional.any-new.iso.piecemap.msgpack" in files_optional
+ assert "data/optional.any.iso.piecemap.msgpack" not in files_optional
+ assert "data/optional.any.iso" not in files_optional
+
+ with Spy.Spy(FileRequest, "route") as requests:
+ site.publish()
+ time.sleep(0.1)
+ site_temp.download(blind_includes=True).join(timeout=5) # Wait for download
+
+ assert len([req[1] for req in requests if req[1] == "streamFile"]) == 0
+
+ with site_temp.storage.openBigfile(inner_path_new, prebuffer=0) as f:
+ f.read(1024)
+
+ # First piece already downloaded
+ assert [req for req in requests if req[1] == "streamFile"] == []
+
+ # Second piece needs to be downloaded + changed piecemap
+ f.seek(self.piece_size)
+ f.read(1024)
+ assert [req[3]["inner_path"] for req in requests if req[1] == "streamFile"] == [inner_path_new + ".piecemap.msgpack", inner_path_new]
+
+ @pytest.mark.parametrize("size", [1024 * 3, 1024 * 1024 * 3, 1024 * 1024 * 30])
+ def testNullFileRead(self, file_server, site, site_temp, size):
+ inner_path = "data/optional.iso"
+
+ f = site.storage.open(inner_path, "w")
+ f.write("\0" * size)
+ f.close()
+ assert site.content_manager.sign("content.json", self.privatekey)
+
+ # Init source server
+ site.connection_server = file_server
+ file_server.sites[site.address] = site
+
+ # Init client server
+ site_temp.connection_server = FileServer(file_server.ip, 1545)
+ site_temp.connection_server.sites[site_temp.address] = site_temp
+ site_temp.addPeer(file_server.ip, 1544)
+
+ # Download site
+ site_temp.download(blind_includes=True).join(timeout=5)
+
+ if "piecemap" in site.content_manager.getFileInfo(inner_path): # Bigfile
+ site_temp.needFile(inner_path + "|all")
+ else:
+ site_temp.needFile(inner_path)
+
+
+ assert site_temp.storage.getSize(inner_path) == size
diff --git a/plugins/Bigfile/Test/conftest.py b/plugins/Bigfile/Test/conftest.py
new file mode 100644
index 00000000..634e66e2
--- /dev/null
+++ b/plugins/Bigfile/Test/conftest.py
@@ -0,0 +1 @@
+from src.Test.conftest import *
diff --git a/plugins/Bigfile/Test/pytest.ini b/plugins/Bigfile/Test/pytest.ini
new file mode 100644
index 00000000..d09210d1
--- /dev/null
+++ b/plugins/Bigfile/Test/pytest.ini
@@ -0,0 +1,5 @@
+[pytest]
+python_files = Test*.py
+addopts = -rsxX -v --durations=6
+markers =
+ webtest: mark a test as a webtest.
\ No newline at end of file
diff --git a/plugins/Bigfile/__init__.py b/plugins/Bigfile/__init__.py
new file mode 100644
index 00000000..cf2dcb49
--- /dev/null
+++ b/plugins/Bigfile/__init__.py
@@ -0,0 +1,2 @@
+from . import BigfilePlugin
+from .BigfilePiecefield import BigfilePiecefield, BigfilePiecefieldPacked
\ No newline at end of file
diff --git a/plugins/Chart/ChartCollector.py b/plugins/Chart/ChartCollector.py
new file mode 100644
index 00000000..776343af
--- /dev/null
+++ b/plugins/Chart/ChartCollector.py
@@ -0,0 +1,181 @@
+import time
+import sys
+import collections
+import itertools
+import logging
+
+import gevent
+from util import helper
+from Config import config
+
+
+class ChartCollector(object):
+ def __init__(self, db):
+ self.db = db
+ if config.action == "main":
+ gevent.spawn_later(60 * 3, self.collector)
+ self.log = logging.getLogger("ChartCollector")
+ self.last_values = collections.defaultdict(dict)
+
+ def setInitialLastValues(self, sites):
+ # Recover last value of site bytes/sent
+ for site in sites:
+ self.last_values["site:" + site.address]["site_bytes_recv"] = site.settings.get("bytes_recv", 0)
+ self.last_values["site:" + site.address]["site_bytes_sent"] = site.settings.get("bytes_sent", 0)
+
+ def getCollectors(self):
+ collectors = {}
+ import main
+ file_server = main.file_server
+ sites = file_server.sites
+ if not sites:
+ return collectors
+ content_db = list(sites.values())[0].content_manager.contents.db
+
+ # Connection stats
+ collectors["connection"] = lambda: len(file_server.connections)
+ collectors["connection_in"] = (
+ lambda: len([1 for connection in file_server.connections if connection.type == "in"])
+ )
+ collectors["connection_onion"] = (
+ lambda: len([1 for connection in file_server.connections if connection.ip.endswith(".onion")])
+ )
+ collectors["connection_ping_avg"] = (
+ lambda: round(1000 * helper.avg(
+ [connection.last_ping_delay for connection in file_server.connections if connection.last_ping_delay]
+ ))
+ )
+ collectors["connection_ping_min"] = (
+ lambda: round(1000 * min(
+ [connection.last_ping_delay for connection in file_server.connections if connection.last_ping_delay]
+ ))
+ )
+ collectors["connection_rev_avg"] = (
+ lambda: helper.avg(
+ [connection.handshake["rev"] for connection in file_server.connections if connection.handshake]
+ )
+ )
+
+ # Request stats
+ collectors["file_bytes_recv|change"] = lambda: file_server.bytes_recv
+ collectors["file_bytes_sent|change"] = lambda: file_server.bytes_sent
+ collectors["request_num_recv|change"] = lambda: file_server.num_recv
+ collectors["request_num_sent|change"] = lambda: file_server.num_sent
+
+ # Limit
+ collectors["optional_limit"] = lambda: content_db.getOptionalLimitBytes()
+ collectors["optional_used"] = lambda: content_db.getOptionalUsedBytes()
+ collectors["optional_downloaded"] = lambda: sum([site.settings.get("optional_downloaded", 0) for site in sites.values()])
+
+ # Peers
+ collectors["peer"] = lambda peers: len(peers)
+ collectors["peer_onion"] = lambda peers: len([True for peer in peers if ".onion" in peer])
+
+ # Size
+ collectors["size"] = lambda: sum([site.settings.get("size", 0) for site in sites.values()])
+ collectors["size_optional"] = lambda: sum([site.settings.get("size_optional", 0) for site in sites.values()])
+ collectors["content"] = lambda: sum([len(site.content_manager.contents) for site in sites.values()])
+
+ return collectors
+
+ def getSiteCollectors(self):
+ site_collectors = {}
+
+ # Size
+ site_collectors["site_size"] = lambda site: site.settings.get("size", 0)
+ site_collectors["site_size_optional"] = lambda site: site.settings.get("size_optional", 0)
+ site_collectors["site_optional_downloaded"] = lambda site: site.settings.get("optional_downloaded", 0)
+ site_collectors["site_content"] = lambda site: len(site.content_manager.contents)
+
+ # Data transfer
+ site_collectors["site_bytes_recv|change"] = lambda site: site.settings.get("bytes_recv", 0)
+ site_collectors["site_bytes_sent|change"] = lambda site: site.settings.get("bytes_sent", 0)
+
+ # Peers
+ site_collectors["site_peer"] = lambda site: len(site.peers)
+ site_collectors["site_peer_onion"] = lambda site: len(
+ [True for peer in site.peers.values() if peer.ip.endswith(".onion")]
+ )
+ site_collectors["site_peer_connected"] = lambda site: len([True for peer in site.peers.values() if peer.connection])
+
+ return site_collectors
+
+ def getUniquePeers(self):
+ import main
+ sites = main.file_server.sites
+ return set(itertools.chain.from_iterable(
+ [site.peers.keys() for site in sites.values()]
+ ))
+
+ def collectDatas(self, collectors, last_values, site=None):
+ if site is None:
+ peers = self.getUniquePeers()
+ datas = {}
+ for key, collector in collectors.items():
+ try:
+ if site:
+ value = collector(site)
+ elif key.startswith("peer"):
+ value = collector(peers)
+ else:
+ value = collector()
+ except Exception as err:
+ self.log.info("Collector %s error: %s" % (key, err))
+ value = None
+
+ if "|change" in key: # Store changes relative to last value
+ key = key.replace("|change", "")
+ last_value = last_values.get(key, 0)
+ last_values[key] = value
+ value = value - last_value
+
+ if value is None:
+ datas[key] = None
+ else:
+ datas[key] = round(value, 3)
+ return datas
+
+ def collectGlobal(self, collectors, last_values):
+ now = int(time.time())
+ s = time.time()
+ datas = self.collectDatas(collectors, last_values["global"])
+ values = []
+ for key, value in datas.items():
+ values.append((self.db.getTypeId(key), value, now))
+ self.log.debug("Global collectors done in %.3fs" % (time.time() - s))
+
+ s = time.time()
+ cur = self.db.getCursor()
+ cur.cursor.executemany("INSERT INTO data (type_id, value, date_added) VALUES (?, ?, ?)", values)
+ cur.close()
+ self.log.debug("Global collectors inserted in %.3fs" % (time.time() - s))
+
+ def collectSites(self, sites, collectors, last_values):
+ now = int(time.time())
+ s = time.time()
+ values = []
+ for address, site in sites.items():
+ site_datas = self.collectDatas(collectors, last_values["site:%s" % address], site)
+ for key, value in site_datas.items():
+ values.append((self.db.getTypeId(key), self.db.getSiteId(address), value, now))
+ time.sleep(0.001)
+ self.log.debug("Site collections done in %.3fs" % (time.time() - s))
+
+ s = time.time()
+ cur = self.db.getCursor()
+ cur.cursor.executemany("INSERT INTO data (type_id, site_id, value, date_added) VALUES (?, ?, ?, ?)", values)
+ cur.close()
+ self.log.debug("Site collectors inserted in %.3fs" % (time.time() - s))
+
+ def collector(self):
+ collectors = self.getCollectors()
+ site_collectors = self.getSiteCollectors()
+ import main
+ sites = main.file_server.sites
+ i = 0
+ while 1:
+ self.collectGlobal(collectors, self.last_values)
+ if i % 12 == 0: # Only collect sites data every hour
+ self.collectSites(sites, site_collectors, self.last_values)
+ time.sleep(60 * 5)
+ i += 1
diff --git a/plugins/Chart/ChartDb.py b/plugins/Chart/ChartDb.py
new file mode 100644
index 00000000..9dd4d3db
--- /dev/null
+++ b/plugins/Chart/ChartDb.py
@@ -0,0 +1,133 @@
+from Config import config
+from Db.Db import Db
+import time
+
+
+class ChartDb(Db):
+ def __init__(self):
+ self.version = 2
+ super(ChartDb, self).__init__(self.getSchema(), "%s/chart.db" % config.data_dir)
+ self.foreign_keys = True
+ self.checkTables()
+ self.sites = self.loadSites()
+ self.types = self.loadTypes()
+
+ def getSchema(self):
+ schema = {}
+ schema["db_name"] = "Chart"
+ schema["tables"] = {}
+ schema["tables"]["data"] = {
+ "cols": [
+ ["data_id", "INTEGER PRIMARY KEY ASC AUTOINCREMENT NOT NULL UNIQUE"],
+ ["type_id", "INTEGER NOT NULL"],
+ ["site_id", "INTEGER"],
+ ["value", "INTEGER"],
+ ["date_added", "DATETIME DEFAULT (CURRENT_TIMESTAMP)"]
+ ],
+ "indexes": [
+ "CREATE INDEX site_id ON data (site_id)",
+ "CREATE INDEX date_added ON data (date_added)"
+ ],
+ "schema_changed": 2
+ }
+ schema["tables"]["type"] = {
+ "cols": [
+ ["type_id", "INTEGER PRIMARY KEY NOT NULL UNIQUE"],
+ ["name", "TEXT"]
+ ],
+ "schema_changed": 1
+ }
+ schema["tables"]["site"] = {
+ "cols": [
+ ["site_id", "INTEGER PRIMARY KEY NOT NULL UNIQUE"],
+ ["address", "TEXT"]
+ ],
+ "schema_changed": 1
+ }
+ return schema
+
+ def getTypeId(self, name):
+ if name not in self.types:
+ self.execute("INSERT INTO type ?", {"name": name})
+ self.types[name] = self.cur.cursor.lastrowid
+
+ return self.types[name]
+
+ def getSiteId(self, address):
+ if address not in self.sites:
+ self.execute("INSERT INTO site ?", {"address": address})
+ self.sites[address] = self.cur.cursor.lastrowid
+
+ return self.sites[address]
+
+ def loadSites(self):
+ sites = {}
+ for row in self.execute("SELECT * FROM site"):
+ sites[row["address"]] = row["site_id"]
+ return sites
+
+ def loadTypes(self):
+ types = {}
+ for row in self.execute("SELECT * FROM type"):
+ types[row["name"]] = row["type_id"]
+ return types
+
+ def deleteSite(self, address):
+ if address in self.sites:
+ site_id = self.sites[address]
+ del self.sites[address]
+ self.execute("DELETE FROM site WHERE ?", {"site_id": site_id})
+ self.execute("DELETE FROM data WHERE ?", {"site_id": site_id})
+
+ def archive(self):
+ week_back = 1
+ while 1:
+ s = time.time()
+ date_added_from = time.time() - 60 * 60 * 24 * 7 * (week_back + 1)
+ date_added_to = date_added_from + 60 * 60 * 24 * 7
+ res = self.execute("""
+ SELECT
+ MAX(date_added) AS date_added,
+ SUM(value) AS value,
+ GROUP_CONCAT(data_id) AS data_ids,
+ type_id,
+ site_id,
+ COUNT(*) AS num
+ FROM data
+ WHERE
+ site_id IS NULL AND
+ date_added > :date_added_from AND
+ date_added < :date_added_to
+ GROUP BY strftime('%Y-%m-%d %H', date_added, 'unixepoch', 'localtime'), type_id
+ """, {"date_added_from": date_added_from, "date_added_to": date_added_to})
+
+ num_archived = 0
+ cur = self.getCursor()
+ for row in res:
+ if row["num"] == 1:
+ continue
+ cur.execute("INSERT INTO data ?", {
+ "type_id": row["type_id"],
+ "site_id": row["site_id"],
+ "value": row["value"],
+ "date_added": row["date_added"]
+ })
+ cur.execute("DELETE FROM data WHERE data_id IN (%s)" % row["data_ids"])
+ num_archived += row["num"]
+ self.log.debug("Archived %s data from %s weeks ago in %.3fs" % (num_archived, week_back, time.time() - s))
+ week_back += 1
+ time.sleep(0.1)
+ if num_archived == 0:
+ break
+ # Only keep 6 month of global stats
+ self.execute(
+ "DELETE FROM data WHERE site_id IS NULL AND date_added < :date_added_limit",
+ {"date_added_limit": time.time() - 60 * 60 * 24 * 30 * 6 }
+ )
+ # Only keep 1 month of site stats
+ self.execute(
+ "DELETE FROM data WHERE site_id IS NOT NULL AND date_added < :date_added_limit",
+ {"date_added_limit": time.time() - 60 * 60 * 24 * 30 }
+ )
+ if week_back > 1:
+ self.execute("VACUUM")
diff --git a/plugins/Chart/ChartPlugin.py b/plugins/Chart/ChartPlugin.py
new file mode 100644
index 00000000..ddc1e609
--- /dev/null
+++ b/plugins/Chart/ChartPlugin.py
@@ -0,0 +1,60 @@
+import time
+import itertools
+
+import gevent
+
+from Config import config
+from util import helper
+from Plugin import PluginManager
+from .ChartDb import ChartDb
+from .ChartCollector import ChartCollector
+
+if "db" not in locals().keys(): # Share on reloads
+ db = ChartDb()
+ gevent.spawn_later(10 * 60, db.archive)
+ helper.timer(60 * 60 * 6, db.archive)
+ collector = ChartCollector(db)
+
+@PluginManager.registerTo("SiteManager")
+class SiteManagerPlugin(object):
+ def load(self, *args, **kwargs):
+ back = super(SiteManagerPlugin, self).load(*args, **kwargs)
+ collector.setInitialLastValues(self.sites.values())
+ return back
+
+ def delete(self, address, *args, **kwargs):
+ db.deleteSite(address)
+ return super(SiteManagerPlugin, self).delete(address, *args, **kwargs)
+
+@PluginManager.registerTo("UiWebsocket")
+class UiWebsocketPlugin(object):
+ def actionChartDbQuery(self, to, query, params=None):
+ if not "ADMIN" in self.permissions:
+ return {"error": "No permission"}
+
+ if config.debug or config.verbose:
+ s = time.time()
+ rows = []
+ try:
+ if not query.strip().upper().startswith("SELECT"):
+ raise Exception("Only SELECT query supported")
+ res = db.execute(query, params)
+ except Exception as err: # Response the error to client
+ self.log.error("ChartDbQuery error: %s" % err)
+ return {"error": str(err)}
+ # Convert result to dict
+ for row in res:
+ rows.append(dict(row))
+ if config.verbose and time.time() - s > 0.1: # Log slow query
+ self.log.debug("Slow query: %s (%.3fs)" % (query, time.time() - s))
+ return rows
+
+ def actionChartGetPeerLocations(self, to):
+ if not "ADMIN" in self.permissions:
+ return {"error": "No permission"}
+
+ peers = {}
+ for site in self.server.sites.values():
+ peers.update(site.peers)
+ peer_locations = self.getPeerLocations(peers)
+ return peer_locations
diff --git a/plugins/Chart/__init__.py b/plugins/Chart/__init__.py
new file mode 100644
index 00000000..2c284609
--- /dev/null
+++ b/plugins/Chart/__init__.py
@@ -0,0 +1 @@
+from . import ChartPlugin
\ No newline at end of file
diff --git a/plugins/ContentFilter/ContentFilterPlugin.py b/plugins/ContentFilter/ContentFilterPlugin.py
new file mode 100644
index 00000000..f6d74e7a
--- /dev/null
+++ b/plugins/ContentFilter/ContentFilterPlugin.py
@@ -0,0 +1,216 @@
+import time
+import re
+import html
+import hashlib
+
+from Plugin import PluginManager
+from Translate import Translate
+from Config import config
+
+from .ContentFilterStorage import ContentFilterStorage
+
+
+if "_" not in locals():
+ _ = Translate("plugins/ContentFilter/languages/")
+
+
+@PluginManager.registerTo("SiteManager")
+class SiteManagerPlugin(object):
+ def load(self, *args, **kwargs):
+ global filter_storage
+ super(SiteManagerPlugin, self).load(*args, **kwargs)
+ filter_storage = ContentFilterStorage(site_manager=self)
+
+
+@PluginManager.registerTo("UiWebsocket")
+class UiWebsocketPlugin(object):
+ # Mute
+ def cbMuteAdd(self, to, auth_address, cert_user_id, reason):
+ filter_storage.file_content["mutes"][auth_address] = {
+ "cert_user_id": cert_user_id, "reason": reason, "source": self.site.address, "date_added": time.time()
+ }
+ filter_storage.save()
+ filter_storage.changeDbs(auth_address, "remove")
+ self.response(to, "ok")
+
+ def actionMuteAdd(self, to, auth_address, cert_user_id, reason):
+ if "ADMIN" in self.getPermissions(to):
+ self.cbMuteAdd(to, auth_address, cert_user_id, reason)
+ else:
+ self.cmd(
+ "confirm",
+ [_["Hide all content from %s?"] % html.escape(cert_user_id), _["Mute"]],
+ lambda res: self.cbMuteAdd(to, auth_address, cert_user_id, reason)
+ )
+
+ def cbMuteRemove(self, to, auth_address):
+ del filter_storage.file_content["mutes"][auth_address]
+ filter_storage.save()
+ filter_storage.changeDbs(auth_address, "load")
+ self.response(to, "ok")
+
+ def actionMuteRemove(self, to, auth_address):
+ if "ADMIN" in self.getPermissions(to):
+ self.cbMuteRemove(to, auth_address)
+ else:
+ self.cmd(
+ "confirm",
+ [_["Unmute %s?"] % html.escape(filter_storage.file_content["mutes"][auth_address]["cert_user_id"]), _["Unmute"]],
+ lambda res: self.cbMuteRemove(to, auth_address)
+ )
+
+ def actionMuteList(self, to):
+ if "ADMIN" in self.getPermissions(to):
+ self.response(to, filter_storage.file_content["mutes"])
+ else:
+ return self.response(to, {"error": "Forbidden: Only ADMIN sites can list mutes"})
+
+ # Siteblock
+ def actionSiteblockAdd(self, to, site_address, reason=None):
+ if "ADMIN" not in self.getPermissions(to):
+ return self.response(to, {"error": "Forbidden: Only ADMIN sites can add to blocklist"})
+ filter_storage.file_content["siteblocks"][site_address] = {"date_added": time.time(), "reason": reason}
+ filter_storage.save()
+ self.response(to, "ok")
+
+ def actionSiteblockRemove(self, to, site_address):
+ if "ADMIN" not in self.getPermissions(to):
+ return self.response(to, {"error": "Forbidden: Only ADMIN sites can remove from blocklist"})
+ del filter_storage.file_content["siteblocks"][site_address]
+ filter_storage.save()
+ self.response(to, "ok")
+
+ def actionSiteblockList(self, to):
+ if "ADMIN" in self.getPermissions(to):
+ self.response(to, filter_storage.file_content["siteblocks"])
+ else:
+ return self.response(to, {"error": "Forbidden: Only ADMIN sites can list blocklists"})
+
+ # Include
+ def actionFilterIncludeAdd(self, to, inner_path, description=None, address=None):
+ if address:
+ if "ADMIN" not in self.getPermissions(to):
+ return self.response(to, {"error": "Forbidden: Only ADMIN sites can manage different site include"})
+ site = self.server.sites[address]
+ else:
+ address = self.site.address
+ site = self.site
+
+ if "ADMIN" in self.getPermissions(to):
+ self.cbFilterIncludeAdd(to, True, address, inner_path, description)
+ else:
+ content = site.storage.loadJson(inner_path)
+ title = _["New shared global content filter: %s (%s sites, %s users)"] % (
+ html.escape(inner_path), len(content.get("siteblocks", {})), len(content.get("mutes", {}))
+ )
+
+ self.cmd(
+ "confirm",
+ [title, "Add"],
+ lambda res: self.cbFilterIncludeAdd(to, res, address, inner_path, description)
+ )
+
+ def cbFilterIncludeAdd(self, to, res, address, inner_path, description):
+ if not res:
+ self.response(to, res)
+ return False
+
+ filter_storage.includeAdd(address, inner_path, description)
+ self.response(to, "ok")
+
+ def actionFilterIncludeRemove(self, to, inner_path, address=None):
+ if address:
+ if "ADMIN" not in self.getPermissions(to):
+ return self.response(to, {"error": "Forbidden: Only ADMIN sites can manage different site include"})
+ else:
+ address = self.site.address
+
+ key = "%s/%s" % (address, inner_path)
+ if key not in filter_storage.file_content["includes"]:
+ self.response(to, {"error": "Include not found"})
+ filter_storage.includeRemove(address, inner_path)
+ self.response(to, "ok")
+
+ def actionFilterIncludeList(self, to, all_sites=False, filters=False):
+ if all_sites and "ADMIN" not in self.getPermissions(to):
+ return self.response(to, {"error": "Forbidden: Only ADMIN sites can list all sites includes"})
+
+ back = []
+ includes = filter_storage.file_content.get("includes", {}).values()
+ for include in includes:
+ if not all_sites and include["address"] != self.site.address:
+ continue
+ if filters:
+ include = dict(include) # Don't modify original file_content
+ include_site = filter_storage.site_manager.get(include["address"])
+ if not include_site:
+ continue
+ content = include_site.storage.loadJson(include["inner_path"])
+ include["mutes"] = content.get("mutes", {})
+ include["siteblocks"] = content.get("siteblocks", {})
+ back.append(include)
+ self.response(to, back)
+
+
+@PluginManager.registerTo("SiteStorage")
+class SiteStoragePlugin(object):
+ def updateDbFile(self, inner_path, file=None, cur=None):
+ if file is not False: # File deletion always allowed
+ # Find for bitcoin addresses in file path
+ matches = re.findall("/(1[A-Za-z0-9]{26,35})/", inner_path)
+ # Check if any of the adresses are in the mute list
+ for auth_address in matches:
+ if filter_storage.isMuted(auth_address):
+ self.log.debug("Mute match: %s, ignoring %s" % (auth_address, inner_path))
+ return False
+
+ return super(SiteStoragePlugin, self).updateDbFile(inner_path, file=file, cur=cur)
+
+ def onUpdated(self, inner_path, file=None):
+ file_path = "%s/%s" % (self.site.address, inner_path)
+ if file_path in filter_storage.file_content["includes"]:
+ self.log.debug("Filter file updated: %s" % inner_path)
+ filter_storage.includeUpdateAll()
+ return super(SiteStoragePlugin, self).onUpdated(inner_path, file=file)
+
+
+@PluginManager.registerTo("UiRequest")
+class UiRequestPlugin(object):
+ def actionWrapper(self, path, extra_headers=None):
+ match = re.match("/(?P[A-Za-z0-9\._-]+)(?P
+
+
+
+
+
+
+
diff --git a/plugins/ContentFilter/media/js/ZeroFrame.js b/plugins/ContentFilter/media/js/ZeroFrame.js
new file mode 100644
index 00000000..d6facdbf
--- /dev/null
+++ b/plugins/ContentFilter/media/js/ZeroFrame.js
@@ -0,0 +1,119 @@
+// Version 1.0.0 - Initial release
+// Version 1.1.0 (2017-08-02) - Added cmdp function that returns promise instead of using callback
+// Version 1.2.0 (2017-08-02) - Added Ajax monkey patch to emulate XMLHttpRequest over ZeroFrame API
+
+const CMD_INNER_READY = 'innerReady'
+const CMD_RESPONSE = 'response'
+const CMD_WRAPPER_READY = 'wrapperReady'
+const CMD_PING = 'ping'
+const CMD_PONG = 'pong'
+const CMD_WRAPPER_OPENED_WEBSOCKET = 'wrapperOpenedWebsocket'
+const CMD_WRAPPER_CLOSE_WEBSOCKET = 'wrapperClosedWebsocket'
+
+class ZeroFrame {
+ constructor(url) {
+ this.url = url
+ this.waiting_cb = {}
+ this.wrapper_nonce = document.location.href.replace(/.*wrapper_nonce=([A-Za-z0-9]+).*/, "$1")
+ this.connect()
+ this.next_message_id = 1
+ this.init()
+ }
+
+ init() {
+ return this
+ }
+
+ connect() {
+ this.target = window.parent
+ window.addEventListener('message', e => this.onMessage(e), false)
+ this.cmd(CMD_INNER_READY)
+ }
+
+ onMessage(e) {
+ let message = e.data
+ let cmd = message.cmd
+ if (cmd === CMD_RESPONSE) {
+ if (this.waiting_cb[message.to] !== undefined) {
+ this.waiting_cb[message.to](message.result)
+ }
+ else {
+ this.log("Websocket callback not found:", message)
+ }
+ } else if (cmd === CMD_WRAPPER_READY) {
+ this.cmd(CMD_INNER_READY)
+ } else if (cmd === CMD_PING) {
+ this.response(message.id, CMD_PONG)
+ } else if (cmd === CMD_WRAPPER_OPENED_WEBSOCKET) {
+ this.onOpenWebsocket()
+ } else if (cmd === CMD_WRAPPER_CLOSE_WEBSOCKET) {
+ this.onCloseWebsocket()
+ } else {
+ this.onRequest(cmd, message)
+ }
+ }
+
+ onRequest(cmd, message) {
+ this.log("Unknown request", message)
+ }
+
+ response(to, result) {
+ this.send({
+ cmd: CMD_RESPONSE,
+ to: to,
+ result: result
+ })
+ }
+
+ cmd(cmd, params={}, cb=null) {
+ this.send({
+ cmd: cmd,
+ params: params
+ }, cb)
+ }
+
+ cmdp(cmd, params={}) {
+ return new Promise((resolve, reject) => {
+ this.cmd(cmd, params, (res) => {
+ if (res && res.error) {
+ reject(res.error)
+ } else {
+ resolve(res)
+ }
+ })
+ })
+ }
+
+ send(message, cb=null) {
+ message.wrapper_nonce = this.wrapper_nonce
+ message.id = this.next_message_id
+ this.next_message_id++
+ this.target.postMessage(message, '*')
+ if (cb) {
+ this.waiting_cb[message.id] = cb
+ }
+ }
+
+ log(...args) {
+ console.log.apply(console, ['[ZeroFrame]'].concat(args))
+ }
+
+ onOpenWebsocket() {
+ this.log('Websocket open')
+ }
+
+ onCloseWebsocket() {
+ this.log('Websocket close')
+ }
+
+ monkeyPatchAjax() {
+ var page = this
+ XMLHttpRequest.prototype.realOpen = XMLHttpRequest.prototype.open
+ this.cmd("wrapperGetAjaxKey", [], (res) => { this.ajax_key = res })
+ var newOpen = function (method, url, async) {
+ url += "?ajax_key=" + page.ajax_key
+ return this.realOpen(method, url, async)
+ }
+ XMLHttpRequest.prototype.open = newOpen
+ }
+}
diff --git a/plugins/Cors/CorsPlugin.py b/plugins/Cors/CorsPlugin.py
new file mode 100644
index 00000000..af501462
--- /dev/null
+++ b/plugins/Cors/CorsPlugin.py
@@ -0,0 +1,104 @@
+import re
+import html
+import copy
+
+from Plugin import PluginManager
+from Translate import Translate
+if "_" not in locals():
+ _ = Translate("plugins/Cors/languages/")
+
+
+def getCorsPath(site, inner_path):
+ match = re.match("^cors-([A-Za-z0-9]{26,35})/(.*)", inner_path)
+ if not match:
+ raise Exception("Invalid cors path: %s" % inner_path)
+ cors_address = match.group(1)
+ cors_inner_path = match.group(2)
+
+ if not "Cors:%s" % cors_address in site.settings["permissions"]:
+ raise Exception("This site has no permission to access site %s" % cors_address)
+
+ return cors_address, cors_inner_path
+
+
+@PluginManager.registerTo("UiWebsocket")
+class UiWebsocketPlugin(object):
+ def hasSitePermission(self, address, cmd=None):
+ if super(UiWebsocketPlugin, self).hasSitePermission(address, cmd=cmd):
+ return True
+
+ if not "Cors:%s" % address in self.site.settings["permissions"] or cmd not in ["fileGet", "fileList", "dirList", "fileRules", "optionalFileInfo", "fileQuery", "dbQuery", "userGetSettings", "siteInfo"]:
+ return False
+ else:
+ return True
+
+ # Add cors support for file commands
+ def corsFuncWrapper(self, func_name, to, inner_path, *args, **kwargs):
+ if inner_path.startswith("cors-"):
+ cors_address, cors_inner_path = getCorsPath(self.site, inner_path)
+
+ req_self = copy.copy(self)
+ req_self.site = self.server.sites.get(cors_address) # Change the site to the merged one
+ if not req_self.site:
+ return {"error": "No site found"}
+
+ func = getattr(super(UiWebsocketPlugin, req_self), func_name)
+ back = func(to, cors_inner_path, *args, **kwargs)
+ return back
+ else:
+ func = getattr(super(UiWebsocketPlugin, self), func_name)
+ return func(to, inner_path, *args, **kwargs)
+
+ def actionFileGet(self, to, inner_path, *args, **kwargs):
+ return self.corsFuncWrapper("actionFileGet", to, inner_path, *args, **kwargs)
+
+ def actionFileList(self, to, inner_path, *args, **kwargs):
+ return self.corsFuncWrapper("actionFileList", to, inner_path, *args, **kwargs)
+
+ def actionDirList(self, to, inner_path, *args, **kwargs):
+ return self.corsFuncWrapper("actionDirList", to, inner_path, *args, **kwargs)
+
+ def actionFileRules(self, to, inner_path, *args, **kwargs):
+ return self.corsFuncWrapper("actionFileRules", to, inner_path, *args, **kwargs)
+
+ def actionOptionalFileInfo(self, to, inner_path, *args, **kwargs):
+ return self.corsFuncWrapper("actionOptionalFileInfo", to, inner_path, *args, **kwargs)
+
+ def actionCorsPermission(self, to, address):
+ site = self.server.sites.get(address)
+ if site:
+ site_name = site.content_manager.contents.get("content.json", {}).get("title")
+ button_title = _["Grant"]
+ else:
+ site_name = address
+ button_title = _["Grant & Add"]
+
+ if site and "Cors:" + address in self.permissions:
+ return "ignored"
+
+ self.cmd(
+ "confirm",
+ [_["This site requests read permission to: %s"] % html.escape(site_name), button_title],
+ lambda res: self.cbCorsPermission(to, address)
+ )
+
+ def cbCorsPermission(self, to, address):
+ self.actionPermissionAdd(to, "Cors:" + address)
+ site = self.server.sites.get(address)
+ if not site:
+ self.server.site_manager.need(address)
+
+
+@PluginManager.registerTo("UiRequest")
+class UiRequestPlugin(object):
+ # Allow to load cross origin files using /cors-address/file.jpg
+ def parsePath(self, path):
+ path_parts = super(UiRequestPlugin, self).parsePath(path)
+ if "cors-" not in path: # Optimization
+ return path_parts
+ site = self.server.sites[path_parts["address"]]
+ try:
+ path_parts["address"], path_parts["inner_path"] = getCorsPath(site, path_parts["inner_path"])
+ except:
+ return None
+ return path_parts
diff --git a/plugins/Cors/__init__.py b/plugins/Cors/__init__.py
new file mode 100644
index 00000000..bcaa502b
--- /dev/null
+++ b/plugins/Cors/__init__.py
@@ -0,0 +1 @@
+from . import CorsPlugin
\ No newline at end of file
diff --git a/plugins/CryptMessage/CryptMessage.py b/plugins/CryptMessage/CryptMessage.py
new file mode 100644
index 00000000..00c5d7c6
--- /dev/null
+++ b/plugins/CryptMessage/CryptMessage.py
@@ -0,0 +1,59 @@
+import hashlib
+import base64
+
+import lib.pybitcointools as btctools
+
+ecc_cache = {}
+
+
+def eciesEncrypt(data, pubkey, ephemcurve=None, ciphername='aes-256-cbc'):
+ import pyelliptic
+ pubkey_openssl = toOpensslPublickey(base64.b64decode(pubkey))
+ curve, pubkey_x, pubkey_y, i = pyelliptic.ECC._decode_pubkey(pubkey_openssl)
+ if ephemcurve is None:
+ ephemcurve = curve
+ ephem = pyelliptic.ECC(curve=ephemcurve)
+ key = hashlib.sha512(ephem.raw_get_ecdh_key(pubkey_x, pubkey_y)).digest()
+ key_e, key_m = key[:32], key[32:]
+ pubkey = ephem.get_pubkey()
+ iv = pyelliptic.OpenSSL.rand(pyelliptic.OpenSSL.get_cipher(ciphername).get_blocksize())
+ ctx = pyelliptic.Cipher(key_e, iv, 1, ciphername)
+ ciphertext = iv + pubkey + ctx.ciphering(data)
+ mac = pyelliptic.hmac_sha256(key_m, ciphertext)
+ return key_e, ciphertext + mac
+
+def eciesDecrypt(encrypted_data, privatekey):
+ ecc_key = getEcc(privatekey)
+ return ecc_key.decrypt(base64.b64decode(encrypted_data))
+
+def split(encrypted):
+ iv = encrypted[0:16]
+ ciphertext = encrypted[16 + 70:-32]
+
+ return iv, ciphertext
+
+
+def getEcc(privatekey=None):
+ import pyelliptic
+ global ecc_cache
+ if privatekey not in ecc_cache:
+ if privatekey:
+ publickey_bin = btctools.encode_pubkey(btctools.privtopub(privatekey), "bin")
+ publickey_openssl = toOpensslPublickey(publickey_bin)
+ privatekey_openssl = toOpensslPrivatekey(privatekey)
+ ecc_cache[privatekey] = pyelliptic.ECC(curve='secp256k1', privkey=privatekey_openssl, pubkey=publickey_openssl)
+ else:
+ ecc_cache[None] = pyelliptic.ECC()
+ return ecc_cache[privatekey]
+
+
+def toOpensslPrivatekey(privatekey):
+ privatekey_bin = btctools.encode_privkey(privatekey, "bin")
+ return b'\x02\xca\x00\x20' + privatekey_bin
+
+
+def toOpensslPublickey(publickey):
+ publickey_bin = btctools.encode_pubkey(publickey, "bin")
+ publickey_bin = publickey_bin[1:]
+ publickey_openssl = b'\x02\xca\x00 ' + publickey_bin[:32] + b'\x00 ' + publickey_bin[32:]
+ return publickey_openssl
diff --git a/plugins/CryptMessage/CryptMessagePlugin.py b/plugins/CryptMessage/CryptMessagePlugin.py
new file mode 100644
index 00000000..9ef2aab1
--- /dev/null
+++ b/plugins/CryptMessage/CryptMessagePlugin.py
@@ -0,0 +1,168 @@
+import base64
+import os
+
+from Plugin import PluginManager
+from Crypt import CryptBitcoin, CryptHash
+import lib.pybitcointools as btctools
+
+from . import CryptMessage
+
+
+@PluginManager.registerTo("UiWebsocket")
+class UiWebsocketPlugin(object):
+ def eciesDecrypt(self, encrypted, privatekey):
+ back = CryptMessage.getEcc(privatekey).decrypt(encrypted)
+ return back.decode("utf8")
+
+ # - Actions -
+
+ # Returns user's public key unique to site
+ # Return: Public key
+ def actionUserPublickey(self, to, index=0):
+ publickey = self.user.getEncryptPublickey(self.site.address, index)
+ self.response(to, publickey)
+
+ # Encrypt a text using the publickey or user's sites unique publickey
+ # Return: Encrypted text using base64 encoding
+ def actionEciesEncrypt(self, to, text, publickey=0, return_aes_key=False):
+ if type(publickey) is int: # Encrypt using user's publickey
+ publickey = self.user.getEncryptPublickey(self.site.address, publickey)
+ aes_key, encrypted = CryptMessage.eciesEncrypt(text.encode("utf8"), publickey)
+ if return_aes_key:
+ self.response(to, [base64.b64encode(encrypted).decode("utf8"), base64.b64encode(aes_key).decode("utf8")])
+ else:
+ self.response(to, base64.b64encode(encrypted).decode("utf8"))
+
+ # Decrypt a text using privatekey or the user's site unique private key
+ # Return: Decrypted text or list of decrypted texts
+ def actionEciesDecrypt(self, to, param, privatekey=0):
+ if type(privatekey) is int: # Decrypt using user's privatekey
+ privatekey = self.user.getEncryptPrivatekey(self.site.address, privatekey)
+
+ if type(param) == list:
+ encrypted_texts = param
+ else:
+ encrypted_texts = [param]
+
+ texts = [] # Decoded texts
+ for encrypted_text in encrypted_texts:
+ try:
+ text = CryptMessage.eciesDecrypt(encrypted_text, privatekey).decode("utf8")
+ texts.append(text)
+ except Exception as err:
+ texts.append(None)
+
+ if type(param) == list:
+ self.response(to, texts)
+ else:
+ self.response(to, texts[0])
+
+ # Encrypt a text using AES
+ # Return: Iv, AES key, Encrypted text
+ def actionAesEncrypt(self, to, text, key=None, iv=None):
+ import pyelliptic
+
+ if key:
+ key = base64.b64decode(key)
+ else:
+ key = os.urandom(32)
+
+ if iv: # Generate new AES key if not definied
+ iv = base64.b64decode(iv)
+ else:
+ iv = pyelliptic.Cipher.gen_IV('aes-256-cbc')
+
+ if text:
+ encrypted = pyelliptic.Cipher(key, iv, 1, ciphername='aes-256-cbc').ciphering(text.encode("utf8"))
+ else:
+ encrypted = b""
+
+ res = [base64.b64encode(item).decode("utf8") for item in [key, iv, encrypted]]
+ self.response(to, res)
+
+ # Decrypt a text using AES
+ # Return: Decrypted text
+ def actionAesDecrypt(self, to, *args):
+ import pyelliptic
+
+ if len(args) == 3: # Single decrypt
+ encrypted_texts = [(args[0], args[1])]
+ keys = [args[2]]
+ else: # Batch decrypt
+ encrypted_texts, keys = args
+
+ texts = [] # Decoded texts
+ for iv, encrypted_text in encrypted_texts:
+ encrypted_text = base64.b64decode(encrypted_text)
+ iv = base64.b64decode(iv)
+ text = None
+ for key in keys:
+ ctx = pyelliptic.Cipher(base64.b64decode(key), iv, 0, ciphername='aes-256-cbc')
+ try:
+ decrypted = ctx.ciphering(encrypted_text)
+ if decrypted and decrypted.decode("utf8"): # Valid text decoded
+ text = decrypted.decode("utf8")
+ except Exception as err:
+ pass
+ texts.append(text)
+
+ if len(args) == 3:
+ self.response(to, texts[0])
+ else:
+ self.response(to, texts)
+
+ # Sign data using ECDSA
+ # Return: Signature
+ def actionEcdsaSign(self, to, data, privatekey=None):
+ if privatekey is None: # Sign using user's privatekey
+ privatekey = self.user.getAuthPrivatekey(self.site.address)
+
+ self.response(to, CryptBitcoin.sign(data, privatekey))
+
+ # Verify data using ECDSA (address is either a address or array of addresses)
+ # Return: bool
+ def actionEcdsaVerify(self, to, data, address, signature):
+ self.response(to, CryptBitcoin.verify(data, address, signature))
+
+ # Gets the publickey of a given privatekey
+ def actionEccPrivToPub(self, to, privatekey):
+ self.response(to, btctools.privtopub(privatekey))
+
+ # Gets the address of a given publickey
+ def actionEccPubToAddr(self, to, publickey):
+ address = btctools.pubtoaddr(btctools.decode_pubkey(publickey))
+ self.response(to, address)
+
+
+@PluginManager.registerTo("User")
+class UserPlugin(object):
+ def getEncryptPrivatekey(self, address, param_index=0):
+ assert param_index >= 0 and param_index <= 1000
+ site_data = self.getSiteData(address)
+
+ if site_data.get("cert"): # Different privatekey for different cert provider
+ index = param_index + self.getAddressAuthIndex(site_data["cert"])
+ else:
+ index = param_index
+
+ if "encrypt_privatekey_%s" % index not in site_data:
+ address_index = self.getAddressAuthIndex(address)
+ crypt_index = address_index + 1000 + index
+ site_data["encrypt_privatekey_%s" % index] = CryptBitcoin.hdPrivatekey(self.master_seed, crypt_index)
+ self.log.debug("New encrypt privatekey generated for %s:%s" % (address, index))
+ return site_data["encrypt_privatekey_%s" % index]
+
+ def getEncryptPublickey(self, address, param_index=0):
+ assert param_index >= 0 and param_index <= 1000
+ site_data = self.getSiteData(address)
+
+ if site_data.get("cert"): # Different privatekey for different cert provider
+ index = param_index + self.getAddressAuthIndex(site_data["cert"])
+ else:
+ index = param_index
+
+ if "encrypt_publickey_%s" % index not in site_data:
+ privatekey = self.getEncryptPrivatekey(address, param_index)
+ publickey = btctools.encode_pubkey(btctools.privtopub(privatekey), "bin_compressed")
+ site_data["encrypt_publickey_%s" % index] = base64.b64encode(publickey).decode("utf8")
+ return site_data["encrypt_publickey_%s" % index]
diff --git a/plugins/CryptMessage/Test/TestCrypt.py b/plugins/CryptMessage/Test/TestCrypt.py
new file mode 100644
index 00000000..681c4790
--- /dev/null
+++ b/plugins/CryptMessage/Test/TestCrypt.py
@@ -0,0 +1,125 @@
+import pytest
+import base64
+from CryptMessage import CryptMessage
+
+
+@pytest.mark.usefixtures("resetSettings")
+class TestCrypt:
+ publickey = "A3HatibU4S6eZfIQhVs2u7GLN5G9wXa9WwlkyYIfwYaj"
+ privatekey = "5JBiKFYBm94EUdbxtnuLi6cvNcPzcKymCUHBDf2B6aq19vvG3rL"
+ utf8_text = '\xc1rv\xedzt\xfbr\xf5t\xfck\xf6rf\xfar\xf3g\xe9'
+ ecies_encrypted_text = "R5J1RFIDOzE5bnWopvccmALKACCk/CRcd/KSE9OgExJKASyMbZ57JVSUenL2TpABMmcT+wAgr2UrOqClxpOWvIUwvwwupXnMbRTzthhIJJrTRW3sCJVaYlGEMn9DAcvbflgEkQX/MVVdLV3tWKySs1Vk8sJC/y+4pGYCrZz7vwDNEEERaqU="
+
+ @pytest.mark.parametrize("text", [b"hello", '\xc1rv\xedzt\xfbr\xf5t\xfck\xf6rf\xfar\xf3g\xe9'.encode("utf8")])
+ @pytest.mark.parametrize("text_repeat", [1, 10, 128, 1024])
+ def testEncryptEcies(self, text, text_repeat):
+ text_repeated = text * text_repeat
+ aes_key, encrypted = CryptMessage.eciesEncrypt(text_repeated, self.publickey)
+ assert len(aes_key) == 32
+ # assert len(encrypted) == 134 + int(len(text) / 16) * 16 # Not always true
+
+ ecc = CryptMessage.getEcc(self.privatekey)
+ assert ecc.decrypt(encrypted) == text_repeated
+
+ def testDecryptEcies(self, user):
+ encrypted = base64.b64decode(self.ecies_encrypted_text)
+ ecc = CryptMessage.getEcc(self.privatekey)
+ assert ecc.decrypt(encrypted) == b"hello"
+
+ def testPublickey(self, ui_websocket):
+ pub = ui_websocket.testAction("UserPublickey", 0)
+ assert len(pub) == 44 # Compressed, b64 encoded publickey
+
+ # Different pubkey for specificed index
+ assert ui_websocket.testAction("UserPublickey", 1) != ui_websocket.testAction("UserPublickey", 0)
+
+ # Same publickey for same index
+ assert ui_websocket.testAction("UserPublickey", 2) == ui_websocket.testAction("UserPublickey", 2)
+
+ # Different publickey for different cert
+ site_data = ui_websocket.user.getSiteData(ui_websocket.site.address)
+ site_data["cert"] = None
+ pub1 = ui_websocket.testAction("UserPublickey", 0)
+
+ site_data = ui_websocket.user.getSiteData(ui_websocket.site.address)
+ site_data["cert"] = "zeroid.bit"
+ pub2 = ui_websocket.testAction("UserPublickey", 0)
+ assert pub1 != pub2
+
+ def testEcies(self, ui_websocket):
+ ui_websocket.actionUserPublickey(0, 0)
+ pub = ui_websocket.ws.getResult()
+
+ ui_websocket.actionEciesEncrypt(0, "hello", pub)
+ encrypted = ui_websocket.ws.getResult()
+ assert len(encrypted) == 180
+
+ # Don't allow decrypt using other privatekey index
+ ui_websocket.actionEciesDecrypt(0, encrypted, 123)
+ decrypted = ui_websocket.ws.getResult()
+ assert decrypted != "hello"
+
+ # Decrypt using correct privatekey
+ ui_websocket.actionEciesDecrypt(0, encrypted)
+ decrypted = ui_websocket.ws.getResult()
+ assert decrypted == "hello"
+
+ # Decrypt batch
+ ui_websocket.actionEciesDecrypt(0, [encrypted, "baad", encrypted])
+ decrypted = ui_websocket.ws.getResult()
+ assert decrypted == ["hello", None, "hello"]
+
+ def testEciesUtf8(self, ui_websocket):
+ # Utf8 test
+ ui_websocket.actionEciesEncrypt(0, self.utf8_text)
+ encrypted = ui_websocket.ws.getResult()
+
+ ui_websocket.actionEciesDecrypt(0, encrypted)
+ assert ui_websocket.ws.getResult() == self.utf8_text
+
+ def testEciesAes(self, ui_websocket):
+ ui_websocket.actionEciesEncrypt(0, "hello", return_aes_key=True)
+ ecies_encrypted, aes_key = ui_websocket.ws.getResult()
+
+ # Decrypt using Ecies
+ ui_websocket.actionEciesDecrypt(0, ecies_encrypted)
+ assert ui_websocket.ws.getResult() == "hello"
+
+ # Decrypt using AES
+ aes_iv, aes_encrypted = CryptMessage.split(base64.b64decode(ecies_encrypted))
+
+ ui_websocket.actionAesDecrypt(0, base64.b64encode(aes_iv), base64.b64encode(aes_encrypted), aes_key)
+ assert ui_websocket.ws.getResult() == "hello"
+
+ def testAes(self, ui_websocket):
+ ui_websocket.actionAesEncrypt(0, "hello")
+ key, iv, encrypted = ui_websocket.ws.getResult()
+
+ assert len(key) == 44
+ assert len(iv) == 24
+ assert len(encrypted) == 24
+
+ # Single decrypt
+ ui_websocket.actionAesDecrypt(0, iv, encrypted, key)
+ assert ui_websocket.ws.getResult() == "hello"
+
+ # Batch decrypt
+ ui_websocket.actionAesEncrypt(0, "hello")
+ key2, iv2, encrypted2 = ui_websocket.ws.getResult()
+
+ assert [key, iv, encrypted] != [key2, iv2, encrypted2]
+
+ # 2 correct key
+ ui_websocket.actionAesDecrypt(0, [[iv, encrypted], [iv, encrypted], [iv, "baad"], [iv2, encrypted2]], [key])
+ assert ui_websocket.ws.getResult() == ["hello", "hello", None, None]
+
+ # 3 key
+ ui_websocket.actionAesDecrypt(0, [[iv, encrypted], [iv, encrypted], [iv, "baad"], [iv2, encrypted2]], [key, key2])
+ assert ui_websocket.ws.getResult() == ["hello", "hello", None, "hello"]
+
+ def testAesUtf8(self, ui_websocket):
+ ui_websocket.actionAesEncrypt(0, self.utf8_text)
+ key, iv, encrypted = ui_websocket.ws.getResult()
+
+ ui_websocket.actionAesDecrypt(0, iv, encrypted, key)
+ assert ui_websocket.ws.getResult() == self.utf8_text
diff --git a/plugins/CryptMessage/Test/conftest.py b/plugins/CryptMessage/Test/conftest.py
new file mode 100644
index 00000000..8c1df5b2
--- /dev/null
+++ b/plugins/CryptMessage/Test/conftest.py
@@ -0,0 +1 @@
+from src.Test.conftest import *
\ No newline at end of file
diff --git a/plugins/CryptMessage/Test/pytest.ini b/plugins/CryptMessage/Test/pytest.ini
new file mode 100644
index 00000000..d09210d1
--- /dev/null
+++ b/plugins/CryptMessage/Test/pytest.ini
@@ -0,0 +1,5 @@
+[pytest]
+python_files = Test*.py
+addopts = -rsxX -v --durations=6
+markers =
+ webtest: mark a test as a webtest.
\ No newline at end of file
diff --git a/plugins/CryptMessage/__init__.py b/plugins/CryptMessage/__init__.py
new file mode 100644
index 00000000..6aeb4e52
--- /dev/null
+++ b/plugins/CryptMessage/__init__.py
@@ -0,0 +1 @@
+from . import CryptMessagePlugin
\ No newline at end of file
diff --git a/plugins/FilePack/FilePackPlugin.py b/plugins/FilePack/FilePackPlugin.py
new file mode 100644
index 00000000..840961b7
--- /dev/null
+++ b/plugins/FilePack/FilePackPlugin.py
@@ -0,0 +1,196 @@
+import os
+import re
+
+import gevent
+
+from Plugin import PluginManager
+from Config import config
+from Debug import Debug
+
+# Keep archive open for faster reponse times for large sites
+archive_cache = {}
+
+
+def closeArchive(archive_path):
+ if archive_path in archive_cache:
+ del archive_cache[archive_path]
+
+
+def openArchive(archive_path, file_obj=None):
+ if archive_path not in archive_cache:
+ if archive_path.endswith("tar.gz"):
+ import tarfile
+ archive_cache[archive_path] = tarfile.open(archive_path, fileobj=file_obj, mode="r:gz")
+ elif archive_path.endswith("tar.bz2"):
+ import tarfile
+ archive_cache[archive_path] = tarfile.open(archive_path, fileobj=file_obj, mode="r:bz2")
+ else:
+ import zipfile
+ archive_cache[archive_path] = zipfile.ZipFile(file_obj or archive_path)
+ gevent.spawn_later(5, lambda: closeArchive(archive_path)) # Close after 5 sec
+
+ archive = archive_cache[archive_path]
+ return archive
+
+
+def openArchiveFile(archive_path, path_within, file_obj=None):
+ archive = openArchive(archive_path, file_obj=file_obj)
+ if archive_path.endswith(".zip"):
+ return archive.open(path_within)
+ else:
+ return archive.extractfile(path_within)
+
+
+@PluginManager.registerTo("UiRequest")
+class UiRequestPlugin(object):
+ def actionSiteMedia(self, path, **kwargs):
+ if ".zip/" in path or ".tar.gz/" in path:
+ file_obj = None
+ path_parts = self.parsePath(path)
+ file_path = "%s/%s/%s" % (config.data_dir, path_parts["address"], path_parts["inner_path"])
+ match = re.match("^(.*\.(?:tar.gz|tar.bz2|zip))/(.*)", file_path)
+ archive_path, path_within = match.groups()
+ if archive_path not in archive_cache:
+ site = self.server.site_manager.get(path_parts["address"])
+ if not site:
+ return self.actionSiteAddPrompt(path)
+ archive_inner_path = site.storage.getInnerPath(archive_path)
+ if not os.path.isfile(archive_path):
+ # Wait until file downloads
+ result = site.needFile(archive_inner_path, priority=10)
+ # Send virutal file path download finished event to remove loading screen
+ site.updateWebsocket(file_done=archive_inner_path)
+ if not result:
+ return self.error404(archive_inner_path)
+ file_obj = site.storage.openBigfile(archive_inner_path)
+ if file_obj == False:
+ file_obj = None
+
+ header_allow_ajax = False
+ if self.get.get("ajax_key"):
+ requester_site = self.server.site_manager.get(path_parts["request_address"])
+ if self.get["ajax_key"] == requester_site.settings["ajax_key"]:
+ header_allow_ajax = True
+ else:
+ return self.error403("Invalid ajax_key")
+
+ try:
+ file = openArchiveFile(archive_path, path_within, file_obj=file_obj)
+ content_type = self.getContentType(file_path)
+ self.sendHeader(200, content_type=content_type, noscript=kwargs.get("header_noscript", False), allow_ajax=header_allow_ajax)
+ return self.streamFile(file)
+ except Exception as err:
+ self.log.debug("Error opening archive file: %s" % Debug.formatException(err))
+ return self.error404(path)
+
+ return super(UiRequestPlugin, self).actionSiteMedia(path, **kwargs)
+
+ def streamFile(self, file):
+ for i in range(100): # Read max 6MB
+ try:
+ block = file.read(60 * 1024)
+ if block:
+ yield block
+ else:
+ raise StopIteration
+ except StopIteration:
+ file.close()
+ break
+
+
+@PluginManager.registerTo("SiteStorage")
+class SiteStoragePlugin(object):
+ def isFile(self, inner_path):
+ if ".zip/" in inner_path or ".tar.gz/" in inner_path:
+ match = re.match("^(.*\.(?:tar.gz|tar.bz2|zip))/(.*)", inner_path)
+ archive_inner_path, path_within = match.groups()
+ return super(SiteStoragePlugin, self).isFile(archive_inner_path)
+ else:
+ return super(SiteStoragePlugin, self).isFile(inner_path)
+
+ def openArchive(self, inner_path):
+ archive_path = self.getPath(inner_path)
+ file_obj = None
+ if archive_path not in archive_cache:
+ if not os.path.isfile(archive_path):
+ result = self.site.needFile(inner_path, priority=10)
+ self.site.updateWebsocket(file_done=inner_path)
+ if not result:
+ raise Exception("Unable to download file")
+ file_obj = self.site.storage.openBigfile(inner_path)
+ if file_obj == False:
+ file_obj = None
+
+ try:
+ archive = openArchive(archive_path, file_obj=file_obj)
+ except Exception as err:
+ raise Exception("Unable to download file: %s" % Debug.formatException(err))
+
+ return archive
+
+ def walk(self, inner_path, *args, **kwags):
+ if ".zip" in inner_path or ".tar.gz" in inner_path:
+ match = re.match("^(.*\.(?:tar.gz|tar.bz2|zip))(.*)", inner_path)
+ archive_inner_path, path_within = match.groups()
+ archive = self.openArchive(archive_inner_path)
+ path_within = path_within.lstrip("/")
+
+ if archive_inner_path.endswith(".zip"):
+ namelist = [name for name in archive.namelist() if not name.endswith("/")]
+ else:
+ namelist = [item.name for item in archive.getmembers() if not item.isdir()]
+
+ namelist_relative = []
+ for name in namelist:
+ if not name.startswith(path_within):
+ continue
+ name_relative = name.replace(path_within, "", 1).rstrip("/")
+ namelist_relative.append(name_relative)
+
+ return namelist_relative
+
+ else:
+ return super(SiteStoragePlugin, self).walk(inner_path, *args, **kwags)
+
+ def list(self, inner_path, *args, **kwags):
+ if ".zip" in inner_path or ".tar.gz" in inner_path:
+ match = re.match("^(.*\.(?:tar.gz|tar.bz2|zip))(.*)", inner_path)
+ archive_inner_path, path_within = match.groups()
+ archive = self.openArchive(archive_inner_path)
+ path_within = path_within.lstrip("/")
+
+ if archive_inner_path.endswith(".zip"):
+ namelist = [name for name in archive.namelist()]
+ else:
+ namelist = [item.name for item in archive.getmembers()]
+
+ namelist_relative = []
+ for name in namelist:
+ if not name.startswith(path_within):
+ continue
+ name_relative = name.replace(path_within, "", 1).rstrip("/")
+
+ if "/" in name_relative: # File is in sub-directory
+ continue
+
+ namelist_relative.append(name_relative)
+ return namelist_relative
+
+ else:
+ return super(SiteStoragePlugin, self).list(inner_path, *args, **kwags)
+
+ def read(self, inner_path, mode="rb"):
+ if ".zip/" in inner_path or ".tar.gz/" in inner_path:
+ match = re.match("^(.*\.(?:tar.gz|tar.bz2|zip))(.*)", inner_path)
+ archive_inner_path, path_within = match.groups()
+ archive = self.openArchive(archive_inner_path)
+ path_within = path_within.lstrip("/")
+
+ if archive_inner_path.endswith(".zip"):
+ return archive.open(path_within).read()
+ else:
+ return archive.extractfile(path_within).read()
+
+ else:
+ return super(SiteStoragePlugin, self).read(inner_path, mode)
+
diff --git a/plugins/FilePack/__init__.py b/plugins/FilePack/__init__.py
new file mode 100644
index 00000000..660a0920
--- /dev/null
+++ b/plugins/FilePack/__init__.py
@@ -0,0 +1 @@
+from . import FilePackPlugin
\ No newline at end of file
diff --git a/plugins/MergerSite/MergerSitePlugin.py b/plugins/MergerSite/MergerSitePlugin.py
new file mode 100644
index 00000000..36c1dbae
--- /dev/null
+++ b/plugins/MergerSite/MergerSitePlugin.py
@@ -0,0 +1,384 @@
+import re
+import time
+import copy
+
+from Plugin import PluginManager
+from Translate import Translate
+from util import RateLimit
+from util import helper
+from Debug import Debug
+try:
+ import OptionalManager.UiWebsocketPlugin # To make optioanlFileInfo merger sites compatible
+except Exception:
+ pass
+
+if "merger_db" not in locals().keys(): # To keep merger_sites between module reloads
+ merger_db = {} # Sites that allowed to list other sites {address: [type1, type2...]}
+ merged_db = {} # Sites that allowed to be merged to other sites {address: type, ...}
+ merged_to_merger = {} # {address: [site1, site2, ...]} cache
+ site_manager = None # Site manager for merger sites
+
+if "_" not in locals():
+ _ = Translate("plugins/MergerSite/languages/")
+
+
+# Check if the site has permission to this merger site
+def checkMergerPath(address, inner_path):
+ merged_match = re.match("^merged-(.*?)/([A-Za-z0-9]{26,35})/", inner_path)
+ if merged_match:
+ merger_type = merged_match.group(1)
+ # Check if merged site is allowed to include other sites
+ if merger_type in merger_db.get(address, []):
+ # Check if included site allows to include
+ merged_address = merged_match.group(2)
+ if merged_db.get(merged_address) == merger_type:
+ inner_path = re.sub("^merged-(.*?)/([A-Za-z0-9]{26,35})/", "", inner_path)
+ return merged_address, inner_path
+ else:
+ raise Exception(
+ "Merger site (%s) does not have permission for merged site: %s (%s)" %
+ (merger_type, merged_address, merged_db.get(merged_address))
+ )
+ else:
+ raise Exception("No merger (%s) permission to load: %s (%s not in %s)" % ( + address, inner_path, merger_type, merger_db.get(address, [])) + ) + else: + raise Exception("Invalid merger path: %s" % inner_path) + + +@PluginManager.registerTo("UiWebsocket") +class UiWebsocketPlugin(object): + # Download new site + def actionMergerSiteAdd(self, to, addresses): + if type(addresses) != list: + # Single site add + addresses = [addresses] + # Check if the site has merger permission + merger_types = merger_db.get(self.site.address) + if not merger_types: + return self.response(to, {"error": "Not a merger site"}) + + if RateLimit.isAllowed(self.site.address + "-MergerSiteAdd", 10) and len(addresses) == 1: + # Without confirmation if only one site address and not called in last 10 sec + self.cbMergerSiteAdd(to, addresses) + else: + self.cmd( + "confirm", + [_["Add %s new site?"] % len(addresses), "Add"], + lambda res: self.cbMergerSiteAdd(to, addresses) + ) + self.response(to, "ok") + + # Callback of adding new site confirmation + def cbMergerSiteAdd(self, to, addresses): + added = 0 + for address in addresses: + added += 1 + site_manager.need(address) + if added: + self.cmd("notification", ["done", _["Added %s new site"] % added, 5000]) + RateLimit.called(self.site.address + "-MergerSiteAdd") + site_manager.updateMergerSites() + + # Delete a merged site + def actionMergerSiteDelete(self, to, address): + site = self.server.sites.get(address) + if not site: + return self.response(to, {"error": "No site found: %s" % address}) + + merger_types = merger_db.get(self.site.address) + if not merger_types: + return self.response(to, {"error": "Not a merger site"}) + if merged_db.get(address) not in merger_types: + return self.response(to, {"error": "Merged type (%s) not in %s" % (merged_db.get(address), merger_types)}) + + self.cmd("notification", ["done", _["Site deleted: %s"] % address, 5000]) + self.response(to, "ok") + + # Lists merged sites + def actionMergerSiteList(self, to, query_site_info=False): + merger_types = merger_db.get(self.site.address) + ret = {} + if not merger_types: + return self.response(to, {"error": "Not a merger site"}) + for address, merged_type in merged_db.items(): + if merged_type not in merger_types: + continue # Site not for us + if query_site_info: + site = self.server.sites.get(address) + ret[address] = self.formatSiteInfo(site, create_user=False) + else: + ret[address] = merged_type + self.response(to, ret) + + def hasSitePermission(self, address, *args, **kwargs): + if super(UiWebsocketPlugin, self).hasSitePermission(address, *args, **kwargs): + return True + else: + if self.site.address in [merger_site.address for merger_site in merged_to_merger.get(address, [])]: + return True + else: + return False + + # Add support merger sites for file commands + def mergerFuncWrapper(self, func_name, to, inner_path, *args, **kwargs): + if inner_path.startswith("merged-"): + merged_address, merged_inner_path = checkMergerPath(self.site.address, inner_path) + + # Set the same cert for merged site + merger_cert = self.user.getSiteData(self.site.address).get("cert") + if merger_cert and self.user.getSiteData(merged_address).get("cert") != merger_cert: + self.user.setCert(merged_address, merger_cert) + + req_self = copy.copy(self) + req_self.site = self.server.sites.get(merged_address) # Change the site to the merged one + + func = getattr(super(UiWebsocketPlugin, req_self), func_name) + return func(to, merged_inner_path, *args, **kwargs) + else: + func = getattr(super(UiWebsocketPlugin, self), func_name) + return func(to, inner_path, *args, **kwargs) + + def actionFileList(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionFileList", to, inner_path, *args, **kwargs) + + def actionDirList(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionDirList", to, inner_path, *args, **kwargs) + + def actionFileGet(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionFileGet", to, inner_path, *args, **kwargs) + + def actionFileWrite(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionFileWrite", to, inner_path, *args, **kwargs) + + def actionFileDelete(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionFileDelete", to, inner_path, *args, **kwargs) + + def actionFileRules(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionFileRules", to, inner_path, *args, **kwargs) + + def actionFileNeed(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionFileNeed", to, inner_path, *args, **kwargs) + + def actionOptionalFileInfo(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionOptionalFileInfo", to, inner_path, *args, **kwargs) + + def actionOptionalFileDelete(self, to, inner_path, *args, **kwargs): + return self.mergerFuncWrapper("actionOptionalFileDelete", to, inner_path, *args, **kwargs) + + def actionBigfileUploadInit(self, to, inner_path, *args, **kwargs): + back = self.mergerFuncWrapper("actionBigfileUploadInit", to, inner_path, *args, **kwargs) + if inner_path.startswith("merged-"): + merged_address, merged_inner_path = checkMergerPath(self.site.address, inner_path) + back["inner_path"] = "merged-%s/%s/%s" % (merged_db[merged_address], merged_address, back["inner_path"]) + return back + + # Add support merger sites for file commands with privatekey parameter + def mergerFuncWrapperWithPrivatekey(self, func_name, to, privatekey, inner_path, *args, **kwargs): + func = getattr(super(UiWebsocketPlugin, self), func_name) + if inner_path.startswith("merged-"): + merged_address, merged_inner_path = checkMergerPath(self.site.address, inner_path) + merged_site = self.server.sites.get(merged_address) + + # Set the same cert for merged site + merger_cert = self.user.getSiteData(self.site.address).get("cert") + if merger_cert: + self.user.setCert(merged_address, merger_cert) + + site_before = self.site # Save to be able to change it back after we ran the command + self.site = merged_site # Change the site to the merged one + try: + back = func(to, privatekey, merged_inner_path, *args, **kwargs) + finally: + self.site = site_before # Change back to original site + return back + else: + return func(to, privatekey, inner_path, *args, **kwargs) + + def actionSiteSign(self, to, privatekey=None, inner_path="content.json", *args, **kwargs): + return self.mergerFuncWrapperWithPrivatekey("actionSiteSign", to, privatekey, inner_path, *args, **kwargs) + + def actionSitePublish(self, to, privatekey=None, inner_path="content.json", *args, **kwargs): + return self.mergerFuncWrapperWithPrivatekey("actionSitePublish", to, privatekey, inner_path, *args, **kwargs) + + def actionPermissionAdd(self, to, permission): + super(UiWebsocketPlugin, self).actionPermissionAdd(to, permission) + if permission.startswith("Merger"): + self.site.storage.rebuildDb() + + def actionPermissionDetails(self, to, permission): + if not permission.startswith("Merger"): + return super(UiWebsocketPlugin, self).actionPermissionDetails(to, permission) + + merger_type = permission.replace("Merger:", "") + if not re.match("^[A-Za-z0-9-]+$", merger_type): + raise Exception("Invalid merger_type: %s" % merger_type) + merged_sites = [] + for address, merged_type in merged_db.items(): + if merged_type != merger_type: + continue + site = self.server.sites.get(address) + try: + merged_sites.append(site.content_manager.contents.get("content.json").get("title", address)) + except Exception as err: + merged_sites.append(address) + + details = _["Read and write permissions to sites with merged type of %s "] % merger_type + details += _["(%s sites)"] % len(merged_sites) + details += "
%s
" % ", ".join(merged_sites)
+ self.response(to, details)
+
+
+@PluginManager.registerTo("UiRequest")
+class UiRequestPlugin(object):
+ # Allow to load merged site files using /merged-ZeroMe/address/file.jpg
+ def parsePath(self, path):
+ path_parts = super(UiRequestPlugin, self).parsePath(path)
+ if "merged-" not in path: # Optimization
+ return path_parts
+ path_parts["address"], path_parts["inner_path"] = checkMergerPath(path_parts["address"], path_parts["inner_path"])
+ return path_parts
+
+
+@PluginManager.registerTo("SiteStorage")
+class SiteStoragePlugin(object):
+ # Also rebuild from merged sites
+ def getDbFiles(self):
+ merger_types = merger_db.get(self.site.address)
+
+ # First return the site's own db files
+ for item in super(SiteStoragePlugin, self).getDbFiles():
+ yield item
+
+ # Not a merger site, that's all
+ if not merger_types:
+ return
+
+ merged_sites = [
+ site_manager.sites[address]
+ for address, merged_type in merged_db.items()
+ if merged_type in merger_types
+ ]
+ found = 0
+ for merged_site in merged_sites:
+ self.log.debug("Loading merged site: %s" % merged_site)
+ merged_type = merged_db[merged_site.address]
+ for content_inner_path, content in merged_site.content_manager.contents.items():
+ # content.json file itself
+ if merged_site.storage.isFile(content_inner_path): # Missing content.json file
+ merged_inner_path = "merged-%s/%s/%s" % (merged_type, merged_site.address, content_inner_path)
+ yield merged_inner_path, merged_site.storage.getPath(content_inner_path)
+ else:
+ merged_site.log.error("[MISSING] %s" % content_inner_path)
+ # Data files in content.json
+ content_inner_path_dir = helper.getDirname(content_inner_path) # Content.json dir relative to site
+ for file_relative_path in list(content.get("files", {}).keys()) + list(content.get("files_optional", {}).keys()):
+ if not file_relative_path.endswith(".json"):
+ continue # We only interesed in json files
+ file_inner_path = content_inner_path_dir + file_relative_path # File Relative to site dir
+ file_inner_path = file_inner_path.strip("/") # Strip leading /
+ if merged_site.storage.isFile(file_inner_path):
+ merged_inner_path = "merged-%s/%s/%s" % (merged_type, merged_site.address, file_inner_path)
+ yield merged_inner_path, merged_site.storage.getPath(file_inner_path)
+ else:
+ merged_site.log.error("[MISSING] %s" % file_inner_path)
+ found += 1
+ if found % 100 == 0:
+ time.sleep(0.001) # Context switch to avoid UI block
+
+ # Also notice merger sites on a merged site file change
+ def onUpdated(self, inner_path, file=None):
+ super(SiteStoragePlugin, self).onUpdated(inner_path, file)
+
+ merged_type = merged_db.get(self.site.address)
+
+ for merger_site in merged_to_merger.get(self.site.address, []):
+ if merger_site.address == self.site.address: # Avoid infinite loop
+ continue
+ virtual_path = "merged-%s/%s/%s" % (merged_type, self.site.address, inner_path)
+ if inner_path.endswith(".json"):
+ if file is not None:
+ merger_site.storage.onUpdated(virtual_path, file=file)
+ else:
+ merger_site.storage.onUpdated(virtual_path, file=self.open(inner_path))
+ else:
+ merger_site.storage.onUpdated(virtual_path)
+
+
+@PluginManager.registerTo("Site")
+class SitePlugin(object):
+ def fileDone(self, inner_path):
+ super(SitePlugin, self).fileDone(inner_path)
+
+ for merger_site in merged_to_merger.get(self.address, []):
+ if merger_site.address == self.address:
+ continue
+ for ws in merger_site.websockets:
+ ws.event("siteChanged", self, {"event": ["file_done", inner_path]})
+
+ def fileFailed(self, inner_path):
+ super(SitePlugin, self).fileFailed(inner_path)
+
+ for merger_site in merged_to_merger.get(self.address, []):
+ if merger_site.address == self.address:
+ continue
+ for ws in merger_site.websockets:
+ ws.event("siteChanged", self, {"event": ["file_failed", inner_path]})
+
+
+@PluginManager.registerTo("SiteManager")
+class SiteManagerPlugin(object):
+ # Update merger site for site types
+ def updateMergerSites(self):
+ global merger_db, merged_db, merged_to_merger, site_manager
+ s = time.time()
+ merger_db = {}
+ merged_db = {}
+ merged_to_merger = {}
+ site_manager = self
+ if not self.sites:
+ return
+ for site in self.sites.values():
+ # Update merged sites
+ try:
+ merged_type = site.content_manager.contents.get("content.json", {}).get("merged_type")
+ except Exception as err:
+ self.log.error("Error loading site %s: %s" % (site.address, Debug.formatException(err)))
+ continue
+ if merged_type:
+ merged_db[site.address] = merged_type
+
+ # Update merger sites
+ for permission in site.settings["permissions"]:
+ if not permission.startswith("Merger:"):
+ continue
+ if merged_type:
+ self.log.error(
+ "Removing permission %s from %s: Merger and merged at the same time." %
+ (permission, site.address)
+ )
+ site.settings["permissions"].remove(permission)
+ continue
+ merger_type = permission.replace("Merger:", "")
+ if site.address not in merger_db:
+ merger_db[site.address] = []
+ merger_db[site.address].append(merger_type)
+ site_manager.sites[site.address] = site
+
+ # Update merged to merger
+ if merged_type:
+ for merger_site in self.sites.values():
+ if "Merger:" + merged_type in merger_site.settings["permissions"]:
+ if site.address not in merged_to_merger:
+ merged_to_merger[site.address] = []
+ merged_to_merger[site.address].append(merger_site)
+ self.log.debug("Updated merger sites in %.3fs" % (time.time() - s))
+
+ def load(self, *args, **kwags):
+ super(SiteManagerPlugin, self).load(*args, **kwags)
+ self.updateMergerSites()
+
+ def save(self, *args, **kwags):
+ super(SiteManagerPlugin, self).save(*args, **kwags)
+ self.updateMergerSites()
diff --git a/plugins/MergerSite/__init__.py b/plugins/MergerSite/__init__.py
new file mode 100644
index 00000000..2cf54611
--- /dev/null
+++ b/plugins/MergerSite/__init__.py
@@ -0,0 +1 @@
+from . import MergerSitePlugin
\ No newline at end of file
diff --git a/plugins/MergerSite/languages/es.json b/plugins/MergerSite/languages/es.json
new file mode 100644
index 00000000..d554c3a9
--- /dev/null
+++ b/plugins/MergerSite/languages/es.json
@@ -0,0 +1,5 @@
+{
+ "Add %s new site?": "¿Agregar %s nuevo sitio?",
+ "Added %s new site": "Sitio %s agregado",
+ "Site deleted: %s": "Sitio removido: %s"
+}
diff --git a/plugins/MergerSite/languages/fr.json b/plugins/MergerSite/languages/fr.json
new file mode 100644
index 00000000..9d59fde9
--- /dev/null
+++ b/plugins/MergerSite/languages/fr.json
@@ -0,0 +1,5 @@
+{
+ "Add %s new site?": "Ajouter le site %s ?",
+ "Added %s new site": "Site %s ajouté",
+ "Site deleted: %s": "Site %s supprimé"
+}
diff --git a/plugins/MergerSite/languages/hu.json b/plugins/MergerSite/languages/hu.json
new file mode 100644
index 00000000..8e377aaa
--- /dev/null
+++ b/plugins/MergerSite/languages/hu.json
@@ -0,0 +1,5 @@
+{
+ "Add %s new site?": "Új oldal hozzáadása: %s?",
+ "Added %s new site": "Új oldal hozzáadva: %s",
+ "Site deleted: %s": "Oldal törölve: %s"
+}
diff --git a/plugins/MergerSite/languages/it.json b/plugins/MergerSite/languages/it.json
new file mode 100644
index 00000000..d56c9817
--- /dev/null
+++ b/plugins/MergerSite/languages/it.json
@@ -0,0 +1,5 @@
+{
+ "Add %s new site?": "Aggiungere %s nuovo sito ?",
+ "Added %s new site": "Sito %s aggiunto",
+ "Site deleted: %s": "Sito %s eliminato"
+}
diff --git a/plugins/MergerSite/languages/pt-br.json b/plugins/MergerSite/languages/pt-br.json
new file mode 100644
index 00000000..cdc298cb
--- /dev/null
+++ b/plugins/MergerSite/languages/pt-br.json
@@ -0,0 +1,5 @@
+{
+ "Add %s new site?": "Adicionar %s novo site?",
+ "Added %s new site": "Site %s adicionado",
+ "Site deleted: %s": "Site removido: %s"
+}
diff --git a/plugins/MergerSite/languages/tr.json b/plugins/MergerSite/languages/tr.json
new file mode 100644
index 00000000..5afb3942
--- /dev/null
+++ b/plugins/MergerSite/languages/tr.json
@@ -0,0 +1,5 @@
+{
+ "Add %s new site?": "%s sitesi eklensin mi?",
+ "Added %s new site": "%s sitesi eklendi",
+ "Site deleted: %s": "%s sitesi silindi"
+}
diff --git a/plugins/MergerSite/languages/zh-tw.json b/plugins/MergerSite/languages/zh-tw.json
new file mode 100644
index 00000000..a0684e63
--- /dev/null
+++ b/plugins/MergerSite/languages/zh-tw.json
@@ -0,0 +1,5 @@
+{
+ "Add %s new site?": "添加新網站: %s？",
+ "Added %s new site": "已添加到新網站：%s",
+ "Site deleted: %s": "網站已刪除：%s"
+}
diff --git a/plugins/MergerSite/languages/zh.json b/plugins/MergerSite/languages/zh.json
new file mode 100644
index 00000000..127044e6
--- /dev/null
+++ b/plugins/MergerSite/languages/zh.json
@@ -0,0 +1,5 @@
+{
+ "Add %s new site?": "添加新站点: %s？",
+ "Added %s new site": "已添加到新站点：%s",
+ "Site deleted: %s": "站点已删除：%s"
+}
diff --git a/plugins/Newsfeed/NewsfeedPlugin.py b/plugins/Newsfeed/NewsfeedPlugin.py
new file mode 100644
index 00000000..461401ac
--- /dev/null
+++ b/plugins/Newsfeed/NewsfeedPlugin.py
@@ -0,0 +1,188 @@
+import time
+import re
+
+from Plugin import PluginManager
+from Db.DbQuery import DbQuery
+from Debug import Debug
+from util import helper
+
+
+@PluginManager.registerTo("UiWebsocket")
+class UiWebsocketPlugin(object):
+ def formatSiteInfo(self, site, create_user=True):
+ site_info = super(UiWebsocketPlugin, self).formatSiteInfo(site, create_user=create_user)
+ feed_following = self.user.sites.get(site.address, {}).get("follow", None)
+ if feed_following == None:
+ site_info["feed_follow_num"] = None
+ else:
+ site_info["feed_follow_num"] = len(feed_following)
+ return site_info
+
+ def actionFeedFollow(self, to, feeds):
+ self.user.setFeedFollow(self.site.address, feeds)
+ self.user.save()
+ self.response(to, "ok")
+
+ def actionFeedListFollow(self, to):
+ feeds = self.user.sites[self.site.address].get("follow", {})
+ self.response(to, feeds)
+
+ def actionFeedQuery(self, to, limit=10, day_limit=3):
+ if "ADMIN" not in self.site.settings["permissions"]:
+ return self.response(to, "FeedQuery not allowed")
+
+ from Site import SiteManager
+ rows = []
+ stats = []
+
+ total_s = time.time()
+ num_sites = 0
+
+ for address, site_data in list(self.user.sites.items()):
+ feeds = site_data.get("follow")
+ if not feeds:
+ continue
+ if type(feeds) is not dict:
+ self.log.debug("Invalid feed for site %s" % address)
+ continue
+ num_sites += 1
+ for name, query_set in feeds.items():
+ site = SiteManager.site_manager.get(address)
+ if not site or not site.storage.has_db:
+ continue
+
+ s = time.time()
+ try:
+ query_raw, params = query_set
+ query_parts = re.split(r"UNION(?:\s+ALL|)", query_raw)
+ for i, query_part in enumerate(query_parts):
+ db_query = DbQuery(query_part)
+ if day_limit:
+ where = " WHERE %s > strftime('%%s', 'now', '-%s day')" % (db_query.fields.get("date_added", "date_added"), day_limit)
+ if "WHERE" in query_part:
+ query_part = re.sub("WHERE (.*?)(?=$| GROUP BY)", where+" AND (\\1)", query_part)
+ else:
+ query_part += where
+ query_parts[i] = query_part
+ query = " UNION ".join(query_parts)
+
+ if ":params" in query:
+ query_params = map(helper.sqlquote, params)
+ query = query.replace(":params", ",".join(query_params))
+
+ res = site.storage.query(query + " ORDER BY date_added DESC LIMIT %s" % limit)
+
+ except Exception as err: # Log error
+ self.log.error("%s feed query %s error: %s" % (address, name, Debug.formatException(err)))
+ stats.append({"site": site.address, "feed_name": name, "error": str(err)})
+ continue
+
+ for row in res:
+ row = dict(row)
+ if not isinstance(row["date_added"], (int, float, complex)):
+ self.log.debug("Invalid date_added from site %s: %r" % (address, row["date_added"]))
+ continue
+ if row["date_added"] > 1000000000000: # Formatted as millseconds
+ row["date_added"] = row["date_added"] / 1000
+ if "date_added" not in row or row["date_added"] > time.time() + 120:
+ self.log.debug("Newsfeed item from the future from from site %s" % address)
+ continue # Feed item is in the future, skip it
+ row["site"] = address
+ row["feed_name"] = name
+ rows.append(row)
+ stats.append({"site": site.address, "feed_name": name, "taken": round(time.time() - s, 3)})
+ time.sleep(0.001)
+ return self.response(to, {"rows": rows, "stats": stats, "num": len(rows), "sites": num_sites, "taken": round(time.time() - total_s, 3)})
+
+ def parseSearch(self, search):
+ parts = re.split("(site|type):", search)
+ if len(parts) > 1: # Found filter
+ search_text = parts[0]
+ parts = [part.strip() for part in parts]
+ filters = dict(zip(parts[1::2], parts[2::2]))
+ else:
+ search_text = search
+ filters = {}
+ return [search_text, filters]
+
+ def actionFeedSearch(self, to, search, limit=30, day_limit=30):
+ if "ADMIN" not in self.site.settings["permissions"]:
+ return self.response(to, "FeedSearch not allowed")
+
+ from Site import SiteManager
+ rows = []
+ stats = []
+ num_sites = 0
+ total_s = time.time()
+
+ search_text, filters = self.parseSearch(search)
+
+ for address, site in SiteManager.site_manager.list().items():
+ if not site.storage.has_db:
+ continue
+
+ if "site" in filters:
+ if filters["site"].lower() not in [site.address, site.content_manager.contents["content.json"].get("title").lower()]:
+ continue
+
+ if site.storage.db: # Database loaded
+ feeds = site.storage.db.schema.get("feeds")
+ else:
+ try:
+ feeds = site.storage.loadJson("dbschema.json").get("feeds")
+ except:
+ continue
+
+ if not feeds:
+ continue
+
+ num_sites += 1
+
+ for name, query in feeds.items():
+ s = time.time()
+ try:
+ db_query = DbQuery(query)
+
+ params = []
+ # Filters
+ if search_text:
+ db_query.wheres.append("(%s LIKE ? OR %s LIKE ?)" % (db_query.fields["body"], db_query.fields["title"]))
+ search_like = "%" + search_text.replace(" ", "%") + "%"
+ params.append(search_like)
+ params.append(search_like)
+ if filters.get("type") and filters["type"] not in query:
+ continue
+
+ if day_limit:
+ db_query.wheres.append(
+ "%s > strftime('%%s', 'now', '-%s day')" % (db_query.fields.get("date_added", "date_added"), day_limit)
+ )
+
+ # Order
+ db_query.parts["ORDER BY"] = "date_added DESC"
+ db_query.parts["LIMIT"] = str(limit)
+
+ res = site.storage.query(str(db_query), params)
+ except Exception as err:
+ self.log.error("%s feed query %s error: %s" % (address, name, Debug.formatException(err)))
+ stats.append({"site": site.address, "feed_name": name, "error": str(err), "query": query})
+ continue
+ for row in res:
+ row = dict(row)
+ if not row["date_added"] or row["date_added"] > time.time() + 120:
+ continue # Feed item is in the future, skip it
+ row["site"] = address
+ row["feed_name"] = name
+ rows.append(row)
+ stats.append({"site": site.address, "feed_name": name, "taken": round(time.time() - s, 3)})
+ return self.response(to, {"rows": rows, "num": len(rows), "sites": num_sites, "taken": round(time.time() - total_s, 3), "stats": stats})
+
+
+@PluginManager.registerTo("User")
+class UserPlugin(object):
+ # Set queries that user follows
+ def setFeedFollow(self, address, feeds):
+ site_data = self.getSiteData(address)
+ site_data["follow"] = feeds
+ self.save()
+ return site_data
diff --git a/plugins/Newsfeed/__init__.py b/plugins/Newsfeed/__init__.py
new file mode 100644
index 00000000..6e624df6
--- /dev/null
+++ b/plugins/Newsfeed/__init__.py
@@ -0,0 +1 @@
+from . import NewsfeedPlugin
\ No newline at end of file
diff --git a/plugins/OptionalManager/ContentDbPlugin.py b/plugins/OptionalManager/ContentDbPlugin.py
new file mode 100644
index 00000000..e7945d93
--- /dev/null
+++ b/plugins/OptionalManager/ContentDbPlugin.py
@@ -0,0 +1,407 @@
+import time
+import collections
+import itertools
+import re
+
+import gevent
+
+from util import helper
+from Plugin import PluginManager
+from Config import config
+from Debug import Debug
+
+if "content_db" not in locals().keys(): # To keep between module reloads
+ content_db = None
+
+
+@PluginManager.registerTo("ContentDb")
+class ContentDbPlugin(object):
+ def __init__(self, *args, **kwargs):
+ global content_db
+ content_db = self
+ self.filled = {} # Site addresses that already filled from content.json
+ self.need_filling = False # file_optional table just created, fill data from content.json files
+ self.time_peer_numbers_updated = 0
+ self.my_optional_files = {} # Last 50 site_address/inner_path called by fileWrite (auto-pinning these files)
+ self.optional_files = collections.defaultdict(dict)
+ self.optional_files_loading = False
+ helper.timer(60 * 5, self.checkOptionalLimit)
+ super(ContentDbPlugin, self).__init__(*args, **kwargs)
+
+ def getSchema(self):
+ schema = super(ContentDbPlugin, self).getSchema()
+
+ # Need file_optional table
+ schema["tables"]["file_optional"] = {
+ "cols": [
+ ["file_id", "INTEGER PRIMARY KEY UNIQUE NOT NULL"],
+ ["site_id", "INTEGER REFERENCES site (site_id) ON DELETE CASCADE"],
+ ["inner_path", "TEXT"],
+ ["hash_id", "INTEGER"],
+ ["size", "INTEGER"],
+ ["peer", "INTEGER DEFAULT 0"],
+ ["uploaded", "INTEGER DEFAULT 0"],
+ ["is_downloaded", "INTEGER DEFAULT 0"],
+ ["is_pinned", "INTEGER DEFAULT 0"],
+ ["time_added", "INTEGER DEFAULT 0"],
+ ["time_downloaded", "INTEGER DEFAULT 0"],
+ ["time_accessed", "INTEGER DEFAULT 0"]
+ ],
+ "indexes": [
+ "CREATE UNIQUE INDEX file_optional_key ON file_optional (site_id, inner_path)",
+ "CREATE INDEX is_downloaded ON file_optional (is_downloaded)"
+ ],
+ "schema_changed": 11
+ }
+
+ return schema
+
+ def initSite(self, site):
+ super(ContentDbPlugin, self).initSite(site)
+ if self.need_filling:
+ self.fillTableFileOptional(site)
+ if not self.optional_files_loading:
+ gevent.spawn_later(1, self.loadFilesOptional)
+ self.optional_files_loading = True
+
+ def checkTables(self):
+ changed_tables = super(ContentDbPlugin, self).checkTables()
+ if "file_optional" in changed_tables:
+ self.need_filling = True
+ return changed_tables
+
+ # Load optional files ending
+ def loadFilesOptional(self):
+ s = time.time()
+ num = 0
+ total = 0
+ total_downloaded = 0
+ res = content_db.execute("SELECT site_id, inner_path, size, is_downloaded FROM file_optional")
+ site_sizes = collections.defaultdict(lambda: collections.defaultdict(int))
+ for row in res:
+ self.optional_files[row["site_id"]][row["inner_path"][-8:]] = 1
+ num += 1
+
+ # Update site size stats
+ site_sizes[row["site_id"]]["size_optional"] += row["size"]
+ if row["is_downloaded"]:
+ site_sizes[row["site_id"]]["optional_downloaded"] += row["size"]
+
+ # Site site size stats to sites.json settings
+ site_ids_reverse = {val: key for key, val in self.site_ids.items()}
+ for site_id, stats in site_sizes.items():
+ site_address = site_ids_reverse.get(site_id)
+ if not site_address:
+ self.log.error("Not found site_id: %s" % site_id)
+ continue
+ site = self.sites[site_address]
+ site.settings["size_optional"] = stats["size_optional"]
+ site.settings["optional_downloaded"] = stats["optional_downloaded"]
+ total += stats["size_optional"]
+ total_downloaded += stats["optional_downloaded"]
+
+ self.log.debug(
+ "Loaded %s optional files: %.2fMB, downloaded: %.2fMB in %.3fs" %
+ (num, float(total) / 1024 / 1024, float(total_downloaded) / 1024 / 1024, time.time() - s)
+ )
+
+ if self.need_filling and self.getOptionalLimitBytes() >= 0 and self.getOptionalLimitBytes() < total_downloaded:
+ limit_bytes = self.getOptionalLimitBytes()
+ limit_new = round((float(total_downloaded) / 1024 / 1024 / 1024) * 1.1, 2) # Current limit + 10%
+ self.log.debug(
+ "First startup after update and limit is smaller than downloaded files size (%.2fGB), increasing it from %.2fGB to %.2fGB" %
+ (float(total_downloaded) / 1024 / 1024 / 1024, float(limit_bytes) / 1024 / 1024 / 1024, limit_new)
+ )
+ config.saveValue("optional_limit", limit_new)
+ config.optional_limit = str(limit_new)
+
+ # Predicts if the file is optional
+ def isOptionalFile(self, site_id, inner_path):
+ return self.optional_files[site_id].get(inner_path[-8:])
+
+ # Fill file_optional table with optional files found in sites
+ def fillTableFileOptional(self, site):
+ s = time.time()
+ site_id = self.site_ids.get(site.address)
+ if not site_id:
+ return False
+ cur = self.getCursor()
+ res = cur.execute("SELECT * FROM content WHERE size_files_optional > 0 AND site_id = %s" % site_id)
+ num = 0
+ for row in res.fetchall():
+ content = site.content_manager.contents[row["inner_path"]]
+ try:
+ num += self.setContentFilesOptional(site, row["inner_path"], content, cur=cur)
+ except Exception as err:
+ self.log.error("Error loading %s into file_optional: %s" % (row["inner_path"], err))
+ cur.close()
+
+ # Set my files to pinned
+ from User import UserManager
+ user = UserManager.user_manager.get()
+ if not user:
+ user = UserManager.user_manager.create()
+ auth_address = user.getAuthAddress(site.address)
+ self.execute(
+ "UPDATE file_optional SET is_pinned = 1 WHERE site_id = :site_id AND inner_path LIKE :inner_path",
+ {"site_id": site_id, "inner_path": "%%/%s/%%" % auth_address}
+ )
+
+ self.log.debug(
+ "Filled file_optional table for %s in %.3fs (loaded: %s, is_pinned: %s)" %
+ (site.address, time.time() - s, num, self.cur.cursor.rowcount)
+ )
+ self.filled[site.address] = True
+
+ def setContentFilesOptional(self, site, content_inner_path, content, cur=None):
+ if not cur:
+ cur = self
+
+ num = 0
+ site_id = self.site_ids[site.address]
+ content_inner_dir = helper.getDirname(content_inner_path)
+ for relative_inner_path, file in content.get("files_optional", {}).items():
+ file_inner_path = content_inner_dir + relative_inner_path
+ hash_id = int(file["sha512"][0:4], 16)
+ if hash_id in site.content_manager.hashfield:
+ is_downloaded = 1
+ else:
+ is_downloaded = 0
+ if site.address + "/" + content_inner_dir in self.my_optional_files:
+ is_pinned = 1
+ else:
+ is_pinned = 0
+ cur.insertOrUpdate("file_optional", {
+ "hash_id": hash_id,
+ "size": int(file["size"])
+ }, {
+ "site_id": site_id,
+ "inner_path": file_inner_path
+ }, oninsert={
+ "time_added": int(time.time()),
+ "time_downloaded": int(time.time()) if is_downloaded else 0,
+ "is_downloaded": is_downloaded,
+ "peer": is_downloaded,
+ "is_pinned": is_pinned
+ })
+ self.optional_files[site_id][file_inner_path[-8:]] = 1
+ num += 1
+
+ return num
+
+ def setContent(self, site, inner_path, content, size=0):
+ super(ContentDbPlugin, self).setContent(site, inner_path, content, size=size)
+ old_content = site.content_manager.contents.get(inner_path, {})
+ if (not self.need_filling or self.filled.get(site.address)) and ("files_optional" in content or "files_optional" in old_content):
+ self.setContentFilesOptional(site, inner_path, content)
+ # Check deleted files
+ if old_content:
+ old_files = old_content.get("files_optional", {}).keys()
+ new_files = content.get("files_optional", {}).keys()
+ content_inner_dir = helper.getDirname(inner_path)
+ deleted = [content_inner_dir + key for key in old_files if key not in new_files]
+ if deleted:
+ site_id = self.site_ids[site.address]
+ self.execute("DELETE FROM file_optional WHERE ?", {"site_id": site_id, "inner_path": deleted})
+
+ def deleteContent(self, site, inner_path):
+ content = site.content_manager.contents.get(inner_path)
+ if content and "files_optional" in content:
+ site_id = self.site_ids[site.address]
+ content_inner_dir = helper.getDirname(inner_path)
+ optional_inner_paths = [
+ content_inner_dir + relative_inner_path
+ for relative_inner_path in content.get("files_optional", {}).keys()
+ ]
+ self.execute("DELETE FROM file_optional WHERE ?", {"site_id": site_id, "inner_path": optional_inner_paths})
+ super(ContentDbPlugin, self).deleteContent(site, inner_path)
+
+ def updatePeerNumbers(self):
+ s = time.time()
+ num_file = 0
+ num_updated = 0
+ num_site = 0
+ for site in list(self.sites.values()):
+ if not site.content_manager.has_optional_files:
+ continue
+ if not site.isServing():
+ continue
+ has_updated_hashfield = next((
+ peer
+ for peer in site.peers.values()
+ if peer.has_hashfield and peer.hashfield.time_changed > self.time_peer_numbers_updated
+ ), None)
+
+ if not has_updated_hashfield and site.content_manager.hashfield.time_changed < self.time_peer_numbers_updated:
+ continue
+
+ hashfield_peers = itertools.chain.from_iterable(
+ peer.hashfield.storage
+ for peer in site.peers.values()
+ if peer.has_hashfield
+ )
+ peer_nums = collections.Counter(
+ itertools.chain(
+ hashfield_peers,
+ site.content_manager.hashfield
+ )
+ )
+
+ site_id = self.site_ids[site.address]
+ if not site_id:
+ continue
+
+ res = self.execute("SELECT file_id, hash_id, peer FROM file_optional WHERE ?", {"site_id": site_id})
+ updates = {}
+ for row in res:
+ peer_num = peer_nums.get(row["hash_id"], 0)
+ if peer_num != row["peer"]:
+ updates[row["file_id"]] = peer_num
+
+ for file_id, peer_num in updates.items():
+ self.execute("UPDATE file_optional SET peer = ? WHERE file_id = ?", (peer_num, file_id))
+
+ num_updated += len(updates)
+ num_file += len(peer_nums)
+ num_site += 1
+
+ self.time_peer_numbers_updated = time.time()
+ self.log.debug("%s/%s peer number for %s site updated in %.3fs" % (num_updated, num_file, num_site, time.time() - s))
+
+ def queryDeletableFiles(self):
+ # First return the files with atleast 10 seeder and not accessed in last week
+ query = """
+ SELECT * FROM file_optional
+ WHERE peer > 10 AND %s
+ ORDER BY time_accessed < %s DESC, uploaded / size
+ """ % (self.getOptionalUsedWhere(), int(time.time() - 60 * 60 * 7))
+ limit_start = 0
+ while 1:
+ num = 0
+ res = self.execute("%s LIMIT %s, 50" % (query, limit_start))
+ for row in res:
+ yield row
+ num += 1
+ if num < 50:
+ break
+ limit_start += 50
+
+ self.log.debug("queryDeletableFiles returning less-seeded files")
+
+ # Then return files less seeder but still not accessed in last week
+ query = """
+ SELECT * FROM file_optional
+ WHERE peer <= 10 AND %s
+ ORDER BY peer DESC, time_accessed < %s DESC, uploaded / size
+ """ % (self.getOptionalUsedWhere(), int(time.time() - 60 * 60 * 7))
+ limit_start = 0
+ while 1:
+ num = 0
+ res = self.execute("%s LIMIT %s, 50" % (query, limit_start))
+ for row in res:
+ yield row
+ num += 1
+ if num < 50:
+ break
+ limit_start += 50
+
+ self.log.debug("queryDeletableFiles returning everyting")
+
+ # At the end return all files
+ query = """
+ SELECT * FROM file_optional
+ WHERE peer <= 10 AND %s
+ ORDER BY peer DESC, time_accessed, uploaded / size
+ """ % self.getOptionalUsedWhere()
+ limit_start = 0
+ while 1:
+ num = 0
+ res = self.execute("%s LIMIT %s, 50" % (query, limit_start))
+ for row in res:
+ yield row
+ num += 1
+ if num < 50:
+ break
+ limit_start += 50
+
+ def getOptionalLimitBytes(self):
+ if config.optional_limit.endswith("%"):
+ limit_percent = float(re.sub("[^0-9.]", "", config.optional_limit))
+ limit_bytes = helper.getFreeSpace() * (limit_percent / 100)
+ else:
+ limit_bytes = float(re.sub("[^0-9.]", "", config.optional_limit)) * 1024 * 1024 * 1024
+ return limit_bytes
+
+ def getOptionalUsedWhere(self):
+ maxsize = config.optional_limit_exclude_minsize * 1024 * 1024
+ query = "is_downloaded = 1 AND is_pinned = 0 AND size < %s" % maxsize
+
+ # Don't delete optional files from owned sites
+ my_site_ids = []
+ for address, site in self.sites.items():
+ if site.settings["own"]:
+ my_site_ids.append(str(self.site_ids[address]))
+
+ if my_site_ids:
+ query += " AND site_id NOT IN (%s)" % ", ".join(my_site_ids)
+ return query
+
+ def getOptionalUsedBytes(self):
+ size = self.execute("SELECT SUM(size) FROM file_optional WHERE %s" % self.getOptionalUsedWhere()).fetchone()[0]
+ if not size:
+ size = 0
+ return size
+
+ def getOptionalNeedDelete(self, size):
+ if config.optional_limit.endswith("%"):
+ limit_percent = float(re.sub("[^0-9.]", "", config.optional_limit))
+ need_delete = size - ((helper.getFreeSpace() + size) * (limit_percent / 100))
+ else:
+ need_delete = size - self.getOptionalLimitBytes()
+ return need_delete
+
+ def checkOptionalLimit(self, limit=None):
+ if not limit:
+ limit = self.getOptionalLimitBytes()
+
+ if limit < 0:
+ self.log.debug("Invalid limit for optional files: %s" % limit)
+ return False
+
+ size = self.getOptionalUsedBytes()
+
+ need_delete = self.getOptionalNeedDelete(size)
+
+ self.log.debug(
+ "Optional size: %.1fMB/%.1fMB, Need delete: %.1fMB" %
+ (float(size) / 1024 / 1024, float(limit) / 1024 / 1024, float(need_delete) / 1024 / 1024)
+ )
+ if need_delete <= 0:
+ return False
+
+ self.updatePeerNumbers()
+
+ site_ids_reverse = {val: key for key, val in self.site_ids.items()}
+ deleted_file_ids = []
+ for row in self.queryDeletableFiles():
+ site_address = site_ids_reverse.get(row["site_id"])
+ site = self.sites.get(site_address)
+ if not site:
+ self.log.error("No site found for id: %s" % row["site_id"])
+ continue
+ site.log.debug("Deleting %s %.3f MB left" % (row["inner_path"], float(need_delete) / 1024 / 1024))
+ deleted_file_ids.append(row["file_id"])
+ try:
+ site.content_manager.optionalRemoved(row["inner_path"], row["hash_id"], row["size"])
+ site.storage.delete(row["inner_path"])
+ need_delete -= row["size"]
+ except Exception as err:
+ site.log.error("Error deleting %s: %s" % (row["inner_path"], err))
+
+ if need_delete <= 0:
+ break
+
+ cur = self.getCursor()
+ for file_id in deleted_file_ids:
+ cur.execute("UPDATE file_optional SET is_downloaded = 0, is_pinned = 0, peer = peer - 1 WHERE ?", {"file_id": file_id})
+ cur.close()
diff --git a/plugins/OptionalManager/OptionalManagerPlugin.py b/plugins/OptionalManager/OptionalManagerPlugin.py
new file mode 100644
index 00000000..909caa31
--- /dev/null
+++ b/plugins/OptionalManager/OptionalManagerPlugin.py
@@ -0,0 +1,239 @@
+import time
+import re
+import collections
+
+import gevent
+
+from util import helper
+from Plugin import PluginManager
+from . import ContentDbPlugin
+
+
+# We can only import plugin host clases after the plugins are loaded
+@PluginManager.afterLoad
+def importPluginnedClasses():
+ global config
+ from Config import config
+
+
+def processAccessLog():
+ if access_log:
+ content_db = ContentDbPlugin.content_db
+ now = int(time.time())
+ num = 0
+ for site_id in access_log:
+ content_db.execute(
+ "UPDATE file_optional SET time_accessed = %s WHERE ?" % now,
+ {"site_id": site_id, "inner_path": list(access_log[site_id].keys())}
+ )
+ num += len(access_log[site_id])
+ access_log.clear()
+
+
+def processRequestLog():
+ if request_log:
+ content_db = ContentDbPlugin.content_db
+ cur = content_db.getCursor()
+ num = 0
+ for site_id in request_log:
+ for inner_path, uploaded in request_log[site_id].items():
+ content_db.execute(
+ "UPDATE file_optional SET uploaded = uploaded + %s WHERE ?" % uploaded,
+ {"site_id": site_id, "inner_path": inner_path}
+ )
+ num += 1
+ request_log.clear()
+
+
+if "access_log" not in locals().keys(): # To keep between module reloads
+ access_log = collections.defaultdict(dict) # {site_id: {inner_path1: 1, inner_path2: 1...}}
+ request_log = collections.defaultdict(lambda: collections.defaultdict(int)) # {site_id: {inner_path1: 1, inner_path2: 1...}}
+ helper.timer(61, processAccessLog)
+ helper.timer(60, processRequestLog)
+
+
+@PluginManager.registerTo("ContentManager")
+class ContentManagerPlugin(object):
+ def __init__(self, *args, **kwargs):
+ self.cache_is_pinned = {}
+ super(ContentManagerPlugin, self).__init__(*args, **kwargs)
+
+ def optionalDownloaded(self, inner_path, hash_id, size=None, own=False):
+ if "|" in inner_path: # Big file piece
+ file_inner_path, file_range = inner_path.split("|")
+ else:
+ file_inner_path = inner_path
+
+ self.contents.db.executeDelayed(
+ "UPDATE file_optional SET time_downloaded = :now, is_downloaded = 1, peer = peer + 1 WHERE site_id = :site_id AND inner_path = :inner_path AND is_downloaded = 0",
+ {"now": int(time.time()), "site_id": self.contents.db.site_ids[self.site.address], "inner_path": file_inner_path}
+ )
+
+ return super(ContentManagerPlugin, self).optionalDownloaded(inner_path, hash_id, size, own)
+
+ def optionalRemoved(self, inner_path, hash_id, size=None):
+ self.contents.db.execute(
+ "UPDATE file_optional SET is_downloaded = 0, is_pinned = 0, peer = peer - 1 WHERE site_id = :site_id AND inner_path = :inner_path AND is_downloaded = 1",
+ {"site_id": self.contents.db.site_ids[self.site.address], "inner_path": inner_path}
+ )
+
+ if self.contents.db.cur.cursor.rowcount > 0:
+ back = super(ContentManagerPlugin, self).optionalRemoved(inner_path, hash_id, size)
+ # Re-add to hashfield if we have other file with the same hash_id
+ if self.isDownloaded(hash_id=hash_id, force_check_db=True):
+ self.hashfield.appendHashId(hash_id)
+ else:
+ back = False
+ self.cache_is_pinned = {}
+ return back
+
+ def optionalRenamed(self, inner_path_old, inner_path_new):
+ back = super(ContentManagerPlugin, self).optionalRenamed(inner_path_old, inner_path_new)
+ self.cache_is_pinned = {}
+ self.contents.db.execute(
+ "UPDATE file_optional SET inner_path = :inner_path_new WHERE site_id = :site_id AND inner_path = :inner_path_old",
+ {"site_id": self.contents.db.site_ids[self.site.address], "inner_path_old": inner_path_old, "inner_path_new": inner_path_new}
+ )
+ return back
+
+ def isDownloaded(self, inner_path=None, hash_id=None, force_check_db=False):
+ if hash_id and not force_check_db and hash_id not in self.hashfield:
+ return False
+
+ if inner_path:
+ res = self.contents.db.execute(
+ "SELECT is_downloaded FROM file_optional WHERE site_id = :site_id AND inner_path = :inner_path LIMIT 1",
+ {"site_id": self.contents.db.site_ids[self.site.address], "inner_path": inner_path}
+ )
+ else:
+ res = self.contents.db.execute(
+ "SELECT is_downloaded FROM file_optional WHERE site_id = :site_id AND hash_id = :hash_id AND is_downloaded = 1 LIMIT 1",
+ {"site_id": self.contents.db.site_ids[self.site.address], "hash_id": hash_id}
+ )
+ row = res.fetchone()
+ if row and row["is_downloaded"]:
+ return True
+ else:
+ return False
+
+ def isPinned(self, inner_path):
+ if inner_path in self.cache_is_pinned:
+ self.site.log.debug("Cached is pinned: %s" % inner_path)
+ return self.cache_is_pinned[inner_path]
+
+ res = self.contents.db.execute(
+ "SELECT is_pinned FROM file_optional WHERE site_id = :site_id AND inner_path = :inner_path LIMIT 1",
+ {"site_id": self.contents.db.site_ids[self.site.address], "inner_path": inner_path}
+ )
+ row = res.fetchone()
+
+ if row and row[0]:
+ is_pinned = True
+ else:
+ is_pinned = False
+
+ self.cache_is_pinned[inner_path] = is_pinned
+ self.site.log.debug("Cache set is pinned: %s %s" % (inner_path, is_pinned))
+
+ return is_pinned
+
+ def setPin(self, inner_path, is_pinned):
+ content_db = self.contents.db
+ site_id = content_db.site_ids[self.site.address]
+ content_db.execute("UPDATE file_optional SET is_pinned = %d WHERE ?" % is_pinned, {"site_id": site_id, "inner_path": inner_path})
+ self.cache_is_pinned = {}
+
+ def optionalDelete(self, inner_path):
+ if self.isPinned(inner_path):
+ self.site.log.debug("Skip deleting pinned optional file: %s" % inner_path)
+ return False
+ else:
+ return super(ContentManagerPlugin, self).optionalDelete(inner_path)
+
+
+@PluginManager.registerTo("WorkerManager")
+class WorkerManagerPlugin(object):
+ def doneTask(self, task):
+ super(WorkerManagerPlugin, self).doneTask(task)
+
+ if task["optional_hash_id"] and not self.tasks: # Execute delayed queries immedietly after tasks finished
+ ContentDbPlugin.content_db.processDelayed()
+
+
+@PluginManager.registerTo("UiRequest")
+class UiRequestPlugin(object):
+ def parsePath(self, path):
+ global access_log
+ path_parts = super(UiRequestPlugin, self).parsePath(path)
+ if path_parts:
+ site_id = ContentDbPlugin.content_db.site_ids.get(path_parts["request_address"])
+ if site_id:
+ if ContentDbPlugin.content_db.isOptionalFile(site_id, path_parts["inner_path"]):
+ access_log[site_id][path_parts["inner_path"]] = 1
+ return path_parts
+
+
+@PluginManager.registerTo("FileRequest")
+class FileRequestPlugin(object):
+ def actionGetFile(self, params):
+ stats = super(FileRequestPlugin, self).actionGetFile(params)
+ self.recordFileRequest(params["site"], params["inner_path"], stats)
+ return stats
+
+ def actionStreamFile(self, params):
+ stats = super(FileRequestPlugin, self).actionStreamFile(params)
+ self.recordFileRequest(params["site"], params["inner_path"], stats)
+ return stats
+
+ def recordFileRequest(self, site_address, inner_path, stats):
+ if not stats:
+ # Only track the last request of files
+ return False
+ site_id = ContentDbPlugin.content_db.site_ids[site_address]
+ if site_id and ContentDbPlugin.content_db.isOptionalFile(site_id, inner_path):
+ request_log[site_id][inner_path] += stats["bytes_sent"]
+
+
+@PluginManager.registerTo("Site")
+class SitePlugin(object):
+ def isDownloadable(self, inner_path):
+ is_downloadable = super(SitePlugin, self).isDownloadable(inner_path)
+ if is_downloadable:
+ return is_downloadable
+
+ for path in self.settings.get("optional_help", {}).keys():
+ if inner_path.startswith(path):
+ return True
+
+ return False
+
+ def fileForgot(self, inner_path):
+ if "|" in inner_path and self.content_manager.isPinned(re.sub(r"\|.*", "", inner_path)):
+ self.log.debug("File %s is pinned, no fileForgot" % inner_path)
+ return False
+ else:
+ return super(SitePlugin, self).fileForgot(inner_path)
+
+ def fileDone(self, inner_path):
+ if "|" in inner_path and self.bad_files.get(inner_path, 0) > 5: # Idle optional file done
+ inner_path_file = re.sub(r"\|.*", "", inner_path)
+ num_changed = 0
+ for key, val in self.bad_files.items():
+ if key.startswith(inner_path_file) and val > 1:
+ self.bad_files[key] = 1
+ num_changed += 1
+ self.log.debug("Idle optional file piece done, changed retry number of %s pieces." % num_changed)
+ if num_changed:
+ gevent.spawn(self.retryBadFiles)
+
+ return super(SitePlugin, self).fileDone(inner_path)
+
+
+@PluginManager.registerTo("ConfigPlugin")
+class ConfigPlugin(object):
+ def createArguments(self):
+ group = self.parser.add_argument_group("OptionalManager plugin")
+ group.add_argument('--optional_limit', help='Limit total size of optional files', default="10%", metavar="GB or free space %")
+ group.add_argument('--optional_limit_exclude_minsize', help='Exclude files larger than this limit from optional size limit calculation', default=20, metavar="MB", type=int)
+
+ return super(ConfigPlugin, self).createArguments()
diff --git a/plugins/OptionalManager/Test/TestOptionalManager.py b/plugins/OptionalManager/Test/TestOptionalManager.py
new file mode 100644
index 00000000..4bd44695
--- /dev/null
+++ b/plugins/OptionalManager/Test/TestOptionalManager.py
@@ -0,0 +1,158 @@
+import copy
+
+import pytest
+
+
+@pytest.mark.usefixtures("resetSettings")
+class TestOptionalManager:
+ def testDbFill(self, site):
+ contents = site.content_manager.contents
+ assert len(site.content_manager.hashfield) > 0
+ assert contents.db.execute("SELECT COUNT(*) FROM file_optional WHERE is_downloaded = 1").fetchone()[0] == len(site.content_manager.hashfield)
+
+ def testSetContent(self, site):
+ contents = site.content_manager.contents
+
+ # Add new file
+ new_content = copy.deepcopy(contents["content.json"])
+ new_content["files_optional"]["testfile"] = {
+ "size": 1234,
+ "sha512": "aaaabbbbcccc"
+ }
+ num_optional_files_before = contents.db.execute("SELECT COUNT(*) FROM file_optional").fetchone()[0]
+ contents["content.json"] = new_content
+ assert contents.db.execute("SELECT COUNT(*) FROM file_optional").fetchone()[0] > num_optional_files_before
+
+ # Remove file
+ new_content = copy.deepcopy(contents["content.json"])
+ del new_content["files_optional"]["testfile"]
+ num_optional_files_before = contents.db.execute("SELECT COUNT(*) FROM file_optional").fetchone()[0]
+ contents["content.json"] = new_content
+ assert contents.db.execute("SELECT COUNT(*) FROM file_optional").fetchone()[0] < num_optional_files_before
+
+ def testDeleteContent(self, site):
+ contents = site.content_manager.contents
+ num_optional_files_before = contents.db.execute("SELECT COUNT(*) FROM file_optional").fetchone()[0]
+ del contents["content.json"]
+ assert contents.db.execute("SELECT COUNT(*) FROM file_optional").fetchone()[0] < num_optional_files_before
+
+ def testVerifyFiles(self, site):
+ contents = site.content_manager.contents
+
+ # Add new file
+ new_content = copy.deepcopy(contents["content.json"])
+ new_content["files_optional"]["testfile"] = {
+ "size": 1234,
+ "sha512": "aaaabbbbcccc"
+ }
+ contents["content.json"] = new_content
+ file_row = contents.db.execute("SELECT * FROM file_optional WHERE inner_path = 'testfile'").fetchone()
+ assert not file_row["is_downloaded"]
+
+ # Write file from outside of ZeroNet
+ site.storage.open("testfile", "wb").write(b"A" * 1234) # For quick check hash does not matter only file size
+
+ hashfield_len_before = len(site.content_manager.hashfield)
+ site.storage.verifyFiles(quick_check=True)
+ assert len(site.content_manager.hashfield) == hashfield_len_before + 1
+
+ file_row = contents.db.execute("SELECT * FROM file_optional WHERE inner_path = 'testfile'").fetchone()
+ assert file_row["is_downloaded"]
+
+ # Delete file outside of ZeroNet
+ site.storage.delete("testfile")
+ site.storage.verifyFiles(quick_check=True)
+ file_row = contents.db.execute("SELECT * FROM file_optional WHERE inner_path = 'testfile'").fetchone()
+ assert not file_row["is_downloaded"]
+
+ def testVerifyFilesSameHashId(self, site):
+ contents = site.content_manager.contents
+
+ new_content = copy.deepcopy(contents["content.json"])
+
+ # Add two files with same hashid (first 4 character)
+ new_content["files_optional"]["testfile1"] = {
+ "size": 1234,
+ "sha512": "aaaabbbbcccc"
+ }
+ new_content["files_optional"]["testfile2"] = {
+ "size": 2345,
+ "sha512": "aaaabbbbdddd"
+ }
+ contents["content.json"] = new_content
+
+ assert site.content_manager.hashfield.getHashId("aaaabbbbcccc") == site.content_manager.hashfield.getHashId("aaaabbbbdddd")
+
+ # Write files from outside of ZeroNet (For quick check hash does not matter only file size)
+ site.storage.open("testfile1", "wb").write(b"A" * 1234)
+ site.storage.open("testfile2", "wb").write(b"B" * 2345)
+
+ site.storage.verifyFiles(quick_check=True)
+
+ # Make sure that both is downloaded
+ assert site.content_manager.isDownloaded("testfile1")
+ assert site.content_manager.isDownloaded("testfile2")
+ assert site.content_manager.hashfield.getHashId("aaaabbbbcccc") in site.content_manager.hashfield
+
+ # Delete one of the files
+ site.storage.delete("testfile1")
+ site.storage.verifyFiles(quick_check=True)
+ assert not site.content_manager.isDownloaded("testfile1")
+ assert site.content_manager.isDownloaded("testfile2")
+ assert site.content_manager.hashfield.getHashId("aaaabbbbdddd") in site.content_manager.hashfield
+
+ def testIsPinned(self, site):
+ assert not site.content_manager.isPinned("data/img/zerotalk-upvote.png")
+ site.content_manager.setPin("data/img/zerotalk-upvote.png", True)
+ assert site.content_manager.isPinned("data/img/zerotalk-upvote.png")
+
+ assert len(site.content_manager.cache_is_pinned) == 1
+ site.content_manager.cache_is_pinned = {}
+ assert site.content_manager.isPinned("data/img/zerotalk-upvote.png")
+
+ def testBigfilePieceReset(self, site):
+ site.bad_files = {
+ "data/fake_bigfile.mp4|0-1024": 10,
+ "data/fake_bigfile.mp4|1024-2048": 10,
+ "data/fake_bigfile.mp4|2048-3064": 10
+ }
+ site.onFileDone("data/fake_bigfile.mp4|0-1024")
+ assert site.bad_files["data/fake_bigfile.mp4|1024-2048"] == 1
+ assert site.bad_files["data/fake_bigfile.mp4|2048-3064"] == 1
+
+ def testOptionalDelete(self, site):
+ contents = site.content_manager.contents
+
+ site.content_manager.setPin("data/img/zerotalk-upvote.png", True)
+ site.content_manager.setPin("data/img/zeroid.png", False)
+ new_content = copy.deepcopy(contents["content.json"])
+ del new_content["files_optional"]["data/img/zerotalk-upvote.png"]
+ del new_content["files_optional"]["data/img/zeroid.png"]
+
+ assert site.storage.isFile("data/img/zerotalk-upvote.png")
+ assert site.storage.isFile("data/img/zeroid.png")
+
+ site.storage.writeJson("content.json", new_content)
+ site.content_manager.loadContent("content.json", force=True)
+
+ assert not site.storage.isFile("data/img/zeroid.png")
+ assert site.storage.isFile("data/img/zerotalk-upvote.png")
+
+ def testOptionalRename(self, site):
+ contents = site.content_manager.contents
+
+ site.content_manager.setPin("data/img/zerotalk-upvote.png", True)
+ new_content = copy.deepcopy(contents["content.json"])
+ new_content["files_optional"]["data/img/zerotalk-upvote-new.png"] = new_content["files_optional"]["data/img/zerotalk-upvote.png"]
+ del new_content["files_optional"]["data/img/zerotalk-upvote.png"]
+
+ assert site.storage.isFile("data/img/zerotalk-upvote.png")
+ assert site.content_manager.isPinned("data/img/zerotalk-upvote.png")
+
+ site.storage.writeJson("content.json", new_content)
+ site.content_manager.loadContent("content.json", force=True)
+
+ assert not site.storage.isFile("data/img/zerotalk-upvote.png")
+ assert not site.content_manager.isPinned("data/img/zerotalk-upvote.png")
+ assert site.content_manager.isPinned("data/img/zerotalk-upvote-new.png")
+ assert site.storage.isFile("data/img/zerotalk-upvote-new.png")
diff --git a/plugins/OptionalManager/Test/conftest.py b/plugins/OptionalManager/Test/conftest.py
new file mode 100644
index 00000000..8c1df5b2
--- /dev/null
+++ b/plugins/OptionalManager/Test/conftest.py
@@ -0,0 +1 @@
+from src.Test.conftest import *
\ No newline at end of file
diff --git a/plugins/OptionalManager/Test/pytest.ini b/plugins/OptionalManager/Test/pytest.ini
new file mode 100644
index 00000000..d09210d1
--- /dev/null
+++ b/plugins/OptionalManager/Test/pytest.ini
@@ -0,0 +1,5 @@
+[pytest]
+python_files = Test*.py
+addopts = -rsxX -v --durations=6
+markers =
+ webtest: mark a test as a webtest.
\ No newline at end of file
diff --git a/plugins/OptionalManager/UiWebsocketPlugin.py b/plugins/OptionalManager/UiWebsocketPlugin.py
new file mode 100644
index 00000000..52b0f749
--- /dev/null
+++ b/plugins/OptionalManager/UiWebsocketPlugin.py
@@ -0,0 +1,382 @@
+import re
+import time
+import html
+
+import gevent
+
+from Plugin import PluginManager
+from Config import config
+from util import helper
+from Translate import Translate
+
+if "_" not in locals():
+ _ = Translate("plugins/OptionalManager/languages/")
+
+bigfile_sha512_cache = {}
+
+
+@PluginManager.registerTo("UiWebsocket")
+class UiWebsocketPlugin(object):
+ def __init__(self, *args, **kwargs):
+ self.time_peer_numbers_updated = 0
+ super(UiWebsocketPlugin, self).__init__(*args, **kwargs)
+
+ def actionSiteSign(self, to, privatekey=None, inner_path="content.json", *args, **kwargs):
+ # Add file to content.db and set it as pinned
+ content_db = self.site.content_manager.contents.db
+ content_inner_dir = helper.getDirname(inner_path)
+ content_db.my_optional_files[self.site.address + "/" + content_inner_dir] = time.time()
+ if len(content_db.my_optional_files) > 50: # Keep only last 50
+ oldest_key = min(
+ iter(content_db.my_optional_files.keys()),
+ key=(lambda key: content_db.my_optional_files[key])
+ )
+ del content_db.my_optional_files[oldest_key]
+
+ return super(UiWebsocketPlugin, self).actionSiteSign(to, privatekey, inner_path, *args, **kwargs)
+
+ def updatePeerNumbers(self):
+ self.site.updateHashfield()
+ content_db = self.site.content_manager.contents.db
+ content_db.updatePeerNumbers()
+ self.site.updateWebsocket(peernumber_updated=True)
+
+ def addBigfileInfo(self, row):
+ global bigfile_sha512_cache
+
+ content_db = self.site.content_manager.contents.db
+ site = content_db.sites[row["address"]]
+ if not site.settings.get("has_bigfile"):
+ return False
+
+ file_key = row["address"] + "/" + row["inner_path"]
+ sha512 = bigfile_sha512_cache.get(file_key)
+ file_info = None
+ if not sha512:
+ file_info = site.content_manager.getFileInfo(row["inner_path"])
+ if not file_info or not file_info.get("piece_size"):
+ return False
+ sha512 = file_info["sha512"]
+ bigfile_sha512_cache[file_key] = sha512
+
+ if sha512 in site.storage.piecefields:
+ piecefield = site.storage.piecefields[sha512].tobytes()
+ else:
+ piecefield = None
+
+ if piecefield:
+ row["pieces"] = len(piecefield)
+ row["pieces_downloaded"] = piecefield.count(b"\x01")
+ row["downloaded_percent"] = 100 * row["pieces_downloaded"] / row["pieces"]
+ if row["pieces_downloaded"]:
+ if row["pieces"] == row["pieces_downloaded"]:
+ row["bytes_downloaded"] = row["size"]
+ else:
+ if not file_info:
+ file_info = site.content_manager.getFileInfo(row["inner_path"])
+ row["bytes_downloaded"] = row["pieces_downloaded"] * file_info.get("piece_size", 0)
+ else:
+ row["bytes_downloaded"] = 0
+
+ row["is_downloading"] = bool(next((inner_path for inner_path in site.bad_files if inner_path.startswith(row["inner_path"])), False))
+
+ # Add leech / seed stats
+ row["peer_seed"] = 0
+ row["peer_leech"] = 0
+ for peer in site.peers.values():
+ if not peer.time_piecefields_updated or sha512 not in peer.piecefields:
+ continue
+ peer_piecefield = peer.piecefields[sha512].tobytes()
+ if not peer_piecefield:
+ continue
+ if peer_piecefield == b"\x01" * len(peer_piecefield):
+ row["peer_seed"] += 1
+ else:
+ row["peer_leech"] += 1
+
+ # Add myself
+ if piecefield:
+ if row["pieces_downloaded"] == row["pieces"]:
+ row["peer_seed"] += 1
+ else:
+ row["peer_leech"] += 1
+
+ return True
+
+ # Optional file functions
+
+ def actionOptionalFileList(self, to, address=None, orderby="time_downloaded DESC", limit=10, filter="downloaded"):
+ if not address:
+ address = self.site.address
+
+ # Update peer numbers if necessary
+ content_db = self.site.content_manager.contents.db
+ if time.time() - content_db.time_peer_numbers_updated > 60 * 1 and time.time() - self.time_peer_numbers_updated > 60 * 5:
+ # Start in new thread to avoid blocking
+ self.time_peer_numbers_updated = time.time()
+ gevent.spawn(self.updatePeerNumbers)
+
+ if address == "all" and "ADMIN" not in self.permissions:
+ return self.response(to, {"error": "Forbidden"})
+
+ if not self.hasSitePermission(address):
+ return self.response(to, {"error": "Forbidden"})
+
+ if not all([re.match("^[a-z_*/+-]+( DESC| ASC|)$", part.strip()) for part in orderby.split(",")]):
+ return self.response(to, "Invalid order_by")
+
+ if type(limit) != int:
+ return self.response(to, "Invalid limit")
+
+ back = []
+ content_db = self.site.content_manager.contents.db
+
+ wheres = {}
+ wheres_raw = []
+ if "bigfile" in filter:
+ wheres["size >"] = 1024 * 1024 * 10
+ if "downloaded" in filter:
+ wheres_raw.append("(is_downloaded = 1 OR is_pinned = 1)")
+ if "pinned" in filter:
+ wheres["is_pinned"] = 1
+
+ if address == "all":
+ join = "LEFT JOIN site USING (site_id)"
+ else:
+ wheres["site_id"] = content_db.site_ids[address]
+ join = ""
+
+ if wheres_raw:
+ query_wheres_raw = "AND" + " AND ".join(wheres_raw)
+ else:
+ query_wheres_raw = ""
+
+ query = "SELECT * FROM file_optional %s WHERE ? %s ORDER BY %s LIMIT %s" % (join, query_wheres_raw, orderby, limit)
+
+ for row in content_db.execute(query, wheres):
+ row = dict(row)
+ if address != "all":
+ row["address"] = address
+
+ if row["size"] > 1024 * 1024:
+ has_info = self.addBigfileInfo(row)
+ else:
+ has_info = False
+
+ if not has_info:
+ if row["is_downloaded"]:
+ row["bytes_downloaded"] = row["size"]
+ row["downloaded_percent"] = 100
+ else:
+ row["bytes_downloaded"] = 0
+ row["downloaded_percent"] = 0
+
+ back.append(row)
+ self.response(to, back)
+
+ def actionOptionalFileInfo(self, to, inner_path):
+ content_db = self.site.content_manager.contents.db
+ site_id = content_db.site_ids[self.site.address]
+
+ # Update peer numbers if necessary
+ if time.time() - content_db.time_peer_numbers_updated > 60 * 1 and time.time() - self.time_peer_numbers_updated > 60 * 5:
+ # Start in new thread to avoid blocking
+ self.time_peer_numbers_updated = time.time()
+ gevent.spawn(self.updatePeerNumbers)
+
+ query = "SELECT * FROM file_optional WHERE site_id = :site_id AND inner_path = :inner_path LIMIT 1"
+ res = content_db.execute(query, {"site_id": site_id, "inner_path": inner_path})
+ row = next(res, None)
+ if row:
+ row = dict(row)
+ if row["size"] > 1024 * 1024:
+ row["address"] = self.site.address
+ self.addBigfileInfo(row)
+ self.response(to, row)
+ else:
+ self.response(to, None)
+
+ def setPin(self, inner_path, is_pinned, address=None):
+ if not address:
+ address = self.site.address
+
+ if not self.hasSitePermission(address):
+ return {"error": "Forbidden"}
+
+ site = self.server.sites[address]
+ site.content_manager.setPin(inner_path, is_pinned)
+
+ return "ok"
+
+ def actionOptionalFilePin(self, to, inner_path, address=None):
+ if type(inner_path) is not list:
+ inner_path = [inner_path]
+ back = self.setPin(inner_path, 1, address)
+ num_file = len(inner_path)
+ if back == "ok":
+ if num_file == 1:
+ self.cmd("notification", ["done", _["Pinned %s"] % html.escape(helper.getFilename(inner_path[0])), 5000])
+ else:
+ self.cmd("notification", ["done", _["Pinned %s files"] % num_file, 5000])
+ self.response(to, back)
+
+ def actionOptionalFileUnpin(self, to, inner_path, address=None):
+ if type(inner_path) is not list:
+ inner_path = [inner_path]
+ back = self.setPin(inner_path, 0, address)
+ num_file = len(inner_path)
+ if back == "ok":
+ if num_file == 1:
+ self.cmd("notification", ["done", _["Removed pin from %s"] % html.escape(helper.getFilename(inner_path[0])), 5000])
+ else:
+ self.cmd("notification", ["done", _["Removed pin from %s files"] % num_file, 5000])
+ self.response(to, back)
+
+ def actionOptionalFileDelete(self, to, inner_path, address=None):
+ if not address:
+ address = self.site.address
+
+ if not self.hasSitePermission(address):
+ return self.response(to, {"error": "Forbidden"})
+
+ site = self.server.sites[address]
+
+ content_db = site.content_manager.contents.db
+ site_id = content_db.site_ids[site.address]
+
+ res = content_db.execute("SELECT * FROM file_optional WHERE ? LIMIT 1", {"site_id": site_id, "inner_path": inner_path, "is_downloaded": 1})
+ row = next(res, None)
+
+ if not row:
+ return self.response(to, {"error": "Not found in content.db"})
+
+ removed = site.content_manager.optionalRemoved(inner_path, row["hash_id"], row["size"])
+ # if not removed:
+ # return self.response(to, {"error": "Not found in hash_id: %s" % row["hash_id"]})
+
+ content_db.execute("UPDATE file_optional SET is_downloaded = 0, is_pinned = 0, peer = peer - 1 WHERE ?", {"site_id": site_id, "inner_path": inner_path})
+
+ try:
+ site.storage.delete(inner_path)
+ except Exception as err:
+ return self.response(to, {"error": "File delete error: %s" % err})
+ site.updateWebsocket(file_delete=inner_path)
+
+ if inner_path in site.content_manager.cache_is_pinned:
+ site.content_manager.cache_is_pinned = {}
+
+ self.response(to, "ok")
+
+ # Limit functions
+
+ def actionOptionalLimitStats(self, to):
+ if "ADMIN" not in self.site.settings["permissions"]:
+ return self.response(to, "Forbidden")
+
+ back = {}
+ back["limit"] = config.optional_limit
+ back["used"] = self.site.content_manager.contents.db.getOptionalUsedBytes()
+ back["free"] = helper.getFreeSpace()
+
+ self.response(to, back)
+
+ def actionOptionalLimitSet(self, to, limit):
+ if "ADMIN" not in self.site.settings["permissions"]:
+ return self.response(to, {"error": "Forbidden"})
+ config.optional_limit = re.sub("\.0+$", "", limit) # Remove unnecessary digits from end
+ config.saveValue("optional_limit", limit)
+ self.response(to, "ok")
+
+ # Distribute help functions
+
+ def actionOptionalHelpList(self, to, address=None):
+ if not address:
+ address = self.site.address
+
+ if not self.hasSitePermission(address):
+ return self.response(to, {"error": "Forbidden"})
+
+ site = self.server.sites[address]
+
+ self.response(to, site.settings.get("optional_help", {}))
+
+ def actionOptionalHelp(self, to, directory, title, address=None):
+ if not address:
+ address = self.site.address
+
+ if not self.hasSitePermission(address):
+ return self.response(to, {"error": "Forbidden"})
+
+ site = self.server.sites[address]
+ content_db = site.content_manager.contents.db
+ site_id = content_db.site_ids[address]
+
+ if "optional_help" not in site.settings:
+ site.settings["optional_help"] = {}
+
+ stats = content_db.execute(
+ "SELECT COUNT(*) AS num, SUM(size) AS size FROM file_optional WHERE site_id = :site_id AND inner_path LIKE :inner_path",
+ {"site_id": site_id, "inner_path": directory + "%"}
+ ).fetchone()
+ stats = dict(stats)
+
+ if not stats["size"]:
+ stats["size"] = 0
+ if not stats["num"]:
+ stats["num"] = 0
+
+ self.cmd("notification", [
+ "done",
+ _["You started to help distribute %s.Directory: %s"] % + (html.escape(title), html.escape(directory)), + 10000 + ]) + + site.settings["optional_help"][directory] = title + + self.response(to, dict(stats)) + + def actionOptionalHelpRemove(self, to, directory, address=None): + if not address: + address = self.site.address + + if not self.hasSitePermission(address): + return self.response(to, {"error": "Forbidden"}) + + site = self.server.sites[address] + + try: + del site.settings["optional_help"][directory] + self.response(to, "ok") + except Exception: + self.response(to, {"error": "Not found"}) + + def cbOptionalHelpAll(self, to, site, value): + site.settings["autodownloadoptional"] = value + self.response(to, value) + + def actionOptionalHelpAll(self, to, value, address=None): + if not address: + address = self.site.address + + if not self.hasSitePermission(address): + return self.response(to, {"error": "Forbidden"}) + + site = self.server.sites[address] + + if value: + if "ADMIN" in self.site.settings["permissions"]: + self.cbOptionalHelpAll(to, site, True) + else: + site_title = site.content_manager.contents["content.json"].get("title", address) + self.cmd( + "confirm", + [ + _["Help distribute all new optional files on site %s"] % html.escape(site_title), + _["Yes, I want to help!"] + ], + lambda res: self.cbOptionalHelpAll(to, site, True) + ) + else: + site.settings["autodownloadoptional"] = False + self.response(to, False) diff --git a/plugins/OptionalManager/__init__.py b/plugins/OptionalManager/__init__.py new file mode 100644 index 00000000..1f0ad2dd --- /dev/null +++ b/plugins/OptionalManager/__init__.py @@ -0,0 +1 @@ +from . import OptionalManagerPlugin \ No newline at end of file diff --git a/plugins/OptionalManager/languages/es.json b/plugins/OptionalManager/languages/es.json new file mode 100644 index 00000000..32ae46ae --- /dev/null +++ b/plugins/OptionalManager/languages/es.json @@ -0,0 +1,7 @@ +{ + "Pinned %s files": "Archivos %s fijados", + "Removed pin from %s files": "Archivos %s que no estan fijados", + "You started to help distribute %s.
Directory: %s": "Tu empezaste a ayudar a distribuir %s.
Directorio: %s", + "Help distribute all new optional files on site %s": "Ayude a distribuir todos los archivos opcionales en el sitio %s", + "Yes, I want to help!": "¡Si, yo quiero ayudar!" +} diff --git a/plugins/OptionalManager/languages/fr.json b/plugins/OptionalManager/languages/fr.json new file mode 100644 index 00000000..47a563dc --- /dev/null +++ b/plugins/OptionalManager/languages/fr.json @@ -0,0 +1,7 @@ +{ + "Pinned %s files": "Fichiers %s épinglés", + "Removed pin from %s files": "Fichiers %s ne sont plus épinglés", + "You started to help distribute %s.
Directory: %s": "Vous avez commencé à aider à distribuer %s.
Dossier : %s", + "Help distribute all new optional files on site %s": "Aider à distribuer tous les fichiers optionnels du site %s", + "Yes, I want to help!": "Oui, je veux aider !" +} diff --git a/plugins/OptionalManager/languages/hu.json b/plugins/OptionalManager/languages/hu.json new file mode 100644 index 00000000..7a23b86c --- /dev/null +++ b/plugins/OptionalManager/languages/hu.json @@ -0,0 +1,7 @@ +{ + "Pinned %s files": "%s fájl rögzítve", + "Removed pin from %s files": "%s fájl rögzítés eltávolítva", + "You started to help distribute %s.
Directory: %s": "Új segítség a terjesztésben: %s.
Könyvtár: %s", + "Help distribute all new optional files on site %s": "Segítség az összes új opcionális fájl terjesztésében az %s oldalon", + "Yes, I want to help!": "Igen, segíteni akarok!" +} diff --git a/plugins/OptionalManager/languages/pt-br.json b/plugins/OptionalManager/languages/pt-br.json new file mode 100644 index 00000000..21d90cc0 --- /dev/null +++ b/plugins/OptionalManager/languages/pt-br.json @@ -0,0 +1,7 @@ +{ + "Pinned %s files": "Arquivos %s fixados", + "Removed pin from %s files": "Arquivos %s não estão fixados", + "You started to help distribute %s.
Directory: %s": "Você começou a ajudar a distribuir %s.
Pasta: %s", + "Help distribute all new optional files on site %s": "Ajude a distribuir todos os novos arquivos opcionais no site %s", + "Yes, I want to help!": "Sim, eu quero ajudar!" +} diff --git a/plugins/OptionalManager/languages/zh-tw.json b/plugins/OptionalManager/languages/zh-tw.json new file mode 100644 index 00000000..dfa9eaf3 --- /dev/null +++ b/plugins/OptionalManager/languages/zh-tw.json @@ -0,0 +1,7 @@ +{ + "Pinned %s files": "已固定 %s 個檔", + "Removed pin from %s files": "已解除固定 %s 個檔", + "You started to help distribute %s.
Directory: %s": "你已經開始幫助分發 %s 。
目錄：%s", + "Help distribute all new optional files on site %s": "你想要幫助分發 %s 網站的所有檔嗎？", + "Yes, I want to help!": "是，我想要幫助！" +} diff --git a/plugins/OptionalManager/languages/zh.json b/plugins/OptionalManager/languages/zh.json new file mode 100644 index 00000000..ae18118e --- /dev/null +++ b/plugins/OptionalManager/languages/zh.json @@ -0,0 +1,7 @@ +{ + "Pinned %s files": "已固定 %s 个文件", + "Removed pin from %s files": "已解除固定 %s 个文件", + "You started to help distribute %s.
Directory: %s": "您已经开始帮助分发 %s 。
目录：%s", + "Help distribute all new optional files on site %s": "您想要帮助分发 %s 站点的所有文件吗？", + "Yes, I want to help!": "是，我想要帮助！" +} diff --git a/plugins/PeerDb/PeerDbPlugin.py b/plugins/PeerDb/PeerDbPlugin.py new file mode 100644 index 00000000..b4c8787b --- /dev/null +++ b/plugins/PeerDb/PeerDbPlugin.py @@ -0,0 +1,100 @@ +import time +import sqlite3 +import random +import atexit + +import gevent +from Plugin import PluginManager + + +@PluginManager.registerTo("ContentDb") +class ContentDbPlugin(object): + def __init__(self, *args, **kwargs): + atexit.register(self.saveAllPeers) + super(ContentDbPlugin, self).__init__(*args, **kwargs) + + def getSchema(self): + schema = super(ContentDbPlugin, self).getSchema() + + schema["tables"]["peer"] = { + "cols": [ + ["site_id", "INTEGER REFERENCES site (site_id) ON DELETE CASCADE"], + ["address", "TEXT NOT NULL"], + ["port", "INTEGER NOT NULL"], + ["hashfield", "BLOB"], + ["reputation", "INTEGER NOT NULL"], + ["time_added", "INTEGER NOT NULL"], + ["time_found", "INTEGER NOT NULL"] + ], + "indexes": [ + "CREATE UNIQUE INDEX peer_key ON peer (site_id, address, port)" + ], + "schema_changed": 2 + } + + return schema + + def loadPeers(self, site): + s = time.time() + site_id = self.site_ids.get(site.address) + res = self.execute("SELECT * FROM peer WHERE site_id = :site_id", {"site_id": site_id}) + num = 0 + num_hashfield = 0 + for row in res: + peer = site.addPeer(str(row["address"]), row["port"]) + if not peer: # Already exist + continue + if row["hashfield"]: + peer.hashfield.replaceFromBytes(row["hashfield"]) + num_hashfield += 1 + peer.time_added = row["time_added"] + peer.time_found = row["time_found"] + peer.reputation = row["reputation"] + if row["address"].endswith(".onion"): + peer.reputation = peer.reputation / 2 - 1 # Onion peers less likely working + num += 1 + if num_hashfield: + site.content_manager.has_optional_files = True + site.log.debug("%s peers (%s with hashfield) loaded in %.3fs" % (num, num_hashfield, time.time() - s)) + + def iteratePeers(self, site): + site_id = self.site_ids.get(site.address) + for key, peer in site.peers.items(): + address, port = key.rsplit(":", 1) + if peer.has_hashfield: + hashfield = sqlite3.Binary(peer.hashfield.tobytes()) + else: + hashfield = "" + yield (site_id, address, port, hashfield, peer.reputation, int(peer.time_added), int(peer.time_found)) + + def savePeers(self, site, spawn=False): + if spawn: + # Save peers every hour (+random some secs to not update very site at same time) + gevent.spawn_later(60 * 60 + random.randint(0, 60), self.savePeers, site, spawn=True) + if not site.peers: + site.log.debug("Peers not saved: No peers found") + return + s = time.time() + site_id = self.site_ids.get(site.address) + cur = self.getCursor() + try: + cur.execute("DELETE FROM peer WHERE site_id = :site_id", {"site_id": site_id}) + cur.cursor.executemany( + "INSERT INTO peer (site_id, address, port, hashfield, reputation, time_added, time_found) VALUES (?, ?, ?, ?, ?, ?, ?)", + self.iteratePeers(site) + ) + except Exception as err: + site.log.error("Save peer error: %s" % err) + site.log.debug("Peers saved in %.3fs" % (time.time() - s)) + + def initSite(self, site): + super(ContentDbPlugin, self).initSite(site) + gevent.spawn_later(0.5, self.loadPeers, site) + gevent.spawn_later(60*60, self.savePeers, site, spawn=True) + + def saveAllPeers(self): + for site in list(self.sites.values()): + try: + self.savePeers(site) + except Exception as err: + site.log.error("Save peer error: %s" % err) diff --git a/plugins/PeerDb/__init__.py b/plugins/PeerDb/__init__.py new file mode 100644 index 00000000..bc8c93b9 --- /dev/null +++ b/plugins/PeerDb/__init__.py @@ -0,0 +1,2 @@ +from . import PeerDbPlugin + diff --git a/plugins/Sidebar/SidebarPlugin.py b/plugins/Sidebar/SidebarPlugin.py new file mode 100644 index 00000000..45287d49 --- /dev/null +++ b/plugins/Sidebar/SidebarPlugin.py @@ -0,0 +1,794 @@ +import re +import os +import html +import sys +import math +import time +import json +import io +import urllib +import urllib.parse + +import gevent + +from Config import config +from Plugin import PluginManager +from Debug import Debug +from Translate import Translate +from util import helper +from .ZipStream import ZipStream + +plugin_dir = "plugins/Sidebar" +media_dir = plugin_dir + "/media" +sys.path.append(plugin_dir) # To able to load geoip lib + +loc_cache = {} +if "_" not in locals(): + _ = Translate(plugin_dir + "/languages/") + + +@PluginManager.registerTo("UiRequest") +class UiRequestPlugin(object): + # Inject our resources to end of original file streams + def actionUiMedia(self, path): + if path == "/uimedia/all.js" or path == "/uimedia/all.css": + # First yield the original file and header + body_generator = super(UiRequestPlugin, self).actionUiMedia(path) + for part in body_generator: + yield part + + # Append our media file to the end + ext = re.match(".*(js|css)$", path).group(1) + plugin_media_file = "%s/all.%s" % (media_dir, ext) + if config.debug: + # If debugging merge *.css to all.css and *.js to all.js + from Debug import DebugMedia + DebugMedia.merge(plugin_media_file) + if ext == "js": + yield _.translateData(open(plugin_media_file).read()).encode("utf8") + else: + for part in self.actionFile(plugin_media_file, send_header=False): + yield part + elif path.startswith("/uimedia/globe/"): # Serve WebGL globe files + file_name = re.match(".*/(.*)", path).group(1) + plugin_media_file = "%s_globe/%s" % (media_dir, file_name) + if config.debug and path.endswith("all.js"): + # If debugging merge *.css to all.css and *.js to all.js + from Debug import DebugMedia + DebugMedia.merge(plugin_media_file) + for part in self.actionFile(plugin_media_file): + yield part + else: + for part in super(UiRequestPlugin, self).actionUiMedia(path): + yield part + + def actionZip(self): + address = self.get["address"] + site = self.server.site_manager.get(address) + if not site: + return self.error404("Site not found") + + title = site.content_manager.contents.get("content.json", {}).get("title", "") + filename = "%s-backup-%s.zip" % (title, time.strftime("%Y-%m-%d_%H_%M")) + filename_quoted = urllib.parse.quote(filename) + self.sendHeader(content_type="application/zip", extra_headers={'Content-Disposition': 'attachment; filename="%s"' % filename_quoted}) + + return self.streamZip(site.storage.getPath(".")) + + def streamZip(self, dir_path): + zs = ZipStream(dir_path) + while 1: + data = zs.read() + if not data: + break + yield data + + +@PluginManager.registerTo("UiWebsocket") +class UiWebsocketPlugin(object): + def __init__(self, *args, **kwargs): + self.async_commands.add("sidebarGetPeers") + return super(UiWebsocketPlugin, self).__init__(*args, **kwargs) + + def sidebarRenderPeerStats(self, body, site): + connected = len([peer for peer in list(site.peers.values()) if peer.connection and peer.connection.connected]) + connectable = len([peer_id for peer_id in list(site.peers.keys()) if not peer_id.endswith(":0")]) + onion = len([peer_id for peer_id in list(site.peers.keys()) if ".onion" in peer_id]) + local = len([peer for peer in list(site.peers.values()) if helper.isPrivateIp(peer.ip)]) + peers_total = len(site.peers) + + # Add myself + if site.isServing(): + peers_total += 1 + if any(site.connection_server.port_opened.values()): + connectable += 1 + if site.connection_server.tor_manager.start_onions: + onion += 1 + + if peers_total: + percent_connected = float(connected) / peers_total + percent_connectable = float(connectable) / peers_total + percent_onion = float(onion) / peers_total + else: + percent_connectable = percent_connected = percent_onion = 0 + + if local: + local_html = _("
-
+
+
+
+
+
-
+
- {_[Connected]}:{connected} +
- {_[Connectable]}:{connectable} +
- {_[Onion]}:{onion} + {local_html} +
- {_[Total]}:{peers_total} +
-
+
+
+
-
+
- {_[Received]}:{recv:.2f}MB +
- {_[Sent]}:{sent:.2f}MB +
-
+ """))
+
+ extensions = (
+ ("html", "yellow"),
+ ("css", "orange"),
+ ("js", "purple"),
+ ("Image", "green"),
+ ("json", "darkblue"),
+ ("User data", "blue"),
+ ("Other", "white"),
+ ("Total", "black")
+ )
+ # Collect stats
+ size_filetypes = {}
+ size_total = 0
+ contents = site.content_manager.listContents() # Without user files
+ for inner_path in contents:
+ content = site.content_manager.contents[inner_path]
+ if "files" not in content or content["files"] is None:
+ continue
+ for file_name, file_details in list(content["files"].items()):
+ size_total += file_details["size"]
+ ext = file_name.split(".")[-1]
+ size_filetypes[ext] = size_filetypes.get(ext, 0) + file_details["size"]
+
+ # Get user file sizes
+ size_user_content = site.content_manager.contents.execute(
+ "SELECT SUM(size) + SUM(size_files) AS size FROM content WHERE ?",
+ {"not__inner_path": contents}
+ ).fetchone()["size"]
+ if not size_user_content:
+ size_user_content = 0
+ size_filetypes["User data"] = size_user_content
+ size_total += size_user_content
+
+ # The missing difference is content.json sizes
+ if "json" in size_filetypes:
+ size_filetypes["json"] += max(0, site.settings["size"] - size_total)
+ size_total = size_other = site.settings["size"]
+
+ # Bar
+ for extension, color in extensions:
+ if extension == "Total":
+ continue
+ if extension == "Other":
+ size = max(0, size_other)
+ elif extension == "Image":
+ size = size_filetypes.get("jpg", 0) + size_filetypes.get("png", 0) + size_filetypes.get("gif", 0)
+ size_other -= size
+ else:
+ size = size_filetypes.get(extension, 0)
+ size_other -= size
+ if size_total == 0:
+ percent = 0
+ else:
+ percent = 100 * (float(size) / size_total)
+ percent = math.floor(percent * 100) / 100 # Floor to 2 digits
+ body.append(
+ """""" %
+ (percent, _[extension], color, _[extension])
+ )
+
+ # Legend
+ body.append("
- ")
+ for extension, color in extensions:
+ if extension == "Other":
+ size = max(0, size_other)
+ elif extension == "Image":
+ size = size_filetypes.get("jpg", 0) + size_filetypes.get("png", 0) + size_filetypes.get("gif", 0)
+ elif extension == "Total":
+ size = size_total
+ else:
+ size = size_filetypes.get(extension, 0)
+
+ if extension == "js":
+ title = "javascript"
+ else:
+ title = extension
+
+ if size > 1024 * 1024 * 10: # Format as mB is more than 10mB
+ size_formatted = "%.0fMB" % (size / 1024 / 1024)
+ else:
+ size_formatted = "%.0fkB" % (size / 1024)
+
+ body.append("
- %s:%s " % (color, _[title], size_formatted)) + + body.append("
-
+
+
+
-
+
- {_[Downloaded]}:{size_formatted_downloaded:.2f}MB +
- {_[Total]}:{size_formatted_total:.2f}MB +
+
+ MB
+ {_[Set]}
+
+ """))
+ body.append("-
+ """))
+
+ i = 0
+ for bad_file, tries in site.bad_files.items():
+ i += 1
+ body.append(_("""
- {bad_filename} """, { + "bad_file_path": bad_file, + "bad_filename": helper.getFilename(bad_file), + "tries": _.pluralize(tries, "{} try", "{} tries") + })) + if i > 30: + break + + if len(site.bad_files) > 30: + num_bad_files = len(site.bad_files) - 30 + body.append(_("""
- {_[+ {num_bad_files} more]} """, nested=True)) + + body.append(""" +
+
+ {_[Reload]}
+ {_[Rebuild]}
+
+
+ {auth_address}
+ {_[Change]}
+
+ +
+ {site_address}
+ """))
+ if donate_key == False or donate_key == "":
+ pass
+ elif (type(donate_key) == str or type(donate_key) == str) and len(donate_key) > 0:
+ body.append(_("""
+
+ +
+ {donate_key}
+ """))
+ else:
+ body.append(_("""
+ {_[Donate]}
+ """))
+ body.append(_("""
+
+ {_[This is my site]}
+ + """)) + + def sidebarRenderOwnSettings(self, body, site): + title = site.content_manager.contents.get("content.json", {}).get("title", "") + description = site.content_manager.contents.get("content.json", {}).get("description", "") + + body.append(_(""" +
+
+ {_[Sign and publish]}
+ \u22EE
+
+ """))
+
+ contents = ["content.json"]
+ contents += list(site.content_manager.contents.get("content.json", {}).get("includes", {}).keys())
+ body.append(_("{_[Choose]}: "))
+ for content in contents:
+ body.append(_("{content} "))
+ body.append("
")
+ body.append("")
+ body.append("×")
+ body.append("
")
+
+ body.append("")
+
+ self.response(to, "".join(body))
+
+ def downloadGeoLiteDb(self, db_path):
+ import gzip
+ import shutil
+ from util import helper
+
+ if config.offline:
+ return False
+
+ self.log.info("Downloading GeoLite2 City database...")
+ self.cmd("progress", ["geolite-info", _["Downloading GeoLite2 City database (one time only, ~20MB)..."], 0])
+ db_urls = [
+ "https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz",
+ "https://raw.githubusercontent.com/texnikru/GeoLite2-Database/master/GeoLite2-City.mmdb.gz"
+ ]
+ for db_url in db_urls:
+ downloadl_err = None
+ try:
+ # Download
+ response = helper.httpRequest(db_url)
+ data_size = response.getheader('content-length')
+ data_recv = 0
+ data = io.BytesIO()
+ while True:
+ buff = response.read(1024 * 512)
+ if not buff:
+ break
+ data.write(buff)
+ data_recv += 1024 * 512
+ if data_size:
+ progress = int(float(data_recv) / int(data_size) * 100)
+ self.cmd("progress", ["geolite-info", _["Downloading GeoLite2 City database (one time only, ~20MB)..."], progress])
+ self.log.info("GeoLite2 City database downloaded (%s bytes), unpacking..." % data.tell())
+ data.seek(0)
+
+ # Unpack
+ with gzip.GzipFile(fileobj=data) as gzip_file:
+ shutil.copyfileobj(gzip_file, open(db_path, "wb"))
+
+ self.cmd("progress", ["geolite-info", _["GeoLite2 City database downloaded!"], 100])
+ time.sleep(2) # Wait for notify animation
+ self.log.info("GeoLite2 City database is ready at: %s" % db_path)
+ return True
+ except Exception as err:
+ download_err = err
+ self.log.error("Error downloading %s: %s" % (db_url, err))
+ pass
+ self.cmd("progress", [
+ "geolite-info",
+ _["GeoLite2 City database download error: {}!%s
" % html.escape(site.content_manager.contents.get("content.json", {}).get("title", ""), True)) + + body.append("") + + body.append("- ")
+
+ self.sidebarRenderPeerStats(body, site)
+ self.sidebarRenderTransferStats(body, site)
+ self.sidebarRenderFileStats(body, site)
+ self.sidebarRenderSizeLimit(body, site)
+ has_optional = self.sidebarRenderOptionalFileStats(body, site)
+ if has_optional:
+ self.sidebarRenderOptionalFileSettings(body, site)
+ self.sidebarRenderDbOptions(body, site)
+ self.sidebarRenderIdentity(body, site)
+ self.sidebarRenderControls(body, site)
+ if site.bad_files:
+ self.sidebarRenderBadFiles(body, site)
+
+ self.sidebarRenderOwnedCheckbox(body, site)
+ body.append("
")
+ self.sidebarRenderOwnSettings(body, site)
+ self.sidebarRenderContents(body, site)
+ body.append("
")
+ body.append("Please download manually and unpack to data dir:
{}"].format(download_err, db_urls[0]), + -100 + ]) + + def getLoc(self, geodb, ip): + global loc_cache + + if ip in loc_cache: + return loc_cache[ip] + else: + try: + loc_data = geodb.get(ip) + except: + loc_data = None + + if not loc_data or "location" not in loc_data: + loc_cache[ip] = None + return None + + loc = { + "lat": loc_data["location"]["latitude"], + "lon": loc_data["location"]["longitude"], + } + if "city" in loc_data: + loc["city"] = loc_data["city"]["names"]["en"] + + if "country" in loc_data: + loc["country"] = loc_data["country"]["names"]["en"] + + loc_cache[ip] = loc + return loc + + def getGeoipDb(self): + db_name = 'GeoLite2-City.mmdb' + + sys_db_paths = [] + if sys.platform == "linux": + sys_db_paths += ['/usr/share/GeoIP/' + db_name] + + data_dir_db_path = os.path.join(config.data_dir, db_name) + + db_paths = sys_db_paths + [data_dir_db_path] + + for path in db_paths: + if os.path.isfile(path) and os.path.getsize(path) > 0: + return path + + self.log.info("GeoIP database not found at [%s]. Downloading to: %s", + " ".join(db_paths), data_dir_db_path) + if self.downloadGeoLiteDb(data_dir_db_path): + return data_dir_db_path + return None + + def getPeerLocations(self, peers): + import maxminddb + + db_path = self.getGeoipDb() + if not db_path: + self.log.debug("Not showing peer locations: no GeoIP database") + return False + + self.log.info("Loading GeoIP database from: %s" % db_path) + geodb = maxminddb.open_database(db_path) + + peers = list(peers.values()) + # Place bars + peer_locations = [] + placed = {} # Already placed bars here + for peer in peers: + # Height of bar + if peer.connection and peer.connection.last_ping_delay: + ping = round(peer.connection.last_ping_delay * 1000) + else: + ping = None + loc = self.getLoc(geodb, peer.ip) + + if not loc: + continue + # Create position array + lat, lon = loc["lat"], loc["lon"] + latlon = "%s,%s" % (lat, lon) + if latlon in placed and helper.getIpType(peer.ip) == "ipv4": # Dont place more than 1 bar to same place, fake repos using ip address last two part + lat += float(128 - int(peer.ip.split(".")[-2])) / 50 + lon += float(128 - int(peer.ip.split(".")[-1])) / 50 + latlon = "%s,%s" % (lat, lon) + placed[latlon] = True + peer_location = {} + peer_location.update(loc) + peer_location["lat"] = lat + peer_location["lon"] = lon + peer_location["ping"] = ping + + peer_locations.append(peer_location) + + # Append myself + for ip in self.site.connection_server.ip_external_list: + my_loc = self.getLoc(geodb, ip) + if my_loc: + my_loc["ping"] = 0 + peer_locations.append(my_loc) + + return peer_locations + + + def actionSidebarGetPeers(self, to): + permissions = self.getPermissions(to) + if "ADMIN" not in permissions: + return self.response(to, "You don't have permission to run this command") + try: + peer_locations = self.getPeerLocations(self.site.peers) + globe_data = [] + ping_times = [ + peer_location["ping"] + for peer_location in peer_locations + if peer_location["ping"] + ] + if ping_times: + ping_avg = sum(ping_times) / float(len(ping_times)) + else: + ping_avg = 0 + + for peer_location in peer_locations: + if peer_location["ping"] == 0: # Me + height = -0.135 + elif peer_location["ping"]: + height = min(0.20, math.log(1 + peer_location["ping"] / ping_avg, 300)) + else: + height = -0.03 + + globe_data += [peer_location["lat"], peer_location["lon"], height] + + self.response(to, globe_data) + except Exception as err: + self.log.debug("sidebarGetPeers error: %s" % Debug.formatException(err)) + self.response(to, {"error": str(err)}) + + def actionSiteSetOwned(self, to, owned): + permissions = self.getPermissions(to) + if "ADMIN" not in permissions: + return self.response(to, "You don't have permission to run this command") + + if self.site.address == config.updatesite: + return self.response(to, "You can't change the ownership of the updater site") + + self.site.settings["own"] = bool(owned) + self.site.updateWebsocket(owned=owned) + + def actionUserSetSitePrivatekey(self, to, privatekey): + permissions = self.getPermissions(to) + if "ADMIN" not in permissions: + return self.response(to, "You don't have permission to run this command") + + site_data = self.user.sites[self.site.address] + site_data["privatekey"] = privatekey + self.site.updateWebsocket(set_privatekey=bool(privatekey)) + + return "ok" + + def actionSiteSetAutodownloadoptional(self, to, owned): + permissions = self.getPermissions(to) + if "ADMIN" not in permissions: + return self.response(to, "You don't have permission to run this command") + + self.site.settings["autodownloadoptional"] = bool(owned) + self.site.bad_files = {} + gevent.spawn(self.site.update, check_files=True) + self.site.worker_manager.removeSolvedFileTasks() + + def actionDbReload(self, to): + permissions = self.getPermissions(to) + if "ADMIN" not in permissions: + return self.response(to, "You don't have permission to run this command") + + self.site.storage.closeDb() + self.site.storage.getDb() + + return self.response(to, "ok") + + def actionDbRebuild(self, to): + permissions = self.getPermissions(to) + if "ADMIN" not in permissions: + return self.response(to, "You don't have permission to run this command") + + result = self.site.storage.rebuildDb() + + if not result: + return self.response(to, {"error": "Failed to rebuild database"}) + + return self.response(to, "ok") diff --git a/plugins/Sidebar/ZipStream.py b/plugins/Sidebar/ZipStream.py new file mode 100644 index 00000000..b6e05b21 --- /dev/null +++ b/plugins/Sidebar/ZipStream.py @@ -0,0 +1,59 @@ +import io +import os +import zipfile + +class ZipStream(object): + def __init__(self, dir_path): + self.dir_path = dir_path + self.pos = 0 + self.buff_pos = 0 + self.zf = zipfile.ZipFile(self, 'w', zipfile.ZIP_DEFLATED, allowZip64=True) + self.buff = io.BytesIO() + self.file_list = self.getFileList() + + def getFileList(self): + for root, dirs, files in os.walk(self.dir_path): + for file in files: + file_path = root + "/" + file + relative_path = os.path.join(os.path.relpath(root, self.dir_path), file) + yield file_path, relative_path + self.zf.close() + + def read(self, size=60 * 1024): + for file_path, relative_path in self.file_list: + self.zf.write(file_path, relative_path) + if self.buff.tell() >= size: + break + self.buff.seek(0) + back = self.buff.read() + self.buff.truncate(0) + self.buff.seek(0) + self.buff_pos += len(back) + return back + + def write(self, data): + self.pos += len(data) + self.buff.write(data) + + def tell(self): + return self.pos + + def seek(self, pos, whence=0): + if pos >= self.buff_pos: + self.buff.seek(pos - self.buff_pos, whence) + self.pos = pos + + def flush(self): + pass + + +if __name__ == "__main__": + zs = ZipStream(".") + out = open("out.zip", "wb") + while 1: + data = zs.read() + print("Write %s" % len(data)) + if not data: + break + out.write(data) + out.close() diff --git a/plugins/Sidebar/__init__.py b/plugins/Sidebar/__init__.py new file mode 100644 index 00000000..f2669d96 --- /dev/null +++ b/plugins/Sidebar/__init__.py @@ -0,0 +1 @@ +from . import SidebarPlugin \ No newline at end of file diff --git a/plugins/Sidebar/languages/da.json b/plugins/Sidebar/languages/da.json new file mode 100644 index 00000000..a421292c --- /dev/null +++ b/plugins/Sidebar/languages/da.json @@ -0,0 +1,81 @@ +{ + "Peers": "Klienter", + "Connected": "Forbundet", + "Connectable": "Mulige", + "Connectable peers": "Mulige klienter", + + "Data transfer": "Data overførsel", + "Received": "Modtaget", + "Received bytes": "Bytes modtaget", + "Sent": "Sendt", + "Sent bytes": "Bytes sendt", + + "Files": "Filer", + "Total": "I alt", + "Image": "Image", + "Other": "Andet", + "User data": "Bruger data", + + "Size limit": "Side max størrelse", + "limit used": "brugt", + "free space": "fri", + "Set": "Opdater", + + "Optional files": "Valgfri filer", + "Downloaded": "Downloadet", + "Download and help distribute all files": "Download og hjælp med at dele filer", + "Total size": "Størrelse i alt", + "Downloaded files": "Filer downloadet", + + "Database": "Database", + "search feeds": "søgninger", + "{feeds} query": "{feeds} søgninger", + "Reload": "Genindlæs", + "Rebuild": "Genopbyg", + "No database found": "Ingen database fundet", + + "Identity address": "Autorisations ID", + "Change": "Skift", + + "Update": "Opdater", + "Pause": "Pause", + "Resume": "Aktiv", + "Delete": "Slet", + "Are you sure?": "Er du sikker?", + + "Site address": "Side addresse", + "Donate": "Doner penge", + + "Missing files": "Manglende filer", + "{} try": "{} forsøg", + "{} tries": "{} forsøg", + "+ {num_bad_files} more": "+ {num_bad_files} mere", + + "This is my site": "Dette er min side", + "Site title": "Side navn", + "Site description": "Side beskrivelse", + "Save site settings": "Gem side opsætning", + + "Content publishing": "Indhold offentliggøres", + "Choose": "Vælg", + "Sign": "Signer", + "Publish": "Offentliggør", + + "This function is disabled on this proxy": "Denne funktion er slået fra på denne ZeroNet proxyEz a funkció ki van kapcsolva ezen a proxy-n", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "GeoLite2 City database kunne ikke downloades: {}!
Download venligst databasen manuelt og udpak i data folder:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "GeoLite2 város adatbázis letöltése (csak egyszer kell, kb 20MB)...", + "GeoLite2 City database downloaded!": "GeoLite2 City database downloadet!", + + "Are you sure?": "Er du sikker?", + "Site storage limit modified!": "Side max størrelse ændret!", + "Database schema reloaded!": "Database definition genindlæst!", + "Database rebuilding....": "Genopbygger database...", + "Database rebuilt!": "Database genopbygget!", + "Site updated!": "Side opdateret!", + "Delete this site": "Slet denne side", + "File write error: ": "Fejl ved skrivning af fil: ", + "Site settings saved!": "Side opsætning gemt!", + "Enter your private key:": "Indtast din private nøgle:", + " Signed!": " Signeret!", + "WebGL not supported": "WebGL er ikke supporteret" +} \ No newline at end of file diff --git a/plugins/Sidebar/languages/de.json b/plugins/Sidebar/languages/de.json new file mode 100644 index 00000000..2f5feacd --- /dev/null +++ b/plugins/Sidebar/languages/de.json @@ -0,0 +1,81 @@ +{ + "Peers": "Peers", + "Connected": "Verbunden", + "Connectable": "Verbindbar", + "Connectable peers": "Verbindbare Peers", + + "Data transfer": "Datei Transfer", + "Received": "Empfangen", + "Received bytes": "Empfangene Bytes", + "Sent": "Gesendet", + "Sent bytes": "Gesendete Bytes", + + "Files": "Dateien", + "Total": "Gesamt", + "Image": "Bilder", + "Other": "Sonstiges", + "User data": "Nutzer Daten", + + "Size limit": "Speicher Limit", + "limit used": "Limit benutzt", + "free space": "freier Speicher", + "Set": "Setzten", + + "Optional files": "Optionale Dateien", + "Downloaded": "Heruntergeladen", + "Download and help distribute all files": "Herunterladen und helfen alle Dateien zu verteilen", + "Total size": "Gesamte Größe", + "Downloaded files": "Heruntergeladene Dateien", + + "Database": "Datenbank", + "search feeds": "Feeds durchsuchen", + "{feeds} query": "{feeds} Abfrage", + "Reload": "Neu laden", + "Rebuild": "Neu bauen", + "No database found": "Keine Datenbank gefunden", + + "Identity address": "Identitäts Adresse", + "Change": "Ändern", + + "Update": "Aktualisieren", + "Pause": "Pausieren", + "Resume": "Fortsetzen", + "Delete": "Löschen", + "Are you sure?": "Bist du sicher?", + + "Site address": "Seiten Adresse", + "Donate": "Spenden", + + "Missing files": "Fehlende Dateien", + "{} try": "{} versuch", + "{} tries": "{} versuche", + "+ {num_bad_files} more": "+ {num_bad_files} mehr", + + "This is my site": "Das ist meine Seite", + "Site title": "Seiten Titel", + "Site description": "Seiten Beschreibung", + "Save site settings": "Einstellungen der Seite speichern", + + "Content publishing": "Inhaltsveröffentlichung", + "Choose": "Wähle", + "Sign": "Signieren", + "Publish": "Veröffentlichen", + + "This function is disabled on this proxy": "Diese Funktion ist auf dieser Proxy deaktiviert", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "GeoLite2 City Datenbank Download Fehler: {}!
Bitte manuell herunterladen und die Datei in das Datei Verzeichnis extrahieren:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "Herunterladen der GeoLite2 City Datenbank (einmalig, ~20MB)...", + "GeoLite2 City database downloaded!": "GeoLite2 City Datenbank heruntergeladen!", + + "Are you sure?": "Bist du sicher?", + "Site storage limit modified!": "Speicher Limit der Seite modifiziert!", + "Database schema reloaded!": "Datebank Schema neu geladen!", + "Database rebuilding....": "Datenbank neu bauen...", + "Database rebuilt!": "Datenbank neu gebaut!", + "Site updated!": "Seite aktualisiert!", + "Delete this site": "Diese Seite löschen", + "File write error: ": "Datei schreib fehler:", + "Site settings saved!": "Seiten Einstellungen gespeichert!", + "Enter your private key:": "Gib deinen privaten Schlüssel ein:", + " Signed!": " Signiert!", + "WebGL not supported": "WebGL nicht unterstützt" +} diff --git a/plugins/Sidebar/languages/es.json b/plugins/Sidebar/languages/es.json new file mode 100644 index 00000000..b9e98c46 --- /dev/null +++ b/plugins/Sidebar/languages/es.json @@ -0,0 +1,79 @@ +{ + "Peers": "Pares", + "Connected": "Conectados", + "Connectable": "Conectables", + "Connectable peers": "Pares conectables", + + "Data transfer": "Transferencia de datos", + "Received": "Recibidos", + "Received bytes": "Bytes recibidos", + "Sent": "Enviados", + "Sent bytes": "Bytes envidados", + + "Files": "Ficheros", + "Total": "Total", + "Image": "Imagen", + "Other": "Otro", + "User data": "Datos del usuario", + + "Size limit": "Límite de tamaño", + "limit used": "Límite utilizado", + "free space": "Espacio libre", + "Set": "Establecer", + + "Optional files": "Ficheros opcionales", + "Downloaded": "Descargado", + "Download and help distribute all files": "Descargar y ayudar a distribuir todos los ficheros", + "Total size": "Tamaño total", + "Downloaded files": "Ficheros descargados", + + "Database": "Base de datos", + "search feeds": "Fuentes de búsqueda", + "{feeds} query": "{feeds} consulta", + "Reload": "Recargar", + "Rebuild": "Reconstruir", + "No database found": "No se ha encontrado la base de datos", + + "Identity address": "Dirección de la identidad", + "Change": "Cambiar", + + "Update": "Actualizar", + "Pause": "Pausar", + "Resume": "Reanudar", + "Delete": "Borrar", + + "Site address": "Dirección del sitio", + "Donate": "Donar", + + "Missing files": "Ficheros perdidos", + "{} try": "{} intento", + "{} tries": "{} intentos", + "+ {num_bad_files} more": "+ {num_bad_files} más", + + "This is my site": "Este es mi sitio", + "Site title": "Título del sitio", + "Site description": "Descripción del sitio", + "Save site settings": "Guardar la configuración del sitio", + + "Content publishing": "Publicación del contenido", + "Choose": "Elegir", + "Sign": "Firmar", + "Publish": "Publicar", + "This function is disabled on this proxy": "Esta función está deshabilitada en este proxy", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "¡Error de la base de datos GeoLite2： {}!
Por favor, descárgalo manualmente y descomprime al directorio de datos：
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "Descargando la base de datos de GeoLite2 (una única vez, ~20MB）...", + "GeoLite2 City database downloaded!": "¡Base de datos de GeoLite2 descargada!", + + "Are you sure?": "¿Estás seguro?", + "Site storage limit modified!": "¡Límite de almacenamiento del sitio modificado!", + "Database schema reloaded!": "¡Esquema de la base de datos recargado!", + "Database rebuilding....": "Reconstruyendo la base de datos...", + "Database rebuilt!": "¡Base de datos reconstruida!", + "Site updated!": "¡Sitio actualizado!", + "Delete this site": "Borrar este sitio", + "File write error: ": "Error de escritura de fichero：", + "Site settings saved!": "¡Configuración del sitio guardada!", + "Enter your private key:": "Introduce tu clave privada：", + " Signed!": " ¡firmado!", + "WebGL not supported": "WebGL no está soportado" +} diff --git a/plugins/Sidebar/languages/fr.json b/plugins/Sidebar/languages/fr.json new file mode 100644 index 00000000..5c4b3ac7 --- /dev/null +++ b/plugins/Sidebar/languages/fr.json @@ -0,0 +1,82 @@ +{ + "Peers": "Pairs", + "Connected": "Connectés", + "Connectable": "Accessibles", + "Connectable peers": "Pairs accessibles", + + "Data transfer": "Données transférées", + "Received": "Reçues", + "Received bytes": "Bytes reçus", + "Sent": "Envoyées", + "Sent bytes": "Bytes envoyés", + + "Files": "Fichiers", + "Total": "Total", + "Image": "Image", + "Other": "Autre", + "User data": "Utilisateurs", + + "Size limit": "Taille maximale", + "limit used": "utlisé", + "free space": "libre", + "Set": "Modifier", + + "Optional files": "Fichiers optionnels", + "Downloaded": "Téléchargé", + "Download and help distribute all files": "Télécharger et distribuer tous les fichiers", + "Total size": "Taille totale", + "Downloaded files": "Fichiers téléchargés", + + "Database": "Base de données", + "search feeds": "recherche", + "{feeds} query": "{feeds} requête", + "Reload": "Recharger", + "Rebuild": "Reconstruire", + "No database found": "Aucune base de données trouvée", + + "Identity address": "Adresse d'identité", + "Change": "Modifier", + + "Site control": "Opérations", + "Update": "Mettre à jour", + "Pause": "Suspendre", + "Resume": "Reprendre", + "Delete": "Supprimer", + "Are you sure?": "Êtes-vous certain?", + + "Site address": "Adresse du site", + "Donate": "Faire un don", + + "Missing files": "Fichiers manquants", + "{} try": "{} essai", + "{} tries": "{} essais", + "+ {num_bad_files} more": "+ {num_bad_files} manquants", + + "This is my site": "Ce site m'appartient", + "Site title": "Nom du site", + "Site description": "Description du site", + "Save site settings": "Enregistrer les paramètres", + + "Content publishing": "Publication du contenu", + "Choose": "Sélectionner", + "Sign": "Signer", + "Publish": "Publier", + + "This function is disabled on this proxy": "Cette fonction est désactivé sur ce proxy", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "Erreur au téléchargement de la base de données GeoLite2: {}!
Téléchargez et décompressez dans le dossier data:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "Téléchargement de la base de données GeoLite2 (une seule fois, ~20MB)...", + "GeoLite2 City database downloaded!": "Base de données GeoLite2 téléchargée!", + + "Are you sure?": "Êtes-vous certain?", + "Site storage limit modified!": "Taille maximale modifiée!", + "Database schema reloaded!": "Base de données rechargée!", + "Database rebuilding....": "Reconstruction de la base de données...", + "Database rebuilt!": "Base de données reconstruite!", + "Site updated!": "Site mis à jour!", + "Delete this site": "Supprimer ce site", + "File write error: ": "Erreur à l'écriture du fichier: ", + "Site settings saved!": "Paramètres du site enregistrés!", + "Enter your private key:": "Entrez votre clé privée:", + " Signed!": " Signé!", + "WebGL not supported": "WebGL n'est pas supporté" +} diff --git a/plugins/Sidebar/languages/hu.json b/plugins/Sidebar/languages/hu.json new file mode 100644 index 00000000..40ed8fab --- /dev/null +++ b/plugins/Sidebar/languages/hu.json @@ -0,0 +1,82 @@ +{ + "Peers": "Csatlakozási pontok", + "Connected": "Csaltakozva", + "Connectable": "Csatlakozható", + "Connectable peers": "Csatlakozható peer-ek", + + "Data transfer": "Adatátvitel", + "Received": "Fogadott", + "Received bytes": "Fogadott byte-ok", + "Sent": "Küldött", + "Sent bytes": "Küldött byte-ok", + + "Files": "Fájlok", + "Total": "Összesen", + "Image": "Kép", + "Other": "Egyéb", + "User data": "Felh. adat", + + "Size limit": "Méret korlát", + "limit used": "felhasznált", + "free space": "szabad hely", + "Set": "Beállít", + + "Optional files": "Opcionális fájlok", + "Downloaded": "Letöltött", + "Download and help distribute all files": "Minden opcionális fájl letöltése", + "Total size": "Teljes méret", + "Downloaded files": "Letöltve", + + "Database": "Adatbázis", + "search feeds": "Keresés források", + "{feeds} query": "{feeds} lekérdezés", + "Reload": "Újratöltés", + "Rebuild": "Újraépítés", + "No database found": "Adatbázis nem található", + + "Identity address": "Azonosító cím", + "Change": "Módosít", + + "Site control": "Oldal műveletek", + "Update": "Frissít", + "Pause": "Szünteltet", + "Resume": "Folytat", + "Delete": "Töröl", + "Are you sure?": "Biztos vagy benne?", + + "Site address": "Oldal címe", + "Donate": "Támogatás", + + "Missing files": "Hiányzó fájlok", + "{} try": "{} próbálkozás", + "{} tries": "{} próbálkozás", + "+ {num_bad_files} more": "+ még {num_bad_files} darab", + + "This is my site": "Ez az én oldalam", + "Site title": "Oldal neve", + "Site description": "Oldal leírása", + "Save site settings": "Oldal beállítások mentése", + + "Content publishing": "Tartalom publikálás", + "Choose": "Válassz", + "Sign": "Aláírás", + "Publish": "Publikálás", + + "This function is disabled on this proxy": "Ez a funkció ki van kapcsolva ezen a proxy-n", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "GeoLite2 város adatbázis letöltési hiba: {}!
A térképhez töltsd le és csomagold ki a data könyvtárba:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "GeoLite2 város adatbázis letöltése (csak egyszer kell, kb 20MB)...", + "GeoLite2 City database downloaded!": "GeoLite2 város adatbázis letöltve!", + + "Are you sure?": "Biztos vagy benne?", + "Site storage limit modified!": "Az oldalt méret korlát módosítva!", + "Database schema reloaded!": "Adatbázis séma újratöltve!", + "Database rebuilding....": "Adatbázis újraépítés...", + "Database rebuilt!": "Adatbázis újraépítve!", + "Site updated!": "Az oldal frissítve!", + "Delete this site": "Az oldal törlése", + "File write error: ": "Fájl írási hiba: ", + "Site settings saved!": "Az oldal beállításai elmentve!", + "Enter your private key:": "Add meg a prviát kulcsod:", + " Signed!": " Aláírva!", + "WebGL not supported": "WebGL nem támogatott" +} \ No newline at end of file diff --git a/plugins/Sidebar/languages/it.json b/plugins/Sidebar/languages/it.json new file mode 100644 index 00000000..6aa0969a --- /dev/null +++ b/plugins/Sidebar/languages/it.json @@ -0,0 +1,81 @@ +{ + "Peers": "Peer", + "Connected": "Connessi", + "Connectable": "Collegabili", + "Connectable peers": "Peer collegabili", + + "Data transfer": "Trasferimento dati", + "Received": "Ricevuti", + "Received bytes": "Byte ricevuti", + "Sent": "Inviati", + "Sent bytes": "Byte inviati", + + "Files": "File", + "Total": "Totale", + "Image": "Imagine", + "Other": "Altro", + "User data": "Dati utente", + + "Size limit": "Limite dimensione", + "limit used": "limite usato", + "free space": "spazio libero", + "Set": "Imposta", + + "Optional files": "File facoltativi", + "Downloaded": "Scaricati", + "Download and help distribute all files": "Scarica e aiuta a distribuire tutti i file", + "Total size": "Dimensione totale", + "Downloaded files": "File scaricati", + + "Database": "Database", + "search feeds": "ricerca di feed", + "{feeds} query": "{feeds} interrogazione", + "Reload": "Ricaricare", + "Rebuild": "Ricostruire", + "No database found": "Nessun database trovato", + + "Identity address": "Indirizzo di identità", + "Change": "Cambia", + + "Update": "Aggiorna", + "Pause": "Sospendi", + "Resume": "Riprendi", + "Delete": "Cancella", + "Are you sure?": "Sei sicuro?", + + "Site address": "Indirizzo sito", + "Donate": "Dona", + + "Missing files": "File mancanti", + "{} try": "{} tenta", + "{} tries": "{} prova", + "+ {num_bad_files} more": "+ {num_bad_files} altri", + + "This is my site": "Questo è il mio sito", + "Site title": "Titolo sito", + "Site description": "Descrizione sito", + "Save site settings": "Salva impostazioni sito", + + "Content publishing": "Pubblicazione contenuto", + "Choose": "Scegli", + "Sign": "Firma", + "Publish": "Pubblica", + + "This function is disabled on this proxy": "Questa funzione è disabilitata su questo proxy", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "Errore scaricamento database GeoLite2 City: {}!
Si prega di scaricarlo manualmente e spacchetarlo nella cartella dir:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "Scaricamento database GeoLite2 City (solo una volta, ~20MB)...", + "GeoLite2 City database downloaded!": "Database GeoLite2 City scaricato!", + + "Are you sure?": "Sei sicuro?", + "Site storage limit modified!": "Limite di archiviazione del sito modificato!", + "Database schema reloaded!": "Schema database ricaricato!", + "Database rebuilding....": "Ricostruzione database...", + "Database rebuilt!": "Database ricostruito!", + "Site updated!": "Sito aggiornato!", + "Delete this site": "Cancella questo sito", + "File write error: ": "Errore scrittura file:", + "Site settings saved!": "Impostazioni sito salvate!", + "Enter your private key:": "Inserisci la tua chiave privata:", + " Signed!": " Firmato!", + "WebGL not supported": "WebGL non supportato" +} diff --git a/plugins/Sidebar/languages/jp.json b/plugins/Sidebar/languages/jp.json new file mode 100644 index 00000000..99b34564 --- /dev/null +++ b/plugins/Sidebar/languages/jp.json @@ -0,0 +1,82 @@ +{ + "Peers": "ピア", + "Connected": "接続済み", + "Connectable": "利用可能", + "Connectable peers": "ピアに接続可能", + + "Data transfer": "データ転送", + "Received": "受信", + "Received bytes": "受信バイト数", + "Sent": "送信", + "Sent bytes": "送信バイト数", + + "Files": "ファイル", + "Total": "合計", + "Image": "画像", + "Other": "その他", + "User data": "ユーザーデータ", + + "Size limit": "サイズ制限", + "limit used": "使用上限", + "free space": "フリースペース", + "Set": "セット", + + "Optional files": "オプション ファイル", + "Downloaded": "ダウンロード済み", + "Download and help distribute all files": "ダウンロードしてすべてのファイルの配布を支援する", + "Total size": "合計サイズ", + "Downloaded files": "ダウンロードされたファイル", + + "Database": "データベース", + "search feeds": "フィードを検索する", + "{feeds} query": "{フィード} お問い合わせ", + "Reload": "再読込", + "Rebuild": "再ビルド", + "No database found": "データベースが見つかりません", + + "Identity address": "Identity address", + "Change": "編集", + + "Site control": "サイト管理", + "Update": "更新", + "Pause": "一時停止", + "Resume": "再開", + "Delete": "削除", + "Are you sure?": "本当によろしいですか？", + + "Site address": "サイトアドレス", + "Donate": "寄付する", + + "Missing files": "ファイルがありません", + "{} try": "{} 試す", + "{} tries": "{} 試行", + "+ {num_bad_files} more": "+ {num_bad_files} more", + + "This is my site": "This is my site", + "Site title": "サイトタイトル", + "Site description": "サイトの説明", + "Save site settings": "サイトの設定を保存する", + + "Content publishing": "コンテンツを公開する", + "Choose": "選択", + "Sign": "Sign", + "Publish": "公開する", + + "This function is disabled on this proxy": "この機能はこのプロキシで無効になっています", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "GeoLite2 Cityデータベースのダウンロードエラー: {}!
手動でダウンロードして、フォルダに解凍してください。:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "GeoLite2 Cityデータベースの読み込み (これは一度だけ行われます, ~20MB)...", + "GeoLite2 City database downloaded!": "GeoLite2 Cityデータベースがダウンロードされました！", + + "Are you sure?": "本当によろしいですか？", + "Site storage limit modified!": "サイトの保存容量の制限が変更されました！", + "Database schema reloaded!": "データベーススキーマがリロードされました！", + "Database rebuilding....": "データベースの再構築中....", + "Database rebuilt!": "データベースが再構築されました！", + "Site updated!": "サイトが更新されました！", + "Delete this site": "このサイトを削除する", + "File write error: ": "ファイル書き込みエラー:", + "Site settings saved!": "サイト設定が保存されました！", + "Enter your private key:": "秘密鍵を入力してください:", + " Signed!": " Signed!", + "WebGL not supported": "WebGLはサポートされていません" +} diff --git a/plugins/Sidebar/languages/pl.json b/plugins/Sidebar/languages/pl.json new file mode 100644 index 00000000..93268507 --- /dev/null +++ b/plugins/Sidebar/languages/pl.json @@ -0,0 +1,82 @@ +{ + "Peers": "Użytkownicy równorzędni", + "Connected": "Połączony", + "Connectable": "Możliwy do podłączenia", + "Connectable peers": "Połączeni użytkownicy równorzędni", + + "Data transfer": "Transfer danych", + "Received": "Odebrane", + "Received bytes": "Odebrany bajty", + "Sent": "Wysłane", + "Sent bytes": "Wysłane bajty", + + "Files": "Pliki", + "Total": "Sumarycznie", + "Image": "Obraz", + "Other": "Inne", + "User data": "Dane użytkownika", + + "Size limit": "Rozmiar limitu", + "limit used": "zużyty limit", + "free space": "wolna przestrzeń", + "Set": "Ustaw", + + "Optional files": "Pliki opcjonalne", + "Downloaded": "Ściągnięte", + "Download and help distribute all files": "Ściągnij i pomóż rozpowszechniać wszystkie pliki", + "Total size": "Rozmiar sumaryczny", + "Downloaded files": "Ściągnięte pliki", + + "Database": "Baza danych", + "search feeds": "przeszukaj zasoby", + "{feeds} query": "{feeds} pytanie", + "Reload": "Odśwież", + "Rebuild": "Odbuduj", + "No database found": "Nie odnaleziono bazy danych", + + "Identity address": "Adres identyfikacyjny", + "Change": "Zmień", + + "Site control": "Kontrola strony", + "Update": "Zaktualizuj", + "Pause": "Wstrzymaj", + "Resume": "Wznów", + "Delete": "Skasuj", + "Are you sure?": "Jesteś pewien?", + + "Site address": "Adres strony", + "Donate": "Wspomóż", + + "Missing files": "Brakujące pliki", + "{} try": "{} próba", + "{} tries": "{} próby", + "+ {num_bad_files} more": "+ {num_bad_files} więcej", + + "This is my site": "To moja strona", + "Site title": "Tytuł strony", + "Site description": "Opis strony", + "Save site settings": "Zapisz ustawienia strony", + + "Content publishing": "Publikowanie treści", + "Choose": "Wybierz", + "Sign": "Podpisz", + "Publish": "Opublikuj", + + "This function is disabled on this proxy": "Ta funkcja jest zablokowana w tym proxy", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "Błąd ściągania bazy danych GeoLite2 City: {}!
Proszę ściągnąć ją recznie i wypakować do katalogu danych:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "Ściąganie bazy danych GeoLite2 City (tylko jednorazowo, ok. 20MB)...", + "GeoLite2 City database downloaded!": "Baza danych GeoLite2 City ściagnięta!", + + "Are you sure?": "Jesteś pewien?", + "Site storage limit modified!": "Limit pamięci strony zmodyfikowany!", + "Database schema reloaded!": "Schemat bazy danych załadowany ponownie!", + "Database rebuilding....": "Przebudowywanie bazy danych...", + "Database rebuilt!": "Baza danych przebudowana!", + "Site updated!": "Strona zaktualizowana!", + "Delete this site": "Usuń tę stronę", + "File write error: ": "Błąd zapisu pliku: ", + "Site settings saved!": "Ustawienia strony zapisane!", + "Enter your private key:": "Wpisz swój prywatny klucz:", + " Signed!": " Podpisane!", + "WebGL not supported": "WebGL nie jest obsługiwany" +} diff --git a/plugins/Sidebar/languages/pt-br.json b/plugins/Sidebar/languages/pt-br.json new file mode 100644 index 00000000..44fe06a2 --- /dev/null +++ b/plugins/Sidebar/languages/pt-br.json @@ -0,0 +1,97 @@ +{ + "Copy to clipboard": "Copiar para área de transferência (clipboard)", + "Peers": "Peers", + "Connected": "Ligados", + "Connectable": "Disponíveis", + "Onion": "Onion", + "Local": "Locais", + "Connectable peers": "Peers disponíveis", + + "Data transfer": "Transferência de dados", + "Received": "Recebidos", + "Received bytes": "Bytes recebidos", + "Sent": "Enviados", + "Sent bytes": "Bytes enviados", + + "Files": "Arquivos", + "Save as .zip": "Salvar como .zip", + "Total": "Total", + "Image": "Imagem", + "Other": "Outros", + "User data": "Dados do usuário", + + "Size limit": "Limite de tamanho", + "limit used": "limite utilizado", + "free space": "espaço livre", + "Set": "Definir", + + "Optional files": "Arquivos opcionais", + "Downloaded": "Baixados", + "Download and help distribute all files": "Baixar e ajudar a distribuir todos os arquivos", + "Total size": "Tamanho total", + "Downloaded files": "Arquivos baixados", + + "Database": "Banco de dados", + "search feeds": "pesquisar feeds", + "{feeds} query": "consulta de {feeds}", + "Reload": "Recarregar", + "Rebuild": "Reconstruir", + "No database found": "Base de dados não encontrada", + + "Identity address": "Endereço de identidade", + "Change": "Alterar", + + "Site control": "Controle do site", + "Update": "Atualizar", + "Pause": "Suspender", + "Resume": "Continuar", + "Delete": "Remover", + "Are you sure?": "Tem certeza?", + + "Site address": "Endereço do site", + "Donate": "Doar", + + "Needs to be updated": "Necessitam ser atualizados", + "{} try": "{} tentativa", + "{} tries": "{} tentativas", + "+ {num_bad_files} more": "+ {num_bad_files} adicionais", + + "This is my site": "Este é o meu site", + "Site title": "Título do site", + "Site description": "Descrição do site", + "Save site settings": "Salvar definições do site", + "Open site directory": "Abrir diretório do site", + + "Content publishing": "内容发布", + "Content publishing": "Publicação do conteúdo", + "Choose": "Escolher", + "Sign": "Assinar", + "Publish": "Publicar", + "Sign and publish": "Assinar e publicar", + "add saved private key": "adicionar privatekey (chave privada) para salvar", + "Private key saved for site signing": "Privatekey foi salva para assinar o site", + "Private key saved.": "Privatekey salva.", + "forgot": "esquecer", + "Saved private key removed": "Privatekey salva foi removida", + "This function is disabled on this proxy": "Esta função encontra-se desativada neste proxy", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "Erro ao baixar a base de dados GeoLite2 City: {}!
Por favor baixe manualmente e descompacte os dados para a seguinte pasta:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "Baixando a base de dados GeoLite2 City (uma única vez, ~20MB)...", + "GeoLite2 City database downloaded!": "A base de dados GeoLite2 City foi baixada!", + + "Are you sure?": "Tem certeza?", + "Site storage limit modified!": "O limite de armazenamento do site foi modificado!", + "Database schema reloaded!": "O esquema da base de dados foi atualizado!", + "Database rebuilding....": "Reconstruindo base de dados...", + "Database rebuilt!": "Base de dados reconstruída!", + "Site updated!": "Site atualizado!", + "Delete this site": "Remover este site", + "Blacklist": "Blacklist", + "Blacklist this site": "Blacklistar este site", + "Reason": "Motivo", + "Delete and Blacklist": "Deletar e blacklistar", + "File write error: ": "Erro de escrita de arquivo: ", + "Site settings saved!": "Definições do site salvas!", + "Enter your private key:": "Digite sua chave privada:", + " Signed!": " Assinado!", + "WebGL not supported": "WebGL não é suportado" +} diff --git a/plugins/Sidebar/languages/ru.json b/plugins/Sidebar/languages/ru.json new file mode 100644 index 00000000..f2eeca04 --- /dev/null +++ b/plugins/Sidebar/languages/ru.json @@ -0,0 +1,82 @@ +{ + "Peers": "Пиры", + "Connected": "Подключенные", + "Connectable": "Доступные", + "Connectable peers": "Пиры доступны для подключения", + + "Data transfer": "Передача данных", + "Received": "Получено", + "Received bytes": "Получено байн", + "Sent": "Отправлено", + "Sent bytes": "Отправлено байт", + + "Files": "Файлы", + "Total": "Всего", + "Image": "Изображений", + "Other": "Другое", + "User data": "Ваш контент", + + "Size limit": "Ограничение по размеру", + "limit used": "Использовано", + "free space": "Доступно", + "Set": "Установить", + + "Optional files": "Опциональные файлы", + "Downloaded": "Загружено", + "Download and help distribute all files": "Загрузить опциональные файлы для помощи сайту", + "Total size": "Объём", + "Downloaded files": "Загруженные файлы", + + "Database": "База данных", + "search feeds": "поиск подписок", + "{feeds} query": "{feeds} запрос", + "Reload": "Перезагрузить", + "Rebuild": "Перестроить", + "No database found": "База данных не найдена", + + "Identity address": "Уникальный адрес", + "Change": "Изменить", + + "Site control": "Управление сайтом", + "Update": "Обновить", + "Pause": "Пауза", + "Resume": "Продолжить", + "Delete": "Удалить", + "Are you sure?": "Вы уверены?", + + "Site address": "Адрес сайта", + "Donate": "Пожертвовать", + + "Missing files": "Отсутствующие файлы", + "{} try": "{} попробовать", + "{} tries": "{} попыток", + "+ {num_bad_files} more": "+ {num_bad_files} ещё", + + "This is my site": "Это мой сайт", + "Site title": "Название сайта", + "Site description": "Описание сайта", + "Save site settings": "Сохранить настройки сайта", + + "Content publishing": "Публикация контента", + "Choose": "Выбрать", + "Sign": "Подписать", + "Publish": "Опубликовать", + + "This function is disabled on this proxy": "Эта функция отключена на этом прокси", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "Ошибка загрузки базы городов GeoLite2: {}!
Пожалуйста, загрузите её вручную и распакуйте в папку:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "Загрузка базы городов GeoLite2 (это делается только 1 раз, ~20MB)...", + "GeoLite2 City database downloaded!": "База GeoLite2 успешно загружена!", + + "Are you sure?": "Вы уверены?", + "Site storage limit modified!": "Лимит хранилища для сайта изменен!", + "Database schema reloaded!": "Схема базы данных перезагружена!", + "Database rebuilding....": "Перестройка базы данных...", + "Database rebuilt!": "База данных перестроена!", + "Site updated!": "Сайт обновлён!", + "Delete this site": "Удалить этот сайт", + "File write error: ": "Ошибка записи файла:", + "Site settings saved!": "Настройки сайта сохранены!", + "Enter your private key:": "Введите свой приватный ключ:", + " Signed!": " Подписано!", + "WebGL not supported": "WebGL не поддерживается" +} diff --git a/plugins/Sidebar/languages/tr.json b/plugins/Sidebar/languages/tr.json new file mode 100644 index 00000000..88fcd6e0 --- /dev/null +++ b/plugins/Sidebar/languages/tr.json @@ -0,0 +1,82 @@ +{ + "Peers": "Eşler", + "Connected": "Bağlı", + "Connectable": "Erişilebilir", + "Connectable peers": "Bağlanılabilir eşler", + + "Data transfer": "Veri aktarımı", + "Received": "Alınan", + "Received bytes": "Bayt alındı", + "Sent": "Gönderilen", + "Sent bytes": "Bayt gönderildi", + + "Files": "Dosyalar", + "Total": "Toplam", + "Image": "Resim", + "Other": "Diğer", + "User data": "Kullanıcı verisi", + + "Size limit": "Boyut sınırı", + "limit used": "kullanılan", + "free space": "boş", + "Set": "Ayarla", + + "Optional files": "İsteğe bağlı dosyalar", + "Downloaded": "İndirilen", + "Download and help distribute all files": "Tüm dosyaları indir ve yayılmalarına yardım et", + "Total size": "Toplam boyut", + "Downloaded files": "İndirilen dosyalar", + + "Database": "Veritabanı", + "search feeds": "kaynak ara", + "{feeds} query": "{feeds} sorgu", + "Reload": "Yenile", + "Rebuild": "Yapılandır", + "No database found": "Veritabanı yok", + + "Identity address": "Kimlik adresi", + "Change": "Değiştir", + + "Site control": "Site kontrolü", + "Update": "Güncelle", + "Pause": "Duraklat", + "Resume": "Sürdür", + "Delete": "Sil", + "Are you sure?": "Emin misin?", + + "Site address": "Site adresi", + "Donate": "Bağış yap", + + "Missing files": "Eksik dosyalar", + "{} try": "{} deneme", + "{} tries": "{} deneme", + "+ {num_bad_files} more": "+ {num_bad_files} tane daha", + + "This is my site": "Bu benim sitem", + "Site title": "Site başlığı", + "Site description": "Site açıklaması", + "Save site settings": "Site ayarlarını kaydet", + + "Content publishing": "İçerik yayımlanıyor", + "Choose": "Seç", + "Sign": "İmzala", + "Publish": "Yayımla", + + "This function is disabled on this proxy": "Bu özellik bu vekilde kullanılamaz", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "GeoLite2 Şehir veritabanı indirme hatası: {}!
Lütfen kendiniz indirip aşağıdaki konuma açınınız:
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "GeoLite2 Şehir veritabanı indiriliyor (sadece bir kere, ~20MB)...", + "GeoLite2 City database downloaded!": "GeoLite2 Şehir veritabanı indirildi!", + + "Are you sure?": "Emin misiniz?", + "Site storage limit modified!": "Site saklama sınırı değiştirildi!", + "Database schema reloaded!": "Veritabanı şeması yeniden yüklendi!", + "Database rebuilding....": "Veritabanı yeniden inşa ediliyor...", + "Database rebuilt!": "Veritabanı yeniden inşa edildi!", + "Site updated!": "Site güncellendi!", + "Delete this site": "Bu siteyi sil", + "File write error: ": "Dosya yazma hatası: ", + "Site settings saved!": "Site ayarları kaydedildi!", + "Enter your private key:": "Özel anahtarınızı giriniz:", + " Signed!": " İmzala!", + "WebGL not supported": "WebGL desteklenmiyor" +} diff --git a/plugins/Sidebar/languages/zh-tw.json b/plugins/Sidebar/languages/zh-tw.json new file mode 100644 index 00000000..9d4ea1be --- /dev/null +++ b/plugins/Sidebar/languages/zh-tw.json @@ -0,0 +1,83 @@ +{ + "Peers": "節點數", + "Connected": "已連線", + "Connectable": "可連線", + "Connectable peers": "可連線節點", + + "Data transfer": "數據傳輸", + "Received": "已接收", + "Received bytes": "已接收位元組", + "Sent": "已傳送", + "Sent bytes": "已傳送位元組", + + "Files": "檔案", + "Total": "共計", + "Image": "圖片", + "Other": "其他", + "User data": "使用者數據", + + "Size limit": "大小限制", + "limit used": "已使用", + "free space": "可用空間", + "Set": "偏好設定", + + "Optional files": "可選檔案", + "Downloaded": "已下載", + "Download and help distribute all files": "下載並幫助分發所有檔案", + "Total size": "總大小", + "Downloaded files": "下載的檔案", + + "Database": "資料庫", + "search feeds": "搜尋供稿", + "{feeds} query": "{feeds} 查詢 ", + "Reload": "重新整理", + "Rebuild": "重建", + "No database found": "未找到資料庫", + + "Identity address": "身分位址", + "Change": "變更", + + "Site control": "網站控制", + "Update": "更新", + "Pause": "暫停", + "Resume": "恢復", + "Delete": "刪除", + "Are you sure?": "你確定？", + + "Site address": "網站位址", + "Donate": "捐贈", + + "Missing files": "缺少的檔案", + "{} try": "{} 嘗試", + "{} tries": "{} 已嘗試", + "+ {num_bad_files} more": "+ {num_bad_files} 更多", + + "This is my site": "這是我的網站", + "Site title": "網站標題", + "Site description": "網站描述", + "Save site settings": "存儲網站設定", + "Open site directory": "打開所在資料夾", + + "Content publishing": "內容發布", + "Choose": "選擇", + "Sign": "簽署", + "Publish": "發布", + "Sign and publish": "簽名並發布", + "This function is disabled on this proxy": "此代理上禁用此功能", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "GeoLite2 地理位置資料庫下載錯誤：{}!
請手動下載並解壓到數據目錄：
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "正在下載 GeoLite2 地理位置資料庫 （僅一次，約 20MB ）...", + "GeoLite2 City database downloaded!": "GeoLite2 地理位置資料庫已下載！", + + "Are you sure?": "你確定？", + "Site storage limit modified!": "網站存儲限制已變更！", + "Database schema reloaded!": "資料庫架構重新加載！", + "Database rebuilding....": "資料庫重建中...", + "Database rebuilt!": "資料庫已重建！", + "Site updated!": "網站已更新！", + "Delete this site": "刪除此網站", + "File write error: ": "檔案寫入錯誤：", + "Site settings saved!": "網站設置已保存！", + "Enter your private key:": "輸入您的私鑰：", + " Signed!": " 已簽署！", + "WebGL not supported": "不支援 WebGL" +} diff --git a/plugins/Sidebar/languages/zh.json b/plugins/Sidebar/languages/zh.json new file mode 100644 index 00000000..696084cf --- /dev/null +++ b/plugins/Sidebar/languages/zh.json @@ -0,0 +1,98 @@ +{ + "Copy to clipboard": "复制到剪切板", + "Peers": "节点数", + "Connected": "已连接", + "Connectable": "可连接", + "Onion": "洋葱点", + "Local": "局域网", + "Connectable peers": "可连接节点", + + "Data transfer": "数据传输", + "Received": "已接收", + "Received bytes": "已接收字节", + "Sent": "已发送", + "Sent bytes": "已发送字节", + + "Files": "文件", + "Save as .zip": "打包成zip文件", + "Total": "总计", + "Image": "图像", + "Other": "其他", + "User data": "用户数据", + + "Size limit": "大小限制", + "limit used": "限额", + "free space": "剩余空间", + "Set": "设置", + + "Optional files": "可选文件", + "Downloaded": "已下载", + "Download and help distribute all files": "下载并帮助分发所有文件", + "Auto download big file size limit": "自动下载大文件大小限制", + "Total size": "总大小", + "Downloaded files": "已下载文件", + + "Database": "数据库", + "search feeds": "搜索数据源", + "{feeds} query": "{feeds} 请求", + "Reload": "重载", + "Rebuild": "重建", + "No database found": "没有找到数据库", + + "Identity address": "身份地址", + "Change": "更改", + + "Site control": "站点控制", + "Update": "更新", + "Pause": "暂停", + "Resume": "恢复", + "Delete": "删除", + "Are you sure?": "您确定吗？", + + "Site address": "站点地址", + "Donate": "捐赠", + + "Needs to be updated": "需要更新", + "{} try": "{} 尝试", + "{} tries": "{} 已尝试", + "+ {num_bad_files} more": "+ {num_bad_files} 更多", + + "This is my site": "这是我的站点", + "Site title": "站点标题", + "Site description": "站点描述", + "Save site settings": "保存站点设置", + "Open site directory": "打开所在文件夹", + + "Content publishing": "内容发布", + "Add saved private key": "添加并保存私钥", + "Save": "保存", + "Private key saved.": "私钥已保存", + "Private key saved for site signing": "已保存用于站点签名的私钥", + "Forgot": "删除私钥", + "Saved private key removed": "保存的私钥已删除", + "Choose": "选择", + "Sign": "签名", + "Publish": "发布", + "Sign and publish": "签名并发布", + "This function is disabled on this proxy": "此功能在代理上被禁用", + "GeoLite2 City database download error: {}!
Please download manually and unpack to data dir:
{}": "GeoLite2 地理位置数据库下载错误：{}!
请手动下载并解压在数据目录：
{}", + "Downloading GeoLite2 City database (one time only, ~20MB)...": "正在下载 GeoLite2 地理位置数据库 （仅需一次，约 20MB ）...", + "GeoLite2 City database downloaded!": "GeoLite2 地理位置数据库已下载！", + + "Are you sure?": "您确定吗？", + "Site storage limit modified!": "站点存储限制已更改！", + "Database schema reloaded!": "数据库模式已重新加载！", + "Database rebuilding....": "数据库重建中...", + "Database rebuilt!": "数据库已重建！", + "Site updated!": "站点已更新！", + "Delete this site": "删除此站点", + "Blacklist": "黑名单", + "Blacklist this site": "拉黑此站点", + "Reason": "原因", + "Delete and Blacklist": "删除并拉黑", + "File write error: ": "文件写入错误：", + "Site settings saved!": "站点设置已保存！", + "Enter your private key:": "输入您的私钥：", + " Signed!": " 已签名！", + "WebGL not supported": "不支持 WebGL" +} diff --git a/plugins/Sidebar/media/Class.coffee b/plugins/Sidebar/media/Class.coffee new file mode 100644 index 00000000..d62ab25c --- /dev/null +++ b/plugins/Sidebar/media/Class.coffee @@ -0,0 +1,23 @@ +class Class + trace: true + + log: (args...) -> + return unless @trace + return if typeof console is 'undefined' + args.unshift("[#{@.constructor.name}]") + console.log(args...) + @ + + logStart: (name, args...) -> + return unless @trace + @logtimers or= {} + @logtimers[name] = +(new Date) + @log "#{name}", args..., "(started)" if args.length > 0 + @ + + logEnd: (name, args...) -> + ms = +(new Date)-@logtimers[name] + @log "#{name}", args..., "(Done in #{ms}ms)" + @ + +window.Class = Class \ No newline at end of file diff --git a/plugins/Sidebar/media/Internals.coffee b/plugins/Sidebar/media/Internals.coffee new file mode 100644 index 00000000..484ecdb7 --- /dev/null +++ b/plugins/Sidebar/media/Internals.coffee @@ -0,0 +1,60 @@ +class Internals extends Class + constructor: (@sidebar) -> + @tag = null + @opened = false + if window.top.location.hash == "#internals" + setTimeout (=> @open()), 10 + + createHtmltag: -> + @when_loaded = $.Deferred() + if not @container + @container = $(""" +
+
+
+ """)
+ @container.appendTo(document.body)
+ @tag = @container.find(".internals")
+
+ open: =>
+ @createHtmltag()
+ @sidebar.fixbutton_targety = @sidebar.page_height
+ @stopDragY()
+
+ onOpened: =>
+ @sidebar.onClosed()
+ @log "onOpened"
+
+ onClosed: =>
+ $(document.body).removeClass("body-internals")
+
+ stopDragY: =>
+ # Animate sidebar and iframe
+ if @sidebar.fixbutton_targety == @sidebar.fixbutton_inity
+ # Closed
+ targety = 0
+ @opened = false
+ else
+ # Opened
+ targety = @sidebar.fixbutton_targety - @sidebar.fixbutton_inity
+ @onOpened()
+ @opened = true
+
+ # Revent sidebar transitions
+ if @tag
+ @tag.css("transition", "0.5s ease-out")
+ @tag.css("transform", "translateY(#{targety}px)").one transitionEnd, =>
+ @tag.css("transition", "")
+ if not @opened
+ @log "cleanup"
+ # Revert body transformations
+ @log "stopdrag", "opened:", @opened, targety
+ if not @opened
+ @onClosed()
+
+window.Internals = Internals
\ No newline at end of file
diff --git a/plugins/Sidebar/media/Internals.css b/plugins/Sidebar/media/Internals.css
new file mode 100644
index 00000000..36b2489e
--- /dev/null
+++ b/plugins/Sidebar/media/Internals.css
@@ -0,0 +1,17 @@
+.internals-container { width: 100%; z-index: 998; position: absolute; top: -100vh; }
+.internals { background-color: #EEE; height: 100vh; transform: translateY(0px); }
+.internals-middle {height: 0px; top: 50%; position: absolute; width: 100%; left: 50%; }
+
+.internals .mynode {
+ border: 0.5px solid #aaa; width: 50px; height: 50px; transform: rotateZ(45deg); margin-top: -25px; margin-left: -25px;
+ opacity: 1; display: inline-block; background-color: #EEE; z-index: 9; position: absolute; outline: 5px solid #EEE;
+}
+.internals .peers { width: 0px; height: 0px; position: absolute; left: -20px; top: -20px; text-align: center; }
+.internals .peer { left: 0px; top: 0px; position: absolute; }
+.internals .peer .icon { width: 20px; height: 20px; padding: 10px; display: inline-block; text-decoration: none; left: 200px; position: absolute; color: #666; }
+.internals .peer .icon:before { content: "\25BC"; position: absolute; margin-top: 3px; margin-left: -1px; opacity: 0; transition: all 0.3s }
+.internals .peer .icon:hover:before { opacity: 1; transition: none }
+.internals .peer .line {
+ width: 187px; border-top: 1px solid #CCC; position: absolute; top: 20px; left: 20px;
+ transform: rotateZ(334deg); transform-origin: bottom left;
+}
\ No newline at end of file
diff --git a/plugins/Sidebar/media/Menu.coffee b/plugins/Sidebar/media/Menu.coffee
new file mode 100644
index 00000000..3e19fd9f
--- /dev/null
+++ b/plugins/Sidebar/media/Menu.coffee
@@ -0,0 +1,49 @@
+class Menu
+ constructor: (@button) ->
+ @elem = $(".menu.template").clone().removeClass("template")
+ @elem.appendTo("body")
+ @items = []
+
+ show: ->
+ if window.visible_menu and window.visible_menu.button[0] == @button[0] # Same menu visible then hide it
+ window.visible_menu.hide()
+ @hide()
+ else
+ button_pos = @button.offset()
+ left = button_pos.left
+ @elem.css({"top": button_pos.top+@button.outerHeight(), "left": left})
+ @button.addClass("menu-active")
+ @elem.addClass("visible")
+ if @elem.position().left + @elem.width() + 20 > window.innerWidth
+ @elem.css("left", window.innerWidth - @elem.width() - 20)
+ if window.visible_menu then window.visible_menu.hide()
+ window.visible_menu = @
+
+
+ hide: ->
+ @elem.removeClass("visible")
+ @button.removeClass("menu-active")
+ window.visible_menu = null
+
+
+ addItem: (title, cb) ->
+ item = $(".menu-item.template", @elem).clone().removeClass("template")
+ item.html(title)
+ item.on "click", =>
+ if not cb(item)
+ @hide()
+ return false
+ item.appendTo(@elem)
+ @items.push item
+ return item
+
+
+ log: (args...) ->
+ console.log "[Menu]", args...
+
+window.Menu = Menu
+
+# Hide menu on outside click
+$("body").on "click", (e) ->
+ if window.visible_menu and e.target != window.visible_menu.button[0] and $(e.target).parent()[0] != window.visible_menu.elem[0]
+ window.visible_menu.hide()
diff --git a/plugins/Sidebar/media/Menu.css b/plugins/Sidebar/media/Menu.css
new file mode 100644
index 00000000..e2afa16e
--- /dev/null
+++ b/plugins/Sidebar/media/Menu.css
@@ -0,0 +1,19 @@
+.menu {
+ background-color: white; padding: 10px 0px; position: absolute; top: 0px; left: 0px; max-height: 0px; overflow: hidden; transform: translate(0px, -30px); pointer-events: none;
+ box-shadow: 0px 2px 8px rgba(0,0,0,0.3); border-radius: 2px; opacity: 0; transition: opacity 0.2s ease-out, transform 1s ease-out, max-height 0.2s ease-in-out;
+}
+.menu.visible { opacity: 1; max-height: 350px; transform: translate(0px, 0px); transition: opacity 0.1s ease-out, transform 0.3s ease-out, max-height 0.3s ease-in-out; pointer-events: all }
+
+.menu-item { display: block; text-decoration: none; color: black; padding: 6px 24px; transition: all 0.2s; border-bottom: none; font-weight: normal; padding-left: 30px; }
+.menu-item-separator { margin-top: 5px; border-top: 1px solid #eee }
+
+.menu-item:hover { background-color: #F6F6F6; transition: none; color: inherit; border: none }
+.menu-item:active, .menu-item:focus { background-color: #AF3BFF; color: white; transition: none }
+.menu-item.selected:before {
+ content: "L"; display: inline-block; transform: rotateZ(45deg) scaleX(-1);
+ font-weight: bold; position: absolute; margin-left: -17px; font-size: 12px; margin-top: 2px;
+}
+
+@media only screen and (max-width: 800px) {
+.menu, .menu.visible { position: absolute; left: unset !important; right: 20px; }
+}
\ No newline at end of file
diff --git a/plugins/Sidebar/media/RateLimit.coffee b/plugins/Sidebar/media/RateLimit.coffee
new file mode 100644
index 00000000..17c67433
--- /dev/null
+++ b/plugins/Sidebar/media/RateLimit.coffee
@@ -0,0 +1,14 @@
+limits = {}
+call_after_interval = {}
+window.RateLimit = (interval, fn) ->
+ if not limits[fn]
+ call_after_interval[fn] = false
+ fn() # First call is not delayed
+ limits[fn] = setTimeout (->
+ if call_after_interval[fn]
+ fn()
+ delete limits[fn]
+ delete call_after_interval[fn]
+ ), interval
+ else # Called within iterval, delay the call
+ call_after_interval[fn] = true
diff --git a/plugins/Sidebar/media/Scrollable.js b/plugins/Sidebar/media/Scrollable.js
new file mode 100644
index 00000000..689a5719
--- /dev/null
+++ b/plugins/Sidebar/media/Scrollable.js
@@ -0,0 +1,91 @@
+/* via http://jsfiddle.net/elGrecode/00dgurnn/ */
+
+window.initScrollable = function () {
+
+ var scrollContainer = document.querySelector('.scrollable'),
+ scrollContentWrapper = document.querySelector('.scrollable .content-wrapper'),
+ scrollContent = document.querySelector('.scrollable .content'),
+ contentPosition = 0,
+ scrollerBeingDragged = false,
+ scroller,
+ topPosition,
+ scrollerHeight;
+
+ function calculateScrollerHeight() {
+ // *Calculation of how tall scroller should be
+ var visibleRatio = scrollContainer.offsetHeight / scrollContentWrapper.scrollHeight;
+ if (visibleRatio == 1)
+ scroller.style.display = "none";
+ else
+ scroller.style.display = "block";
+ return visibleRatio * scrollContainer.offsetHeight;
+ }
+
+ function moveScroller(evt) {
+ // Move Scroll bar to top offset
+ var scrollPercentage = evt.target.scrollTop / scrollContentWrapper.scrollHeight;
+ topPosition = scrollPercentage * (scrollContainer.offsetHeight - 5); // 5px arbitrary offset so scroll bar doesn't move too far beyond content wrapper bounding box
+ scroller.style.top = topPosition + 'px';
+ }
+
+ function startDrag(evt) {
+ normalizedPosition = evt.pageY;
+ contentPosition = scrollContentWrapper.scrollTop;
+ scrollerBeingDragged = true;
+ window.addEventListener('mousemove', scrollBarScroll);
+ return false;
+ }
+
+ function stopDrag(evt) {
+ scrollerBeingDragged = false;
+ window.removeEventListener('mousemove', scrollBarScroll);
+ }
+
+ function scrollBarScroll(evt) {
+ if (scrollerBeingDragged === true) {
+ evt.preventDefault();
+ var mouseDifferential = evt.pageY - normalizedPosition;
+ var scrollEquivalent = mouseDifferential * (scrollContentWrapper.scrollHeight / scrollContainer.offsetHeight);
+ scrollContentWrapper.scrollTop = contentPosition + scrollEquivalent;
+ }
+ }
+
+ function updateHeight() {
+ scrollerHeight = calculateScrollerHeight() - 10;
+ scroller.style.height = scrollerHeight + 'px';
+ }
+
+ function createScroller() {
+ // *Creates scroller element and appends to '.scrollable' div
+ // create scroller element
+ scroller = document.createElement("div");
+ scroller.className = 'scroller';
+
+ // determine how big scroller should be based on content
+ scrollerHeight = calculateScrollerHeight() - 10;
+
+ if (scrollerHeight / scrollContainer.offsetHeight < 1) {
+ // *If there is a need to have scroll bar based on content size
+ scroller.style.height = scrollerHeight + 'px';
+
+ // append scroller to scrollContainer div
+ scrollContainer.appendChild(scroller);
+
+ // show scroll path divot
+ scrollContainer.className += ' showScroll';
+
+ // attach related draggable listeners
+ scroller.addEventListener('mousedown', startDrag);
+ window.addEventListener('mouseup', stopDrag);
+ }
+
+ }
+
+ createScroller();
+
+
+ // *** Listeners ***
+ scrollContentWrapper.addEventListener('scroll', moveScroller);
+
+ return updateHeight;
+};
\ No newline at end of file
diff --git a/plugins/Sidebar/media/Scrollbable.css b/plugins/Sidebar/media/Scrollbable.css
new file mode 100644
index 00000000..6e3e0b6a
--- /dev/null
+++ b/plugins/Sidebar/media/Scrollbable.css
@@ -0,0 +1,44 @@
+.scrollable {
+ overflow: hidden;
+}
+
+.scrollable.showScroll::after {
+ position: absolute;
+ content: '';
+ top: 5%;
+ right: 7px;
+ height: 90%;
+ width: 3px;
+ background: rgba(224, 224, 255, .3);
+}
+
+.scrollable .content-wrapper {
+ width: 100%;
+ height: 100%;
+ padding-right: 50%;
+ overflow-y: scroll;
+}
+.scroller {
+ margin-top: 5px;
+ z-index: 5;
+ cursor: pointer;
+ position: absolute;
+ width: 7px;
+ border-radius: 5px;
+ background: #3A3A3A;
+ top: 0px;
+ left: 395px;
+ -webkit-transition: top .08s;
+ -moz-transition: top .08s;
+ -ms-transition: top .08s;
+ -o-transition: top .08s;
+ transition: top .08s;
+}
+.scroller {
+ -webkit-touch-callout: none;
+ -webkit-user-select: none;
+ -khtml-user-select: none;
+ -moz-user-select: none;
+ -ms-user-select: none;
+ user-select: none;
+}
diff --git a/plugins/Sidebar/media/Sidebar.coffee b/plugins/Sidebar/media/Sidebar.coffee
new file mode 100644
index 00000000..ab9f5316
--- /dev/null
+++ b/plugins/Sidebar/media/Sidebar.coffee
@@ -0,0 +1,616 @@
+class Sidebar extends Class
+ constructor: (@wrapper) ->
+ @tag = null
+ @container = null
+ @opened = false
+ @width = 410
+ @internals = new Internals(@)
+ @fixbutton = $(".fixbutton")
+ @fixbutton_addx = 0
+ @fixbutton_addy = 0
+ @fixbutton_initx = 0
+ @fixbutton_inity = 15
+ @fixbutton_targetx = 0
+ @move_lock = null
+ @page_width = $(window).width()
+ @page_height = $(window).height()
+ @frame = $("#inner-iframe")
+ @initFixbutton()
+ @dragStarted = 0
+ @globe = null
+ @preload_html = null
+
+ @original_set_site_info = @wrapper.setSiteInfo # We going to override this, save the original
+
+ # Start in opened state for debugging
+ if false
+ @startDrag()
+ @moved()
+ @fixbutton_targetx = @fixbutton_initx - @width
+ @stopDrag()
+
+
+ initFixbutton: ->
+
+ # Detect dragging
+ @fixbutton.on "mousedown touchstart", (e) =>
+ if e.button > 0 # Right or middle click
+ return
+ e.preventDefault()
+
+ # Disable previous listeners
+ @fixbutton.off "click touchend touchcancel"
+
+ # Make sure its not a click
+ @dragStarted = (+ new Date)
+
+ # Fullscreen drag bg to capture mouse events over iframe
+ $(".drag-bg").remove()
+ $("").appendTo(document.body)
+
+ $("body").one "mousemove touchmove", (e) =>
+ mousex = e.pageX
+ mousey = e.pageY
+ if not mousex
+ mousex = e.originalEvent.touches[0].pageX
+ mousey = e.originalEvent.touches[0].pageY
+
+ @fixbutton_addx = @fixbutton.offset().left - mousex
+ @fixbutton_addy = @fixbutton.offset().top - mousey
+ @startDrag()
+ @fixbutton.parent().on "click touchend touchcancel", (e) =>
+ if (+ new Date) - @dragStarted < 100
+ window.top.location = @fixbutton.find(".fixbutton-bg").attr("href")
+ @stopDrag()
+ @resized()
+ $(window).on "resize", @resized
+
+ resized: =>
+ @page_width = $(window).width()
+ @page_height = $(window).height()
+ @fixbutton_initx = @page_width - 75 # Initial x position
+ if @opened
+ @fixbutton.css
+ left: @fixbutton_initx - @width
+ else
+ @fixbutton.css
+ left: @fixbutton_initx
+
+ # Start dragging the fixbutton
+ startDrag: ->
+ @move_lock = "x" # Temporary until internals not finished
+ @log "startDrag"
+ @fixbutton_targetx = @fixbutton_initx # Fallback x position
+
+ @fixbutton.addClass("dragging")
+
+ # IE position wrap fix
+ if navigator.userAgent.indexOf('MSIE') != -1 or navigator.appVersion.indexOf('Trident/') > 0
+ @fixbutton.css("pointer-events", "none")
+
+ # Don't go to homepage
+ @fixbutton.one "click", (e) =>
+ @stopDrag()
+ @fixbutton.removeClass("dragging")
+ moved_x = Math.abs(@fixbutton.offset().left - @fixbutton_initx)
+ moved_y = Math.abs(@fixbutton.offset().top - @fixbutton_inity)
+ if moved_x > 5 or moved_y > 10
+ # If moved more than some pixel the button then don't go to homepage
+ e.preventDefault()
+
+ # Animate drag
+ @fixbutton.parents().on "mousemove touchmove", @animDrag
+ @fixbutton.parents().on "mousemove touchmove" ,@waitMove
+
+ # Stop dragging listener
+ @fixbutton.parents().one "mouseup touchend touchcancel", (e) =>
+ e.preventDefault()
+ @stopDrag()
+
+
+ # Wait for moving the fixbutton
+ waitMove: (e) =>
+ document.body.style.perspective = "1000px"
+ document.body.style.height = "100%"
+ document.body.style.willChange = "perspective"
+ document.documentElement.style.height = "100%"
+ #$(document.body).css("backface-visibility", "hidden").css("perspective", "1000px").css("height", "900px")
+ # $("iframe").css("backface-visibility", "hidden")
+
+ moved_x = Math.abs(parseInt(@fixbutton[0].style.left) - @fixbutton_targetx)
+ moved_y = Math.abs(parseInt(@fixbutton[0].style.top) - @fixbutton_targety)
+ if moved_x > 5 and (+ new Date) - @dragStarted + moved_x > 50
+ @moved("x")
+ @fixbutton.stop().animate {"top": @fixbutton_inity}, 1000
+ @fixbutton.parents().off "mousemove touchmove" ,@waitMove
+
+ else if moved_y > 5 and (+ new Date) - @dragStarted + moved_y > 50
+ @moved("y")
+ @fixbutton.parents().off "mousemove touchmove" ,@waitMove
+
+ moved: (direction) ->
+ @log "Moved", direction
+ @move_lock = direction
+ if direction == "y"
+ $(document.body).addClass("body-internals")
+ return @internals.createHtmltag()
+ @createHtmltag()
+ $(document.body).addClass("body-sidebar")
+ @container.on "mousedown touchend touchcancel", (e) =>
+ if e.target != e.currentTarget
+ return true
+ @log "closing"
+ if $(document.body).hasClass("body-sidebar")
+ @close()
+ return true
+
+ $(window).off "resize"
+ $(window).on "resize", =>
+ $(document.body).css "height", $(window).height()
+ @scrollable()
+ @resized()
+
+ # Override setsiteinfo to catch changes
+ @wrapper.setSiteInfo = (site_info) =>
+ @setSiteInfo(site_info)
+ @original_set_site_info.apply(@wrapper, arguments)
+
+ # Preload world.jpg
+ img = new Image();
+ img.src = "/uimedia/globe/world.jpg";
+
+ setSiteInfo: (site_info) ->
+ RateLimit 1500, =>
+ @updateHtmlTag()
+ RateLimit 30000, =>
+ @displayGlobe()
+
+ # Create the sidebar html tag
+ createHtmltag: ->
+ @when_loaded = $.Deferred()
+ if not @container
+ @container = $("""
+
+ """)
+ @container.appendTo(document.body)
+ @tag = @container.find(".sidebar")
+ @updateHtmlTag()
+ @scrollable = window.initScrollable()
+
+
+ updateHtmlTag: ->
+ if @preload_html
+ @setHtmlTag(@preload_html)
+ @preload_html = null
+ else
+ @wrapper.ws.cmd "sidebarGetHtmlTag", {}, @setHtmlTag
+
+ setHtmlTag: (res) =>
+ if @tag.find(".content").children().length == 0 # First update
+ @log "Creating content"
+ @container.addClass("loaded")
+ morphdom(@tag.find(".content")[0], '
+
+
+
+
+ '+res+'
')
+ # @scrollable()
+ @when_loaded.resolve()
+
+ else # Not first update, patch the html to keep unchanged dom elements
+ morphdom @tag.find(".content")[0], ''+res+'
', {
+ onBeforeMorphEl: (from_el, to_el) -> # Ignore globe loaded state
+ if from_el.className == "globe" or from_el.className.indexOf("noupdate") >= 0
+ return false
+ else
+ return true
+ }
+
+ # Save and forgot privatekey for site signing
+ @tag.find("#privatekey-add").off("click, touchend").on "click touchend", (e) =>
+ @wrapper.displayPrompt "Enter your private key:", "password", "Save", "", (privatekey) =>
+ @wrapper.ws.cmd "userSetSitePrivatekey", [privatekey], (res) =>
+ @wrapper.notifications.add "privatekey", "done", "Private key saved for site signing", 5000
+ return false
+
+ @tag.find("#privatekey-forgot").off("click, touchend").on "click touchend", (e) =>
+ @wrapper.displayConfirm "Remove saved private key for this site?", "Forgot", (res) =>
+ if not res
+ return false
+ @wrapper.ws.cmd "userSetSitePrivatekey", [""], (res) =>
+ @wrapper.notifications.add "privatekey", "done", "Saved private key removed", 5000
+ return false
+
+
+
+ animDrag: (e) =>
+ mousex = e.pageX
+ mousey = e.pageY
+ if not mousex and e.originalEvent.touches
+ mousex = e.originalEvent.touches[0].pageX
+ mousey = e.originalEvent.touches[0].pageY
+
+ overdrag = @fixbutton_initx - @width - mousex
+ if overdrag > 0 # Overdragged
+ overdrag_percent = 1 + overdrag/300
+ mousex = (mousex + (@fixbutton_initx-@width)*overdrag_percent)/(1+overdrag_percent)
+ targetx = @fixbutton_initx - mousex - @fixbutton_addx
+ targety = @fixbutton_inity - mousey - @fixbutton_addy
+
+ if @move_lock == "x"
+ targety = @fixbutton_inity
+ else if @move_lock == "y"
+ targetx = @fixbutton_initx
+
+ if not @move_lock or @move_lock == "x"
+ @fixbutton[0].style.left = (mousex + @fixbutton_addx) + "px"
+ if @tag
+ @tag[0].style.transform = "translateX(#{0 - targetx}px)"
+
+ if not @move_lock or @move_lock == "y"
+ @fixbutton[0].style.top = (mousey + @fixbutton_addy) + "px"
+ if @internals.tag
+ @internals.tag[0].style.transform = "translateY(#{0 - targety}px)"
+
+ #if @move_lock == "x"
+ # @fixbutton[0].style.left = "#{@fixbutton_targetx} px"
+ #@fixbutton[0].style.top = "#{@fixbutton_inity}px"
+ #if @move_lock == "y"
+ # @fixbutton[0].style.top = "#{@fixbutton_targety} px"
+
+ # Check if opened
+ if (not @opened and targetx > @width/3) or (@opened and targetx > @width*0.9)
+ @fixbutton_targetx = @fixbutton_initx - @width # Make it opened
+ else
+ @fixbutton_targetx = @fixbutton_initx
+
+ if (not @internals.opened and 0 - targety > @page_height/10) or (@internals.opened and 0 - targety > @page_height*0.95)
+ @fixbutton_targety = @page_height - @fixbutton_inity - 50
+ else
+ @fixbutton_targety = @fixbutton_inity
+
+
+ # Stop dragging the fixbutton
+ stopDrag: ->
+ @fixbutton.parents().off "mousemove touchmove"
+ @fixbutton.off "mousemove touchmove"
+ @fixbutton.css("pointer-events", "")
+ $(".drag-bg").remove()
+ if not @fixbutton.hasClass("dragging")
+ return
+ @fixbutton.removeClass("dragging")
+
+ # Move back to initial position
+ if @fixbutton_targetx != @fixbutton.offset().left
+ # Animate fixbutton
+ if @move_lock == "y"
+ top = @fixbutton_targety
+ left = @fixbutton_initx
+ if @move_lock == "x"
+ top = @fixbutton_inity
+ left = @fixbutton_targetx
+ @fixbutton.stop().animate {"left": left, "top": top}, 500, "easeOutBack", =>
+ # Switch back to auto align
+ if @fixbutton_targetx == @fixbutton_initx # Closed
+ @fixbutton.css("left", "auto")
+ else # Opened
+ @fixbutton.css("left", left)
+
+ $(".fixbutton-bg").trigger "mouseout" # Switch fixbutton back to normal status
+
+ @stopDragX()
+ @internals.stopDragY()
+ @move_lock = null
+
+ stopDragX: ->
+ # Animate sidebar and iframe
+ if @fixbutton_targetx == @fixbutton_initx or @move_lock == "y"
+ # Closed
+ targetx = 0
+ @opened = false
+ else
+ # Opened
+ targetx = @width
+ if @opened
+ @onOpened()
+ else
+ @when_loaded.done =>
+ @onOpened()
+ @opened = true
+
+ # Revent sidebar transitions
+ if @tag
+ @tag.css("transition", "0.4s ease-out")
+ @tag.css("transform", "translateX(-#{targetx}px)").one transitionEnd, =>
+ @tag.css("transition", "")
+ if not @opened
+ @container.remove()
+ @container = null
+ if @tag
+ @tag.remove()
+ @tag = null
+
+ # Revert body transformations
+ @log "stopdrag", "opened:", @opened
+ if not @opened
+ @onClosed()
+
+ sign: (inner_path, privatekey) ->
+ @wrapper.displayProgress("sign", "Signing: #{inner_path}...", 0)
+ @wrapper.ws.cmd "siteSign", {privatekey: privatekey, inner_path: inner_path, update_changed_files: true}, (res) =>
+ if res == "ok"
+ @wrapper.displayProgress("sign", "#{inner_path} signed!", 100)
+ else
+ @wrapper.displayProgress("sign", "Error signing #{inner_path}", -1)
+
+ publish: (inner_path, privatekey) ->
+ @wrapper.ws.cmd "sitePublish", {privatekey: privatekey, inner_path: inner_path, sign: true, update_changed_files: true}, (res) =>
+ if res == "ok"
+ @wrapper.notifications.add "sign", "done", "#{inner_path} Signed and published!", 5000
+
+ onOpened: ->
+ @log "Opened"
+ @scrollable()
+
+ # Re-calculate height when site admin opened or closed
+ @tag.find("#checkbox-owned, #checkbox-autodownloadoptional").off("click touchend").on "click touchend", =>
+ setTimeout (=>
+ @scrollable()
+ ), 300
+
+ # Site limit button
+ @tag.find("#button-sitelimit").off("click touchend").on "click touchend", =>
+ @wrapper.ws.cmd "siteSetLimit", $("#input-sitelimit").val(), (res) =>
+ if res == "ok"
+ @wrapper.notifications.add "done-sitelimit", "done", "Site storage limit modified!", 5000
+ @updateHtmlTag()
+ return false
+
+ # Site autodownload limit button
+ @tag.find("#button-autodownload_bigfile_size_limit").off("click touchend").on "click touchend", =>
+ @wrapper.ws.cmd "siteSetAutodownloadBigfileLimit", $("#input-autodownload_bigfile_size_limit").val(), (res) =>
+ if res == "ok"
+ @wrapper.notifications.add "done-bigfilelimit", "done", "Site bigfile auto download limit modified!", 5000
+ @updateHtmlTag()
+ return false
+
+ # Database reload
+ @tag.find("#button-dbreload").off("click touchend").on "click touchend", =>
+ @wrapper.ws.cmd "dbReload", [], =>
+ @wrapper.notifications.add "done-dbreload", "done", "Database schema reloaded!", 5000
+ @updateHtmlTag()
+ return false
+
+ # Database rebuild
+ @tag.find("#button-dbrebuild").off("click touchend").on "click touchend", =>
+ @wrapper.notifications.add "done-dbrebuild", "info", "Database rebuilding...."
+ @wrapper.ws.cmd "dbRebuild", [], =>
+ @wrapper.notifications.add "done-dbrebuild", "done", "Database rebuilt!", 5000
+ @updateHtmlTag()
+ return false
+
+ # Update site
+ @tag.find("#button-update").off("click touchend").on "click touchend", =>
+ @tag.find("#button-update").addClass("loading")
+ @wrapper.ws.cmd "siteUpdate", @wrapper.site_info.address, =>
+ @wrapper.notifications.add "done-updated", "done", "Site updated!", 5000
+ @tag.find("#button-update").removeClass("loading")
+ return false
+
+ # Pause site
+ @tag.find("#button-pause").off("click touchend").on "click touchend", =>
+ @tag.find("#button-pause").addClass("hidden")
+ @wrapper.ws.cmd "sitePause", @wrapper.site_info.address
+ return false
+
+ # Resume site
+ @tag.find("#button-resume").off("click touchend").on "click touchend", =>
+ @tag.find("#button-resume").addClass("hidden")
+ @wrapper.ws.cmd "siteResume", @wrapper.site_info.address
+ return false
+
+ # Delete site
+ @tag.find("#button-delete").off("click touchend").on "click touchend", =>
+ @wrapper.displayConfirm "Are you sure?", ["Delete this site", "Blacklist"], (confirmed) =>
+ if confirmed == 1
+ @tag.find("#button-delete").addClass("loading")
+ @wrapper.ws.cmd "siteDelete", @wrapper.site_info.address, ->
+ document.location = $(".fixbutton-bg").attr("href")
+ else if confirmed == 2
+ @wrapper.displayPrompt "Blacklist this site", "text", "Delete and Blacklist", "Reason", (reason) =>
+ @tag.find("#button-delete").addClass("loading")
+ @wrapper.ws.cmd "siteblockAdd", [@wrapper.site_info.address, reason]
+ @wrapper.ws.cmd "siteDelete", @wrapper.site_info.address, ->
+ document.location = $(".fixbutton-bg").attr("href")
+
+
+ return false
+
+ # Owned checkbox
+ @tag.find("#checkbox-owned").off("click touchend").on "click touchend", =>
+ @wrapper.ws.cmd "siteSetOwned", [@tag.find("#checkbox-owned").is(":checked")]
+
+ # Owned checkbox
+ @tag.find("#checkbox-autodownloadoptional").off("click touchend").on "click touchend", =>
+ @wrapper.ws.cmd "siteSetAutodownloadoptional", [@tag.find("#checkbox-autodownloadoptional").is(":checked")]
+
+ # Change identity button
+ @tag.find("#button-identity").off("click touchend").on "click touchend", =>
+ @wrapper.ws.cmd "certSelect"
+ return false
+
+ # Save settings
+ @tag.find("#button-settings").off("click touchend").on "click touchend", =>
+ @wrapper.ws.cmd "fileGet", "content.json", (res) =>
+ data = JSON.parse(res)
+ data["title"] = $("#settings-title").val()
+ data["description"] = $("#settings-description").val()
+ json_raw = unescape(encodeURIComponent(JSON.stringify(data, undefined, '\t')))
+ @wrapper.ws.cmd "fileWrite", ["content.json", btoa(json_raw), true], (res) =>
+ if res != "ok" # fileWrite failed
+ @wrapper.notifications.add "file-write", "error", "File write error: #{res}"
+ else
+ @wrapper.notifications.add "file-write", "done", "Site settings saved!", 5000
+ if @wrapper.site_info.privatekey
+ @wrapper.ws.cmd "siteSign", {privatekey: "stored", inner_path: "content.json", update_changed_files: true}
+ @updateHtmlTag()
+ return false
+
+
+ # Open site directory
+ @tag.find("#link-directory").off("click touchend").on "click touchend", =>
+ @wrapper.ws.cmd "serverShowdirectory", ["site", @wrapper.site_info.address]
+ return false
+
+ # Copy site with peers
+ @tag.find("#link-copypeers").off("click touchend").on "click touchend", (e) =>
+ copy_text = e.currentTarget.href
+ handler = (e) =>
+ e.clipboardData.setData('text/plain', copy_text)
+ e.preventDefault()
+ @wrapper.notifications.add "copy", "done", "Site address with peers copied to your clipboard", 5000
+ document.removeEventListener('copy', handler, true)
+
+ document.addEventListener('copy', handler, true)
+ document.execCommand('copy')
+ return false
+
+ # Sign and publish content.json
+ $(document).on "click touchend", =>
+ @tag?.find("#button-sign-publish-menu").removeClass("visible")
+ @tag?.find(".contents + .flex").removeClass("sign-publish-flex")
+
+ @tag.find(".contents-content").off("click touchend").on "click touchend", (e) =>
+ $("#input-contents").val(e.currentTarget.innerText);
+ return false;
+
+ menu = new Menu(@tag.find("#menu-sign-publish"))
+ menu.elem.css("margin-top", "-130px") # Open upwards
+ menu.addItem "Sign", =>
+ inner_path = @tag.find("#input-contents").val()
+
+ @wrapper.ws.cmd "fileRules", {inner_path: inner_path}, (rules) =>
+ if @wrapper.site_info.auth_address in rules.signers
+ # ZeroID or other ID provider
+ @sign(inner_path)
+ else if @wrapper.site_info.privatekey
+ # Privatekey stored in users.json
+ @sign(inner_path, "stored")
+ else
+ # Ask the user for privatekey
+ @wrapper.displayPrompt "Enter your private key:", "password", "Sign", "", (privatekey) => # Prompt the private key
+ @sign(inner_path, privatekey)
+
+ @tag.find(".contents + .flex").removeClass "active"
+ menu.hide()
+
+ menu.addItem "Publish", =>
+ inner_path = @tag.find("#input-contents").val()
+ @wrapper.ws.cmd "sitePublish", {"inner_path": inner_path, "sign": false}
+
+ @tag.find(".contents + .flex").removeClass "active"
+ menu.hide()
+
+ @tag.find("#menu-sign-publish").off("click touchend").on "click touchend", =>
+ if window.visible_menu == menu
+ @tag.find(".contents + .flex").removeClass "active"
+ menu.hide()
+ else
+ @tag.find(".contents + .flex").addClass "active"
+ @tag.find(".content-wrapper").prop "scrollTop", 10000
+ menu.show()
+ return false
+
+ $("body").on "click", =>
+ if @tag
+ @tag.find(".contents + .flex").removeClass "active"
+
+ @tag.find("#button-sign-publish").off("click touchend").on "click touchend", =>
+ inner_path = @tag.find("#input-contents").val()
+
+ @wrapper.ws.cmd "fileRules", {inner_path: inner_path}, (rules) =>
+ if @wrapper.site_info.auth_address in rules.signers
+ # ZeroID or other ID provider
+ @publish(inner_path, null)
+ else if @wrapper.site_info.privatekey
+ # Privatekey stored in users.json
+ @publish(inner_path, "stored")
+ else
+ # Ask the user for privatekey
+ @wrapper.displayPrompt "Enter your private key:", "password", "Sign", "", (privatekey) => # Prompt the private key
+ @publish(inner_path, privatekey)
+ return false
+
+ # Close
+ @tag.find(".close").off("click touchend").on "click touchend", (e) =>
+ @close()
+ return false
+
+ @loadGlobe()
+
+ close: ->
+ @move_lock = "x"
+ @startDrag()
+ @stopDrag()
+
+
+ onClosed: ->
+ $(window).off "resize"
+ $(window).on "resize", @resized
+ $(document.body).css("transition", "0.6s ease-in-out").removeClass("body-sidebar").on transitionEnd, (e) =>
+ if e.target == document.body and not $(document.body).hasClass("body-sidebar") and not $(document.body).hasClass("body-internals")
+ $(document.body).css("height", "auto").css("perspective", "").css("will-change", "").css("transition", "").off transitionEnd
+ @unloadGlobe()
+
+ # We dont need site info anymore
+ @wrapper.setSiteInfo = @original_set_site_info
+
+
+ loadGlobe: =>
+ if @tag.find(".globe").hasClass("loading")
+ setTimeout (=>
+ if typeof(DAT) == "undefined" # Globe script not loaded, do it first
+ script_tag = $("
+