Merge branch 'master' into no-muted-propagation

This commit is contained in:
caryoscelus 2023-09-16 12:08:49 +00:00
commit 90de9c6500
10 changed files with 73 additions and 48 deletions

View file

@ -6,15 +6,14 @@
[по-русски](README-ru.md) | [em português](README-ptbr.md) | [简体中文](README-zh-cn.md)
zeronet-conservancy is a fork/continuation of [ZeroNet](https://github.com/HelloZeroNet/ZeroNet) project
`zeronet-conservancy` is a fork/continuation of [ZeroNet](https://github.com/HelloZeroNet/ZeroNet) project
(that has been abandoned by its creator) that is dedicated to sustaining existing p2p network and developing
its values of decentralization and freedom, while gradually switching to a better designed network
## No active maintainer warning
This fork was created and maintained by @caryoscelus, but due to vanishing interest and in order to avoid having
another one-person project, they stepped down. This means there currently is no active maintainer (you're are
welcome to become one!), however some development might still happen.
another one-person project, the development is limitted.
## Why fork?
@ -75,21 +74,15 @@ Following links relate to original ZeroNet:
### Install from your distribution repository
- NixOS: https://search.nixos.org/packages?channel=22.05&show=zeronet-conservancy&type=packages&query=zeronet-conservancy (and see below)
- NixOS: [zeronet-conservancy packages search](https://search.nixos.org/packages?from=0&size=50&sort=relevance&type=packages&query=zeronet-conservancy) (and see below)
- ArchLinux: [latest release](https://aur.archlinux.org/packages/zeronet-conservancy), [fresh git version](https://aur.archlinux.org/packages/zeronet-conservancy-git)
### Install from Nix package manager (Linux or MacOS)
```
# install & configure nix package manager
nix-env -iA nixpkgs.zeronet-conservancy
```
- install & configure nix package manager (if needed)
- `nix-env -iA nixpkgs.zeronet-conservancy`
or
`nix-env -iA nixos.zeronet-conservancy`
if you're on NixOS
or add `zeronet-conservancy` to your system configuration if you're on NixOS
(thanks @fgaz for making & maintaining the package)
@ -100,11 +93,25 @@ if you're on NixOS
##### Generic unix-like (including mac os x)
Install autoconf and other basic development tools, python3 and pip, then proceed to "building python dependencies"
(if running fails due to missing dependency, please report it/make pull request to fix dependency list)
##### Apt-based (debian, ubuntu, etc)
- `sudo apt update`
- `sudo apt install pkg-config libffi-dev python3-pip python3-venv python3-dev build-essential`
##### Red Hat and Fedora based
- `yum install epel-release -y 2>/dev/null`
- `yum install git python3 python3-wheel`
##### Fedora based dandified
- `sudo dnf install git python3-pip python3-wheel -y`
##### openSUSE
- `sudo zypper install python3-pip python3-setuptools python3-wheel`
##### Arch and Manjaro based
- `sudo pacman -S git python-pip -v --no-confirm`
##### Android/Termux
- install [Termux](https://termux.com/) (in Termux you can install packages via `pkg install <package-names>`)
- `pkg update`
@ -114,7 +121,7 @@ Install autoconf and other basic development tools, python3 and pip, then procee
- (optional) `pkg install tor`
- (optional) run tor via `tor --ControlPort 9051 --CookieAuthentication 1` command (you can then open new session by swiping to the right)
#### Building python dependencies venv & running
#### Building python dependencies, venv & running
- clone this repo (NOTE: on Android/Termux you should clone it into "home" folder of Termux, because virtual environment cannot live in `storage/`)
- `python3 -m venv venv` (make python virtual environment, the last `venv` is just a name, if you use different you should replace it in later commands)
- `source venv/bin/activate` (activate environment)
@ -139,13 +146,30 @@ Install autoconf and other basic development tools, python3 and pip, then procee
- or: `docker compose up -d 0net-tor` for run 0net and tor in one container.
(please check if these instructions are still accurate)
#### Alternative one-liner (by @ssdifnskdjfnsdjk) (installing python dependencies globally)
Clone Github repository and install required Python modules. First
edit zndir path at the begining of the command, to be the path where
you want to store `zeronet-conservancy`:
`zndir="/home/user/myapps/zeronet" ; if [[ ! -d "$zndir" ]]; then git clone --recursive "https://github.com/zeronet-conservancy/zeronet-conservancy.git" "$zndir" && cd "$zndir"||exit; else cd "$zndir";git pull origin master; fi; cd "$zndir" && pip install -r requirements.txt|grep -v "already satisfied"; echo "Try to run: python3 $(pwd)/zeronet.py"`
(This command can also be used to keep `zeronet-conservancy` up to date)
#### Alternative script
- after installing general dependencies and cloning repo (as above), run `start-venv.sh` which will create a virtual env for you and install python requirements
- after installing general dependencies and cloning repo (as above),
run `start-venv.sh` which will create a virtual env for you and
install python requirements
- more convenience scripts to be added soon
### Building under windows os
### (unofficial) Windows OS build
(this instruction is work-in-progress, please help us test it and improve it!)
Download and extract .zip archive
[zeronet-conservancy-0.7.10-unofficial-win64.zip](https://github.com/zeronet-conservancy/zeronet-conservancy/releases/download/v0.7.10/zeronet-conservancy-0.7.10-unofficial-win64.zip)
### Building under Windows OS
(These instructions are work-in-progress, please help us test it and improve it!)
- install python from https://www.python.org/downloads/
- install some windows compiler suitable for python , this proved to be the most difficult part for me as non-windows user (see here https://wiki.python.org/moin/WindowsCompilers and i'll link more references later)

View file

@ -1 +1 @@
https://github.com/zeronet-conservancy/zeronet-conservancy/releases/download/v0.7.10/data-default-2023-07-26.zip
https://github.com/zeronet-conservancy/zeronet-conservancy/releases/download/v0.7.10/data-default-2023-09-03.zip

View file

@ -14,8 +14,7 @@ class WsLogStreamer(logging.StreamHandler):
self.ui_websocket = ui_websocket
if filter:
if not SafeRe.isSafePattern(filter):
raise Exception("Not a safe prex pattern")
SafeRe.guard(filter)
self.filter_re = re.compile(".*" + filter)
else:
self.filter_re = None
@ -55,7 +54,7 @@ class UiWebsocketPlugin(object):
pos_start = log_file.tell()
lines = []
if filter:
assert SafeRe.isSafePattern(filter)
SafeRe.guard(filter)
filter_re = re.compile(".*" + filter)
last_match = False

View file

@ -256,6 +256,7 @@ class Config(object):
self.parser.add_argument('--ip_local', help='My local ips', default=ip_local, type=int, metavar='ip', nargs='*')
self.parser.add_argument('--ip_external', help='Set reported external ip (tested on start if None)', metavar='ip', nargs='*')
self.parser.add_argument('--offline', help='Disable network communication', action='store_true')
self.parser.add_argument('--disable_port_check', help='Disable checking port', action='store_true')
self.parser.add_argument('--disable_udp', help='Disable UDP connections', action='store_true')
self.parser.add_argument('--proxy', help='Socks proxy address', metavar='ip:port')

View file

@ -152,9 +152,13 @@ class FileServer(ConnectionServer):
FileRequest = imp.load_source("FileRequest", "src/File/FileRequest.py").FileRequest
def portCheck(self):
if config.offline or config.tor == 'always':
msg = "Offline mode" if config.offline else "Tor-only"
self.log.info(f'{msg}: port check disabled')
if config.offline or config.tor == 'always' or config.disable_port_check:
if config.offline:
self.log.info(f'Offline mode: port check disabled')
elif config.tor == 'always':
self.log.info('Tor-only mode: port check disabled')
else:
self.log.info('Port check disabled')
res = {"ipv4": None, "ipv6": None}
self.port_opened = res
return res

View file

@ -15,11 +15,9 @@ class TestThreadPool:
@pool.wrap
def blocker():
events.append("S")
out = 0
for i in range(10000000):
if i == 3000000:
events.append("M")
out += 1
time.sleep(0.001)
events.append("M")
time.sleep(0.001)
events.append("D")
return out
@ -30,9 +28,6 @@ class TestThreadPool:
assert events == ["S"] * 3 + ["M"] * 3 + ["D"] * 3
res = blocker()
assert res == 10000000
def testLockBlockingSameThread(self):
lock = ThreadPool.Lock()

View file

@ -63,9 +63,6 @@ config.debug = True
os.chdir(os.path.abspath(os.path.dirname(__file__) + "/../..")) # Set working dir
all_loaded = PluginManager.plugin_manager.loadPlugins()
assert all_loaded, "Not all plugin loaded successfully"
config.loadPlugins()
config.parse(parse_config=False) # Parse again to add plugin configuration options

View file

@ -1,6 +1,6 @@
[pytest]
python_files = Test*.py
addopts = -rsxX -v --durations=6 --no-print-logs --capture=fd
addopts = -rsxX -v --durations=6 --capture=fd
markers =
slow: mark a tests as slow.
webtest: mark a test as a webtest.

View file

@ -7,6 +7,8 @@ from Debug import Debug
from Config import config
from Content.ContentManager import VerifyError
import traceback
class WorkerDownloadError(Exception):
pass
@ -119,13 +121,15 @@ class Worker(object):
self.manager.log.error("%s: Error writing: %s (%s: %s)" % (self.key, task["inner_path"], type(err), err))
raise WorkerIOError(str(err))
def onTaskVerifyFail(self, task, error_message):
def onTaskVerifyFail(self, task, error):
self.num_failed += 1
if self.manager.started_task_num < 50 or config.verbose:
self.manager.log.debug(
self.manager.log.info(
"%s: Verify failed: %s, error: %s, failed peers: %s, workers: %s" %
(self.key, task["inner_path"], error_message, len(task["failed"]), task["workers_num"])
(self.key, task["inner_path"], error, len(task["failed"]), task["workers_num"])
)
# traceback.format_
self.manager.log.debug(''.join(traceback.format_exception(error)))
task["failed"].append(self.peer)
self.peer.hash_failed += 1
if self.peer.hash_failed >= max(len(self.manager.tasks), 3) or self.peer.connection_error > 10:

View file

@ -7,7 +7,8 @@ class UnsafePatternError(Exception):
cached_patterns = {}
def isSafePattern(pattern):
def guard(pattern):
'''Checks if pattern is safe and raises exception if it isn't'''
if len(pattern) > 255:
raise UnsafePatternError("Pattern too long: %s characters in %s" % (len(pattern), pattern))
@ -15,18 +16,18 @@ def isSafePattern(pattern):
if unsafe_pattern_match:
raise UnsafePatternError("Potentially unsafe part of the pattern: %s in %s" % (unsafe_pattern_match.group(0), pattern))
repetitions = re.findall(r"\.[\*\{\+]", pattern)
if len(repetitions) >= 10:
raise UnsafePatternError("More than 10 repetitions of %s in %s" % (repetitions[0], pattern))
return True
repetitions1 = re.findall(r"\.[\*\{\+]", pattern)
repetitions2 = re.findall(r"[^(][?]", pattern)
if len(repetitions1) + len(repetitions2) >= 10:
raise UnsafePatternError("More than 10 repetitions in %s" % pattern)
def match(pattern, *args, **kwargs):
'''Guard for safety, compile, cache and match regexp'''
cached_pattern = cached_patterns.get(pattern)
if cached_pattern:
return cached_pattern.match(*args, **kwargs)
else:
if isSafePattern(pattern):
cached_patterns[pattern] = re.compile(pattern)
return cached_patterns[pattern].match(*args, **kwargs)
guard(pattern)
cached_patterns[pattern] = re.compile(pattern)
return cached_patterns[pattern].match(*args, **kwargs)