Merge branch 'master' into no-muted-propagation

README.md

@@ -6,15 +6,14 @@
 
 [по-русски](README-ru.md) | [em português](README-ptbr.md) | [简体中文](README-zh-cn.md)
 
-zeronet-conservancy is a fork/continuation of [ZeroNet](https://github.com/HelloZeroNet/ZeroNet) project
+`zeronet-conservancy` is a fork/continuation of [ZeroNet](https://github.com/HelloZeroNet/ZeroNet) project
 (that has been abandoned by its creator) that is dedicated to sustaining existing p2p network and developing
 its values of decentralization and freedom, while gradually switching to a better designed network
 
 ## No active maintainer warning
 
 This fork was created and maintained by @caryoscelus, but due to vanishing interest and in order to avoid having
-another one-person project, they stepped down. This means there currently is no active maintainer (you're are
+another one-person project, the development is limitted.
-welcome to become one!), however some development might still happen.
 
 ## Why fork?
 
@@ -75,21 +74,15 @@ Following links relate to original ZeroNet:
 
 ### Install from your distribution repository
 
-- NixOS: https://search.nixos.org/packages?channel=22.05&show=zeronet-conservancy&type=packages&query=zeronet-conservancy (and see below)
+- NixOS: [zeronet-conservancy packages search](https://search.nixos.org/packages?from=0&size=50&sort=relevance&type=packages&query=zeronet-conservancy) (and see below)
 - ArchLinux: [latest release](https://aur.archlinux.org/packages/zeronet-conservancy), [fresh git version](https://aur.archlinux.org/packages/zeronet-conservancy-git)
 
 ### Install from Nix package manager (Linux or MacOS)
 
-```
+ - install & configure nix package manager (if needed)
-# install & configure nix package manager
+ - `nix-env -iA nixpkgs.zeronet-conservancy`
-nix-env -iA nixpkgs.zeronet-conservancy
-```
 
-or
+or add `zeronet-conservancy` to your system configuration if you're on NixOS
-
-`nix-env -iA nixos.zeronet-conservancy`
-
-if you're on NixOS
 
 (thanks @fgaz for making & maintaining the package)
 
@@ -100,11 +93,25 @@ if you're on NixOS
 ##### Generic unix-like (including mac os x)
 
 Install autoconf and other basic development tools, python3 and pip, then proceed to "building python dependencies"
+(if running fails due to missing dependency, please report it/make pull request to fix dependency list)
 
 ##### Apt-based (debian, ubuntu, etc)
  - `sudo apt update`
  - `sudo apt install pkg-config libffi-dev python3-pip python3-venv python3-dev build-essential`
 
+##### Red Hat and Fedora based
+ - `yum install epel-release -y 2>/dev/null`
+ - `yum install git python3 python3-wheel`
+
+##### Fedora based dandified
+ - `sudo dnf install git python3-pip python3-wheel -y`
+
+##### openSUSE
+ - `sudo zypper install python3-pip python3-setuptools python3-wheel`
+
+##### Arch and Manjaro based
+ - `sudo pacman -S git python-pip -v --no-confirm`
+
 ##### Android/Termux
  - install [Termux](https://termux.com/) (in Termux you can install packages via `pkg install <package-names>`)
  - `pkg update`
@@ -114,7 +121,7 @@ Install autoconf and other basic development tools, python3 and pip, then procee
  - (optional) `pkg install tor`
  - (optional) run tor via `tor --ControlPort 9051 --CookieAuthentication 1` command (you can then open new session by swiping to the right)
 
-#### Building python dependencies venv & running
+#### Building python dependencies, venv & running
  - clone this repo (NOTE: on Android/Termux you should clone it into "home" folder of Termux, because virtual environment cannot live in `storage/`)
  - `python3 -m venv venv` (make python virtual environment, the last `venv` is just a name, if you use different you should replace it in later commands)
  - `source venv/bin/activate` (activate environment)
@@ -139,13 +146,30 @@ Install autoconf and other basic development tools, python3 and pip, then procee
 - or: `docker compose up -d 0net-tor` for run 0net and tor in one container.
 (please check if these instructions are still accurate)
 
+#### Alternative one-liner (by @ssdifnskdjfnsdjk) (installing python dependencies globally)
+
+Clone Github repository and install required Python modules. First
+edit zndir path at the begining of the command, to be the path where
+you want to store `zeronet-conservancy`:
+
+`zndir="/home/user/myapps/zeronet" ; if [[ ! -d "$zndir" ]]; then git clone --recursive "https://github.com/zeronet-conservancy/zeronet-conservancy.git" "$zndir" && cd "$zndir"||exit; else cd "$zndir";git pull origin master; fi; cd "$zndir" && pip install -r requirements.txt|grep -v "already satisfied"; echo "Try to run: python3 $(pwd)/zeronet.py"`
+
+(This command can also be used to keep `zeronet-conservancy` up to date)
+
 #### Alternative script
- - after installing general dependencies and cloning repo (as above), run `start-venv.sh` which will create a virtual env for you and install python requirements
+ - after installing general dependencies and cloning repo (as above),
+   run `start-venv.sh` which will create a virtual env for you and
+   install python requirements
  - more convenience scripts to be added soon
 
-### Building under windows os
+### (unofficial) Windows OS build
 
-(this instruction is work-in-progress, please help us test it and improve it!)
+Download and extract .zip archive
+[zeronet-conservancy-0.7.10-unofficial-win64.zip](https://github.com/zeronet-conservancy/zeronet-conservancy/releases/download/v0.7.10/zeronet-conservancy-0.7.10-unofficial-win64.zip)
+
+### Building under Windows OS
+
+(These instructions are work-in-progress, please help us test it and improve it!)
 
 - install python from https://www.python.org/downloads/
 - install some windows compiler suitable for python , this proved to be the most difficult part for me as non-windows user (see here https://wiki.python.org/moin/WindowsCompilers and i'll link more references later)

bootstrap.url

@@ -1 +1 @@
-https://github.com/zeronet-conservancy/zeronet-conservancy/releases/download/v0.7.10/data-default-2023-07-26.zip
+https://github.com/zeronet-conservancy/zeronet-conservancy/releases/download/v0.7.10/data-default-2023-09-03.zip

plugins/Sidebar/ConsolePlugin.py

@@ -14,8 +14,7 @@ class WsLogStreamer(logging.StreamHandler):
         self.ui_websocket = ui_websocket
 
         if filter:
-            if not SafeRe.isSafePattern(filter):
+            SafeRe.guard(filter)
-                raise Exception("Not a safe prex pattern")
             self.filter_re = re.compile(".*" + filter)
         else:
             self.filter_re = None
@@ -55,7 +54,7 @@ class UiWebsocketPlugin(object):
         pos_start = log_file.tell()
         lines = []
         if filter:
-            assert SafeRe.isSafePattern(filter)
+            SafeRe.guard(filter)
             filter_re = re.compile(".*" + filter)
 
         last_match = False

src/Config.py

@@ -256,6 +256,7 @@ class Config(object):
         self.parser.add_argument('--ip_local', help='My local ips', default=ip_local, type=int, metavar='ip', nargs='*')
         self.parser.add_argument('--ip_external', help='Set reported external ip (tested on start if None)', metavar='ip', nargs='*')
         self.parser.add_argument('--offline', help='Disable network communication', action='store_true')
+        self.parser.add_argument('--disable_port_check', help='Disable checking port', action='store_true')
 
         self.parser.add_argument('--disable_udp', help='Disable UDP connections', action='store_true')
         self.parser.add_argument('--proxy', help='Socks proxy address', metavar='ip:port')

src/File/FileServer.py

@@ -152,9 +152,13 @@ class FileServer(ConnectionServer):
         FileRequest = imp.load_source("FileRequest", "src/File/FileRequest.py").FileRequest
 
     def portCheck(self):
-        if config.offline or config.tor == 'always':
+        if config.offline or config.tor == 'always' or config.disable_port_check:
-            msg = "Offline mode" if config.offline else "Tor-only"
+            if config.offline:
-            self.log.info(f'{msg}: port check disabled')
+                self.log.info(f'Offline mode: port check disabled')
+            elif config.tor == 'always':
+                self.log.info('Tor-only mode: port check disabled')
+            else:
+                self.log.info('Port check disabled')
             res = {"ipv4": None, "ipv6": None}
             self.port_opened = res
             return res

src/Test/TestThreadPool.py

@@ -15,11 +15,9 @@ class TestThreadPool:
             @pool.wrap
             def blocker():
                 events.append("S")
-                out = 0
+                time.sleep(0.001)
-                for i in range(10000000):
+                events.append("M")
-                    if i == 3000000:
+                time.sleep(0.001)
-                        events.append("M")
-                    out += 1
                 events.append("D")
                 return out
 
@@ -30,9 +28,6 @@ class TestThreadPool:
 
             assert events == ["S"] * 3 + ["M"] * 3 + ["D"] * 3
 
-            res = blocker()
-            assert res == 10000000
-
     def testLockBlockingSameThread(self):
         lock = ThreadPool.Lock()
 

src/Test/conftest.py

@@ -63,9 +63,6 @@ config.debug = True
 
 os.chdir(os.path.abspath(os.path.dirname(__file__) + "/../.."))  # Set working dir
 
-all_loaded = PluginManager.plugin_manager.loadPlugins()
-assert all_loaded, "Not all plugin loaded successfully"
-
 config.loadPlugins()
 config.parse(parse_config=False)  # Parse again to add plugin configuration options
 

src/Test/pytest.ini

@@ -1,6 +1,6 @@
 [pytest]
 python_files = Test*.py
-addopts = -rsxX -v --durations=6 --no-print-logs --capture=fd
+addopts = -rsxX -v --durations=6 --capture=fd
 markers =
     slow: mark a tests as slow.
     webtest: mark a test as a webtest.

src/Worker/Worker.py

@@ -7,6 +7,8 @@ from Debug import Debug
 from Config import config
 from Content.ContentManager import VerifyError
 
+import traceback
+
 
 class WorkerDownloadError(Exception):
     pass
@@ -119,13 +121,15 @@ class Worker(object):
                 self.manager.log.error("%s: Error writing: %s (%s: %s)" % (self.key, task["inner_path"], type(err), err))
             raise WorkerIOError(str(err))
 
-    def onTaskVerifyFail(self, task, error_message):
+    def onTaskVerifyFail(self, task, error):
         self.num_failed += 1
         if self.manager.started_task_num < 50 or config.verbose:
-            self.manager.log.debug(
+            self.manager.log.info(
                 "%s: Verify failed: %s, error: %s, failed peers: %s, workers: %s" %
-                (self.key, task["inner_path"], error_message, len(task["failed"]), task["workers_num"])
+                (self.key, task["inner_path"], error, len(task["failed"]), task["workers_num"])
             )
+            # traceback.format_
+            self.manager.log.debug(''.join(traceback.format_exception(error)))
         task["failed"].append(self.peer)
         self.peer.hash_failed += 1
         if self.peer.hash_failed >= max(len(self.manager.tasks), 3) or self.peer.connection_error > 10:

src/util/SafeRe.py

@@ -7,7 +7,8 @@ class UnsafePatternError(Exception):
 cached_patterns = {}
 
 
-def isSafePattern(pattern):
+def guard(pattern):
+    '''Checks if pattern is safe and raises exception if it isn't'''
     if len(pattern) > 255:
         raise UnsafePatternError("Pattern too long: %s characters in %s" % (len(pattern), pattern))
 
@@ -15,18 +16,18 @@ def isSafePattern(pattern):
     if unsafe_pattern_match:
         raise UnsafePatternError("Potentially unsafe part of the pattern: %s in %s" % (unsafe_pattern_match.group(0), pattern))
 
-    repetitions = re.findall(r"\.[\*\{\+]", pattern)
+    repetitions1 = re.findall(r"\.[\*\{\+]", pattern)
-    if len(repetitions) >= 10:
+    repetitions2 = re.findall(r"[^(][?]", pattern)
-        raise UnsafePatternError("More than 10 repetitions of %s in %s" % (repetitions[0], pattern))
+    if len(repetitions1) + len(repetitions2) >= 10:
-
+        raise UnsafePatternError("More than 10 repetitions in %s" % pattern)
-    return True
 
 
 def match(pattern, *args, **kwargs):
+    '''Guard for safety, compile, cache and match regexp'''
     cached_pattern = cached_patterns.get(pattern)
     if cached_pattern:
         return cached_pattern.match(*args, **kwargs)
     else:
-        if isSafePattern(pattern):
+        guard(pattern)
-            cached_patterns[pattern] = re.compile(pattern)
+        cached_patterns[pattern] = re.compile(pattern)
-            return cached_patterns[pattern].match(*args, **kwargs)
+        return cached_patterns[pattern].match(*args, **kwargs)