diff --git a/README.md b/README.md index b58fc90a..b0179849 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,34 @@ # ZeroNet -Decentralized websites using Bitcoin crypto and BitTorrent network +Decentralized websites using Bitcoin crypto and the BitTorrent network ## Why? - - We believe in open, free and uncensored network and communication. - - No single point of failure: Site goes on until at least 1 peer serving it. - - No hosting costs: Site served by visitors. - - Impossible to shut down: It's nowhere because it's everywhere. - - Fast and works offline: You can access the site even if your internet is gone. + +* We believe in open, free, and uncensored network and communication. +* No single point of failure: Site remains online so long as at least 1 peer + serving it. +* No hosting costs: Sites are served by visitors. +* Impossible to shut down: It's nowhere because it's everywhere. +* Fast and works offline: You can access the site even if your internet is + unavailable. ## How does it work? - - After starting `zeronet.py` you will be able to visit zeronet sites using http://127.0.0.1:43110/{zeronet_address} (eg. http://127.0.0.1:43110/1EU1tbG9oC1A8jz2ouVwGZyQ5asrNsE4Vr). - - When you visit a new zeronet site, it's trying to find peers using BitTorrent network and download the site files (html, css, js...) from them. - - Each visited sites become also served by You. - - Every site containing a `site.json` which holds all other files sha512 hash and a sign generated using site's private key. - - If the site owner (who has the private key for the site address) modifies the site, then he/she signs the new `content.json` and publish it to the peers. After the peers have verified the `content.json` integrity (using the sign), they download the modified files and publish the new content to other peers. + +* After starting `zeronet.py` you will be able to visit zeronet sites using + `http://127.0.0.1:43110/{zeronet_address}` (eg. + `http://127.0.0.1:43110/1EU1tbG9oC1A8jz2ouVwGZyQ5asrNsE4Vr`). +* When you visit a new zeronet site, it tries to find peers using the BitTorrent + network so it can download the site files (html, css, js...) from them. +* Each visited site becomes also served by you. +* Every site contains a `site.json` which holds all other files in a sha512 hash + and a signature generated using site's private key. +* If the site owner (who has the private key for the site address) modifies the + site, then he/she signs the new `content.json` and publishes it to the peers. + After the peers have verified the `content.json` integrity (using the + signature), they download the modified files and publish the new content to + other peers. ## Screenshot @@ -25,39 +37,45 @@ Decentralized websites using Bitcoin crypto and BitTorrent network ## How to join? -Windows: - - [Install Python 2.7](https://www.python.org/ftp/python/2.7.9/python-2.7.9.msi) - - [Install Python ZeroMQ](http://zeronet.io/files/windows/pyzmq-14.4.1.win32-py2.7.exe) - - [Install Python Greenlet](http://zeronet.io/files/windows/greenlet-0.4.5.win32-py2.7.exe) - - [Install Python Gevent](http://zeronet.io/files/windows/gevent-1.0.1.win32-py2.7.exe) - - [Install Python MsgPack](http://zeronet.io/files/windows/msgpack-python-0.4.2.win32-py2.7.exe) - - start `start.py` -Linux (Debian): - - `apt-get install python-pip` - - `pip install pyzmq` (if it drops a compile error then run `apt-get install python-dev` and try again) - - `pip install gevent` - - `pip install msgpack-python` - - start using `python zeronet.py` +### Windows -Linux (Without root access): - - `wget https://bootstrap.pypa.io/get-pip.py` - - `python get-pip.py --user pyzmq gevent msgpack-python` - - start using `python zeronet.py` +* [Install Python 2.7](https://www.python.org/ftp/python/2.7.9/python-2.7.9.msi) +* [Install Python ZeroMQ](http://zeronet.io/files/windows/pyzmq-14.4.1.win32-py2.7.exe) +* [Install Python Greenlet](http://zeronet.io/files/windows/greenlet-0.4.5.win32-py2.7.exe) +* [Install Python Gevent](http://zeronet.io/files/windows/gevent-1.0.1.win32-py2.7.exe) +* [Install Python MsgPack](http://zeronet.io/files/windows/msgpack-python-0.4.2.win32-py2.7.exe) +* Start `start.py` +### Linux + +#### Debian + +* `apt-get install python-dev python-pip` +* `pip install pyzmq gevent msgpack-python` +* Start with `python zeronet.py` + +#### Without root access + +* `wget https://bootstrap.pypa.io/get-pip.py` +* `python get-pip.py --user pyzmq gevent msgpack-python` +* Start with `python zeronet.py` ## Current limitations - - No torrent-like, file splitting big file support - - Just as anonymous as the bittorrent - - File transactions not compressed or encrypted yet - - No private sites - - You must have an open port to publish new changes - - Timeout problems on slow connections + +* No torrent-like, file splitting for big file support +* No more anonymous than Bittorrent +* File transactions are not compressed or encrypted yet +* No private sites +* You must have an open port to publish new changes +* Timeout problems on slow connections ## How can I create a ZeroNet site? + Shut down zeronet if you are running it already -``` + +```bash $ zeronet.py siteCreate ... - Site private key: 23DKQpzxhbVBrAtvLEc2uvk7DZweh4qL3fn3jpM3LgHDczMK2TtYUq @@ -67,27 +85,35 @@ $ zeronet.py siteCreate $ zeronet.py ... ``` -Congratulations, you are done! Now anyone can access your site using http://localhost:43110/13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2 + +Congratulations, you're finished! Now anyone can access your site using +`http://localhost:43110/13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2` Next steps: [ZeroNet Developer Documentation](https://github.com/HelloZeroNet/ZeroNet/wiki/ZeroNet-Developer-Documentation) ## How can I modify a ZeroNet site? -- Modify files located in data/13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2 directory. After you done: -``` + +* Modify files located in data/13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2 directory. + After you're finished: + +```bash $ zeronet.py siteSign 13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2 - Signing site: 13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2... Private key (input hidden): ``` -- Enter your private key you got when created the site, then: -``` + +* Enter the private key you got when created the site, then: + +```bash $ zeronet.py sitePublish 13DNDkMUExRf9Xa9ogwPKqp7zyHFEqbhC2 ... Site:13DNDk..bhC2 Publishing to 3/10 peers... Site:13DNDk..bhC2 Successfuly published to 3 peers - Serving files.... ``` -- That's it! You successfuly signed and published your modifications. + +* That's it! You've successfully signed and published your modifications. ## If you want to help keep this project alive @@ -97,5 +123,5 @@ Bitcoin: 1QDhxQ6PraUZa21ET5fYUCPgdrwBomnFgX #### Thank you! -- More info, help, changelog, zeronet sites: http://www.reddit.com/r/zeronet/ -- Come, chat with us: [#zeronet @ FreeNode](https://kiwiirc.com/client/irc.freenode.net/zeronet) +* More info, help, changelog, zeronet sites: http://www.reddit.com/r/zeronet/ +* Come, chat with us: [#zeronet @ FreeNode](https://kiwiirc.com/client/irc.freenode.net/zeronet) diff --git a/src/Config.py b/src/Config.py index 99ad9567..2b0b0538 100644 --- a/src/Config.py +++ b/src/Config.py @@ -3,7 +3,7 @@ import ConfigParser class Config(object): def __init__(self): - self.version = "0.2.1" + self.version = "0.2.3" self.parser = self.createArguments() argv = sys.argv[:] # Copy command line arguments argv = self.parseConfig(argv) # Add arguments from config file @@ -19,7 +19,7 @@ class Config(object): def createArguments(self): # Platform specific if sys.platform.startswith("win"): - upnpc = "tools\\upnpc\\upnpc-static.exe" + upnpc = "tools\\upnpc\\upnpc-shared.exe" coffeescript = "type %s | tools\\coffee\\coffee.cmd" else: upnpc = None diff --git a/src/File/FileServer.py b/src/File/FileServer.py index 4f2c2b6c..5ae5b732 100644 --- a/src/File/FileServer.py +++ b/src/File/FileServer.py @@ -18,6 +18,7 @@ class FileServer: else: self.port_opened = None # Is file server opened on router self.sites = SiteManager.list() + self.running = True # Handle request to fileserver @@ -168,17 +169,26 @@ class FileServer: if check_sites: # Open port, Update sites, Check files integrity gevent.spawn(self.checkSites) - gevent.spawn(self.announceSites) - gevent.spawn(self.wakeupWatcher) + thread_announce_sites = gevent.spawn(self.announceSites) + thread_wakeup_watcher = gevent.spawn(self.wakeupWatcher) - while True: + while self.running: try: ret = {} req = msgpack.unpackb(socket.recv()) self.handleRequest(req) except Exception, err: self.log.error(err) - self.socket.send(msgpack.packb({"error": "%s" % Debug.formatException(err)}, use_bin_type=True)) + if self.running: self.socket.send(msgpack.packb({"error": "%s" % Debug.formatException(err)}, use_bin_type=True)) if config.debug: # Raise exception import sys - sys.excepthook(*sys.exc_info()) + sys.modules["src.main"].DebugHook.handleError() + thread_wakeup_watcher.kill(exception=Debug.Notify("Stopping FileServer")) + thread_announce_sites.kill(exception=Debug.Notify("Stopping FileServer")) + self.log.debug("Stopped.") + + + def stop(self): + self.running = False + self.socket.close() + diff --git a/src/Site/Site.py b/src/Site/Site.py index 21cf7105..590ed856 100644 --- a/src/Site/Site.py +++ b/src/Site/Site.py @@ -194,9 +194,6 @@ class Site: self.log.info("[OK] %s: %s" % (peer.key, result["ok"])) else: self.log.info("[ERROR] %s: %s" % (peer.key, result)) - - - # Update content.json on peers diff --git a/src/Ui/UiServer.py b/src/Ui/UiServer.py index cc15d8e2..0592a019 100644 --- a/src/Ui/UiServer.py +++ b/src/Ui/UiServer.py @@ -11,20 +11,25 @@ from Debug import Debug # Skip websocket handler if not necessary class UiWSGIHandler(WSGIHandler): def __init__(self, *args, **kwargs): + self.server = args[2] super(UiWSGIHandler, self).__init__(*args, **kwargs) - self.ws_handler = WebSocketHandler(*args, **kwargs) + self.args = args + self.kwargs = kwargs def run_application(self): + self.server.sockets[self.client_address] = self.socket if "HTTP_UPGRADE" in self.environ: # Websocket request - self.ws_handler.__dict__ = self.__dict__ # Match class variables - self.ws_handler.run_application() + ws_handler = WebSocketHandler(*self.args, **self.kwargs) + ws_handler.__dict__ = self.__dict__ # Match class variables + ws_handler.run_application() else: # Standard HTTP request #print self.application.__class__.__name__ try: return super(UiWSGIHandler, self).run_application() except Exception, err: logging.debug("UiWSGIHandler error: %s" % err) + del self.server.sockets[self.client_address] class UiServer: @@ -89,4 +94,27 @@ class UiServer: browser = webbrowser.get(config.open_browser) browser.open("http://%s:%s" % (config.ui_ip, config.ui_port), new=2) - WSGIServer((self.ip, self.port), handler, handler_class=UiWSGIHandler, log=self.log).serve_forever() + self.server = WSGIServer((self.ip, self.port), handler, handler_class=UiWSGIHandler, log=self.log) + self.server.sockets = {} + self.server.serve_forever() + self.log.debug("Stopped.") + + def stop(self): + # Close WS sockets + for client in self.server.clients.values(): + client.ws.close() + # Close http sockets + sock_closed = 0 + for sock in self.server.sockets.values(): + try: + sock._sock.close() + sock.close() + sock_closed += 1 + except Exception, err: + pass + self.log.debug("Socket closed: %s" % sock_closed) + + self.server.socket.close() + self.server.stop() + time.sleep(1) + diff --git a/src/Ui/UiWebsocket.py b/src/Ui/UiWebsocket.py index 38614a53..dec12fb5 100644 --- a/src/Ui/UiWebsocket.py +++ b/src/Ui/UiWebsocket.py @@ -121,6 +121,8 @@ class UiWebsocket: func = self.actionSiteSetLimit elif cmd == "channelJoinAllsite" and "ADMIN" in permissions: func = self.actionChannelJoinAllsite + elif cmd == "serverUpdate" and "ADMIN" in permissions: + func = self.actionServerUpdate # Unknown command else: self.response(req["id"], "Unknown command: %s" % cmd) @@ -361,3 +363,12 @@ class UiWebsocket: self.site.saveSettings() self.response(to, "Site size limit changed to %sMB" % size_limit) self.site.download() + + + def actionServerUpdate(self, to): + import sys + self.cmd("updating") + sys.modules["src.main"].update_after_shutdown = True + sys.modules["src.main"].file_server.stop() + sys.modules["src.main"].ui_server.stop() + diff --git a/src/Ui/media/Wrapper.coffee b/src/Ui/media/Wrapper.coffee index 4935b141..2b87f6c1 100644 --- a/src/Ui/media/Wrapper.coffee +++ b/src/Ui/media/Wrapper.coffee @@ -46,6 +46,9 @@ class Wrapper @sendInner message # Pass to inner frame if message.params.address == window.address # Current page @setSiteInfo message.params + else if cmd == "updating" # Close connection + @ws.ws.close() + @ws.onCloseWebsocket(null, 4000) else @sendInner message # Pass message to inner frame @@ -69,6 +72,10 @@ class Wrapper @actionWrapperPrompt(message) else if cmd == "wrapperSetViewport" # Set the viewport @actionSetViewport(message) + else if cmd == "wrapperGetLocalStorage" + @actionGetLocalStorage(message) + else if cmd == "wrapperSetLocalStorage" + @actionSetLocalStorage(message) else # Send to websocket if message.id < 1000000 @ws.send(message) # Pass message to websocket @@ -126,6 +133,16 @@ class Wrapper $('').attr("content", @toHtmlSafe message.params).appendTo("head") + actionGetLocalStorage: (message) -> + data = localStorage.getItem "site.#{window.address}" + if data then data = JSON.parse(data) + @sendInner {"cmd": "response", "to": message.id, "result": data} + + + actionSetLocalStorage: (message) -> + back = localStorage.setItem "site.#{window.address}", JSON.stringify(message.params) + + # EOF actions @@ -152,7 +169,7 @@ class Wrapper @wrapperWsInited = false setTimeout (=> # Wait a bit, maybe its page closing @sendInner {"cmd": "wrapperClosedWebsocket"} # Send to inner frame - if e.code == 1000 # Server error please reload page + if e and e.code == 1000 and e.wasClean == false # Server error please reload page @ws_error = @notifications.add("connection", "error", "UiServer Websocket error, please reload the page.") else if not @ws_error @ws_error = @notifications.add("connection", "error", "Connection with UiServer Websocket was lost. Reconnecting...") diff --git a/src/Ui/media/all.js b/src/Ui/media/all.js index c29fd146..401459dd 100644 --- a/src/Ui/media/all.js +++ b/src/Ui/media/all.js @@ -120,17 +120,20 @@ } }; - ZeroWebsocket.prototype.onCloseWebsocket = function(e) { + ZeroWebsocket.prototype.onCloseWebsocket = function(e, reconnect) { + if (reconnect == null) { + reconnect = 10000; + } this.log("Closed", e); - if (e.code === 1000) { - this.log("Server error, please reload the page"); + if (e && e.code === 1000 && e.wasClean === false) { + this.log("Server error, please reload the page", e.wasClean); } else { setTimeout(((function(_this) { return function() { _this.log("Reconnecting..."); return _this.connect(); }; - })(this)), 10000); + })(this)), reconnect); } if (this.onClose != null) { return this.onClose(e); @@ -780,6 +783,9 @@ jQuery.extend( jQuery.easing, if (message.params.address === window.address) { return this.setSiteInfo(message.params); } + } else if (cmd === "updating") { + this.ws.ws.close(); + return this.ws.onCloseWebsocket(null, 4000); } else { return this.sendInner(message); } @@ -807,6 +813,10 @@ jQuery.extend( jQuery.easing, return this.actionWrapperPrompt(message); } else if (cmd === "wrapperSetViewport") { return this.actionSetViewport(message); + } else if (cmd === "wrapperGetLocalStorage") { + return this.actionGetLocalStorage(message); + } else if (cmd === "wrapperSetLocalStorage") { + return this.actionSetLocalStorage(message); } else { if (message.id < 1000000) { return this.ws.send(message); @@ -891,6 +901,24 @@ jQuery.extend( jQuery.easing, } }; + Wrapper.prototype.actionGetLocalStorage = function(message) { + var data; + data = localStorage.getItem("site." + window.address); + if (data) { + data = JSON.parse(data); + } + return this.sendInner({ + "cmd": "response", + "to": message.id, + "result": data + }); + }; + + Wrapper.prototype.actionSetLocalStorage = function(message) { + var back; + return back = localStorage.setItem("site." + window.address, JSON.stringify(message.params)); + }; + Wrapper.prototype.onOpenWebsocket = function(e) { this.ws.cmd("channelJoin", { "channel": "siteChanged" @@ -925,7 +953,7 @@ jQuery.extend( jQuery.easing, _this.sendInner({ "cmd": "wrapperClosedWebsocket" }); - if (e.code === 1000) { + if (e && e.code === 1000 && e.wasClean === false) { return _this.ws_error = _this.notifications.add("connection", "error", "UiServer Websocket error, please reload the page."); } else if (!_this.ws_error) { return _this.ws_error = _this.notifications.add("connection", "error", "Connection with UiServer Websocket was lost. Reconnecting..."); diff --git a/src/Ui/media/lib/ZeroWebsocket.coffee b/src/Ui/media/lib/ZeroWebsocket.coffee index daa5228f..eebafa0a 100644 --- a/src/Ui/media/lib/ZeroWebsocket.coffee +++ b/src/Ui/media/lib/ZeroWebsocket.coffee @@ -66,15 +66,15 @@ class ZeroWebsocket if @onError? then @onError(e) - onCloseWebsocket: (e) => + onCloseWebsocket: (e, reconnect=10000) => @log "Closed", e - if e.code == 1000 - @log "Server error, please reload the page" + if e and e.code == 1000 and e.wasClean == false + @log "Server error, please reload the page", e.wasClean else # Connection error setTimeout (=> @log "Reconnecting..." @connect() - ), 10000 + ), reconnect if @onClose? then @onClose(e) diff --git a/src/User/User.py b/src/User/User.py index 46642515..14b7072e 100644 --- a/src/User/User.py +++ b/src/User/User.py @@ -25,9 +25,9 @@ class User: self.log.debug("Saved") - # Get BIP32 address from site address - # Return: BIP32 auth address - def getAuthAddress(self, address): + # Get user site data + # Return: {"auth_address": "xxx", "auth_privatekey": "xxx"} + def getSiteData(self, address): if not address in self.sites: # Genreate new BIP32 child key based on site address s = time.time() address_id = int(address.encode("hex"), 16) # Convert site address to int @@ -38,12 +38,17 @@ class User: } self.save() self.log.debug("Added new site: %s in %.3fs" % (address, time.time()-s)) + return self.sites[address] - return self.sites[address]["auth_address"] + + # Get BIP32 address from site address + # Return: BIP32 auth address + def getAuthAddress(self, address): + return self.getSiteData(address)["auth_address"] def getAuthPrivatekey(self, address): - return self.sites[address]["auth_privatekey"] + return self.getSiteData(address)["auth_privatekey"] @@ -51,3 +56,4 @@ class User: def setData(self, data): for key, val in data.items(): setattr(self, key, val) + diff --git a/src/User/UserManager.py b/src/User/UserManager.py index e4983c45..2926027c 100644 --- a/src/User/UserManager.py +++ b/src/User/UserManager.py @@ -60,6 +60,6 @@ def getCurrent(): def reload(): import imp global users, User - users.clear() # Remove all items User = imp.load_source("User", "src/User/User.py").User # Reload source + users.clear() # Remove all items load() diff --git a/src/Worker/WorkerManager.py b/src/Worker/WorkerManager.py index 7c970639..e5bf3915 100644 --- a/src/Worker/WorkerManager.py +++ b/src/Worker/WorkerManager.py @@ -1,5 +1,5 @@ from Worker import Worker -import gevent, time, logging +import gevent, time, logging, random MAX_WORKERS = 10 @@ -87,10 +87,12 @@ class WorkerManager: def startWorkers(self, peers=None): if len(self.workers) >= MAX_WORKERS and not peers: return False # Workers number already maxed if not self.tasks: return False # No task for workers - for key, peer in self.site.peers.iteritems(): # One worker for every peer + peers = self.site.peers.values() + random.shuffle(peers) + for peer in peers: # One worker for every peer if peers and peer not in peers: continue # If peers definied and peer not valid worker = self.addWorker(peer) - if worker: self.log.debug("Added worker: %s, workers: %s/%s" % (key, len(self.workers), MAX_WORKERS)) + if worker: self.log.debug("Added worker: %s, workers: %s/%s" % (peer.key, len(self.workers), MAX_WORKERS)) # Stop all worker diff --git a/src/main.py b/src/main.py index 03deb90e..f1385eb2 100644 --- a/src/main.py +++ b/src/main.py @@ -1,4 +1,5 @@ import os, sys +update_after_shutdown = False sys.path.insert(0, os.path.dirname(__file__)) # Imports relative to main.py # Create necessary files and dirs @@ -43,7 +44,6 @@ else: import gevent import time - logging.debug("Starting... %s" % config) # Starts here when running zeronet.py @@ -56,6 +56,7 @@ def start(): # Start serving UiServer and PeerServer def main(): + global ui_server, file_server from File import FileServer from Ui import UiServer logging.info("Creating UiServer....") diff --git a/update.py b/update.py new file mode 100644 index 00000000..e27cda45 --- /dev/null +++ b/update.py @@ -0,0 +1,39 @@ +from gevent import monkey; monkey.patch_all() +import urllib, zipfile, os, ssl, httplib, socket +import cStringIO as StringIO + +def update(): + # Gevent https bug workaround (https://github.com/gevent/gevent/issues/477) + reload(socket) + reload(httplib) + reload(ssl) + + print "Downloading.", + file = urllib.urlopen("https://github.com/HelloZeroNet/ZeroNet/archive/master.zip") + data = StringIO.StringIO() + while True: + buff = file.read(1024*16) + if not buff: break + data.write(buff) + print ".", + + print "Extracting...", + zip = zipfile.ZipFile(data) + for inner_path in zip.namelist(): + print ".", + dest_path = inner_path.replace("ZeroNet-master/", "") + if not dest_path: continue + + dest_dir = os.path.dirname(dest_path) + if dest_dir and not os.path.isdir(dest_dir): + os.makedirs(dest_dir) + + if dest_dir != dest_path.strip("/"): + data = zip.read(inner_path) + open(dest_path, 'wb').write(data) + + print "Done." + + +if __name__ == "__main__": + update() \ No newline at end of file diff --git a/zeronet.py b/zeronet.py index f8a83fa5..e8156209 100644 --- a/zeronet.py +++ b/zeronet.py @@ -1,13 +1,37 @@ #!/usr/bin/env python def main(): - try: - from src import main - main.start() - except Exception, err: # Prevent closing - import traceback - traceback.print_exc() - raw_input("-- Error happened, press enter to close --") + try: + from src import main + main.start() + if main.update_after_shutdown: # Updater + import update, sys, os, gc + # Update + update.update() + + # Close log files + logger = sys.modules["src.main"].logging.getLogger() + + for handler in logger.handlers[:]: + handler.flush() + handler.close() + logger.removeHandler(handler) + + except Exception, err: # Prevent closing + import traceback + traceback.print_exc() + raw_input("-- Error happened, press enter to close --") + + if main.update_after_shutdown: # Updater + # Restart + gc.collect() # Garbage collect + print "Restarting..." + args = sys.argv[:] + args.insert(0, sys.executable) + if sys.platform == 'win32': + args = ['"%s"' % arg for arg in args] + os.execv(sys.executable, args) + print "Bye." if __name__ == '__main__': - main() + main()