Commit graph

330 commits

Author SHA1 Message Date
ZeroNet
0e2f742b7e Merge pull request #842 from mishfit/refactorListAndWalk
Rename SiteStorage.list to SiteStorage.walk
2017-03-04 12:39:06 +01:00
shortcutme
4e846774b4
Log formatting 2017-02-27 00:13:30 +01:00
shortcutme
a6be8ddc2d
Connectable peers ordered 2017-02-27 00:13:18 +01:00
shortcutme
b65670b9e4
Don't remove peers with atleast 5 sites 2017-02-27 00:12:55 +01:00
shortcutme
ea635d4630
Only totally connected peers counted to peer cleanup 2017-02-27 00:12:36 +01:00
shortcutme
2e9737c149
Faster and better getConnectedpeers 2017-02-27 00:11:57 +01:00
shortcutme
0f8de2b7c1
Use latest added peers and use reputation for listModifications and publishing 2017-02-27 00:11:45 +01:00
shortcutme
2188c5d9fa
Use getRecentPeers for publishing 2017-02-27 00:11:02 +01:00
shortcutme
b5d879c1cd
Start site update check on site re-connect 2017-02-27 00:10:27 +01:00
shortcutme
fd365a19f7
Use getRecentPeers for listModification 2017-02-27 00:09:59 +01:00
shortcutme
adbaadae81
Allow 100 sec timeout for listModification 2017-02-27 00:09:41 +01:00
shortcutme
09ad4133fb
20 sec timeout for listModified 2017-02-27 00:08:46 +01:00
shortcutme
bad2a79f05
Reason for connection closing 2017-02-27 00:02:24 +01:00
mishfit
656c818211 rename SiteStorage.list to SiteStorage.walk
also, add SiteStorage.list as non-recursive
directory listing (`os.listdir`)
2017-02-24 21:58:46 -07:00
shortcutme
26bad8d05b
Fix needConnections bug 2017-02-24 01:16:22 +01:00
shortcutme
4aff146812
Log giving up on a file 2017-02-22 11:29:35 +01:00
shortcutme
221aa16433
Rev1929, Fix file filed on not bad files 2017-02-19 12:32:41 +01:00
shortcutme
63b609f725
Give up files after 30 tries 2017-02-19 00:52:31 +01:00
shortcutme
2eef0bded1
Only increment bad_file if internet connection is active 2017-02-19 00:52:20 +01:00
shortcutme
cfad85b30e
Log file object on db load 2017-02-13 16:12:56 +01:00
shortcutme
6149f5c6ee
Download everything on add 2017-02-11 18:26:37 +01:00
shortcutme
56efa02820
GetDbFiles return inner path 2017-02-09 01:57:12 +01:00
shortcutme
f3c71f3e6c
Move SiteStorage database updating to separate function 2017-02-09 01:56:20 +01:00
shortcutme
66e2bfd76b
Reset bad_files on full update 2017-01-29 19:03:22 +01:00
shortcutme
db7dad6731
Don't delete content.json files from bad_files on update 2017-01-29 19:03:08 +01:00
shortcutme
2b389e4e85
Support since parameter for siteUpdate 2017-01-29 19:02:08 +01:00
shortcutme
a0c3d7f8a6
Rev1833, Fix utf8 system paths 2017-01-22 21:22:53 +01:00
shortcutme
2dc981496b
Rev1821, Show progress bar on publishing 2017-01-16 13:41:57 +01:00
shortcutme
4e96ddfb9e
Allow .. in data_dir path 2017-01-11 13:14:02 +01:00
shortcutme
d9f8e83429 Safe disk write for content.json files 2017-01-05 02:26:44 +01:00
shortcutme
c94e73a029 Skip sites with broken content.json on startup 2017-01-05 02:26:05 +01:00
shortcutme
f89f351b3c Dont copy content.json-default file 2016-12-14 00:43:08 +01:00
shortcutme
34ab2bad49 Clone inner path dest fix 2016-12-14 00:42:52 +01:00
shortcutme
62501d00b1 Use content.json-default if exists 2016-12-14 00:42:31 +01:00
shortcutme
10772a704e Root path parameter for cloning 2016-12-08 17:14:55 +01:00
shortcutme
20ed04af84 Also delete orphan sites from site_id cache 2016-12-04 18:54:43 +01:00
shortcutme
2e08d68d87 Remove content.json files from bad file list 2016-11-16 11:19:08 +01:00
shortcutme
2cdde7dd46 Retry file deletion on error 2016-11-07 23:47:09 +01:00
shortcutme
9765afc286 Modify checkfiles to use optionalRemove, optionalDownloaded and isDownloadable functions 2016-11-07 23:46:54 +01:00
shortcutme
4890c4cb2f Fix big site download 2016-11-07 23:43:08 +01:00
shortcutme
73b3c8db55 Use site object on delete from contentdb 2016-11-07 23:42:28 +01:00
shortcutme
713a2f88a9 Query and send more hashfield 2016-11-07 23:42:13 +01:00
shortcutme
30a2116abf Only announce if trackers 2016-11-07 23:42:00 +01:00
shortcutme
6acc81cb50 Retry bad files in separate thread 2016-11-07 23:41:25 +01:00
shortcutme
af571deb16 Peer cleanup ttl based on peer numbers 2016-11-07 23:40:04 +01:00
shortcutme
e6a82d4c64 Pooled file downloads to handle large ammount of files better 2016-11-07 23:39:45 +01:00
shortcutme
83b3dc8fbe Autodownload optional pattern check to separate function to allow plugins 2016-11-07 23:39:04 +01:00
shortcutme
d1ad7e0850 Stats for total optional files and downloaded optional files in a site 2016-11-07 23:36:04 +01:00
shortcutme
0418a3e6de Only save sites.json if fully loaded to avoid data loss 2016-11-07 22:40:12 +01:00
shortcutme
8f3dfd7435 Fix site deletion errors on startup 2016-11-07 22:38:01 +01:00
shortcutme
d6da0c421c Better json compressing 2016-10-02 14:23:05 +02:00
shortcutme
bb23bc7b3c Send diff to second batch of peers 2016-10-02 14:22:32 +02:00
shortcutme
d760a71b76 Rev1529, Fix json write bug with [] characters in value 2016-09-29 13:01:02 +02:00
shortcutme
117a82c059 Delete bad_files without size 2016-09-28 20:35:53 +02:00
shortcutme
67c9921ace Only add non-connected peers to publish if necessary 2016-09-17 21:01:16 +02:00
shortcutme
4690a111bf Push updates to newer clients first 2016-09-17 21:00:40 +02:00
shortcutme
65e5d05a9e Delete cleanup bad files without file info on update 2016-09-17 21:00:20 +02:00
shortcutme
9fd19bf382 Fix content deletion 2016-09-14 10:52:41 +02:00
shortcutme
b2b91c9aff Remove save error message when no sites loaded yet 2016-09-07 17:41:55 +02:00
shortcutme
a6c4a77e13 Rev1510, Fix site save without loaded SiteManager 2016-09-06 14:52:55 +02:00
shortcutme
d9bbb20743 Rev1509, Fix site delete 2016-09-06 10:16:09 +02:00
shortcutme
dae5cd3969 Fix startup with 1000+ sites 2016-09-05 13:57:12 +02:00
shortcutme
59205088d2 Move site deletion to dedicted function 2016-09-05 13:55:51 +02:00
shortcutme
37a3b4678d Fix user file pre-content.json downloading 2016-09-04 18:01:09 +02:00
shortcutme
365f0cd646 Reset files on cloning 2016-09-04 18:00:51 +02:00
shortcutme
f698ec2502 Also delete sites from content.db 2016-09-04 17:59:47 +02:00
shortcutme
a1f0c20afc Rename checkFiles to updateBadFiles 2016-09-04 17:59:29 +02:00
shortcutme
b0a85b0669 Dont check every file on update 2016-09-04 17:58:18 +02:00
shortcutme
9528d333bd Retry failed files on download end 2016-09-04 17:57:44 +02:00
shortcutme
4437a99330 Use pool to download large ammount of content.json files 2016-09-04 17:57:32 +02:00
shortcutme
8f158cbc8f Make listmodified query use content.db cache 2016-09-04 17:56:33 +02:00
shortcutme
135a9b8f5c Fix content.json retry 2016-09-04 17:54:57 +02:00
shortcutme
1dd488ef89 Remove deleted sites from content.db 2016-09-04 17:52:14 +02:00
shortcutme
a614545467 Faster site settings load 2016-09-04 17:51:55 +02:00
shortcutme
3724e0f376 Save sites.json on exit 2016-09-04 17:51:20 +02:00
shortcutme
1bd6e2f493 Cache bad_files and optional hashfield in sites.json 2016-09-04 17:43:30 +02:00
shortcutme
ecb5885dba Cache file sizes and modification date in sqlite db 2016-09-04 17:41:04 +02:00
shortcutme
709dc854e7 Add site to site manager if required 2016-08-29 11:17:19 +02:00
shortcutme
887a07c3fb Rev1427, Skip invalid files on dbRebuild 2016-08-27 11:01:41 +02:00
shortcutme
d76c1ca98e Also cleanup connections for smaller sites 2016-08-26 11:42:23 +02:00
shortcutme
7edb784305 Fix user file updating on owned sites 2016-08-26 11:40:22 +02:00
ZeroNet
3c7a9f0f82 Merge pull request #537 from megfault/patch-1
Fix typos in log message
2016-08-17 11:32:25 +02:00
shortcutme
274414a54b Give up user's content.json after 5 retrys 2016-08-16 19:56:18 +02:00
shortcutme
b40b3e2d85 Fix config size_limit parsing 2016-08-16 19:55:55 +02:00
Ana Barroso
c9660ed20e Fix typos in log message 2016-08-16 18:14:02 +02:00
shortcutme
92e4dd64d1 Dont recalculate site size on update 2016-08-15 13:56:17 +02:00
ZeroNet
67f75c4bef Skip archived user files 2016-08-10 12:42:29 +02:00
ZeroNet
0275954fb7 Tracker log total peers 2016-08-10 12:42:13 +02:00
ZeroNet
44dc4c41c9 Wait for user's content.json on file request 2016-08-10 12:41:35 +02:00
ZeroNet
e5e76a4ef5 Do not reload the whole site on update command 2016-08-10 12:41:04 +02:00
ZeroNet
021a084796 Peers query optimize 2016-08-10 12:40:17 +02:00
ZeroNet
1ec00947ad Save site on content.json arrive 2016-08-10 12:36:20 +02:00
ZeroNet
f38f4c3758 Use SiteManager to save site 2016-08-10 12:35:40 +02:00
ZeroNet
2852bbc657 Remove unnecessary site savings 2016-08-10 12:35:17 +02:00
ZeroNet
0d565dc64d Better site settings saving 2016-08-10 12:34:28 +02:00
ZeroNet
1f3a4f0e2f Remove orphan sites from sites.json 2016-08-10 12:33:34 +02:00
ZeroNet
53249f4f07 Add own logger to SiteManager 2016-08-10 12:32:28 +02:00
ZeroNet
0569697d98 Update database on delete 2016-08-10 12:31:54 +02:00
ZeroNet
acf24aab6f Allow to pass file object on file updated 2016-08-10 12:31:30 +02:00
ZeroNet
3ff75c6602 Also trigger onUpdated on file delete 2016-08-10 12:30:35 +02:00
ZeroNet
0be6cb6602 Move database files listing to separate funcion 2016-08-10 12:30:04 +02:00
ZeroNet
9def3b58eb Allow plugins on SiteStorage 2016-08-10 12:28:57 +02:00
HelloZeroNet
36e969ed1f Fix siteSign without connection server 2016-05-05 12:11:35 +02:00
HelloZeroNet
639a834658 Keep minimum 5 connections per site 2016-04-29 02:58:00 +02:00
HelloZeroNet
93e598cc59 Rev1275, Fix dictionary changed size during iteration error 2016-04-26 10:46:53 +02:00
HelloZeroNet
bf384922a6 Use iteritems to save memory 2016-04-25 02:26:29 +02:00
HelloZeroNet
e53f56ed30 Cleanup connections with less sites first 2016-04-25 02:25:11 +02:00
HelloZeroNet
73c69d54bc Dont save settings on every announce 2016-04-20 23:32:43 +02:00
HelloZeroNet
23389dc31e Formatting 2016-04-20 23:30:25 +02:00
HelloZeroNet
38057c2e03 Log site config save time 2016-04-20 23:29:26 +02:00
HelloZeroNet
cd52a61c4c Save sites.json every 10 minute 2016-04-20 23:28:58 +02:00
HelloZeroNet
55d76c8c5c Faster SiteStorage.getSize 2016-04-20 23:28:21 +02:00
HelloZeroNet
cd5cdd580a Re-publish modifications to less peers 2016-04-18 00:42:26 +02:00
HelloZeroNet
3bdde14821 Log patch size 2016-04-10 23:12:34 +02:00
HelloZeroNet
bbf1bc163d Extra debug messages, cleanup unused lines 2016-04-10 13:25:11 +02:00
HelloZeroNet
561bd80aa3 Fallback to simple file copy if rename failed 2016-04-09 19:43:44 +02:00
HelloZeroNet
6388973292 Retry file rename later if resource not ready yet 2016-04-07 10:36:50 +02:00
HelloZeroNet
9c1ea71d2f Try to use diff before downloading the file 2016-04-06 14:01:03 +02:00
HelloZeroNet
7ee8eead23 Remove duplicate peers from publish queue 2016-04-06 14:00:40 +02:00
HelloZeroNet
82b960c641 Pass diffs to other users 2016-04-06 13:59:53 +02:00
HelloZeroNet
ab1aaea7f5 Calculate and send diffs on publish 2016-04-06 13:57:19 +02:00
HelloZeroNet
84e78859dc SiteStorage.rename function 2016-04-06 13:50:20 +02:00
HelloZeroNet
8f39ed2467 Cloned site updating 2016-04-06 13:50:05 +02:00
HelloZeroNet
ec70c47aa0 Optimize path modifying functions 2016-04-06 13:48:13 +02:00
HelloZeroNet
5a34448ba1 Less verbose logging 2016-04-06 13:39:17 +02:00
HelloZeroNet
c8b949cef5 Save last received update time from peer to avoid send the same update on publish 2016-03-30 23:05:43 +02:00
HelloZeroNet
09197ef71e Dont publish twice 2016-03-26 01:33:37 +01:00
HelloZeroNet
6d222b6ed7 Cleanup empty dirs 2016-03-26 00:22:27 +01:00
HelloZeroNet
437a9b79a8 Less verbose site downloading 2016-03-19 18:06:07 +01:00
HelloZeroNet
ae092dc757 Disable DB updating command line argument 2016-03-19 18:05:49 +01:00
HelloZeroNet
a692640e46 Ability to skip check files on update 2016-03-18 19:19:16 +01:00
HelloZeroNet
2c4f582437 Mark site as failed updating if query modifications failed 2016-03-18 19:18:59 +01:00
HelloZeroNet
cd2aa033b4 Allow the site owner to modify banned users file 2016-03-17 01:56:25 +01:00
HelloZeroNet
77ac1ae796 Keep connections opened regardless prot opened status 2016-03-16 23:24:57 +01:00
HelloZeroNet
6187abe56f Remove active peers discrimination in peer cleanup 2016-03-16 21:11:34 +01:00
HelloZeroNet
6f4b896bce Always publish to 5 peers and remove discrimination of active peers 2016-03-16 21:11:04 +01:00
HelloZeroNet
f372fe371a Context switch for every 50 checked files 2016-03-16 00:40:19 +01:00
HelloZeroNet
1cf0e0eb21 Always prefer publishing to connected peers 2016-03-16 00:40:01 +01:00
HelloZeroNet
11e09adda7 Longer timeout for non connected peers publishing 2016-03-16 00:39:09 +01:00
HelloZeroNet
f735862977 Switch back to 3 thread publishing on large sites 2016-03-12 23:07:41 +01:00
HelloZeroNet
6ff91cd2e8 Reduce hangs on file validation process 2016-03-12 23:07:06 +01:00
HelloZeroNet
0897154584 Rev966, New Greenlet debug hook to support Gevent 1.1.0, Publish to 8 peers on external publish request, Dont try to download banned user files, Check user files even if its own site 2016-03-11 12:39:39 +01:00
HelloZeroNet
48db062b49 Rev966, Display user quota in sidebar, Fix non-root content.json publishing in sidebar, Publish to same ammount of passive peers as limit, Fix site address case bug 2016-03-09 15:30:04 +01:00
HelloZeroNet
e891a10e54 Rev957, Sidebar displays onion peers in graph, Sidebar display bad file retry number, Sidebar site Update/Pause/Delete, Ratelimit sidebar update, Encoded typo, Fix onion findHashId, More retry for bad files, Log file path errors, Testcase for self findhashIds, Testcase for Tor findHashId, Better Tor version parse, UiWebsocket callback on update/pause/resume/delete, Skip invalid postMessage messages 2016-03-09 00:48:57 +01:00
HelloZeroNet
dba42f5f5e Rev943, Less verbose logging by default, Load ZeroBundle SSL lib, Log loaded SSL lib 2016-03-06 00:55:50 +01:00
HelloZeroNet
08005499fb Rev936, Fix Bootstrapper test without tormanager, Bootstrapper return latest announced peers, Only create 3 threads for passive peers 2016-03-04 19:59:59 +01:00
Barrabin Fc. ⚑
58ce98e0fc Merge branch 'master' of github.com:HelloZeroNet/ZeroNet 2016-03-03 20:56:43 -03:00
Barrabin Fc. ⚑
99aecedd0b print syntax error when json dbschema.json failed 2016-03-03 20:47:26 -03:00
Barrabin Fc. ⚑
aee83605ab Added exception handling for invalid dbschema.json files 2016-03-03 20:40:26 -03:00
HelloZeroNet
3f6f273fb1 Rev932, Skip news event from future, Disable siteSetOwned and setAutodownloadoptional when multiuser plugin enabled, Fix sidebar double click handlers, Log peer sent commands, Send modification to less peer if have enought peers, Close peers if more than 10 per site 2016-03-03 21:12:16 +01:00