Commit graph

454 commits

Author SHA1 Message Date
shortcutme
2575476086
Some trackers does not accept port 0, so we send port 1 as not-connectable 2018-02-10 01:08:31 +01:00
shortcutme
211d68f590
Only parse dict result from udp tracker 2018-02-10 01:08:20 +01:00
shortcutme
3e51b888cb
Add left bytes to request to improve tracker compatibility 2018-02-10 01:04:23 +01:00
shortcutme
ff8f41c1eb
Request less peers in update mode 2018-02-10 01:03:49 +01:00
shortcutme
a6e67fa9c3
Generalized tracker result log format for pex 2018-02-08 18:06:45 +01:00
shortcutme
b6a401d697
Get loading status based on property 2018-02-08 18:06:19 +01:00
shortcutme
23be1d18a4
Store site list change time 2018-02-08 18:05:50 +01:00
shortcutme
d721d459eb
Sites always a dict to keep the reference 2018-02-08 18:05:23 +01:00
shortcutme
e1dd0f0979
Only recover sites from sites.json that had peers 2018-02-08 18:04:41 +01:00
shortcutme
866947c445
Fix symlinked data site deletion 2018-02-08 18:03:55 +01:00
shortcutme
64b4789bc1
Generalize tracker logging 2018-02-08 17:54:09 +01:00
shortcutme
13e651c822
Store and prefer recent peers from trackers and local peer discovery 2018-02-08 17:53:17 +01:00
shortcutme
7d3fd6f5b6
Only add recent peers if there is more 2018-02-08 17:51:44 +01:00
shortcutme
0c91fd396e
De-duplicate peers before publishing 2018-02-08 17:51:10 +01:00
shortcutme
1827e1e033
Force announce on startup 2018-02-08 17:50:40 +01:00
shortcutme
771dce4c8c
Pre-generate site address hash 2018-02-08 17:50:23 +01:00
shortcutme
c2edbb30b5
Don't send private ip addresses on pex 2018-01-30 13:58:01 +01:00
shortcutme
48e8b956d2
Only add myself to connect blacklist if running other cli action 2018-01-28 16:42:23 +01:00
shortcutme
005358f4bf
Use separate db connection for rebuilding 2018-01-19 02:31:46 +01:00
shortcutme
ea1cd63929
Log hashfield change time on send 2018-01-19 02:30:52 +01:00
shortcutme
c8ac4c0278
Update content update time after successful listmodifications 2018-01-19 02:29:18 +01:00
shortcutme
49a332724e
No connected peers if no connection server 2018-01-19 02:27:40 +01:00
shortcutme
a21c417c0b
Better announce logging 2018-01-19 02:27:19 +01:00
shortcutme
ccd002b97f
Rev3177, Import missing SafeRe module 2017-12-20 23:35:49 +01:00
shortcutme
c7d067ea3c
Rev3176, Skip listing ignored directories on signing 2017-12-20 23:25:25 +01:00
shortcutme
7ee5cbe473
Rev3175, Make getConnectablePeers return not-connected peers by default 2017-12-19 16:14:13 +01:00
shortcutme
b5452bce6b
Rev3171, Allow 127.0.0.1 peers for testing reasons 2017-12-15 12:28:48 +01:00
shortcutme
ca42740e1b
Don't add peer 127.0.0.1 and 0.0.0.0 2017-12-15 12:10:36 +01:00
shortcutme
74e50e209a
Rev3158, Fix site clone with sites larger that 10MB 2017-12-04 16:04:10 +01:00
shortcutme
90ff9ac7fb
Rev3155, Avoid UI hang during db rebuild 2017-12-02 02:38:17 +01:00
shortcutme
6b92d011d2
Rev3153, Download missing sites from sites.json 2017-11-30 19:38:56 +01:00
shortcutme
24372470f6
My_ip no longer required to be sent via updates 2017-11-19 18:21:47 +01:00
shortcutme
51f1781df4
Move bad file checking to separate function, root content.json does not requires file_info 2017-11-19 18:21:10 +01:00
shortcutme
9d511ba165
Rev3122, Auto ignore database file 2017-10-22 11:13:09 +02:00
shortcutme
e048fa6c6a
Target 6 connections per site 2017-10-21 11:02:40 +02:00
shortcutme
940d583f9a
Fix site violation bug when using Tor 2017-10-21 11:02:27 +02:00
shortcutme
68ea4d02cb
Don't reset bad file counter after restart, but limit to 20 2017-10-11 14:29:54 +02:00
shortcutme
8e3e96fe65
Create storage object after site settings loaded 2017-10-04 12:50:24 +02:00
shortcutme
c91011e6ea
Pass file info for addTask call 2017-10-03 15:49:08 +02:00
shortcutme
9b8eeb6a73
WorkerManager addTask returns task instead of event 2017-10-03 15:47:46 +02:00
shortcutme
468fe8f266
Move json formatting to separate function 2017-10-03 15:30:10 +02:00
shortcutme
5ccdfbd40a
Update site optional size on save 2017-10-03 15:29:52 +02:00
shortcutme
ad969dcca7
Move file download allowed checking to separate function 2017-10-03 15:29:26 +02:00
shortcutme
08b31416f3
Move needFileInfo to separate function 2017-10-03 15:28:37 +02:00
shortcutme
32ca6ac1a7
Move site cache generation to separate function to allow plugins to extend it 2017-10-03 14:40:13 +02:00
shortcutme
9f762a0230
Bootstrapper use python date functions instead of sqlite 2017-08-18 14:43:28 +02:00
shortcutme
1db2327b3d
Rev2180, Allow ajax requests using ajax_key http get parameter 2017-08-15 02:40:38 +02:00
shortcutme
29aa1b7b93
Log file diffing timings 2017-08-09 14:21:59 +02:00
shortcutme
5a08545e2d
Only recalculate site sizes on exit 2017-08-09 14:21:44 +02:00
shortcutme
b503d59c49
Support gzip compressed database files 2017-08-09 14:19:39 +02:00
shortcutme
4cd393e4d8
Auto download optional files command line argument 2017-07-31 14:39:21 +02:00
shortcutme
ac230219ee
DB cleanup error is not fatal 2017-07-31 14:35:48 +02:00
shortcutme
c96dce3d0b
Fix site cleanup from content.db 2017-07-31 14:35:01 +02:00
shortcutme
467cd480cf
Rev2168, Only avoid reload site on every missing file 2017-07-29 12:49:38 +02:00
shortcutme
6610914fdb
Rev2167, Skip downloading if no connection server specified 2017-07-28 22:57:41 +02:00
shortcutme
2aba9cc3c2
Only download content.json files as content 2017-07-19 16:45:12 +02:00
shortcutme
bb27e3124f
Remove files without info on retry bad files 2017-07-18 20:54:39 +02:00
shortcutme
066f54f521
Only log listModification request in verbose mode 2017-06-19 16:12:41 +02:00
shortcutme
64199c7ded
Reduce logging on tracker error 2017-06-19 16:12:24 +02:00
shortcutme
ff69b04216
Catch verify exceptions on manual site files verification process 2017-06-19 15:57:02 +02:00
shortcutme
fec3ff7d8e
Remove bad files that has no info 2017-06-19 15:40:13 +02:00
shortcutme
4b346243eb
On clone sign content.json after all other files copied 2017-06-19 15:39:14 +02:00
shortcutme
f506ac9701
Faster getPath 2017-06-15 13:30:36 +02:00
shortcutme
6167dda01d
Compact json list output 2017-06-03 00:53:59 +02:00
shortcutme
aacde33614
File size limit default to 10MB 2017-05-23 12:41:37 +02:00
shortcutme
3f59727ab6
Formatting 2017-05-23 12:40:42 +02:00
shortcutme
84b535767b
By default nothing is domain 2017-05-11 18:02:05 +02:00
shortcutme
438bdbc141
Protect peers for low traffic sites from closing 2017-05-07 18:15:23 +02:00
shortcutme
728e497057
Display database rebuilding progressbar 2017-05-07 18:14:13 +02:00
shortcutme
169cf2e6ae
Support optional db files 2017-04-12 17:36:03 +02:00
shortcutme
16bfb35109
No positive preference for peers with more than 5 shared sites to avoid centralization 2017-04-09 11:57:35 +02:00
shortcutme
03f3e9e4fd
Rev2025, Publish remotely outdated files on modification check 2017-04-07 21:17:25 +02:00
shortcutme
8ea8e8c28a
Publish incoming files to more peers 2017-04-07 18:17:30 +02:00
shortcutme
909b9da9da
Unnecessary log and comment 2017-03-06 15:30:42 +01:00
ZeroNet
0e2f742b7e Merge pull request #842 from mishfit/refactorListAndWalk
Rename SiteStorage.list to SiteStorage.walk
2017-03-04 12:39:06 +01:00
shortcutme
4e846774b4
Log formatting 2017-02-27 00:13:30 +01:00
shortcutme
a6be8ddc2d
Connectable peers ordered 2017-02-27 00:13:18 +01:00
shortcutme
b65670b9e4
Don't remove peers with atleast 5 sites 2017-02-27 00:12:55 +01:00
shortcutme
ea635d4630
Only totally connected peers counted to peer cleanup 2017-02-27 00:12:36 +01:00
shortcutme
2e9737c149
Faster and better getConnectedpeers 2017-02-27 00:11:57 +01:00
shortcutme
0f8de2b7c1
Use latest added peers and use reputation for listModifications and publishing 2017-02-27 00:11:45 +01:00
shortcutme
2188c5d9fa
Use getRecentPeers for publishing 2017-02-27 00:11:02 +01:00
shortcutme
b5d879c1cd
Start site update check on site re-connect 2017-02-27 00:10:27 +01:00
shortcutme
fd365a19f7
Use getRecentPeers for listModification 2017-02-27 00:09:59 +01:00
shortcutme
adbaadae81
Allow 100 sec timeout for listModification 2017-02-27 00:09:41 +01:00
shortcutme
09ad4133fb
20 sec timeout for listModified 2017-02-27 00:08:46 +01:00
shortcutme
bad2a79f05
Reason for connection closing 2017-02-27 00:02:24 +01:00
mishfit
656c818211 rename SiteStorage.list to SiteStorage.walk
also, add SiteStorage.list as non-recursive
directory listing (`os.listdir`)
2017-02-24 21:58:46 -07:00
shortcutme
26bad8d05b
Fix needConnections bug 2017-02-24 01:16:22 +01:00
shortcutme
4aff146812
Log giving up on a file 2017-02-22 11:29:35 +01:00
shortcutme
221aa16433
Rev1929, Fix file filed on not bad files 2017-02-19 12:32:41 +01:00
shortcutme
63b609f725
Give up files after 30 tries 2017-02-19 00:52:31 +01:00
shortcutme
2eef0bded1
Only increment bad_file if internet connection is active 2017-02-19 00:52:20 +01:00
shortcutme
cfad85b30e
Log file object on db load 2017-02-13 16:12:56 +01:00
shortcutme
6149f5c6ee
Download everything on add 2017-02-11 18:26:37 +01:00
shortcutme
56efa02820
GetDbFiles return inner path 2017-02-09 01:57:12 +01:00
shortcutme
f3c71f3e6c
Move SiteStorage database updating to separate function 2017-02-09 01:56:20 +01:00
shortcutme
66e2bfd76b
Reset bad_files on full update 2017-01-29 19:03:22 +01:00
shortcutme
db7dad6731
Don't delete content.json files from bad_files on update 2017-01-29 19:03:08 +01:00
shortcutme
2b389e4e85
Support since parameter for siteUpdate 2017-01-29 19:02:08 +01:00
shortcutme
a0c3d7f8a6
Rev1833, Fix utf8 system paths 2017-01-22 21:22:53 +01:00
shortcutme
2dc981496b
Rev1821, Show progress bar on publishing 2017-01-16 13:41:57 +01:00
shortcutme
4e96ddfb9e
Allow .. in data_dir path 2017-01-11 13:14:02 +01:00
shortcutme
d9f8e83429 Safe disk write for content.json files 2017-01-05 02:26:44 +01:00
shortcutme
c94e73a029 Skip sites with broken content.json on startup 2017-01-05 02:26:05 +01:00
shortcutme
f89f351b3c Dont copy content.json-default file 2016-12-14 00:43:08 +01:00
shortcutme
34ab2bad49 Clone inner path dest fix 2016-12-14 00:42:52 +01:00
shortcutme
62501d00b1 Use content.json-default if exists 2016-12-14 00:42:31 +01:00
shortcutme
10772a704e Root path parameter for cloning 2016-12-08 17:14:55 +01:00
shortcutme
20ed04af84 Also delete orphan sites from site_id cache 2016-12-04 18:54:43 +01:00
shortcutme
2e08d68d87 Remove content.json files from bad file list 2016-11-16 11:19:08 +01:00
shortcutme
2cdde7dd46 Retry file deletion on error 2016-11-07 23:47:09 +01:00
shortcutme
9765afc286 Modify checkfiles to use optionalRemove, optionalDownloaded and isDownloadable functions 2016-11-07 23:46:54 +01:00
shortcutme
4890c4cb2f Fix big site download 2016-11-07 23:43:08 +01:00
shortcutme
73b3c8db55 Use site object on delete from contentdb 2016-11-07 23:42:28 +01:00
shortcutme
713a2f88a9 Query and send more hashfield 2016-11-07 23:42:13 +01:00
shortcutme
30a2116abf Only announce if trackers 2016-11-07 23:42:00 +01:00
shortcutme
6acc81cb50 Retry bad files in separate thread 2016-11-07 23:41:25 +01:00
shortcutme
af571deb16 Peer cleanup ttl based on peer numbers 2016-11-07 23:40:04 +01:00
shortcutme
e6a82d4c64 Pooled file downloads to handle large ammount of files better 2016-11-07 23:39:45 +01:00
shortcutme
83b3dc8fbe Autodownload optional pattern check to separate function to allow plugins 2016-11-07 23:39:04 +01:00
shortcutme
d1ad7e0850 Stats for total optional files and downloaded optional files in a site 2016-11-07 23:36:04 +01:00
shortcutme
0418a3e6de Only save sites.json if fully loaded to avoid data loss 2016-11-07 22:40:12 +01:00
shortcutme
8f3dfd7435 Fix site deletion errors on startup 2016-11-07 22:38:01 +01:00
shortcutme
d6da0c421c Better json compressing 2016-10-02 14:23:05 +02:00
shortcutme
bb23bc7b3c Send diff to second batch of peers 2016-10-02 14:22:32 +02:00
shortcutme
d760a71b76 Rev1529, Fix json write bug with [] characters in value 2016-09-29 13:01:02 +02:00
shortcutme
117a82c059 Delete bad_files without size 2016-09-28 20:35:53 +02:00
shortcutme
67c9921ace Only add non-connected peers to publish if necessary 2016-09-17 21:01:16 +02:00
shortcutme
4690a111bf Push updates to newer clients first 2016-09-17 21:00:40 +02:00
shortcutme
65e5d05a9e Delete cleanup bad files without file info on update 2016-09-17 21:00:20 +02:00
shortcutme
9fd19bf382 Fix content deletion 2016-09-14 10:52:41 +02:00
shortcutme
b2b91c9aff Remove save error message when no sites loaded yet 2016-09-07 17:41:55 +02:00
shortcutme
a6c4a77e13 Rev1510, Fix site save without loaded SiteManager 2016-09-06 14:52:55 +02:00
shortcutme
d9bbb20743 Rev1509, Fix site delete 2016-09-06 10:16:09 +02:00
shortcutme
dae5cd3969 Fix startup with 1000+ sites 2016-09-05 13:57:12 +02:00
shortcutme
59205088d2 Move site deletion to dedicted function 2016-09-05 13:55:51 +02:00
shortcutme
37a3b4678d Fix user file pre-content.json downloading 2016-09-04 18:01:09 +02:00
shortcutme
365f0cd646 Reset files on cloning 2016-09-04 18:00:51 +02:00
shortcutme
f698ec2502 Also delete sites from content.db 2016-09-04 17:59:47 +02:00
shortcutme
a1f0c20afc Rename checkFiles to updateBadFiles 2016-09-04 17:59:29 +02:00
shortcutme
b0a85b0669 Dont check every file on update 2016-09-04 17:58:18 +02:00
shortcutme
9528d333bd Retry failed files on download end 2016-09-04 17:57:44 +02:00
shortcutme
4437a99330 Use pool to download large ammount of content.json files 2016-09-04 17:57:32 +02:00
shortcutme
8f158cbc8f Make listmodified query use content.db cache 2016-09-04 17:56:33 +02:00
shortcutme
135a9b8f5c Fix content.json retry 2016-09-04 17:54:57 +02:00
shortcutme
1dd488ef89 Remove deleted sites from content.db 2016-09-04 17:52:14 +02:00
shortcutme
a614545467 Faster site settings load 2016-09-04 17:51:55 +02:00
shortcutme
3724e0f376 Save sites.json on exit 2016-09-04 17:51:20 +02:00
shortcutme
1bd6e2f493 Cache bad_files and optional hashfield in sites.json 2016-09-04 17:43:30 +02:00
shortcutme
ecb5885dba Cache file sizes and modification date in sqlite db 2016-09-04 17:41:04 +02:00
shortcutme
709dc854e7 Add site to site manager if required 2016-08-29 11:17:19 +02:00
shortcutme
887a07c3fb Rev1427, Skip invalid files on dbRebuild 2016-08-27 11:01:41 +02:00
shortcutme
d76c1ca98e Also cleanup connections for smaller sites 2016-08-26 11:42:23 +02:00
shortcutme
7edb784305 Fix user file updating on owned sites 2016-08-26 11:40:22 +02:00
ZeroNet
3c7a9f0f82 Merge pull request #537 from megfault/patch-1
Fix typos in log message
2016-08-17 11:32:25 +02:00
shortcutme
274414a54b Give up user's content.json after 5 retrys 2016-08-16 19:56:18 +02:00
shortcutme
b40b3e2d85 Fix config size_limit parsing 2016-08-16 19:55:55 +02:00
Ana Barroso
c9660ed20e Fix typos in log message 2016-08-16 18:14:02 +02:00
shortcutme
92e4dd64d1 Dont recalculate site size on update 2016-08-15 13:56:17 +02:00
ZeroNet
67f75c4bef Skip archived user files 2016-08-10 12:42:29 +02:00
ZeroNet
0275954fb7 Tracker log total peers 2016-08-10 12:42:13 +02:00
ZeroNet
44dc4c41c9 Wait for user's content.json on file request 2016-08-10 12:41:35 +02:00
ZeroNet
e5e76a4ef5 Do not reload the whole site on update command 2016-08-10 12:41:04 +02:00
ZeroNet
021a084796 Peers query optimize 2016-08-10 12:40:17 +02:00
ZeroNet
1ec00947ad Save site on content.json arrive 2016-08-10 12:36:20 +02:00
ZeroNet
f38f4c3758 Use SiteManager to save site 2016-08-10 12:35:40 +02:00
ZeroNet
2852bbc657 Remove unnecessary site savings 2016-08-10 12:35:17 +02:00
ZeroNet
0d565dc64d Better site settings saving 2016-08-10 12:34:28 +02:00
ZeroNet
1f3a4f0e2f Remove orphan sites from sites.json 2016-08-10 12:33:34 +02:00
ZeroNet
53249f4f07 Add own logger to SiteManager 2016-08-10 12:32:28 +02:00
ZeroNet
0569697d98 Update database on delete 2016-08-10 12:31:54 +02:00
ZeroNet
acf24aab6f Allow to pass file object on file updated 2016-08-10 12:31:30 +02:00
ZeroNet
3ff75c6602 Also trigger onUpdated on file delete 2016-08-10 12:30:35 +02:00
ZeroNet
0be6cb6602 Move database files listing to separate funcion 2016-08-10 12:30:04 +02:00
ZeroNet
9def3b58eb Allow plugins on SiteStorage 2016-08-10 12:28:57 +02:00
HelloZeroNet
36e969ed1f Fix siteSign without connection server 2016-05-05 12:11:35 +02:00
HelloZeroNet
639a834658 Keep minimum 5 connections per site 2016-04-29 02:58:00 +02:00
HelloZeroNet
93e598cc59 Rev1275, Fix dictionary changed size during iteration error 2016-04-26 10:46:53 +02:00
HelloZeroNet
bf384922a6 Use iteritems to save memory 2016-04-25 02:26:29 +02:00
HelloZeroNet
e53f56ed30 Cleanup connections with less sites first 2016-04-25 02:25:11 +02:00
HelloZeroNet
73c69d54bc Dont save settings on every announce 2016-04-20 23:32:43 +02:00
HelloZeroNet
23389dc31e Formatting 2016-04-20 23:30:25 +02:00
HelloZeroNet
38057c2e03 Log site config save time 2016-04-20 23:29:26 +02:00
HelloZeroNet
cd52a61c4c Save sites.json every 10 minute 2016-04-20 23:28:58 +02:00
HelloZeroNet
55d76c8c5c Faster SiteStorage.getSize 2016-04-20 23:28:21 +02:00
HelloZeroNet
cd5cdd580a Re-publish modifications to less peers 2016-04-18 00:42:26 +02:00
HelloZeroNet
3bdde14821 Log patch size 2016-04-10 23:12:34 +02:00
HelloZeroNet
bbf1bc163d Extra debug messages, cleanup unused lines 2016-04-10 13:25:11 +02:00
HelloZeroNet
561bd80aa3 Fallback to simple file copy if rename failed 2016-04-09 19:43:44 +02:00
HelloZeroNet
6388973292 Retry file rename later if resource not ready yet 2016-04-07 10:36:50 +02:00
HelloZeroNet
9c1ea71d2f Try to use diff before downloading the file 2016-04-06 14:01:03 +02:00
HelloZeroNet
7ee8eead23 Remove duplicate peers from publish queue 2016-04-06 14:00:40 +02:00
HelloZeroNet
82b960c641 Pass diffs to other users 2016-04-06 13:59:53 +02:00
HelloZeroNet
ab1aaea7f5 Calculate and send diffs on publish 2016-04-06 13:57:19 +02:00
HelloZeroNet
84e78859dc SiteStorage.rename function 2016-04-06 13:50:20 +02:00
HelloZeroNet
8f39ed2467 Cloned site updating 2016-04-06 13:50:05 +02:00
HelloZeroNet
ec70c47aa0 Optimize path modifying functions 2016-04-06 13:48:13 +02:00
HelloZeroNet
5a34448ba1 Less verbose logging 2016-04-06 13:39:17 +02:00
HelloZeroNet
c8b949cef5 Save last received update time from peer to avoid send the same update on publish 2016-03-30 23:05:43 +02:00