87a67db680
Remove unused conn piece inclinations
2021-11-01 11:49:27 +11:00
487352fa5b
fixup! [trackerscraper] Add custom DNS lookup function
2021-11-01 11:26:58 +11:00
2a6152b832
[trackerscraper] Add custom DNS lookup function
2021-11-01 11:26:58 +11:00
98957b6e8e
Only assert pending requests when check is enabled
...
Uses too much CPU for production from its current callsites.
2021-10-26 14:42:42 +11:00
37373864e4
Add more comprehensive pending requests assertions
2021-10-25 16:16:56 +11:00
d331419604
Demote Torrent.runHandshookConnLoggingErr level to debug
2021-10-21 11:40:06 +11:00
ff3c186396
Add low requests check
...
This is an optimization for webseeds, which have 10 synchronous request routines, and if the request count dips below 10, some sit idle. There is probably something similar to be done with PeerConns, which won't update until there are zero requests, but there there is a timer to refresh updates, and the queues are very long (typically 512-2048).
2021-10-21 10:48:43 +11:00
8eec0b665e
Update webseeds for peer requesting
2021-10-21 10:28:57 +11:00
a7d3296003
Use roaring.Bitmap.IsEmpty where appropriate
2021-10-19 14:08:56 +11:00
de87fd3eb1
Avoid allocation putting undirtiedChunksIter into interface
2021-10-19 14:08:56 +11:00
529aaa9b1f
Use roaring bitmap for pending pieces
...
Also makes a significant optimization for peerHasWantedPieces possible.
2021-10-19 14:08:56 +11:00
5e8f61abb5
Optimize request updates due to conn and reader state changes
2021-10-19 14:08:56 +11:00
7d0be0ac65
Optimize chunk calculations in request strategy
2021-10-19 14:08:56 +11:00
10b9e88356
Implement pending requests using BSI
2021-10-19 14:08:56 +11:00
9aff9f3592
Extract pendingRequests
2021-10-19 14:08:56 +11:00
2be5a95b06
Add reasons for updateRequests to be triggered
2021-10-19 14:08:56 +11:00
16c571b58b
Add pprof labels for request updates and remove Client-wide requester
2021-10-19 14:08:13 +11:00
30b99e7102
Remove peer piece inclination and piece request order
...
These are vestigial data structures used with old request strategy implementations.
2021-10-19 14:08:13 +11:00
3ef8cfba27
Log handshook conn run errors with level warning
...
(cherry picked from commit 7ca7890e9f30307a1f4942e6d987791a142c3cb3)
2021-10-19 14:07:48 +11:00
6f196c8d33
Update piece priority after processing piece hash result
...
Seems to avoid some silly performance issues in triggering request updates due to a piece still pending while being marked.
2021-10-19 14:03:03 +11:00
637045c19a
Add DisableInitialPieceCheck option ( #677 )
2021-10-11 11:33:40 +11:00
72a06d88fc
Add TorrentStats.PiecesComplete
...
Fixes https://github.com/anacrolix/torrent/issues/352 .
2021-10-07 13:50:13 +11:00
28e9064288
Fix comment that seems to mess with GoLand's package structure
2021-10-07 12:09:52 +11:00
c9c137d8f2
Add a flag for Complete
2021-10-07 12:09:20 +11:00
9720fd576d
Update chansync
2021-10-07 11:45:09 +11:00
da1221dd50
Change Peer._peerPieces to use raw roaring Bitmap type
...
The wrapper type was from when roaring didn't support zero-alloc initialization.
2021-10-05 17:48:34 +11:00
86fe6cc872
Inlineable `(*Torrent).BytesMissing()` ( #633 )
...
Honestly a name like `BytesLeft` would have been more suitable, but it's too late for that I guess.
2021-09-27 12:43:09 +10:00
63b3d2d211
Track dirty chunks in a single bitmap on Torrent
2021-09-20 18:52:54 +10:00
510877ea43
Reduce the diff to master and add peerRequesting feature const
2021-09-20 15:09:28 +10:00
1d2d1a9cde
Store peer requests in a bitmap
2021-09-19 15:16:37 +10:00
e80b989f8e
Clarify maximum value of "metadata_size" ( #609 )
2021-09-14 22:36:19 +10:00
a8db640c62
Drop bradfitz/iter dependency ( #605 )
...
* Drop bradfitz/iter dependency
`range iter.N` looks nice and doesn't allocate, but unfortunately using a `range` expression blocks a function from being inlined wherever it's used (for now). It's not that we need inlining in all cases, but I do think a C-style for loop looks just as nice and is probably clearer to the majority. There also aren't any clear disadvantages to changing (unless you just happen to dislike the look of C)
* Update misc_test.go
* Update rlreader_test.go
* Update torrent_test.go
* Update bench_test.go
* Update client_test.go
* Update iplist_test.go
* Update mse_test.go
* Update peerconn_test.go
* Update peerconn.go
* Update order_test.go
* Update decoder_test.go
* Update main.go
* Update bench-piece-mark-complete.go
* Update main.go
* Update torrent.go
* Update iplist_test.go
* Update main.go
2021-09-14 13:46:50 +10:00
5c440e8929
Simplify (*Torrent).gotMetainfo ( #581 )
2021-09-13 11:41:11 +10:00
0d10a1b53a
Optimize Torrent.worstBadConn
2021-09-10 23:07:10 +10:00
62c6fbc8f8
Clarify ownership of (*Torrent).chunkPool ( #583 )
...
Basically bind the lifetime of chunkPool to the torrent by using `sync.Pool` in lieu of `*sync.Pool`. Gives the GC ever so slightly less work to do.
2021-09-09 22:27:16 +10:00
ccb90f1252
Improve Closing ( #559 )
...
- Prevent double-closing in cmd/torrent
- Move async closing from storage to torrents, and wait on them to finish tidying up before returning/exiting.
2021-09-05 11:33:41 +10:00
175b826e73
Rework Reader waiting
2021-09-04 23:07:32 +10:00
bf6e64a652
Fix some DeepSource lints
2021-08-30 11:48:34 +10:00
72c74f7800
Remove unused code
2021-08-19 16:06:46 +10:00
16176b762e
Add linter CI ( #542 )
...
* Add linter CI
Signed-off-by: Bora M. Alper <bora@boramalper.org>
* Make gosec CI ignore SHA1 and upload sarif
Signed-off-by: Bora M. Alper <bora@boramalper.org>
* Fix formatting of source files
Signed-off-by: Bora M. Alper <bora@boramalper.org>
* Make go vet ignore unkeyed composite literals and fix other warnings
Signed-off-by: Bora M. Alper <bora@boramalper.org>
* Make staticcheck ignore unused methods and fix other warnings
Signed-off-by: Bora M. Alper <bora@boramalper.org>
* Use golangci-lint
Signed-off-by: Bora M. Alper <bora@boramalper.org>
2021-08-16 11:11:31 +10:00
22c5a94a6a
Use roaring.Bitmap directly for completed pieces
...
Looking at improving the performance around this per https://github.com/anacrolix/torrent/discussions/547#discussion-3522317 .
2021-08-16 11:07:10 +10:00
1d53c170b0
Close torrent storage asynchronously on drop
2021-07-17 19:10:07 +10:00
5eab39001f
Add debug level to piece hash failure log message
2021-07-17 18:52:41 +10:00
c3c982b621
Fixes for wasm
2021-06-23 17:24:50 +10:00
15f8e77519
Add explicit metadata extension types
2021-06-21 12:14:06 +10:00
af1ca91e04
Remove conntrack, expose Torrent.AnnounceToDht, ClientConfig.PeriodicallyAnnounceTorrentsToDht
2021-06-21 12:13:53 +10:00
ebd19af795
Merge branch 'request-strategy-rewrite'
2021-06-21 12:04:06 +10:00
e2a559e243
Allow Storage Backends to do their own Hashing ( #518 )
...
* Allow Storage Backends to do their own Hashing
- Describes an optional interface 'SelfHashing' that a storage backend's type implementing 'PieceImpl' may also implement in order to calculate piece hashsums itself.
- Alters the 'hashPiece' function in the torrent package to look for types implementing 'SelfHashing' . If not implemented, calculate the hash as usual.
2021-06-08 15:45:35 +10:00
47c49735fc
Fix race in piece availability runs
2021-06-07 13:01:40 +10:00
b43987fcc8
Fix up some crashes around piece availability
2021-06-07 13:01:40 +10:00
95d808d3c5
Use rejiggered missinggo bitmap
2021-06-07 13:01:39 +10:00
42bf6b58a6
Fix in decPieceAvailability when we don't have torrent info
...
Happens when we close a Peer before getting Torrent info.
2021-06-07 13:01:39 +10:00
fcc4ef2b95
Fix sqlite piece completion
2021-06-07 13:01:39 +10:00
f19564d597
Improve piece availability status output
2021-06-07 13:01:39 +10:00
4577e8a3c1
Forgot to remove initial have all state for webseeds
...
Later when the webseed was closed, it was assumed that it had contributed to the piece availability count.
2021-06-07 13:01:39 +10:00
99a7cb9291
Missed piece availability code
2021-06-07 13:01:39 +10:00
2d2456249a
Remove unused pieceRequestOrder in Client
2021-06-07 13:01:39 +10:00
56e2a8a3a6
Fix download rate, status output
2021-06-07 13:01:39 +10:00
5f8471e21b
Rework storage.TorrentImpl to support shared capacity key
2021-06-07 13:01:39 +10:00
0830589b0a
Pass tests with new full-client request strategy implementation
2021-06-07 13:01:39 +10:00
f5d2ba37bc
Count peers added from DHT for logging
2021-06-07 12:57:32 +10:00
d7c549d2e4
Add TODOs around setting info bytes
2021-06-07 12:57:09 +10:00
47284cf426
Rename Torrent.{add,delete}Connection
2021-05-14 15:39:01 +10:00
4ab20f60b6
Improve some lock handling under panics
2021-05-14 15:36:53 +10:00
305ba51dc2
Big logging cleanup to improve experience from README
2021-05-04 12:44:51 +10:00
28f17be267
Close all peer types when Torrent closed
2021-02-19 12:58:20 +11:00
7e2c2106e6
Include webseed URLs in output of Torrent.Metainfo
2021-02-18 14:37:06 +11:00
927866e881
Limit conns per host across webseed clients
2021-02-09 19:22:38 +11:00
6bc2a2a6ac
Revert "Fix stalls for responsive transfer tests"
...
This reverts commit d48c6ae6dc
2021-02-03 18:55:54 +11:00
d48c6ae6dc
Fix stalls for responsive transfer tests
...
Looks like there's a timing issue between marking things complete, and already having decided to read the data from incomplete chunks.
2021-02-02 10:41:39 +11:00
60e992ec97
Rework webseed peers to use a pool of requesters
...
This fixes the limitation that the max outstanding requests for a webseed peer must match the request concurrency. It should mean less recalculation, and more pipelining.
2021-01-29 16:01:35 +11:00
4078644db9
Pass callbacks to webseed peers
2021-01-28 18:25:06 +11:00
1ac5811990
Remove requests as soon as chunk data is received
...
Note that this breaks the backpressure on webseed responses again, and should be fixed shortly.
2021-01-28 16:36:35 +11:00
456a2f7c5d
Expose more callbacks and Request and ChunkSpec
2021-01-28 14:23:22 +11:00
b5215ea7e8
Expose Peer.Network to replace Peer.RemoteAddr.Network
2021-01-25 15:49:48 +11:00
955e337b78
Generate a RemoteAddr for webseed Peers
2021-01-25 15:49:48 +11:00
27108bd2f7
Export Peer
2021-01-25 15:49:48 +11:00
52bb7f7a39
connection drop log debug level
2021-01-07 00:21:59 +11:00
ded6c19edb
Add the DropMutuallyCompletePeers ClientConfig field
2021-01-05 16:58:45 +11:00
d713aaf280
Rework active announce limiter to reduce contention
2020-12-15 15:29:45 +11:00
99ddad31c9
Track piece marking state separately
2020-11-21 13:44:23 +11:00
6cceb5749d
Mark piece complete without Client lock
2020-11-16 16:37:11 +11:00
e6da640bb2
Tidy up doc, file names, naming
2020-11-09 10:56:45 +11:00
7cb74b158f
Replace storage.IncompletePieceToWriter with io.Writer
...
It was incorrect to assume piece hashing only operates on incomplete chunk data. This actually uncovered a bug where duplicate hash checks occurred, and the redundant checks would fail due to not reading the completed data.
2020-11-06 16:23:38 +11:00
636b20b860
Add write incomplete/consecutive chunks interfaces
2020-11-02 15:35:07 +11:00
43891309ad
Return errors from Reader if data downloading won't occur
...
Chunk write errors to storage can disable data download. Previously Readers would wait indefinitely for the data to become available. This change returns an error instead of stalling.
2020-10-27 14:10:17 +11:00
1bfca56e94
Read from more than a single piece in each read to Torrent storage
2020-10-27 12:24:43 +11:00
c1d189ed31
Performance improvements to PEX
2020-10-23 08:58:55 +11:00
417d7d1d48
Tweaks to storage error and completion handling
2020-10-11 12:54:03 +11:00
01dfc28484
Set tracker announce timeout back to default
...
Add lots of comments about how this works and why.
2020-10-01 11:45:47 +10:00
a3827099c4
Limit simultaneous announces to the same URL
2020-10-01 10:43:10 +10:00
a2c7b384df
Limit half-open connections at the Client level
2020-09-30 16:56:27 +10:00
1cdae13700
Fix conn status string for WebRTC connections
2020-09-29 16:21:54 +10:00
9d0ea15044
Expose peer.RemoteAddr
2020-07-15 16:16:14 +10:00
12ace95493
Tidy up webseed peer naming and unused types
2020-06-04 11:58:18 +10:00
62e7f29a0c
Fix panic in webseed request rejection logging
...
Also use a new helper in Logger to simplify things.
2020-06-04 11:51:17 +10:00
645901636d
torrent : add udp scheme to 'switch case'
2020-06-04 10:37:50 +10:00
a7689f52bd
Add DisallowDataDownload/Upload to TorrentSpec and small log cleanup
2020-06-04 10:36:56 +10:00
8606385985
Merge pull request #410 from anacrolix/webseeds
...
* Rename Peer to PeerInfo, and unexport PeerInfos
* Break peer out from PeerConn
* Abstract out segments mapping and use it in mmap storage
* Got file storage working with segment index
* Fix race in webtorrent.TrackerClient.Run
* storage file implementation: Error on short writes
* Remove debug logging from storage file implementation
* cmd/torrent-verify: Fix piece hash output
* Support disabling webtorrent
* Further progress on webseeding
* Handle webseed Client events
* Rename fastestConn->fastestPeer
* Add webseeds from magnet links
* Remove events from webseed
Manage this stuff inside the webseed peer instead.
* Make use of magnet source fields and expose Torrent.MergeSpec
* Add option to disable webseeds
* Fix webseeds when info isn't available immediately
* Handle webseed request errors
* Tidy up the interface changes
2020-06-03 11:25:08 +10:00
8a2ff43246
Tidy up the interface changes
2020-06-02 17:41:59 +10:00
Matt Joiner