b69bb34eaf
Rejig some storage options
2021-01-25 15:54:37 +11:00
28682f6ae6
Fix unused opts param
2021-01-25 15:54:37 +11:00
19f5984aec
Include context.Context and pprof label in withConn
2021-01-25 15:54:37 +11:00
2ddd3169ef
Move storage piece benchmarks to storage/test and add a lot more dials
2021-01-25 15:54:37 +11:00
cb5f80ec11
Improve configurability and add PutSized to sqlite storage
2021-01-25 15:54:37 +11:00
434dfdf6e0
Add MarkComplete benchmark for sqlite
2021-01-25 15:54:37 +11:00
cb1d6bfc8f
Fix boundary conditions trimming sqlite3 storage cache
2021-01-18 14:48:30 +11:00
388e6899a1
piece resource storage: Delete incomplete chunks concurrently after writing complete piece
2020-11-21 13:44:23 +11:00
b35c204238
Use ConsecutiveChunkWriter in MarkComplete for piece resource storage if available
2020-11-21 13:44:23 +11:00
41ef267c1b
sqlite storage: Ensure that chunks are consecutive
2020-11-21 13:44:23 +11:00
8ef37cb2e6
sqlite storage: Remove num batched write queries log message
2020-11-16 16:37:11 +11:00
d84367c9a6
Update to race fixes in filecache and sqlite
2020-11-16 16:37:11 +11:00
258585fd8b
sqlite storage: Buffer write requests
2020-11-16 16:37:11 +11:00
b0ebc856d9
sqlite storage: Add some expvars
2020-11-16 16:37:11 +11:00
e65fac26ac
sqlite storage: Add error return from withConn
2020-11-16 16:37:11 +11:00
7afa4a2de4
Fix panic in when bbolt storage has Closed
2020-11-16 16:37:11 +11:00
7cb74b158f
Replace storage.IncompletePieceToWriter with io.Writer
...
It was incorrect to assume piece hashing only operates on incomplete chunk data. This actually uncovered a bug where duplicate hash checks occurred, and the redundant checks would fail due to not reading the completed data.
2020-11-06 16:23:38 +11:00
96b640065a
sqlite storage: Add capacity management
2020-11-03 15:17:12 +11:00
8afb05a8a5
sqlite storage: Batch writes synchronously instead of buffering them first
2020-11-03 15:16:57 +11:00
3c102393d1
Comment on storage.IncompletePieceToWriter
2020-11-03 14:28:16 +11:00
784345e9f7
sqlite storage: Init schema in NewPool instead of NewProvider and add an option to disable
2020-11-03 13:11:44 +11:00
d04622e4b3
sqlite storage: Track total blob data size manually
...
Works around possible full table scan at startup, and possible lack of caching around cast(data as blob).
2020-11-03 13:10:17 +11:00
636b20b860
Add write incomplete/consecutive chunks interfaces
2020-11-02 15:35:07 +11:00
17e22516ff
sqlite storage: Add batched writes
2020-10-30 19:46:51 +11:00
ba70ad9b41
sqlite storage: Set mmap_size to a very large size
...
Seems to fall back to ~2GiB on my system.
2020-10-30 18:40:47 +11:00
b75ebbf9e7
sqlite storage: Provide helpers and reasonable defaults
2020-10-30 12:20:54 +11:00
fc039262d9
sqlite storage: Force data to be used as a blob
...
There's a bug in crawshaw.io/sqlite, and some forks where inserting []byte results in a text type instead of blob. To ensure things work correctly, we coerce data to blob wherever we can. See https://github.com/crawshaw/sqlite/issues/94 and the fork that fixes it.
2020-10-30 10:47:50 +11:00
8e1a8440bf
Performance fiddling on sqlite storage
2020-10-27 17:07:49 +11:00
e30084223d
sqlite storage: Include capacity management
2020-10-27 11:08:37 +11:00
55d4bcaf26
sqlite storage: Do ReadAt without using incremental I/O
2020-10-27 11:08:08 +11:00
7fa8c604ab
sqlite storage: Working recursive CTE for excess data
2020-10-23 11:03:42 +11:00
1fc63bdded
sqlite storage: Add NewProviderPool
2020-10-23 09:03:44 +11:00
bb2ce15114
sqlite storage: Rename table to blob
...
Avoiding plural table names.
2020-10-23 09:01:15 +11:00
8088002ba6
Fix error handling for bad torrent adds with safe file handling
2020-10-15 16:03:43 +11:00
89235e180f
Sanitize metainfo file paths for file-based storage
...
Fixes exploit where specially crafted infos can cause the client to write files to arbitrary locations on local storage when using file-based storages like mmap and file.
2020-10-15 15:45:08 +11:00
3820017a0c
Add last_used to sqlite storage
2020-10-13 09:36:58 +11:00
d820f7861c
Add a sqlite data storage implementation
2020-10-11 12:58:27 +11:00
00ac333ea2
Panic on bad piece per resource chunk names
...
This was masking a bug in a coming sqlite storage implementation. Maybe it could be relaxed to checking for no filepath.Dir in the future.
2020-10-11 12:57:33 +11:00
417d7d1d48
Tweaks to storage error and completion handling
2020-10-11 12:54:03 +11:00
898260c177
Tidy up some storage close handling
2020-10-11 12:40:43 +11:00
7455b98a0f
Switch to crawshaw.io/sqlite
...
I'm using it elsewhere, and it seems to conflict when linking with github.com/mattn/go-sqlite3.
2020-07-15 16:16:14 +10:00
235c67a6ec
Remove debug logging from storage file implementation
2020-06-01 11:12:51 +10:00
604af2be82
storage file implementation: Error on short writes
2020-06-01 11:12:33 +10:00
69a0f2f1e4
Got file storage working with segment index
2020-06-01 00:14:51 +10:00
7fec6785bf
Abstract out segments mapping and use it in mmap storage
2020-05-31 21:00:19 +10:00
c04f09ee42
Fix panic closing torrent that failed to open storage
2020-05-21 12:17:10 +10:00
ae9aaaaa20
Use anacrolix/log in storage completion
2020-05-03 14:10:46 +10:00
2ea04e9083
Use default bbolt package alias
2020-03-24 12:55:03 +11:00
7175dba614
Fix bbolt compatibility with go1.14
2020-03-24 12:55:03 +11:00
19ce53e69f
Make io.EOF an expected error from storage.Piece.ReadAt
...
Fixes #381 .
2020-02-27 16:45:57 +11:00
c855c7a366
Differentiate between storage.ClientImpl{,Closer}
2020-02-21 14:12:44 +11:00
72e54fb353
Resource per piece storage: Store incomplete chunks separately
2020-01-14 10:51:09 +11:00
83867dbbe9
Move entirely to etcd-io/bbolt
2019-11-04 09:19:07 +11:00
e7bd5c2991
Replace github.com/boltdb/bolt with github.com/etcd-io/bbolt
2019-10-12 11:07:49 +11:00
e3420e0b60
Don't close shared client piece completion in mmap storage
...
Fixes #335 .
2019-10-11 17:37:22 +11:00
cecd83f966
Don't ignore error getting mmap piece completion
2019-10-11 17:36:46 +11:00
cb1bf0f413
goimports -local
2019-08-21 20:58:40 +10:00
fc4fab91f5
Switch to goimports import sorting
...
Used to use sortimports, but it's old, and goimports seems to have an opinion now.
2018-11-02 23:12:01 +11:00
bd5ae2cb6d
Fix doc comment
2018-07-15 12:56:28 +10:00
ffd41230a6
Fix file permissions
2018-06-09 15:11:28 +02:00
a101ebb07e
package assert->expect
2018-04-12 11:34:24 +10:00
6441e98f62
Implement go vet recommendations
2018-02-02 19:28:09 +11:00
e686523178
storage: Export NewSqlitePieceCompletion
2018-01-12 11:09:33 +11:00
d5e2d7ce99
storage: Don't check completion state on every write
2018-01-12 10:45:19 +11:00
f3ff3821ec
file storage: Report errors getting completion
2018-01-12 10:41:42 +11:00
78e23734d4
Clarify the PieceCompletion.Set method bool argument
2018-01-11 22:10:52 +11:00
a82c76defb
Update sqlite3 piece completion
2018-01-11 22:10:00 +11:00
09218898e9
Use NoSync with bolt storage and piece completion
...
Fixes terrible slowness on Linux.
2018-01-09 23:11:34 +11:00
57fec52371
mmap_span: Synchronize access to memory maps to avoid race when unmapping
2018-01-06 16:39:52 +11:00
6bcf2a58b2
storage: Rename mmapStorage->mmapClientImpl
2018-01-06 16:38:41 +11:00
b9a1cb5b8b
missinggo.LimitLen changed
2017-12-03 13:44:08 +11:00
520da249a1
Test that mmap size doesn't exceed system memory address width
...
Should fix http://paste.ubuntu.com/26074620/
2017-12-02 10:26:09 +11:00
4190856b6c
sortimports
2017-12-01 18:12:29 +11:00
cecc70f4cc
More megacheck
2017-11-08 00:12:34 +11:00
dd083a4e11
Track completion known to implementation state
...
Addresses #193
2017-10-12 16:09:32 +11:00
018afed5a7
storage: Remove incorrect comment
2017-09-23 15:27:26 +10:00
9e6cdff175
storage: Don't add empty mmaps to the mmap span
2017-09-18 13:41:19 +10:00
19adb2cac3
Improve mmap storage error messages
...
Trying to map unusual Info's gave unhelpful messages.
2017-09-18 12:19:34 +10:00
cee24fe21b
storage: Add TODO about handling Close errors
2017-09-12 18:31:56 +10:00
edcb9deb95
bug fix: initial infoHash for mmapStoragePiece
2017-07-10 19:33:39 +08:00
49648b9ae1
Close implicit Client default storage on Client.Close
...
Fixes #158
2017-06-01 22:57:08 +10:00
c2f77fc84a
Make bolt completion DB directory if necessary
2017-06-01 11:20:50 +10:00
d8c7f37d2b
make pieceCompletion configurable
2017-05-22 10:23:37 +08:00
65a7c3daa7
storage: Fix race condition in mapPieceCompletion
2017-03-30 21:38:44 +11:00
9bd463ffc0
storage: Add package doc
2017-03-18 13:03:02 +11:00
9d96cd659f
fix `Spec.Storage` and allow per-torrent dir
...
`TorrentSpec.Storage` was not honored when calling `Client.AddTorrentSpec`
and the configured `cfg.DefaultStorage` was always used. Now if you construct
your `TorrentSpec` you can specify any `StorageImpl`
Also, the most common use case for custom storage being per-torrent paths for
FileStorage, this adds a `pathMaker` function to the File implementation that
allows customization, along with the default (always use base path) and my use
case (which seemed common enough from the Gitter chat) which is infohash based
subdirectories.
All Public methods have not changed signature, but 1 private method did, hence
the test update.
2017-03-16 14:40:21 +00:00
1c37903a74
Remove FileStorePieces storage backend
...
ResourcePIeces is now preferred.
2017-01-05 17:00:59 +11:00
5e83287890
storage: ClientImpl interface now includes Close
...
This is now necessary because bolt doesn't allow multiple DB handles, so the bolt piece completion DB instance must be at the client level. The new method is not currently used from the Client however, this maybe necessary to avoid leaks.
2016-10-25 20:00:09 +11:00
3814a50fa0
storage: Fix spelling mistake
2016-10-25 19:57:35 +11:00
27893acf34
storage: Only include sqlite3 piece completion implementation if cgo is enabled
...
Issue #124 .
2016-10-25 19:54:09 +11:00
18309b866b
storage: Remove sqlite3 import from bolt implementation
...
#124
2016-10-25 19:52:44 +11:00
5c5a26afed
Add bolt piece completion DB
...
This means it can be persistent without needing cgo. Fixes issues #115 and #124 .
2016-10-25 19:07:26 +11:00
69ffbe6a48
storage: Move sqlite3 import to appropriate file
2016-09-16 11:19:26 +10:00
d4e30f84f2
Rename some of the unexported file storage types
2016-09-12 17:26:39 +10:00
b5ed171ac5
Create zero-length files in the file storage when the torrent storage is opened
...
Fixes #111 .
2016-09-12 17:01:00 +10:00
1e919dd6b1
Rework storage interfaces to make them simpler to implement
...
This allows lots of behaviour to be baked into the new Client, Torrent and Piece wrappers, rather than duplicating (badly) them in all the backend implementations.
2016-09-02 15:10:57 +10:00
db3be3441f
storage.boltDB: Some comments, and use global completedValue
2016-08-31 21:00:44 +10:00
c121615fab
Extract chunkSize as a const
2016-08-31 20:11:04 +10:00
e51cb07f6e
Uncomment the error handling in storage.boltDBPiece.ReadAt
2016-08-31 18:02:45 +10:00
3c2db92a83
Add boltdb storage backend
2016-08-31 17:48:50 +10:00
2a1cef7c9e
Remove the InfoEx type, and don't generate its infohash on the fly
...
Fixes #106 .
2016-08-26 20:29:05 +10:00
a1ba463750
Fix issue #97
2016-07-12 16:45:22 +10:00
f055abe2fc
Fix issue #96
...
In the native file-based storage, mark pieces incomplete if the necessary file data is missing, or there's a read error on a piece.
2016-07-10 23:03:59 +10:00
9c837a03d0
Move issue #95 tests into their own file
2016-07-09 00:36:53 +10:00
11a53fa732
Fix issue #95 : Closing torrent storage also closed client storage for some storage types
2016-07-09 00:36:32 +10:00
408686b440
Add a test for issue #95
2016-07-09 00:28:12 +10:00
cd1c1a9fed
mmap storage: Close pieceCompletion on Close
2016-07-06 14:02:15 +10:00
dbb3c25b1a
storage: Move pieceCompletion interface into correct file
2016-07-06 13:38:16 +10:00
ce0a3d56f1
storage: Remove debug log
2016-06-20 18:21:16 +10:00
d80d0db9d5
storage: Also use completion DB in mmap implementation
2016-06-20 18:20:59 +10:00
cee200a5a2
Add piece completion storage
...
Toward fixing https://github.com/anacrolix/torrent/issues/50 .
2016-06-20 17:51:05 +10:00
b75e85d187
storage.MMapTorrent should not be exported
2016-06-20 16:11:43 +10:00
869fb25cbc
Changes to missinggo resource package
2016-05-16 22:32:00 +10:00
c1e0d37266
Rename some storage funcs
2016-05-16 22:02:03 +10:00
02ea8b1081
Rename storage.I->Client
2016-05-16 21:50:43 +10:00
61e43f4261
Add piece-per-resource storage
2016-05-16 18:44:58 +10:00
0aabd37e6e
Add some comments to file storage
2016-05-09 15:46:48 +10:00
9f4bf70e17
refactor struct identifiers to follow conventional go names
2016-04-18 23:11:11 -05:00
86e34977de
piece-file storage: Use the receiver-local FileStore field
...
Because aliens, mainly.
2016-04-14 17:28:30 +10:00
b80bb6393a
Use metainfo.Hash for piece and info hashes
2016-04-04 13:01:31 +10:00
e3a7433154
piece file storage: Don't write to completed pieces
2016-04-03 16:37:50 +10:00
f19e8b32aa
storage: Don't read bad completed piece files
2016-04-03 16:33:31 +10:00
7638d678a8
storage: Fix io.EOF on short piece files
2016-04-02 16:57:22 +11:00
7790e72832
Move storage.FileStore interface to missinggo
2016-03-30 19:12:33 +11:00
baa8c1aed2
Few fixes for downstream projects
2016-03-29 12:13:03 +11:00
ee22446440
Reimplement piece storage
2016-03-29 11:14:34 +11:00
a5b54f21a1
Make opening a torrent in storage an explicit method
...
This is storage types where opening can fail, like mmap
2016-03-28 22:40:29 +11:00
775cf53809
Get mmap storage working
2016-03-28 21:57:04 +11:00
b97b50aca9
New storage interface
2016-03-28 20:38:30 +11:00
Matt Joiner