c3c982b621
Fixes for wasm
2021-06-23 17:24:50 +10:00
5fc42e8105
Remove sqlite piece-resource storage
2021-06-21 12:29:37 +10:00
ebd19af795
Merge branch 'request-strategy-rewrite'
2021-06-21 12:04:06 +10:00
e2a559e243
Allow Storage Backends to do their own Hashing ( #518 )
...
* Allow Storage Backends to do their own Hashing
- Describes an optional interface 'SelfHashing' that a storage backend's type implementing 'PieceImpl' may also implement in order to calculate piece hashsums itself.
- Alters the 'hashPiece' function in the torrent package to look for types implementing 'SelfHashing' . If not implemented, calculate the hash as usual.
2021-06-08 15:45:35 +10:00
a68f040ea6
Fix nil context being passed to function
2021-06-07 19:32:02 +10:00
f183acefe2
Add default param name in TorrentImpl.Piece func
2021-06-07 13:01:40 +10:00
ef39f408fe
Add storage/disabled
...
The default storage will create empty files on torrent open, which is undesirable in some circumstances. This storage implementation is explicit about not storing anything.
2021-06-07 13:01:40 +10:00
5f8471e21b
Rework storage.TorrentImpl to support shared capacity key
2021-06-07 13:01:39 +10:00
8f6f47a2e0
Added build tags to disable few packages if necessary ( #499 )
2021-05-25 18:48:59 +10:00
a60a4c9117
Progress testing without cgo a bit
2021-05-21 15:22:26 +10:00
b21aebeaae
Make wal the default for sqlite resource pieces
...
Fixes broken test as a result.
2021-05-17 11:56:10 +10:00
7798e2a306
Make synchronous=off the default
2021-05-17 11:56:10 +10:00
c585b84126
Set smarter defaults
2021-05-17 11:56:10 +10:00
fc0f2d146d
Set page size before initializing connections
...
Setting page_size seems to be ignored if done after setting journal_mode, specifically to WAL I think. There's huge performance benefits to getting it right.
2021-05-17 11:56:10 +10:00
ad421ea2d0
Default to sqlite piece completion for dir if cgo enabled
2021-05-14 15:51:26 +10:00
a4e84688ae
Fix sqlite piece completion
2021-05-14 15:45:54 +10:00
961cbfd765
Add CustomDirect BenchmarkMarkComplete sub-test
2021-05-14 15:42:54 +10:00
7b3e48b915
Expose SetSynchronous as an option
2021-05-14 15:41:56 +10:00
5030804b01
Fix race in MarkNotComplete
2021-05-14 15:41:28 +10:00
e5d21dbf34
Don't create blobs when reading
2021-05-14 15:41:23 +10:00
b151514154
Tidy up sqlite direct piece receiver name
2021-05-14 15:40:59 +10:00
9feb85d0d1
Remove unused method
2021-05-14 15:40:38 +10:00
627b75c8b4
Merge branch 'sqlite-direct'
2021-05-14 15:36:35 +10:00
b8e1c95399
Allows one to use Custom PieceCompletion ( #486 )
...
* Change ClientImpl to ClientImplCloser
We cannot close underlying Boltdb PieceCompletion if it is not closable
* Update file.go
2021-05-09 19:34:41 +10:00
5e53f35b72
Change ClientImpl to ClientImplCloser
...
We cannot close underlying Boltdb PieceCompletion if it is not closable
2021-05-09 11:04:43 +10:00
8dcc52ccf9
Don't set the page size by default
2021-05-06 15:25:01 +10:00
96574468c5
Expose a variety of blob cleanup styles
2021-05-06 15:17:31 +10:00
abe003b6b3
Benchmark different mmap sizes and journal modes
2021-05-06 12:00:20 +10:00
20c00dedc0
Rework lots of option handling
2021-05-05 21:36:36 +10:00
acfe0ba87e
Rename new.go
2021-05-05 15:55:08 +10:00
b583fe3d47
Use bench timer more effectively, and use Piece.WriteTo
2021-05-05 15:53:43 +10:00
c3768da38c
Use sync.Map in map piece completion
2021-05-05 12:38:26 +10:00
cf8bd49abe
Big rename of files and types in storage
2021-05-05 12:05:02 +10:00
745a34b43e
Handle expired blobs as they occur
2021-05-05 11:47:39 +10:00
675a0ab0dc
Rework to use a pool of blobs
2021-05-05 10:02:15 +10:00
855144212c
Get benchmarks working
2021-05-04 22:56:43 +10:00
afea28091f
Implement sqlite directly without using piece resources
2021-05-04 19:51:42 +10:00
256101768e
piece resource storage: Do sized puts for WriteAts if possible
2021-05-04 12:44:51 +10:00
0021c2a70c
Use BenchmarkMarkComplete for non-sqlite storages too
2021-05-04 12:44:51 +10:00
f00f513706
Add missing closeMu use in ReadConsecutiveChunks
2021-02-22 16:39:21 +11:00
ac05b69e11
Only write update bolt piece completion if the existing value differs
2021-02-22 14:51:34 +11:00
86a2b71c47
Code comment
2021-02-19 12:37:26 +11:00
579925fa94
Fix panic in benchmark
2021-02-05 10:01:33 +11:00
bc0936c44a
Use locks on piece per resource pieces to prevent races in reading incomplete data
2021-02-04 19:18:59 +11:00
78c77c0b45
Expose mmap_size in sqlite storage, and change default to 8 MiB
2021-02-04 18:29:40 +11:00
371d54ab76
Fix Close race in sqlite storage when batch writes disabled
2021-02-03 19:27:26 +11:00
6f9c2fed61
Add batch writes cases to sqlite storage benchmarks
2021-02-03 18:55:14 +11:00
48843c0b87
Fix for go<1.16
2021-02-02 10:41:39 +11:00
15ccbc95c6
Fix sqlite storage for numconns 1
2021-02-02 10:41:39 +11:00
270a2ba1ae
Switch to reading consecutive incomplete chunks
...
This fixes a race where a sqlite conn isn't reserved for the read part of a MarkComplete operation after the write has already begun.
2021-02-02 10:41:38 +11:00
fbc9a77cd8
Escape URI path
2021-01-26 21:49:53 +11:00
9d6bf7a4f0
Remove cast(data as blob) workaround
...
Upstream merged the fix. Good performance boost.
2021-01-25 15:54:37 +11:00
c424a2510e
Create index on blob(last_used)
2021-01-25 15:54:37 +11:00
3e34763678
Add sqlite-storage-cli
2021-01-25 15:54:37 +11:00
5920bcae89
Remove obsolete reference to piece storage opts
2021-01-25 15:54:37 +11:00
25c60b72eb
Fix race in writes to closed provider
2021-01-25 15:54:37 +11:00
beb9ec8ffe
Fix transfer tests involving sqlite memory storage
2021-01-25 15:54:37 +11:00
b69bb34eaf
Rejig some storage options
2021-01-25 15:54:37 +11:00
28682f6ae6
Fix unused opts param
2021-01-25 15:54:37 +11:00
19f5984aec
Include context.Context and pprof label in withConn
2021-01-25 15:54:37 +11:00
2ddd3169ef
Move storage piece benchmarks to storage/test and add a lot more dials
2021-01-25 15:54:37 +11:00
cb5f80ec11
Improve configurability and add PutSized to sqlite storage
2021-01-25 15:54:37 +11:00
434dfdf6e0
Add MarkComplete benchmark for sqlite
2021-01-25 15:54:37 +11:00
cb1d6bfc8f
Fix boundary conditions trimming sqlite3 storage cache
2021-01-18 14:48:30 +11:00
388e6899a1
piece resource storage: Delete incomplete chunks concurrently after writing complete piece
2020-11-21 13:44:23 +11:00
b35c204238
Use ConsecutiveChunkWriter in MarkComplete for piece resource storage if available
2020-11-21 13:44:23 +11:00
41ef267c1b
sqlite storage: Ensure that chunks are consecutive
2020-11-21 13:44:23 +11:00
8ef37cb2e6
sqlite storage: Remove num batched write queries log message
2020-11-16 16:37:11 +11:00
d84367c9a6
Update to race fixes in filecache and sqlite
2020-11-16 16:37:11 +11:00
258585fd8b
sqlite storage: Buffer write requests
2020-11-16 16:37:11 +11:00
b0ebc856d9
sqlite storage: Add some expvars
2020-11-16 16:37:11 +11:00
e65fac26ac
sqlite storage: Add error return from withConn
2020-11-16 16:37:11 +11:00
7afa4a2de4
Fix panic in when bbolt storage has Closed
2020-11-16 16:37:11 +11:00
7cb74b158f
Replace storage.IncompletePieceToWriter with io.Writer
...
It was incorrect to assume piece hashing only operates on incomplete chunk data. This actually uncovered a bug where duplicate hash checks occurred, and the redundant checks would fail due to not reading the completed data.
2020-11-06 16:23:38 +11:00
96b640065a
sqlite storage: Add capacity management
2020-11-03 15:17:12 +11:00
8afb05a8a5
sqlite storage: Batch writes synchronously instead of buffering them first
2020-11-03 15:16:57 +11:00
3c102393d1
Comment on storage.IncompletePieceToWriter
2020-11-03 14:28:16 +11:00
784345e9f7
sqlite storage: Init schema in NewPool instead of NewProvider and add an option to disable
2020-11-03 13:11:44 +11:00
d04622e4b3
sqlite storage: Track total blob data size manually
...
Works around possible full table scan at startup, and possible lack of caching around cast(data as blob).
2020-11-03 13:10:17 +11:00
636b20b860
Add write incomplete/consecutive chunks interfaces
2020-11-02 15:35:07 +11:00
17e22516ff
sqlite storage: Add batched writes
2020-10-30 19:46:51 +11:00
ba70ad9b41
sqlite storage: Set mmap_size to a very large size
...
Seems to fall back to ~2GiB on my system.
2020-10-30 18:40:47 +11:00
b75ebbf9e7
sqlite storage: Provide helpers and reasonable defaults
2020-10-30 12:20:54 +11:00
fc039262d9
sqlite storage: Force data to be used as a blob
...
There's a bug in crawshaw.io/sqlite, and some forks where inserting []byte results in a text type instead of blob. To ensure things work correctly, we coerce data to blob wherever we can. See https://github.com/crawshaw/sqlite/issues/94 and the fork that fixes it.
2020-10-30 10:47:50 +11:00
8e1a8440bf
Performance fiddling on sqlite storage
2020-10-27 17:07:49 +11:00
e30084223d
sqlite storage: Include capacity management
2020-10-27 11:08:37 +11:00
55d4bcaf26
sqlite storage: Do ReadAt without using incremental I/O
2020-10-27 11:08:08 +11:00
7fa8c604ab
sqlite storage: Working recursive CTE for excess data
2020-10-23 11:03:42 +11:00
1fc63bdded
sqlite storage: Add NewProviderPool
2020-10-23 09:03:44 +11:00
bb2ce15114
sqlite storage: Rename table to blob
...
Avoiding plural table names.
2020-10-23 09:01:15 +11:00
8088002ba6
Fix error handling for bad torrent adds with safe file handling
2020-10-15 16:03:43 +11:00
89235e180f
Sanitize metainfo file paths for file-based storage
...
Fixes exploit where specially crafted infos can cause the client to write files to arbitrary locations on local storage when using file-based storages like mmap and file.
2020-10-15 15:45:08 +11:00
3820017a0c
Add last_used to sqlite storage
2020-10-13 09:36:58 +11:00
d820f7861c
Add a sqlite data storage implementation
2020-10-11 12:58:27 +11:00
00ac333ea2
Panic on bad piece per resource chunk names
...
This was masking a bug in a coming sqlite storage implementation. Maybe it could be relaxed to checking for no filepath.Dir in the future.
2020-10-11 12:57:33 +11:00
417d7d1d48
Tweaks to storage error and completion handling
2020-10-11 12:54:03 +11:00
898260c177
Tidy up some storage close handling
2020-10-11 12:40:43 +11:00
7455b98a0f
Switch to crawshaw.io/sqlite
...
I'm using it elsewhere, and it seems to conflict when linking with github.com/mattn/go-sqlite3.
2020-07-15 16:16:14 +10:00
235c67a6ec
Remove debug logging from storage file implementation
2020-06-01 11:12:51 +10:00
604af2be82
storage file implementation: Error on short writes
2020-06-01 11:12:33 +10:00
Matt Joiner