dlss
/
FedP2P
4
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
2,582 Commits 1 Branch 0 Tags 9.2 MiB
Commit Graph

145 Commits

Author SHA1 Message Date
Matt Joiner 86a2b71c47 Code comment 2021-02-19 12:37:26 +11:00
Matt Joiner 579925fa94 Fix panic in benchmark 2021-02-05 10:01:33 +11:00
Matt Joiner bc0936c44a Use locks on piece per resource pieces to prevent races in reading incomplete data 2021-02-04 19:18:59 +11:00
Matt Joiner 78c77c0b45 Expose mmap_size in sqlite storage, and change default to 8 MiB 2021-02-04 18:29:40 +11:00
Matt Joiner 371d54ab76 Fix Close race in sqlite storage when batch writes disabled 2021-02-03 19:27:26 +11:00
Matt Joiner 6f9c2fed61 Add batch writes cases to sqlite storage benchmarks 2021-02-03 18:55:14 +11:00
Matt Joiner 48843c0b87 Fix for go<1.16 2021-02-02 10:41:39 +11:00
Matt Joiner 15ccbc95c6 Fix sqlite storage for numconns 1 2021-02-02 10:41:39 +11:00
Matt Joiner 270a2ba1ae Switch to reading consecutive incomplete chunks 
This fixes a race where a sqlite conn isn't reserved for the read part of a MarkComplete operation after the write has already begun.
 2021-02-02 10:41:38 +11:00
Matt Joiner fbc9a77cd8 Escape URI path 2021-01-26 21:49:53 +11:00
Matt Joiner 9d6bf7a4f0 Remove cast(data as blob) workaround 
Upstream merged the fix. Good performance boost.
 2021-01-25 15:54:37 +11:00
Matt Joiner c424a2510e Create index on blob(last_used) 2021-01-25 15:54:37 +11:00
Matt Joiner 3e34763678 Add sqlite-storage-cli 2021-01-25 15:54:37 +11:00
Matt Joiner 5920bcae89 Remove obsolete reference to piece storage opts 2021-01-25 15:54:37 +11:00
Matt Joiner 25c60b72eb Fix race in writes to closed provider 2021-01-25 15:54:37 +11:00
Matt Joiner beb9ec8ffe Fix transfer tests involving sqlite memory storage 2021-01-25 15:54:37 +11:00
Matt Joiner b69bb34eaf Rejig some storage options 2021-01-25 15:54:37 +11:00
Matt Joiner 28682f6ae6 Fix unused opts param 2021-01-25 15:54:37 +11:00
Matt Joiner 19f5984aec Include context.Context and pprof label in withConn 2021-01-25 15:54:37 +11:00
Matt Joiner 2ddd3169ef Move storage piece benchmarks to storage/test and add a lot more dials 2021-01-25 15:54:37 +11:00
Matt Joiner cb5f80ec11 Improve configurability and add PutSized to sqlite storage 2021-01-25 15:54:37 +11:00
Matt Joiner 434dfdf6e0 Add MarkComplete benchmark for sqlite 2021-01-25 15:54:37 +11:00
Matt Joiner cb1d6bfc8f Fix boundary conditions trimming sqlite3 storage cache 2021-01-18 14:48:30 +11:00
Matt Joiner 388e6899a1 piece resource storage: Delete incomplete chunks concurrently after writing complete piece 2020-11-21 13:44:23 +11:00
Matt Joiner b35c204238 Use ConsecutiveChunkWriter in MarkComplete for piece resource storage if available 2020-11-21 13:44:23 +11:00
Matt Joiner 41ef267c1b sqlite storage: Ensure that chunks are consecutive 2020-11-21 13:44:23 +11:00
Matt Joiner 8ef37cb2e6 sqlite storage: Remove num batched write queries log message 2020-11-16 16:37:11 +11:00
Matt Joiner d84367c9a6 Update to race fixes in filecache and sqlite 2020-11-16 16:37:11 +11:00
Matt Joiner 258585fd8b sqlite storage: Buffer write requests 2020-11-16 16:37:11 +11:00
Matt Joiner b0ebc856d9 sqlite storage: Add some expvars 2020-11-16 16:37:11 +11:00
Matt Joiner e65fac26ac sqlite storage: Add error return from withConn 2020-11-16 16:37:11 +11:00
Matt Joiner 7afa4a2de4 Fix panic in when bbolt storage has Closed 2020-11-16 16:37:11 +11:00
Matt Joiner 7cb74b158f Replace storage.IncompletePieceToWriter with io.Writer 
It was incorrect to assume piece hashing only operates on incomplete chunk data. This actually uncovered a bug where duplicate hash checks occurred, and the redundant checks would fail due to not reading the completed data.
 2020-11-06 16:23:38 +11:00
Matt Joiner 96b640065a sqlite storage: Add capacity management 2020-11-03 15:17:12 +11:00
Matt Joiner 8afb05a8a5 sqlite storage: Batch writes synchronously instead of buffering them first 2020-11-03 15:16:57 +11:00
Matt Joiner 3c102393d1 Comment on storage.IncompletePieceToWriter 2020-11-03 14:28:16 +11:00
Matt Joiner 784345e9f7 sqlite storage: Init schema in NewPool instead of NewProvider and add an option to disable 2020-11-03 13:11:44 +11:00
Matt Joiner d04622e4b3 sqlite storage: Track total blob data size manually 
Works around possible full table scan at startup, and possible lack of caching around cast(data as blob).
 2020-11-03 13:10:17 +11:00
Matt Joiner 636b20b860 Add write incomplete/consecutive chunks interfaces 2020-11-02 15:35:07 +11:00
Matt Joiner 17e22516ff sqlite storage: Add batched writes 2020-10-30 19:46:51 +11:00
Matt Joiner ba70ad9b41 sqlite storage: Set mmap_size to a very large size 
Seems to fall back to ~2GiB on my system.
 2020-10-30 18:40:47 +11:00
Matt Joiner b75ebbf9e7 sqlite storage: Provide helpers and reasonable defaults 2020-10-30 12:20:54 +11:00
Matt Joiner fc039262d9 sqlite storage: Force data to be used as a blob 
There's a bug in crawshaw.io/sqlite, and some forks where inserting []byte results in a text type instead of blob. To ensure things work correctly, we coerce data to blob wherever we can. See https://github.com/crawshaw/sqlite/issues/94 and the fork that fixes it.
 2020-10-30 10:47:50 +11:00
Matt Joiner 8e1a8440bf Performance fiddling on sqlite storage 2020-10-27 17:07:49 +11:00
Matt Joiner e30084223d sqlite storage: Include capacity management 2020-10-27 11:08:37 +11:00
Matt Joiner 55d4bcaf26 sqlite storage: Do ReadAt without using incremental I/O 2020-10-27 11:08:08 +11:00
Matt Joiner 7fa8c604ab sqlite storage: Working recursive CTE for excess data 2020-10-23 11:03:42 +11:00
Matt Joiner 1fc63bdded sqlite storage: Add NewProviderPool 2020-10-23 09:03:44 +11:00
Matt Joiner bb2ce15114 sqlite storage: Rename table to blob 
Avoiding plural table names.
 2020-10-23 09:01:15 +11:00
Matt Joiner 8088002ba6 Fix error handling for bad torrent adds with safe file handling 2020-10-15 16:03:43 +11:00