dlss
/
FedP2P
4
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
2,622 Commits 1 Branch 0 Tags 9.2 MiB
Commit Graph

154 Commits

Author SHA1 Message Date
Matt Joiner cf8bd49abe Big rename of files and types in storage 2021-05-05 12:05:02 +10:00
Matt Joiner 745a34b43e Handle expired blobs as they occur 2021-05-05 11:47:39 +10:00
Matt Joiner 675a0ab0dc Rework to use a pool of blobs 2021-05-05 10:02:15 +10:00
Matt Joiner 855144212c Get benchmarks working 2021-05-04 22:56:43 +10:00
Matt Joiner afea28091f Implement sqlite directly without using piece resources 2021-05-04 19:51:42 +10:00
Matt Joiner 256101768e piece resource storage: Do sized puts for WriteAts if possible 2021-05-04 12:44:51 +10:00
Matt Joiner 0021c2a70c Use BenchmarkMarkComplete for non-sqlite storages too 2021-05-04 12:44:51 +10:00
Matt Joiner f00f513706 Add missing closeMu use in ReadConsecutiveChunks 2021-02-22 16:39:21 +11:00
Matt Joiner ac05b69e11 Only write update bolt piece completion if the existing value differs 2021-02-22 14:51:34 +11:00
Matt Joiner 86a2b71c47 Code comment 2021-02-19 12:37:26 +11:00
Matt Joiner 579925fa94 Fix panic in benchmark 2021-02-05 10:01:33 +11:00
Matt Joiner bc0936c44a Use locks on piece per resource pieces to prevent races in reading incomplete data 2021-02-04 19:18:59 +11:00
Matt Joiner 78c77c0b45 Expose mmap_size in sqlite storage, and change default to 8 MiB 2021-02-04 18:29:40 +11:00
Matt Joiner 371d54ab76 Fix Close race in sqlite storage when batch writes disabled 2021-02-03 19:27:26 +11:00
Matt Joiner 6f9c2fed61 Add batch writes cases to sqlite storage benchmarks 2021-02-03 18:55:14 +11:00
Matt Joiner 48843c0b87 Fix for go<1.16 2021-02-02 10:41:39 +11:00
Matt Joiner 15ccbc95c6 Fix sqlite storage for numconns 1 2021-02-02 10:41:39 +11:00
Matt Joiner 270a2ba1ae Switch to reading consecutive incomplete chunks 
This fixes a race where a sqlite conn isn't reserved for the read part of a MarkComplete operation after the write has already begun.
 2021-02-02 10:41:38 +11:00
Matt Joiner fbc9a77cd8 Escape URI path 2021-01-26 21:49:53 +11:00
Matt Joiner 9d6bf7a4f0 Remove cast(data as blob) workaround 
Upstream merged the fix. Good performance boost.
 2021-01-25 15:54:37 +11:00
Matt Joiner c424a2510e Create index on blob(last_used) 2021-01-25 15:54:37 +11:00
Matt Joiner 3e34763678 Add sqlite-storage-cli 2021-01-25 15:54:37 +11:00
Matt Joiner 5920bcae89 Remove obsolete reference to piece storage opts 2021-01-25 15:54:37 +11:00
Matt Joiner 25c60b72eb Fix race in writes to closed provider 2021-01-25 15:54:37 +11:00
Matt Joiner beb9ec8ffe Fix transfer tests involving sqlite memory storage 2021-01-25 15:54:37 +11:00
Matt Joiner b69bb34eaf Rejig some storage options 2021-01-25 15:54:37 +11:00
Matt Joiner 28682f6ae6 Fix unused opts param 2021-01-25 15:54:37 +11:00
Matt Joiner 19f5984aec Include context.Context and pprof label in withConn 2021-01-25 15:54:37 +11:00
Matt Joiner 2ddd3169ef Move storage piece benchmarks to storage/test and add a lot more dials 2021-01-25 15:54:37 +11:00
Matt Joiner cb5f80ec11 Improve configurability and add PutSized to sqlite storage 2021-01-25 15:54:37 +11:00
Matt Joiner 434dfdf6e0 Add MarkComplete benchmark for sqlite 2021-01-25 15:54:37 +11:00
Matt Joiner cb1d6bfc8f Fix boundary conditions trimming sqlite3 storage cache 2021-01-18 14:48:30 +11:00
Matt Joiner 388e6899a1 piece resource storage: Delete incomplete chunks concurrently after writing complete piece 2020-11-21 13:44:23 +11:00
Matt Joiner b35c204238 Use ConsecutiveChunkWriter in MarkComplete for piece resource storage if available 2020-11-21 13:44:23 +11:00
Matt Joiner 41ef267c1b sqlite storage: Ensure that chunks are consecutive 2020-11-21 13:44:23 +11:00
Matt Joiner 8ef37cb2e6 sqlite storage: Remove num batched write queries log message 2020-11-16 16:37:11 +11:00
Matt Joiner d84367c9a6 Update to race fixes in filecache and sqlite 2020-11-16 16:37:11 +11:00
Matt Joiner 258585fd8b sqlite storage: Buffer write requests 2020-11-16 16:37:11 +11:00
Matt Joiner b0ebc856d9 sqlite storage: Add some expvars 2020-11-16 16:37:11 +11:00
Matt Joiner e65fac26ac sqlite storage: Add error return from withConn 2020-11-16 16:37:11 +11:00
Matt Joiner 7afa4a2de4 Fix panic in when bbolt storage has Closed 2020-11-16 16:37:11 +11:00
Matt Joiner 7cb74b158f Replace storage.IncompletePieceToWriter with io.Writer 
It was incorrect to assume piece hashing only operates on incomplete chunk data. This actually uncovered a bug where duplicate hash checks occurred, and the redundant checks would fail due to not reading the completed data.
 2020-11-06 16:23:38 +11:00
Matt Joiner 96b640065a sqlite storage: Add capacity management 2020-11-03 15:17:12 +11:00
Matt Joiner 8afb05a8a5 sqlite storage: Batch writes synchronously instead of buffering them first 2020-11-03 15:16:57 +11:00
Matt Joiner 3c102393d1 Comment on storage.IncompletePieceToWriter 2020-11-03 14:28:16 +11:00
Matt Joiner 784345e9f7 sqlite storage: Init schema in NewPool instead of NewProvider and add an option to disable 2020-11-03 13:11:44 +11:00
Matt Joiner d04622e4b3 sqlite storage: Track total blob data size manually 
Works around possible full table scan at startup, and possible lack of caching around cast(data as blob).
 2020-11-03 13:10:17 +11:00
Matt Joiner 636b20b860 Add write incomplete/consecutive chunks interfaces 2020-11-02 15:35:07 +11:00
Matt Joiner 17e22516ff sqlite storage: Add batched writes 2020-10-30 19:46:51 +11:00
Matt Joiner ba70ad9b41 sqlite storage: Set mmap_size to a very large size 
Seems to fall back to ~2GiB on my system.
 2020-10-30 18:40:47 +11:00