Commit Graph

224 Commits

Author SHA1 Message Date
Matt Joiner c585b84126 Set smarter defaults 2021-05-17 11:56:10 +10:00
Matt Joiner fc0f2d146d Set page size before initializing connections
Setting page_size seems to be ignored if done after setting journal_mode, specifically to WAL I think. There's huge performance benefits to getting it right.
2021-05-17 11:56:10 +10:00
Matt Joiner ad421ea2d0 Default to sqlite piece completion for dir if cgo enabled 2021-05-14 15:51:26 +10:00
Matt Joiner a4e84688ae Fix sqlite piece completion 2021-05-14 15:45:54 +10:00
Matt Joiner 961cbfd765 Add CustomDirect BenchmarkMarkComplete sub-test 2021-05-14 15:42:54 +10:00
Matt Joiner 7b3e48b915 Expose SetSynchronous as an option 2021-05-14 15:41:56 +10:00
Matt Joiner 5030804b01 Fix race in MarkNotComplete 2021-05-14 15:41:28 +10:00
Matt Joiner e5d21dbf34 Don't create blobs when reading 2021-05-14 15:41:23 +10:00
Matt Joiner b151514154 Tidy up sqlite direct piece receiver name 2021-05-14 15:40:59 +10:00
Matt Joiner 9feb85d0d1 Remove unused method 2021-05-14 15:40:38 +10:00
Matt Joiner 627b75c8b4 Merge branch 'sqlite-direct' 2021-05-14 15:36:35 +10:00
gitpubber b8e1c95399
Allows one to use Custom PieceCompletion (#486)
* Change ClientImpl to ClientImplCloser

We cannot close underlying Boltdb PieceCompletion if it is not closable

* Update file.go
2021-05-09 19:34:41 +10:00
gitpubber 5e53f35b72 Change ClientImpl to ClientImplCloser
We cannot close underlying Boltdb PieceCompletion if it is not closable
2021-05-09 11:04:43 +10:00
Matt Joiner 8dcc52ccf9 Don't set the page size by default 2021-05-06 15:25:01 +10:00
Matt Joiner 96574468c5 Expose a variety of blob cleanup styles 2021-05-06 15:17:31 +10:00
Matt Joiner abe003b6b3 Benchmark different mmap sizes and journal modes 2021-05-06 12:00:20 +10:00
Matt Joiner 20c00dedc0 Rework lots of option handling 2021-05-05 21:36:36 +10:00
Matt Joiner acfe0ba87e Rename new.go 2021-05-05 15:55:08 +10:00
Matt Joiner b583fe3d47 Use bench timer more effectively, and use Piece.WriteTo 2021-05-05 15:53:43 +10:00
Matt Joiner c3768da38c Use sync.Map in map piece completion 2021-05-05 12:38:26 +10:00
Matt Joiner cf8bd49abe Big rename of files and types in storage 2021-05-05 12:05:02 +10:00
Matt Joiner 745a34b43e Handle expired blobs as they occur 2021-05-05 11:47:39 +10:00
Matt Joiner 675a0ab0dc Rework to use a pool of blobs 2021-05-05 10:02:15 +10:00
Matt Joiner 855144212c Get benchmarks working 2021-05-04 22:56:43 +10:00
Matt Joiner afea28091f Implement sqlite directly without using piece resources 2021-05-04 19:51:42 +10:00
Matt Joiner 256101768e piece resource storage: Do sized puts for WriteAts if possible 2021-05-04 12:44:51 +10:00
Matt Joiner 0021c2a70c Use BenchmarkMarkComplete for non-sqlite storages too 2021-05-04 12:44:51 +10:00
Matt Joiner f00f513706 Add missing closeMu use in ReadConsecutiveChunks 2021-02-22 16:39:21 +11:00
Matt Joiner ac05b69e11 Only write update bolt piece completion if the existing value differs 2021-02-22 14:51:34 +11:00
Matt Joiner 86a2b71c47 Code comment 2021-02-19 12:37:26 +11:00
Matt Joiner 579925fa94 Fix panic in benchmark 2021-02-05 10:01:33 +11:00
Matt Joiner bc0936c44a Use locks on piece per resource pieces to prevent races in reading incomplete data 2021-02-04 19:18:59 +11:00
Matt Joiner 78c77c0b45 Expose mmap_size in sqlite storage, and change default to 8 MiB 2021-02-04 18:29:40 +11:00
Matt Joiner 371d54ab76 Fix Close race in sqlite storage when batch writes disabled 2021-02-03 19:27:26 +11:00
Matt Joiner 6f9c2fed61 Add batch writes cases to sqlite storage benchmarks 2021-02-03 18:55:14 +11:00
Matt Joiner 48843c0b87 Fix for go<1.16 2021-02-02 10:41:39 +11:00
Matt Joiner 15ccbc95c6 Fix sqlite storage for numconns 1 2021-02-02 10:41:39 +11:00
Matt Joiner 270a2ba1ae Switch to reading consecutive incomplete chunks
This fixes a race where a sqlite conn isn't reserved for the read part of a MarkComplete operation after the write has already begun.
2021-02-02 10:41:38 +11:00
Matt Joiner fbc9a77cd8 Escape URI path 2021-01-26 21:49:53 +11:00
Matt Joiner 9d6bf7a4f0 Remove cast(data as blob) workaround
Upstream merged the fix. Good performance boost.
2021-01-25 15:54:37 +11:00
Matt Joiner c424a2510e Create index on blob(last_used) 2021-01-25 15:54:37 +11:00
Matt Joiner 3e34763678 Add sqlite-storage-cli 2021-01-25 15:54:37 +11:00
Matt Joiner 5920bcae89 Remove obsolete reference to piece storage opts 2021-01-25 15:54:37 +11:00
Matt Joiner 25c60b72eb Fix race in writes to closed provider 2021-01-25 15:54:37 +11:00
Matt Joiner beb9ec8ffe Fix transfer tests involving sqlite memory storage 2021-01-25 15:54:37 +11:00
Matt Joiner b69bb34eaf Rejig some storage options 2021-01-25 15:54:37 +11:00
Matt Joiner 28682f6ae6 Fix unused opts param 2021-01-25 15:54:37 +11:00
Matt Joiner 19f5984aec Include context.Context and pprof label in withConn 2021-01-25 15:54:37 +11:00
Matt Joiner 2ddd3169ef Move storage piece benchmarks to storage/test and add a lot more dials 2021-01-25 15:54:37 +11:00
Matt Joiner cb5f80ec11 Improve configurability and add PutSized to sqlite storage 2021-01-25 15:54:37 +11:00
Matt Joiner 434dfdf6e0 Add MarkComplete benchmark for sqlite 2021-01-25 15:54:37 +11:00
Matt Joiner cb1d6bfc8f Fix boundary conditions trimming sqlite3 storage cache 2021-01-18 14:48:30 +11:00
Matt Joiner 388e6899a1 piece resource storage: Delete incomplete chunks concurrently after writing complete piece 2020-11-21 13:44:23 +11:00
Matt Joiner b35c204238 Use ConsecutiveChunkWriter in MarkComplete for piece resource storage if available 2020-11-21 13:44:23 +11:00
Matt Joiner 41ef267c1b sqlite storage: Ensure that chunks are consecutive 2020-11-21 13:44:23 +11:00
Matt Joiner 8ef37cb2e6 sqlite storage: Remove num batched write queries log message 2020-11-16 16:37:11 +11:00
Matt Joiner d84367c9a6 Update to race fixes in filecache and sqlite 2020-11-16 16:37:11 +11:00
Matt Joiner 258585fd8b sqlite storage: Buffer write requests 2020-11-16 16:37:11 +11:00
Matt Joiner b0ebc856d9 sqlite storage: Add some expvars 2020-11-16 16:37:11 +11:00
Matt Joiner e65fac26ac sqlite storage: Add error return from withConn 2020-11-16 16:37:11 +11:00
Matt Joiner 7afa4a2de4 Fix panic in when bbolt storage has Closed 2020-11-16 16:37:11 +11:00
Matt Joiner 7cb74b158f Replace storage.IncompletePieceToWriter with io.Writer
It was incorrect to assume piece hashing only operates on incomplete chunk data. This actually uncovered a bug where duplicate hash checks occurred, and the redundant checks would fail due to not reading the completed data.
2020-11-06 16:23:38 +11:00
Matt Joiner 96b640065a sqlite storage: Add capacity management 2020-11-03 15:17:12 +11:00
Matt Joiner 8afb05a8a5 sqlite storage: Batch writes synchronously instead of buffering them first 2020-11-03 15:16:57 +11:00
Matt Joiner 3c102393d1 Comment on storage.IncompletePieceToWriter 2020-11-03 14:28:16 +11:00
Matt Joiner 784345e9f7 sqlite storage: Init schema in NewPool instead of NewProvider and add an option to disable 2020-11-03 13:11:44 +11:00
Matt Joiner d04622e4b3 sqlite storage: Track total blob data size manually
Works around possible full table scan at startup, and possible lack of caching around cast(data as blob).
2020-11-03 13:10:17 +11:00
Matt Joiner 636b20b860 Add write incomplete/consecutive chunks interfaces 2020-11-02 15:35:07 +11:00
Matt Joiner 17e22516ff sqlite storage: Add batched writes 2020-10-30 19:46:51 +11:00
Matt Joiner ba70ad9b41 sqlite storage: Set mmap_size to a very large size
Seems to fall back to ~2GiB on my system.
2020-10-30 18:40:47 +11:00
Matt Joiner b75ebbf9e7 sqlite storage: Provide helpers and reasonable defaults 2020-10-30 12:20:54 +11:00
Matt Joiner fc039262d9 sqlite storage: Force data to be used as a blob
There's a bug in crawshaw.io/sqlite, and some forks where inserting []byte results in a text type instead of blob. To ensure things work correctly, we coerce data to blob wherever we can. See https://github.com/crawshaw/sqlite/issues/94 and the fork that fixes it.
2020-10-30 10:47:50 +11:00
Matt Joiner 8e1a8440bf Performance fiddling on sqlite storage 2020-10-27 17:07:49 +11:00
Matt Joiner e30084223d sqlite storage: Include capacity management 2020-10-27 11:08:37 +11:00
Matt Joiner 55d4bcaf26 sqlite storage: Do ReadAt without using incremental I/O 2020-10-27 11:08:08 +11:00
Matt Joiner 7fa8c604ab sqlite storage: Working recursive CTE for excess data 2020-10-23 11:03:42 +11:00
Matt Joiner 1fc63bdded sqlite storage: Add NewProviderPool 2020-10-23 09:03:44 +11:00
Matt Joiner bb2ce15114 sqlite storage: Rename table to blob
Avoiding plural table names.
2020-10-23 09:01:15 +11:00
Matt Joiner 8088002ba6 Fix error handling for bad torrent adds with safe file handling 2020-10-15 16:03:43 +11:00
Matt Joiner 89235e180f Sanitize metainfo file paths for file-based storage
Fixes exploit where specially crafted infos can cause the client to write files to arbitrary locations on local storage when using file-based storages like mmap and file.
2020-10-15 15:45:08 +11:00
Matt Joiner 3820017a0c Add last_used to sqlite storage 2020-10-13 09:36:58 +11:00
Matt Joiner d820f7861c Add a sqlite data storage implementation 2020-10-11 12:58:27 +11:00
Matt Joiner 00ac333ea2 Panic on bad piece per resource chunk names
This was masking a bug in a coming sqlite storage implementation. Maybe it could be relaxed to checking for no filepath.Dir in the future.
2020-10-11 12:57:33 +11:00
Matt Joiner 417d7d1d48 Tweaks to storage error and completion handling 2020-10-11 12:54:03 +11:00
Matt Joiner 898260c177 Tidy up some storage close handling 2020-10-11 12:40:43 +11:00
Matt Joiner 7455b98a0f Switch to crawshaw.io/sqlite
I'm using it elsewhere, and it seems to conflict when linking with github.com/mattn/go-sqlite3.
2020-07-15 16:16:14 +10:00
Matt Joiner 235c67a6ec Remove debug logging from storage file implementation 2020-06-01 11:12:51 +10:00
Matt Joiner 604af2be82 storage file implementation: Error on short writes 2020-06-01 11:12:33 +10:00
Matt Joiner 69a0f2f1e4 Got file storage working with segment index 2020-06-01 00:14:51 +10:00
Matt Joiner 7fec6785bf Abstract out segments mapping and use it in mmap storage 2020-05-31 21:00:19 +10:00
Matt Joiner c04f09ee42 Fix panic closing torrent that failed to open storage 2020-05-21 12:17:10 +10:00
Matt Joiner ae9aaaaa20 Use anacrolix/log in storage completion 2020-05-03 14:10:46 +10:00
Matt Joiner 2ea04e9083 Use default bbolt package alias 2020-03-24 12:55:03 +11:00
Matt Joiner 7175dba614 Fix bbolt compatibility with go1.14 2020-03-24 12:55:03 +11:00
Matt Joiner 19ce53e69f Make io.EOF an expected error from storage.Piece.ReadAt
Fixes #381.
2020-02-27 16:45:57 +11:00
Matt Joiner c855c7a366 Differentiate between storage.ClientImpl{,Closer} 2020-02-21 14:12:44 +11:00
Matt Joiner 72e54fb353 Resource per piece storage: Store incomplete chunks separately 2020-01-14 10:51:09 +11:00
Matt Joiner 83867dbbe9 Move entirely to etcd-io/bbolt 2019-11-04 09:19:07 +11:00
Matt Joiner e7bd5c2991 Replace github.com/boltdb/bolt with github.com/etcd-io/bbolt 2019-10-12 11:07:49 +11:00
Matt Joiner e3420e0b60 Don't close shared client piece completion in mmap storage
Fixes #335.
2019-10-11 17:37:22 +11:00