2009-09-22 12:56:53 +08:00
|
|
|
/*
|
2010-06-01 16:01:25 +08:00
|
|
|
* Compressed RAM block device
|
2009-09-22 12:56:53 +08:00
|
|
|
*
|
2010-01-28 23:51:35 +08:00
|
|
|
* Copyright (C) 2008, 2009, 2010 Nitin Gupta
|
2014-01-31 07:45:55 +08:00
|
|
|
* 2012, 2013 Minchan Kim
|
2009-09-22 12:56:53 +08:00
|
|
|
*
|
|
|
|
* This code is released using a dual license strategy: BSD/GPL
|
|
|
|
* You can choose the licence that better fits your requirements.
|
|
|
|
*
|
|
|
|
* Released under the terms of 3-clause BSD License
|
|
|
|
* Released under the terms of GNU General Public License Version 2.0
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
2010-06-01 16:01:25 +08:00
|
|
|
#ifndef _ZRAM_DRV_H_
|
|
|
|
#define _ZRAM_DRV_H_
|
2009-09-22 12:56:53 +08:00
|
|
|
|
zram: use crypto api to check alg availability
There is no way to get a string with all the crypto comp algorithms
supported by the crypto comp engine, so we need to maintain our own
backends list. At the same time we additionally need to use
crypto_has_comp() to make sure that the user has requested a compression
algorithm that is recognized by the crypto comp engine. Relying on
/proc/crypto is not an options here, because it does not show
not-yet-inserted compression modules.
Example:
modprobe zram
cat /proc/crypto | grep -i lz4
modprobe lz4
cat /proc/crypto | grep -i lz4
name : lz4
driver : lz4-generic
module : lz4
So the user can't tell exactly if the lz4 is really supported from
/proc/crypto output, unless someone or something has loaded it.
This patch also adds crypto_has_comp() to zcomp_available_show(). We
store all the compression algorithms names in zcomp's `backends' array,
regardless the CONFIG_CRYPTO_FOO configuration, but show only those that
are also supported by crypto engine. This helps user to know the exact
list of compression algorithms that can be used.
Example:
module lz4 is not loaded yet, but is supported by the crypto
engine. /proc/crypto has no information on this module, while
zram's `comp_algorithm' lists it:
cat /proc/crypto | grep -i lz4
cat /sys/block/zram0/comp_algorithm
[lzo] lz4 deflate lz4hc 842
We still use the `backends' array to determine if the requested
compression backend is known to crypto api. This array, however, may not
contain some entries, therefore as the last step we call crypto_has_comp()
function which attempts to insmod the requested compression algorithm to
determine if crypto api supports it. The advantage of this method is that
now we permit the usage of out-of-tree crypto compression modules
(implementing S/W or H/W compression).
[sergey.senozhatsky@gmail.com: zram-use-crypto-api-to-check-alg-availability-v3]
Link: http://lkml.kernel.org/r/20160604024902.11778-4-sergey.senozhatsky@gmail.com
Link: http://lkml.kernel.org/r/20160531122017.2878-5-sergey.senozhatsky@gmail.com
Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Acked-by: Minchan Kim <minchan@kernel.org>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-07-27 06:22:48 +08:00
|
|
|
#include <linux/rwsem.h>
|
2014-01-31 07:45:50 +08:00
|
|
|
#include <linux/zsmalloc.h>
|
zram: use crypto api to check alg availability
There is no way to get a string with all the crypto comp algorithms
supported by the crypto comp engine, so we need to maintain our own
backends list. At the same time we additionally need to use
crypto_has_comp() to make sure that the user has requested a compression
algorithm that is recognized by the crypto comp engine. Relying on
/proc/crypto is not an options here, because it does not show
not-yet-inserted compression modules.
Example:
modprobe zram
cat /proc/crypto | grep -i lz4
modprobe lz4
cat /proc/crypto | grep -i lz4
name : lz4
driver : lz4-generic
module : lz4
So the user can't tell exactly if the lz4 is really supported from
/proc/crypto output, unless someone or something has loaded it.
This patch also adds crypto_has_comp() to zcomp_available_show(). We
store all the compression algorithms names in zcomp's `backends' array,
regardless the CONFIG_CRYPTO_FOO configuration, but show only those that
are also supported by crypto engine. This helps user to know the exact
list of compression algorithms that can be used.
Example:
module lz4 is not loaded yet, but is supported by the crypto
engine. /proc/crypto has no information on this module, while
zram's `comp_algorithm' lists it:
cat /proc/crypto | grep -i lz4
cat /sys/block/zram0/comp_algorithm
[lzo] lz4 deflate lz4hc 842
We still use the `backends' array to determine if the requested
compression backend is known to crypto api. This array, however, may not
contain some entries, therefore as the last step we call crypto_has_comp()
function which attempts to insmod the requested compression algorithm to
determine if crypto api supports it. The advantage of this method is that
now we permit the usage of out-of-tree crypto compression modules
(implementing S/W or H/W compression).
[sergey.senozhatsky@gmail.com: zram-use-crypto-api-to-check-alg-availability-v3]
Link: http://lkml.kernel.org/r/20160604024902.11778-4-sergey.senozhatsky@gmail.com
Link: http://lkml.kernel.org/r/20160531122017.2878-5-sergey.senozhatsky@gmail.com
Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Acked-by: Minchan Kim <minchan@kernel.org>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-07-27 06:22:48 +08:00
|
|
|
#include <linux/crypto.h>
|
2009-09-22 12:56:53 +08:00
|
|
|
|
2014-04-08 06:38:12 +08:00
|
|
|
#include "zcomp.h"
|
|
|
|
|
2009-09-22 12:56:53 +08:00
|
|
|
/*-- Configurable parameters */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Pages that compress to size greater than this are stored
|
|
|
|
* uncompressed in memory.
|
|
|
|
*/
|
2011-09-10 07:01:00 +08:00
|
|
|
static const size_t max_zpage_size = PAGE_SIZE / 4 * 3;
|
2009-09-22 12:56:53 +08:00
|
|
|
|
|
|
|
/*
|
2010-05-13 16:54:21 +08:00
|
|
|
* NOTE: max_zpage_size must be less than or equal to:
|
2012-10-10 07:49:52 +08:00
|
|
|
* ZS_MAX_ALLOC_SIZE. Otherwise, zs_malloc() would
|
|
|
|
* always return failure.
|
2009-09-22 12:56:53 +08:00
|
|
|
*/
|
|
|
|
|
|
|
|
/*-- End of configurable params */
|
|
|
|
|
|
|
|
#define SECTOR_SHIFT 9
|
|
|
|
#define SECTORS_PER_PAGE_SHIFT (PAGE_SHIFT - SECTOR_SHIFT)
|
|
|
|
#define SECTORS_PER_PAGE (1 << SECTORS_PER_PAGE_SHIFT)
|
2011-06-10 21:28:48 +08:00
|
|
|
#define ZRAM_LOGICAL_BLOCK_SHIFT 12
|
|
|
|
#define ZRAM_LOGICAL_BLOCK_SIZE (1 << ZRAM_LOGICAL_BLOCK_SHIFT)
|
|
|
|
#define ZRAM_SECTOR_PER_LOGICAL_BLOCK \
|
|
|
|
(1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT))
|
2009-09-22 12:56:53 +08:00
|
|
|
|
zram: replace global tb_lock with fine grain lock
Currently, we use a rwlock tb_lock to protect concurrent access to the
whole zram meta table. However, according to the actual access model,
there is only a small chance for upper user to access the same
table[index], so the current lock granularity is too big.
The idea of optimization is to change the lock granularity from whole
meta table to per table entry (table -> table[index]), so that we can
protect concurrent access to the same table[index], meanwhile allow the
maximum concurrency.
With this in mind, several kinds of locks which could be used as a
per-entry lock were tested and compared:
Test environment:
x86-64 Intel Core2 Q8400, system memory 4GB, Ubuntu 12.04,
kernel v3.15.0-rc3 as base, zram with 4 max_comp_streams LZO.
iozone test:
iozone -t 4 -R -r 16K -s 200M -I +Z
(1GB zram with ext4 filesystem, take the average of 10 tests, KB/s)
Test base CAS spinlock rwlock bit_spinlock
-------------------------------------------------------------------
Initial write 1381094 1425435 1422860 1423075 1421521
Rewrite 1529479 1641199 1668762 1672855 1654910
Read 8468009 11324979 11305569 11117273 10997202
Re-read 8467476 11260914 11248059 11145336 10906486
Reverse Read 6821393 8106334 8282174 8279195 8109186
Stride read 7191093 8994306 9153982 8961224 9004434
Random read 7156353 8957932 9167098 8980465 8940476
Mixed workload 4172747 5680814 5927825 5489578 5972253
Random write 1483044 1605588 1594329 1600453 1596010
Pwrite 1276644 1303108 1311612 1314228 1300960
Pread 4324337 4632869 4618386 4457870 4500166
To enhance the possibility of access the same table[index] concurrently,
set zram a small disksize(10MB) and let threads run with large loop
count.
fio test:
fio --bs=32k --randrepeat=1 --randseed=100 --refill_buffers
--scramble_buffers=1 --direct=1 --loops=3000 --numjobs=4
--filename=/dev/zram0 --name=seq-write --rw=write --stonewall
--name=seq-read --rw=read --stonewall --name=seq-readwrite
--rw=rw --stonewall --name=rand-readwrite --rw=randrw --stonewall
(10MB zram raw block device, take the average of 10 tests, KB/s)
Test base CAS spinlock rwlock bit_spinlock
-------------------------------------------------------------
seq-write 933789 999357 1003298 995961 1001958
seq-read 5634130 6577930 6380861 6243912 6230006
seq-rw 1405687 1638117 1640256 1633903 1634459
rand-rw 1386119 1614664 1617211 1609267 1612471
All the optimization methods show a higher performance than the base,
however, it is hard to say which method is the most appropriate.
On the other hand, zram is mostly used on small embedded system, so we
don't want to increase any memory footprint.
This patch pick the bit_spinlock method, pack object size and page_flag
into an unsigned long table.value, so as to not increase any memory
overhead on both 32-bit and 64-bit system.
On the third hand, even though different kinds of locks have different
performances, we can ignore this difference, because: if zram is used as
zram swapfile, the swap subsystem can prevent concurrent access to the
same swapslot; if zram is used as zram-blk for set up filesystem on it,
the upper filesystem and the page cache also prevent concurrent access
of the same block mostly. So we can ignore the different performances
among locks.
Acked-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Reviewed-by: Davidlohr Bueso <davidlohr@hp.com>
Signed-off-by: Weijie Yang <weijie.yang@samsung.com>
Signed-off-by: Minchan Kim <minchan@kernel.org>
Cc: Jerome Marchand <jmarchan@redhat.com>
Cc: Nitin Gupta <ngupta@vflare.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-08-07 07:08:31 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The lower ZRAM_FLAG_SHIFT bits of table.value is for
|
|
|
|
* object size (excluding header), the higher bits is for
|
|
|
|
* zram_pageflags.
|
|
|
|
*
|
|
|
|
* zram is mainly used for memory efficiency so we want to keep memory
|
|
|
|
* footprint small so we can squeeze size and flags into a field.
|
|
|
|
* The lower ZRAM_FLAG_SHIFT bits is for object size (excluding header),
|
|
|
|
* the higher bits is for zram_pageflags.
|
|
|
|
*/
|
|
|
|
#define ZRAM_FLAG_SHIFT 24
|
|
|
|
|
|
|
|
/* Flags for zram pages (table[page_no].value) */
|
2010-06-01 16:01:25 +08:00
|
|
|
enum zram_pageflags {
|
2009-09-22 12:56:53 +08:00
|
|
|
/* Page consists entirely of zeros */
|
2017-02-25 06:59:27 +08:00
|
|
|
ZRAM_SAME = ZRAM_FLAG_SHIFT,
|
2014-12-13 08:57:04 +08:00
|
|
|
ZRAM_ACCESS, /* page is now accessed */
|
2009-09-22 12:56:53 +08:00
|
|
|
|
2010-06-01 16:01:25 +08:00
|
|
|
__NR_ZRAM_PAGEFLAGS,
|
2009-09-22 12:56:53 +08:00
|
|
|
};
|
|
|
|
|
|
|
|
/*-- Data structures */
|
|
|
|
|
2010-06-01 16:01:25 +08:00
|
|
|
/* Allocated for each disk page */
|
2014-08-07 07:08:25 +08:00
|
|
|
struct zram_table_entry {
|
2017-02-25 06:59:27 +08:00
|
|
|
union {
|
|
|
|
unsigned long handle;
|
|
|
|
unsigned long element;
|
|
|
|
};
|
zram: replace global tb_lock with fine grain lock
Currently, we use a rwlock tb_lock to protect concurrent access to the
whole zram meta table. However, according to the actual access model,
there is only a small chance for upper user to access the same
table[index], so the current lock granularity is too big.
The idea of optimization is to change the lock granularity from whole
meta table to per table entry (table -> table[index]), so that we can
protect concurrent access to the same table[index], meanwhile allow the
maximum concurrency.
With this in mind, several kinds of locks which could be used as a
per-entry lock were tested and compared:
Test environment:
x86-64 Intel Core2 Q8400, system memory 4GB, Ubuntu 12.04,
kernel v3.15.0-rc3 as base, zram with 4 max_comp_streams LZO.
iozone test:
iozone -t 4 -R -r 16K -s 200M -I +Z
(1GB zram with ext4 filesystem, take the average of 10 tests, KB/s)
Test base CAS spinlock rwlock bit_spinlock
-------------------------------------------------------------------
Initial write 1381094 1425435 1422860 1423075 1421521
Rewrite 1529479 1641199 1668762 1672855 1654910
Read 8468009 11324979 11305569 11117273 10997202
Re-read 8467476 11260914 11248059 11145336 10906486
Reverse Read 6821393 8106334 8282174 8279195 8109186
Stride read 7191093 8994306 9153982 8961224 9004434
Random read 7156353 8957932 9167098 8980465 8940476
Mixed workload 4172747 5680814 5927825 5489578 5972253
Random write 1483044 1605588 1594329 1600453 1596010
Pwrite 1276644 1303108 1311612 1314228 1300960
Pread 4324337 4632869 4618386 4457870 4500166
To enhance the possibility of access the same table[index] concurrently,
set zram a small disksize(10MB) and let threads run with large loop
count.
fio test:
fio --bs=32k --randrepeat=1 --randseed=100 --refill_buffers
--scramble_buffers=1 --direct=1 --loops=3000 --numjobs=4
--filename=/dev/zram0 --name=seq-write --rw=write --stonewall
--name=seq-read --rw=read --stonewall --name=seq-readwrite
--rw=rw --stonewall --name=rand-readwrite --rw=randrw --stonewall
(10MB zram raw block device, take the average of 10 tests, KB/s)
Test base CAS spinlock rwlock bit_spinlock
-------------------------------------------------------------
seq-write 933789 999357 1003298 995961 1001958
seq-read 5634130 6577930 6380861 6243912 6230006
seq-rw 1405687 1638117 1640256 1633903 1634459
rand-rw 1386119 1614664 1617211 1609267 1612471
All the optimization methods show a higher performance than the base,
however, it is hard to say which method is the most appropriate.
On the other hand, zram is mostly used on small embedded system, so we
don't want to increase any memory footprint.
This patch pick the bit_spinlock method, pack object size and page_flag
into an unsigned long table.value, so as to not increase any memory
overhead on both 32-bit and 64-bit system.
On the third hand, even though different kinds of locks have different
performances, we can ignore this difference, because: if zram is used as
zram swapfile, the swap subsystem can prevent concurrent access to the
same swapslot; if zram is used as zram-blk for set up filesystem on it,
the upper filesystem and the page cache also prevent concurrent access
of the same block mostly. So we can ignore the different performances
among locks.
Acked-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Reviewed-by: Davidlohr Bueso <davidlohr@hp.com>
Signed-off-by: Weijie Yang <weijie.yang@samsung.com>
Signed-off-by: Minchan Kim <minchan@kernel.org>
Cc: Jerome Marchand <jmarchan@redhat.com>
Cc: Nitin Gupta <ngupta@vflare.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-08-07 07:08:31 +08:00
|
|
|
unsigned long value;
|
|
|
|
};
|
2009-09-22 12:56:53 +08:00
|
|
|
|
2010-06-01 16:01:25 +08:00
|
|
|
struct zram_stats {
|
2014-04-08 06:38:03 +08:00
|
|
|
atomic64_t compr_data_size; /* compressed size of pages stored */
|
2013-06-07 00:07:31 +08:00
|
|
|
atomic64_t num_reads; /* failed + successful */
|
|
|
|
atomic64_t num_writes; /* --do-- */
|
2014-08-30 06:18:37 +08:00
|
|
|
atomic64_t failed_reads; /* can happen when memory is too low */
|
2013-06-07 00:07:31 +08:00
|
|
|
atomic64_t failed_writes; /* can happen when memory is too low */
|
|
|
|
atomic64_t invalid_io; /* non-page-aligned I/O requests */
|
|
|
|
atomic64_t notify_free; /* no. of swap slot free notifications */
|
2017-02-25 06:59:27 +08:00
|
|
|
atomic64_t same_pages; /* no. of same element filled pages */
|
2014-04-08 06:38:03 +08:00
|
|
|
atomic64_t pages_stored; /* no. of pages currently stored */
|
2014-10-10 06:29:55 +08:00
|
|
|
atomic_long_t max_used_pages; /* no. of maximum pages stored */
|
2016-05-21 08:00:02 +08:00
|
|
|
atomic64_t writestall; /* no. of write slow paths */
|
2009-09-22 12:56:53 +08:00
|
|
|
};
|
|
|
|
|
2017-05-04 05:55:47 +08:00
|
|
|
struct zram {
|
2014-08-07 07:08:25 +08:00
|
|
|
struct zram_table_entry *table;
|
2013-02-06 07:48:53 +08:00
|
|
|
struct zs_pool *mem_pool;
|
2015-02-13 07:00:45 +08:00
|
|
|
struct zcomp *comp;
|
2009-09-22 12:56:53 +08:00
|
|
|
struct gendisk *disk;
|
2015-02-13 07:00:45 +08:00
|
|
|
/* Prevent concurrent execution of device init */
|
2011-09-06 21:02:11 +08:00
|
|
|
struct rw_semaphore init_lock;
|
2009-09-22 12:56:53 +08:00
|
|
|
/*
|
2015-02-13 07:00:45 +08:00
|
|
|
* the number of pages zram can consume for storing compressed data
|
2009-09-22 12:56:53 +08:00
|
|
|
*/
|
2015-02-13 07:00:45 +08:00
|
|
|
unsigned long limit_pages;
|
|
|
|
|
2010-06-01 16:01:25 +08:00
|
|
|
struct zram_stats stats;
|
2014-10-10 06:29:53 +08:00
|
|
|
/*
|
2015-02-13 07:00:45 +08:00
|
|
|
* This is the limit on amount of *uncompressed* worth of data
|
|
|
|
* we can store in a disk.
|
2014-10-10 06:29:53 +08:00
|
|
|
*/
|
2015-02-13 07:00:45 +08:00
|
|
|
u64 disksize; /* bytes */
|
zram: use crypto api to check alg availability
There is no way to get a string with all the crypto comp algorithms
supported by the crypto comp engine, so we need to maintain our own
backends list. At the same time we additionally need to use
crypto_has_comp() to make sure that the user has requested a compression
algorithm that is recognized by the crypto comp engine. Relying on
/proc/crypto is not an options here, because it does not show
not-yet-inserted compression modules.
Example:
modprobe zram
cat /proc/crypto | grep -i lz4
modprobe lz4
cat /proc/crypto | grep -i lz4
name : lz4
driver : lz4-generic
module : lz4
So the user can't tell exactly if the lz4 is really supported from
/proc/crypto output, unless someone or something has loaded it.
This patch also adds crypto_has_comp() to zcomp_available_show(). We
store all the compression algorithms names in zcomp's `backends' array,
regardless the CONFIG_CRYPTO_FOO configuration, but show only those that
are also supported by crypto engine. This helps user to know the exact
list of compression algorithms that can be used.
Example:
module lz4 is not loaded yet, but is supported by the crypto
engine. /proc/crypto has no information on this module, while
zram's `comp_algorithm' lists it:
cat /proc/crypto | grep -i lz4
cat /sys/block/zram0/comp_algorithm
[lzo] lz4 deflate lz4hc 842
We still use the `backends' array to determine if the requested
compression backend is known to crypto api. This array, however, may not
contain some entries, therefore as the last step we call crypto_has_comp()
function which attempts to insmod the requested compression algorithm to
determine if crypto api supports it. The advantage of this method is that
now we permit the usage of out-of-tree crypto compression modules
(implementing S/W or H/W compression).
[sergey.senozhatsky@gmail.com: zram-use-crypto-api-to-check-alg-availability-v3]
Link: http://lkml.kernel.org/r/20160604024902.11778-4-sergey.senozhatsky@gmail.com
Link: http://lkml.kernel.org/r/20160531122017.2878-5-sergey.senozhatsky@gmail.com
Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Acked-by: Minchan Kim <minchan@kernel.org>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-07-27 06:22:48 +08:00
|
|
|
char compressor[CRYPTO_MAX_ALG_NAME];
|
2015-06-26 06:00:21 +08:00
|
|
|
/*
|
|
|
|
* zram is claimed so open request will be failed
|
|
|
|
*/
|
|
|
|
bool claim; /* Protected by bdev->bd_mutex */
|
2017-09-07 07:19:54 +08:00
|
|
|
#ifdef CONFIG_ZRAM_WRITEBACK
|
|
|
|
struct file *backing_dev;
|
|
|
|
struct block_device *bdev;
|
|
|
|
unsigned int old_block_size;
|
2017-09-07 07:19:57 +08:00
|
|
|
unsigned long *bitmap;
|
|
|
|
unsigned long nr_pages;
|
|
|
|
spinlock_t bitmap_lock;
|
2017-09-07 07:19:54 +08:00
|
|
|
#endif
|
2009-09-22 12:56:53 +08:00
|
|
|
};
|
2010-01-28 23:43:37 +08:00
|
|
|
#endif
|