Commit Graph

515 Commits

Author SHA1 Message Date
Gerald Schaefer 1378a68344 s390/dcssblk: fix possible deadlock in remove vs. per-device attributes
dcssblk_remove_store() holds the dcssblk_devices_sem semaphore while
calling device_unregister(), which in turn tries to acquire the kernfs
kn->dev_map rwsem for the device sysfs subtree. The same rwsem is also
acquired when using the per-device sysfs attributes in the device sub-tree,
and the attribute handlers then also acquire the dcssblk_devices_sem.

This can lead to a deadlock when removing a DCSS while concurrently
reading from / writing to one of its sysfs attributes. The following
lockdep warning hinted towards the issue (CPU0 = dcssblk_remove_store,
CPU1 = dcssblk_shared_store):

[   76.496047]  Possible unsafe locking scenario:

[   76.496054]        CPU0                    CPU1
[   76.496059]        ----                    ----
[   76.496087]   lock(&dcssblk_devices_sem);
[   76.496090]                                lock(s_active#175);
[   76.496106]                                lock(&dcssblk_devices_sem);
[   76.496110]   lock(s_active#175);
[   76.496115]
 *** DEADLOCK ***

Fix this by releasing the dcssblk_devices_sem semaphore, which only
protects internal DCSS data, before calling device_unregister().

Signed-off-by: Gerald Schaefer <gerald.schaefer@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-04-15 18:01:44 +02:00
Sebastian Ott b707c65ae7 s390/scm_blk: fix deadlock for requests != REQ_TYPE_FS
When we refuse a non REQ_TYPE_FS request in the build request function
we already hold the queue lock. Thus we must not call blk_end_request_all
but __blk_end_request_all.

Reported-by: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>
Fixes: de9587a ('s390/scm_blk: fix endless loop for requests != REQ_TYPE_FS')
Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-04-01 17:20:54 +02:00
Stefan Haberland 59a9ed5f87 s390/dasd: reorder lcu and device lock
Reorder lcu and device lock to get rid of the error-prone trylock
mechanism.

The locking order is lcu lock -> device lock.
This protects against changes to the lcu device lists and enables us
to iterate over the devices, take the cdev lock and make changes to
the device structures.

The complicated part is the summary unit check handler that gets an
interrupt on one device of the lcu that leads to structural changes of
the whole lcu itself. This work needs to be done even if devices on
the lcu disappear. So a device independent worker is used.
The old approach tried to update some lcu structures and set up the
lcu worker in the interrupt context with the device lock held.
But this forced the lock order "cdev lock -> lcu lock" that made it
hard to have the lcu lock held and iterate over all devices and change
them.

The new approach is to schedule a device specific worker that gets
out of the interrupt context and rid of the device lock for summary
unit checks. This worker is able to take the lcu lock and schedule the
lcu worker that updates all devices. The time between interrupt and
worker execution is no problem because the devices in the lcu reject
all I/O in this time with an appropriate error. The dasd driver can
deal with this situation and re-drive the I/O later on.

Signed-off-by: Stefan Haberland <sth@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-17 13:18:25 +01:00
Linus Torvalds 72aafdf01d Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
Pull s390 updates from Martin Schwidefsky:

 - Add the CPU id for the new z13s machine

 - Add a s390 specific XOR template for RAID-5 checksumming based on the
   XC instruction.  Remove all other alternatives, XC is always faster

 - The merge of our four different stack tracers into a single one

 - Tidy up the code related to page tables, several large inline
   functions are now out-of-line.  Bloat-o-meter reports ~11K text size
   reduction

 - A binary interface for the priviledged CLP instruction to retrieve
   the hardware view of the installed PCI functions

 - Improvements for the dasd format code

 - Bug fixes and cleanups

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux: (31 commits)
  s390/pci: enforce fmb page boundary rule
  s390: fix floating pointer register corruption (again)
  s390/cpumf: add missing lpp magic initialization
  s390: Fix misspellings in comments
  s390/mm: split arch/s390/mm/pgtable.c
  s390/mm: uninline pmdp_xxx functions from pgtable.h
  s390/mm: uninline ptep_xxx functions from pgtable.h
  s390/pci: add ioctl interface for CLP
  s390: Use pr_warn instead of pr_warning
  s390/dasd: remove casts to dasd_*_private
  s390/dasd: Refactor dasd format functions
  s390/dasd: Simplify code in format logic
  s390/dasd: Improve dasd format code
  s390/percpu: remove this_cpu_cmpxchg_double_4
  s390/cpumf: Improve guest detection heuristics
  s390/fault: merge report_user_fault implementations
  s390/dis: use correct escape sequence for '%' character
  s390/kvm: simplify set_guest_storage_key
  s390/oprofile: add z13/z13s model numbers
  s390: add z13s model number to z13 elf platform
  ...
2016-03-16 10:47:45 -07:00
Joe Perches baebc70a4d s390: Use pr_warn instead of pr_warning
Convert the uses of pr_warning to pr_warn so there are fewer
uses of the old pr_warning.

Miscellanea:

o Align arguments
o Coalesce formats

Signed-off-by: Joe Perches <joe@perches.com>
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-07 13:12:04 +01:00
Sebastian Ott 543691a4e1 s390/dasd: remove casts to dasd_*_private
Convert dasd_device.private to be a void pointer to get
rid of a lot of explicit casts.

Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Reviewed-by: Stefan Haberland <sth@linux.vnet.ibm.com>
Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-07 13:12:02 +01:00
Jan Höppner 570d237c19 s390/dasd: Refactor dasd format functions
Prepare for new format checking function by renaming functions and
moving reusable code to separate functions:

- Move sanity checks into a new function and make it reusable.
- Move common format code to a new function called
  dasd_eckd_format_process_data.
- Create the generic function dasd_eckd_format_build_ccw_req, which
  itself will then decide what ccw request is being built according to
  the input data. (with upcoming functionality).

Signed-off-by: Jan Höppner <hoeppner@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-07 13:12:00 +01:00
Jan Höppner 8542885bf5 s390/dasd: Simplify code in format logic
Currently, dasd_format is calling the format logic of a DASD discipline
with PAV enabled. If that fails with an error code of -EAGAIN the value
of retries is decremented and the discipline function is called with PAV
turned off.

The loop is supposed to try this up to 255 times until success. However,
-EAGAIN can only occur once here and therefore the loop will never reach
the 255 retries.

So, replace the unnecessarily complicated loop logic and simply try again
without PAV enabled in case of an -EAGAIN error.

Signed-off-by: Jan Höppner <hoeppner@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-07 13:11:58 +01:00
Jan Höppner 46d1c03c82 s390/dasd: Improve dasd format code
- Make sure a calling function can rely on data in fdata by resetting to
  its initial values
- Move special treatment for track 0 and 1 to dasd_eckd_build_format
- Replace dangerous backward goto with a loop logic
- Add define for number that specifies the maximum amount of CCWs per
  request and is used for format_step calculation
- Remove unused variable

Signed-off-by: Jan Höppner <hoeppner@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-07 13:11:56 +01:00
Heiko Carstens ce0c12b633 s390/dasd: fix diag 0x250 inline assembly
git commit 1ec2772e0c ("s390/diag: add a statistic for diagnose
calls") added function calls to gather diagnose statistics.

In case of the dasd diag driver the function call was added between a
register asm statement which initialized register r2 and the inline
assembly itself.  The function call clobbers the contents of register
r2 and therefore the diag 0x250 call behaves in a more or less random
way.

Fix this by extracting the function call into a separate function like
we do everywhere else.

Fixes: 1ec2772e0c ("s390/diag: add a statistic for diagnose calls")
Cc: <stable@vger.kernel.org> # 4.4+
Reported-and-tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-03-02 06:43:48 -06:00
Stefan Haberland 9bfefde718 s390/dasd: fix incorrect locking order for LCU device add/remove
The correct lock order for LCU lock and cdev lock is to take the cdev
lock first and afterwards the LCU lock. This is caused by the fact
that LCU functions are called in an interrupt context with the cdev
lock implicitly hold by CIO.

To assure the right locking order but also be able to iterate over
devices in a LCU introduce a trylock block that can be called with
the device lock for one device hold and then takes the LCU lock and
try to lock all devices accounted to this LCU. Afterwards all devices
and the LCU itself are locked.

Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-02-23 08:56:19 +01:00
Stefan Haberland 12d319b920 s390/dasd: fix performance drop
Commit ca369d51b ("sd: Fix device-imposed transfer length limits")
introduced a new queue limit max_dev_sectors which limits the maximum
sectors for requests. The default value leads to small dasd requests
and therefor to a performance drop.
Set the max_dev_sectors value to the same value as the max_hw_sectors
to use the maximum available request size for DASD devices.

Signed-off-by: Stefan Haberland <sth@linux.vnet.ibm.com>
Cc: stable@vger.kernel.org # 4.4+
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-02-17 09:24:07 +01:00
Stefan Haberland 9d862ababb s390/dasd: fix refcount for PAV reassignment
Add refcount to the DASD device when a summary unit check worker is
scheduled. This prevents that the device is set offline with worker
in place.

Cc: stable@vger.kernel.org
Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-02-11 13:05:55 +01:00
Stefan Haberland 020bf042e5 s390/dasd: prevent incorrect length error under z/VM after PAV changes
The channel checks the specified length and the provided amount of
data for CCWs and provides an incorrect length error if the size does
not match. Under z/VM with simulation activated the length may get
changed. Having the suppress length indication bit set is stated as
good CCW coding practice and avoids errors under z/VM.

Cc: stable@vger.kernel.org
Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2016-02-11 13:05:53 +01:00
Dan Williams 34c0fd540e mm, dax, pmem: introduce pfn_t
For the purpose of communicating the optional presence of a 'struct
page' for the pfn returned from ->direct_access(), introduce a type that
encapsulates a page-frame-number plus flags.  These flags contain the
historical "page_link" encoding for a scatterlist entry, but can also
denote "device memory".  Where "device memory" is a set of pfns that are
not part of the kernel's linear mapping by default, but are accessed via
the same memory controller as ram.

The motivation for this new type is large capacity persistent memory
that needs struct page entries in the 'memmap' to support 3rd party DMA
(i.e.  O_DIRECT I/O with a persistent memory source/target).  However,
we also need it in support of maintaining a list of mapped inodes which
need to be unmapped at driver teardown or freeze_bdev() time.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Dave Hansen <dave@sr71.net>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-01-15 17:56:32 -08:00
Stefan Haberland c6fc7b6f8c s390/dasd: fix failfast for disconnected devices
Enabling failfast should let request fail immediately if either an
error occurred or the device gets disconnected.
For disconnected devices new requests are not fetches from the block
queue and therefore failfast is not triggered.

Fix by letting the DASD driver fetch requests for disconnected devices
with failfast active.

Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-12-30 10:34:41 +01:00
Jens Axboe dece16353e block: change ->make_request_fn() and users to return a queue cookie
No functional changes in this patch, but it prepares us for returning
a more useful cookie related to the IO that was queued up.

Signed-off-by: Jens Axboe <axboe@fb.com>
Acked-by: Christoph Hellwig <hch@lst.de>
Acked-by: Keith Busch <keith.busch@intel.com>
2015-11-07 10:40:46 -07:00
Stefan Haberland ccc0e7dc70 s390/dasd: fix disconnected device with valid path mask
Path verification is either done via dasd_eckd_read_conf() which is
triggered during online processing and resume or via
do_path_verification_work() which is triggered after path events.
The dasd_eckd_read_conf() version added paths unconditionally and did
not check if the path mask was empty. This led to devices having the
disconnected stop flag set but a valid path mask. So they where not
working although they had paths validated successfully. After a resume
this state could even not be solved with additional paths added.

Fix by checking for an empty path mask in dasd_eckd_read_conf() and
clearing the device stop bits for a newly added channel path.

Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-11-03 14:40:47 +01:00
Stefan Haberland 251afd69e3 s390/dasd: fix invalid PAV assignment after suspend/resume
For a valid PAV assignment the DASD driver needs to notice possibly
changed configuration data. Thus the failing of read configuration
data should also fail the device restore to prevent invalid PAV
assignment. The failed device may get restored after additional paths
get available later on.
If the restore fails after the device was added to the lcu alias
handling it needs to be removed from the alias handling before exiting
the restore function.

Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-11-03 14:40:46 +01:00
Stefan Haberland c7c0c9deb0 s390/dasd: fix double free in dasd_eckd_read_conf
The configuration data is stored per path and also the first valid
configuration data per device. When dasd_eckd_read_conf is called
again after a path got lost the device configuration data is cleared
but possibly not the per path configuration data. This might lead to a
double free when the lost path gets operational again.

Fix by clearing all per path configuration data when the first valid
configuration data is received and stored.

Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-11-03 14:40:44 +01:00
Stefan Haberland 6933c35a30 s390/dasd: fix list_del corruption after lcu changes
A summary unit check occurs when the lcu updates the PAV configuration
e.g. base PAV assignment or PAV mode at all. This requires the reset
of the drivers internal pavgroups. Therefore the alias devices are
flushed and moved via a temporary list to the active_devices list
where they are not associated with a pavgroup. In conjunction with
updates to the base device the pavgroup may be removed since both
base_list and alias_list are empty. Unfortunately during alias flush
and move to the active_device list from alias_list the pavgroup
pointer is not deleted in the device private structure. This leads to
a list del_corruption if another lcu_update tries to move the device
in the non existent pavgroup.

Fix by removing the pavgroup pointer after the alias device was moved
to the active_devices list.

Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-15 09:28:15 +02:00
Martin Schwidefsky 1ec2772e0c s390/diag: add a statistic for diagnose calls
Introduce /sys/debug/kernel/diag_stat with a statistic how many diagnose
calls have been done by each CPU in the system.

Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-14 14:32:06 +02:00
Sebastian Ott 9d49f86dab s390/cio: introduce pathmask_to_pos
We often need to correlate an 8 bit path mask with the position
in a channel path array. Introduce and use pathmask_to_pos for
that task.

Reviewed-by: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>
Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-14 14:32:04 +02:00
Christian Borntraeger 9a21268360 s390/dasd: mark DASD devices as non rotational
We were able to reduce the CPU overhead of big paging scenarios
when announcing our paging disks as non-rotational.
Almost all dasd devices are implemented in storage servers with
cache, raid, striping and lots of magic. There is no point in
optimizing the disk schedulers and swap code for a single platter
moving arm rotational disks. Given the complexity of the setup
and the fact that this change is mostly to disable the additional
overhead in swap code, lets keep the other functionality unchanged
and do not disable the this device as entropy source - unlike other
non-rotational devices.

Suggested-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Reviewed-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-10-14 14:32:00 +02:00
Linus Torvalds 12f03ee606 libnvdimm for 4.3:
1/ Introduce ZONE_DEVICE and devm_memremap_pages() as a generic
    mechanism for adding device-driver-discovered memory regions to the
    kernel's direct map.  This facility is used by the pmem driver to
    enable pfn_to_page() operations on the page frames returned by DAX
    ('direct_access' in 'struct block_device_operations'). For now, the
    'memmap' allocation for these "device" pages comes from "System
    RAM".  Support for allocating the memmap from device memory will
    arrive in a later kernel.
 
 2/ Introduce memremap() to replace usages of ioremap_cache() and
    ioremap_wt().  memremap() drops the __iomem annotation for these
    mappings to memory that do not have i/o side effects.  The
    replacement of ioremap_cache() with memremap() is limited to the
    pmem driver to ease merging the api change in v4.3.  Completion of
    the conversion is targeted for v4.4.
 
 3/ Similar to the usage of memcpy_to_pmem() + wmb_pmem() in the pmem
    driver, update the VFS DAX implementation and PMEM api to provide
    persistence guarantees for kernel operations on a DAX mapping.
 
 4/ Convert the ACPI NFIT 'BLK' driver to map the block apertures as
    cacheable to improve performance.
 
 5/ Miscellaneous updates and fixes to libnvdimm including support
    for issuing "address range scrub" commands, clarifying the optimal
    'sector size' of pmem devices, a clarification of the usage of the
    ACPI '_STA' (status) property for DIMM devices, and other minor
    fixes.
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQIcBAABAgAGBQJV6Nx7AAoJEB7SkWpmfYgCWyYQAI5ju6Gvw27RNFtPovHcZUf5
 JGnxXejI6/AqeTQ+IulgprxtEUCrXOHjCDA5dkjr1qvsoqK1qxug+vJHOZLgeW0R
 OwDtmdW4Qrgeqm+CPoxETkorJ8wDOc8mol81kTiMgeV3UqbYeeHIiTAmwe7VzZ0C
 nNdCRDm5g8dHCjTKcvK3rvozgyoNoWeBiHkPe76EbnxDICxCB5dak7XsVKNMIVFQ
 NuYlnw6IYN7+rMHgpgpRux38NtIW8VlYPWTmHExejc2mlioWMNBG/bmtwLyJ6M3e
 zliz4/cnonTMUaizZaVozyinTa65m7wcnpjK+vlyGV2deDZPJpDRvSOtB0lH30bR
 1gy+qrKzuGKpaN6thOISxFLLjmEeYwzYd7SvC9n118r32qShz+opN9XX0WmWSFlA
 sajE1ehm4M7s5pkMoa/dRnAyR8RUPu4RNINdQ/Z9jFfAOx+Q26rLdQXwf9+uqbEb
 bIeSQwOteK5vYYCstvpAcHSMlJAglzIX5UfZBvtEIJN7rlb0VhmGWfxAnTu+ktG1
 o9cqAt+J4146xHaFwj5duTsyKhWb8BL9+xqbKPNpXEp+PbLsrnE/+WkDLFD67jxz
 dgIoK60mGnVXp+16I2uMqYYDgAyO5zUdmM4OygOMnZNa1mxesjbDJC6Wat1Wsndn
 slsw6DkrWT60CRE42nbK
 =o57/
 -----END PGP SIGNATURE-----

Merge tag 'libnvdimm-for-4.3' of git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm

Pull libnvdimm updates from Dan Williams:
 "This update has successfully completed a 0day-kbuild run and has
  appeared in a linux-next release.  The changes outside of the typical
  drivers/nvdimm/ and drivers/acpi/nfit.[ch] paths are related to the
  removal of IORESOURCE_CACHEABLE, the introduction of memremap(), and
  the introduction of ZONE_DEVICE + devm_memremap_pages().

  Summary:

   - Introduce ZONE_DEVICE and devm_memremap_pages() as a generic
     mechanism for adding device-driver-discovered memory regions to the
     kernel's direct map.

     This facility is used by the pmem driver to enable pfn_to_page()
     operations on the page frames returned by DAX ('direct_access' in
     'struct block_device_operations').

     For now, the 'memmap' allocation for these "device" pages comes
     from "System RAM".  Support for allocating the memmap from device
     memory will arrive in a later kernel.

   - Introduce memremap() to replace usages of ioremap_cache() and
     ioremap_wt().  memremap() drops the __iomem annotation for these
     mappings to memory that do not have i/o side effects.  The
     replacement of ioremap_cache() with memremap() is limited to the
     pmem driver to ease merging the api change in v4.3.

     Completion of the conversion is targeted for v4.4.

   - Similar to the usage of memcpy_to_pmem() + wmb_pmem() in the pmem
     driver, update the VFS DAX implementation and PMEM api to provide
     persistence guarantees for kernel operations on a DAX mapping.

   - Convert the ACPI NFIT 'BLK' driver to map the block apertures as
     cacheable to improve performance.

   - Miscellaneous updates and fixes to libnvdimm including support for
     issuing "address range scrub" commands, clarifying the optimal
     'sector size' of pmem devices, a clarification of the usage of the
     ACPI '_STA' (status) property for DIMM devices, and other minor
     fixes"

* tag 'libnvdimm-for-4.3' of git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm: (34 commits)
  libnvdimm, pmem: direct map legacy pmem by default
  libnvdimm, pmem: 'struct page' for pmem
  libnvdimm, pfn: 'struct page' provider infrastructure
  x86, pmem: clarify that ARCH_HAS_PMEM_API implies PMEM mapped WB
  add devm_memremap_pages
  mm: ZONE_DEVICE for "device memory"
  mm: move __phys_to_pfn and __pfn_to_phys to asm/generic/memory_model.h
  dax: drop size parameter to ->direct_access()
  nd_blk: change aperture mapping from WC to WB
  nvdimm: change to use generic kvfree()
  pmem, dax: have direct_access use __pmem annotation
  dax: update I/O path to do proper PMEM flushing
  pmem: add copy_from_iter_pmem() and clear_pmem()
  pmem, x86: clean up conditional pmem includes
  pmem: remove layer when calling arch_has_wmb_pmem()
  pmem, x86: move x86 PMEM API to new pmem.h header
  libnvdimm, e820: make CONFIG_X86_PMEM_LEGACY a tristate option
  pmem: switch to devm_ allocations
  devres: add devm_memremap
  libnvdimm, btt: write and validate parent_uuid
  ...
2015-09-08 14:35:59 -07:00
Linus Torvalds 1081230b74 Merge branch 'for-4.3/core' of git://git.kernel.dk/linux-block
Pull core block updates from Jens Axboe:
 "This first core part of the block IO changes contains:

   - Cleanup of the bio IO error signaling from Christoph.  We used to
     rely on the uptodate bit and passing around of an error, now we
     store the error in the bio itself.

   - Improvement of the above from myself, by shrinking the bio size
     down again to fit in two cachelines on x86-64.

   - Revert of the max_hw_sectors cap removal from a revision again,
     from Jeff Moyer.  This caused performance regressions in various
     tests.  Reinstate the limit, bump it to a more reasonable size
     instead.

   - Make /sys/block/<dev>/queue/discard_max_bytes writeable, by me.
     Most devices have huge trim limits, which can cause nasty latencies
     when deleting files.  Enable the admin to configure the size down.
     We will look into having a more sane default instead of UINT_MAX
     sectors.

   - Improvement of the SGP gaps logic from Keith Busch.

   - Enable the block core to handle arbitrarily sized bios, which
     enables a nice simplification of bio_add_page() (which is an IO hot
     path).  From Kent.

   - Improvements to the partition io stats accounting, making it
     faster.  From Ming Lei.

   - Also from Ming Lei, a basic fixup for overflow of the sysfs pending
     file in blk-mq, as well as a fix for a blk-mq timeout race
     condition.

   - Ming Lin has been carrying Kents above mentioned patches forward
     for a while, and testing them.  Ming also did a few fixes around
     that.

   - Sasha Levin found and fixed a use-after-free problem introduced by
     the bio->bi_error changes from Christoph.

   - Small blk cgroup cleanup from Viresh Kumar"

* 'for-4.3/core' of git://git.kernel.dk/linux-block: (26 commits)
  blk: Fix bio_io_vec index when checking bvec gaps
  block: Replace SG_GAPS with new queue limits mask
  block: bump BLK_DEF_MAX_SECTORS to 2560
  Revert "block: remove artifical max_hw_sectors cap"
  blk-mq: fix race between timeout and freeing request
  blk-mq: fix buffer overflow when reading sysfs file of 'pending'
  Documentation: update notes in biovecs about arbitrarily sized bios
  block: remove bio_get_nr_vecs()
  fs: use helper bio_add_page() instead of open coding on bi_io_vec
  block: kill merge_bvec_fn() completely
  md/raid5: get rid of bio_fits_rdev()
  md/raid5: split bio for chunk_aligned_read
  block: remove split code in blkdev_issue_{discard,write_same}
  btrfs: remove bio splitting and merge_bvec_fn() calls
  bcache: remove driver private bio splitting code
  block: simplify bio_add_page()
  block: make generic_make_request handle arbitrarily sized bios
  blk-cgroup: Drop unlikely before IS_ERR(_OR_NULL)
  block: don't access bio->bi_error after bio_put()
  block: shrink struct bio down to 2 cache lines again
  ...
2015-09-02 13:10:25 -07:00
Dan Williams cb389b9c0e dax: drop size parameter to ->direct_access()
None of the implementations currently use it.  The common
bdev_direct_access() entry point handles all the size checks before
calling ->direct_access().

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2015-08-27 19:40:58 -04:00
Ross Zwisler e2e05394e4 pmem, dax: have direct_access use __pmem annotation
Update the annotation for the kaddr pointer returned by direct_access()
so that it is a __pmem pointer.  This is consistent with the PMEM driver
and with how this direct_access() pointer is used in the DAX code.

Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
2015-08-20 14:07:24 -04:00
Martin Schwidefsky 42cfc6b590 s390/dcssblk: correct out of bounds array indexes
Fix a couple of warnings like this:

[linux-4.2-rc7/drivers/s390/block/dcssblk.c:553]:
   (style) Array index 'j' is used before limits check.

Reported-by: David Binderman <dcb314@hotmail.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-08-19 10:48:49 +02:00
Heiko Carstens 92d6289105 s390: remove unneeded sizeof(void *) comparisons
Remove two more statements which always evaluate to 'false'.
These are more leftovers from the 31 bit era.

Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Reviewed-by: David Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-08-19 10:03:44 +02:00
Kent Overstreet 54efd50bfd block: make generic_make_request handle arbitrarily sized bios
The way the block layer is currently written, it goes to great lengths
to avoid having to split bios; upper layer code (such as bio_add_page())
checks what the underlying device can handle and tries to always create
bios that don't need to be split.

But this approach becomes unwieldy and eventually breaks down with
stacked devices and devices with dynamic limits, and it adds a lot of
complexity. If the block layer could split bios as needed, we could
eliminate a lot of complexity elsewhere - particularly in stacked
drivers. Code that creates bios can then create whatever size bios are
convenient, and more importantly stacked drivers don't have to deal with
both their own bio size limitations and the limitations of the
(potentially multiple) devices underneath them.  In the future this will
let us delete merge_bvec_fn and a bunch of other code.

We do this by adding calls to blk_queue_split() to the various
make_request functions that need it - a few can already handle arbitrary
size bios. Note that we add the call _after_ any call to
blk_queue_bounce(); this means that blk_queue_split() and
blk_recalc_rq_segments() don't need to be concerned with bouncing
affecting segment merging.

Some make_request_fn() callbacks were simple enough to audit and verify
they don't need blk_queue_split() calls. The skipped ones are:

 * nfhd_make_request (arch/m68k/emu/nfblock.c)
 * axon_ram_make_request (arch/powerpc/sysdev/axonram.c)
 * simdisk_make_request (arch/xtensa/platforms/iss/simdisk.c)
 * brd_make_request (ramdisk - drivers/block/brd.c)
 * mtip_submit_request (drivers/block/mtip32xx/mtip32xx.c)
 * loop_make_request
 * null_queue_bio
 * bcache's make_request fns

Some others are almost certainly safe to remove now, but will be left
for future patches.

Cc: Jens Axboe <axboe@kernel.dk>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Ming Lei <ming.lei@canonical.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: dm-devel@redhat.com
Cc: Lars Ellenberg <drbd-dev@lists.linbit.com>
Cc: drbd-user@lists.linbit.com
Cc: Jiri Kosina <jkosina@suse.cz>
Cc: Geoff Levand <geoff@infradead.org>
Cc: Jim Paris <jim@jtan.com>
Cc: Philip Kelleher <pjk1939@linux.vnet.ibm.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Nitin Gupta <ngupta@vflare.org>
Cc: Oleg Drokin <oleg.drokin@intel.com>
Cc: Andreas Dilger <andreas.dilger@intel.com>
Acked-by: NeilBrown <neilb@suse.de> (for the 'md/md.c' bits)
Acked-by: Mike Snitzer <snitzer@redhat.com>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: Kent Overstreet <kent.overstreet@gmail.com>
[dpark: skip more mq-based drivers, resolve merge conflicts, etc.]
Signed-off-by: Dongsu Park <dpark@posteo.net>
Signed-off-by: Ming Lin <ming.l@ssi.samsung.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-08-13 12:31:33 -06:00
Stefan Haberland b179b037e5 s390/dasd: enhance CUIR scope detection
This patch adds an enhanced detection for control unit initiated
reconfiguration request scope.
The first approach assumed the scope of the reconfiguration request
to be restricted to the path on which the message was received.
The enhanced approach determines the full scope of the reconfiguration
request by evaluating additional path and device selection information
contained in the reconfiguration message.

Reviewed-by: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>
Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-08-09 21:08:10 +02:00
Stefan Haberland 1eb38023dd s390/dasd: fix failing path verification
DASD path verification requires the usage of sleep_on_immediatly to
ensure that no other I/O request is blocking the recovery of
disconnected devices. But two concurrent path verification workers for
the same device may kill each others requests due to the usage of the
immediate sleep_on function. This may lead to unsuccessful path
verifications.

Prevent that two parallel path verification workers conflict with
each other by implementing a device flag signalling a already running
worker.

Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-08-09 21:07:41 +02:00
Peter Senna Tschudin 3b97487445 s390: remove unneeded semicolon
Remove unneeded semicolon.

The semantic patch that detects this change is available
at scripts/coccinelle/misc/semicolon.cocci.

Signed-off-by: Peter Senna Tschudin <peter.senna@gmail.com>
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-08-07 09:57:00 +02:00
Christoph Hellwig 4246a0b63b block: add a bi_error field to struct bio
Currently we have two different ways to signal an I/O error on a BIO:

 (1) by clearing the BIO_UPTODATE flag
 (2) by returning a Linux errno value to the bi_end_io callback

The first one has the drawback of only communicating a single possible
error (-EIO), and the second one has the drawback of not beeing persistent
when bios are queued up, and are not passed along from child to parent
bio in the ever more popular chaining scenario.  Having both mechanisms
available has the additional drawback of utterly confusing driver authors
and introducing bugs where various I/O submitters only deal with one of
them, and the others have to add boilerplate code to deal with both kinds
of error returns.

So add a new bi_error field to store an errno value directly in struct
bio and remove the existing mechanisms to clean all this up.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-07-29 08:55:15 -06:00
Stefan Haberland f81a49d13b s390/dasd: fix kernel panic when alias is set offline
The dasd device driver selects which (alias or base) device is used
for a given requests when the request is build. If the chosen alias
device is set offline before the request gets queued to the device
queue the starting function may use device structures that are
already freed. This might lead to a hanging offline process or a
kernel panic.

Add a check to the starting function that returns the request to the
upper layer if the device is already in offline processing.

In addition to that prevent that an alias device that's already in
offline processing gets chosen as start device.

Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Reviewed-by: Peter Oberparleiter <peter.oberparleiter@linux.vnet.ibm.com>
Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-07-13 11:02:14 +02:00
Linus Torvalds 6a398a3ef4 Merge branch 'for-4.2/drivers' of git://git.kernel.dk/linux-block
Pull block driver updates from Jens Axboe:
 "This contains:

   - a few race fixes for null_blk, from Akinobu Mita.

   - a series of fixes for mtip32xx, from Asai Thambi and Selvan Mani at
     Micron.

   - NVMe:
        * Fix for missing error return on allocation failure, from Axel
          Lin.

        * Code consolidation and cleanups from Christoph.

        * Memory barrier addition, syncing queue count and queue
          pointers. From Jon Derrick.

        * Various fixes from Keith, an addition to support user
          issue reset from sysfs or ioctl, and automatic namespace
          rescan.

        * Fix from Matias, avoiding losing some request flags when
          marking the request failfast.

   - small cleanups and sparse fixups for ps3vram.  From Geert
     Uytterhoeven and Geoff Lavand.

   - s390/dasd dead code removal, from Jarod Wilson.

   - a set of fixes and optimizations for loop, from Ming Lei.

   - conversion to blkdev_reread_part() of loop, dasd, ndb.  From Ming
     Lei.

   - updates to cciss.  From Tomas Henzl"

* 'for-4.2/drivers' of git://git.kernel.dk/linux-block: (44 commits)
  mtip32xx: Fix accessing freed memory
  block: nvme-scsi: Catch kcalloc failure
  NVMe: Fix IO for extended metadata formats
  nvme: don't overwrite req->cmd_flags on sync cmd
  mtip32xx: increase wait time for hba reset
  mtip32xx: fix minor number
  mtip32xx: remove unnecessary sleep in mtip_ftl_rebuild_poll()
  mtip32xx: fix crash on surprise removal of the drive
  mtip32xx: Abort I/O during secure erase operation
  mtip32xx: fix incorrectly setting MTIP_DDF_SEC_LOCK_BIT
  mtip32xx: remove unused variable 'port->allocated'
  mtip32xx: fix rmmod issue
  MAINTAINERS: Update ps3vram block driver
  block/ps3vram: Remove obsolete reference to MTD
  block/ps3vram: Fix sparse warnings
  NVMe: Automatic namespace rescan
  NVMe: Memory barrier before queue_count is incremented
  NVMe: add sysfs and ioctl controller reset
  null_blk: restart request processing on completion handler
  null_blk: prevent timer handler running on a different CPU where started
  ...
2015-06-25 15:12:50 -07:00
Jarod Wilson a05e578055 s390/block/dasd: remove obsolete while -EBUSY loop
With the mutex_trylock bit gone from blkdev_reread_part(), the retry logic
in dasd_scan_partitions() shouldn't be necessary.

CC: Christoph Hellwig <hch@infradead.org>
CC: Jens Axboe <axboe@kernel.dk>
CC: Tejun Heo <tj@kernel.org>
CC: Alexander Viro <viro@zeniv.linux.org.uk>
CC: Markus Pargmann <mpa@pengutronix.de>
CC: Stefan Weinhuber <wein@de.ibm.com>
CC: Stefan Haberland <stefan.haberland@de.ibm.com>
CC: Sebastian Ott <sebott@linux.vnet.ibm.com>
CC: Fabian Frederick <fabf@skynet.be>
CC: Ming Lei <ming.lei@canonical.com>
CC: David Herrmann <dh.herrmann@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: Peter Zijlstra <peterz@infradead.org>
CC: nbd-general@lists.sourceforge.net
CC: linux-s390@vger.kernel.org
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Ming Lei <ming.lei@canonical.com>
Signed-off-by: Jarod Wilson <jarod@redhat.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-20 09:06:17 -06:00
Ming Lei 6029a06c88 block: dasd_genhd: convert to blkdev_reread_part
Also remove the obsolete comment.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Tested-by: Jarod Wilson <jarod@redhat.com>
Acked-by: Jarod Wilson <jarod@redhat.com>
Acked-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Ming Lei <ming.lei@canonical.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-20 09:06:15 -06:00
Peter Oberparleiter 7c53fcb39f s390/dasd: Enable automatic loading of dasd_diag_mod
Enabling a DASD that was configured to use the DIAG250 access method
while the corresponding kernel module dasd_diag_mod has not been loaded
fails with an error message. To fix this, users need to manually load
the dasd_diag_mod module.

This procedure can be simplified by automatically loading the
dasd_diag_mod from within the kernel when a DASD configured for DIAG250
is set online.

Signed-off-by: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-05-13 09:58:19 +02:00
Stefan Haberland df3044f1ef s390/dasd: Fix unresumed device after suspend/resume having no paths
The DASD device driver prevents I/O from being started on stopped
devices. This also prevented channel paths to be verified and so
the device was unable to be resumed.
Fix by allowing path verification requests on stopped devices.

Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-04-15 12:23:51 +02:00
Stefan Haberland a3147a7bc2 s390/dasd: fix unresumed device after suspend/resume
The DASD device driver only has a limited amount of memory to build
I/O requests.
This memory was used by blocklayer requests leading to an inability
to build needed internal requests to resume the device.
Fix by preventing the DASD driver to fetch requests for a stopped
device.

Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Reference-ID: RQM 2520
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-04-15 12:23:51 +02:00
Stefan Haberland f2608cd4a3 s390/dasd: fix inability to set a DASD device offline
Fix ref counting for DASD devices leading to an inability to set a
DASD device offline.
Before a worker is scheduled the DASD device driver takes a reference
to the device. If the worker was already scheduled this reference was
never freed.
Fix by giving the reference to the DASD device free when
schedule_work() returns false.

Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-04-15 12:23:50 +02:00
Stefan Haberland a5fd8ddce2 s390/dasd: remove setting of scheduler from driver
Remove the hard coded scheduler for the DASD device driver to enable
change of the scheduler during runtime. Set recommended deadline
scheduler via additional udev rule.

Signed-off-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-03-25 11:49:40 +01:00
Heiko Carstens 5a79859ae0 s390: remove 31 bit support
Remove the 31 bit support in order to reduce maintenance cost and
effectively remove dead code. Since a couple of years there is no
distribution left that comes with a 31 bit kernel.

The 31 bit kernel also has been broken since more than a year before
anybody noticed. In addition I added a removal warning to the kernel
shown at ipl for 5 minutes: a960062e58 ("s390: add 31 bit warning
message") which let everybody know about the plan to remove 31 bit
code. We didn't get any response.

Given that the last 31 bit only machine was introduced in 1999 let's
remove the code.
Anybody with 31 bit user space code can still use the compat mode.

Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-03-25 11:49:33 +01:00
Ameen Ali 3a9f9183bd s390/dcss: array index 'i' is used before limits check.
Avoid out-of-bounds-read by checking count before indexing.

Signed-off-by : Ameen Ali <Ameenali023@gmail.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-02-26 09:24:48 +01:00
Sebastian Ott a13ccb04af s390/scm_block: fix off by one during cluster reservation
We increase the msb_count after we're finished building the request.
That way we can always access the current request via
scmrq->request[msb_count] . But once the request is started we need
to make sure that the array index stays below msb_count.

Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-02-26 09:24:47 +01:00
Linus Torvalds 3e12cefbe1 Merge branch 'for-3.20/core' of git://git.kernel.dk/linux-block
Pull core block IO changes from Jens Axboe:
 "This contains:

   - A series from Christoph that cleans up and refactors various parts
     of the REQ_BLOCK_PC handling.  Contributions in that series from
     Dongsu Park and Kent Overstreet as well.

   - CFQ:
        - A bug fix for cfq for realtime IO scheduling from Jeff Moyer.
        - A stable patch fixing a potential crash in CFQ in OOM
          situations.  From Konstantin Khlebnikov.

   - blk-mq:
        - Add support for tag allocation policies, from Shaohua. This is
          a prep patch enabling libata (and other SCSI parts) to use the
          blk-mq tagging, instead of rolling their own.
        - Various little tweaks from Keith and Mike, in preparation for
          DM blk-mq support.
        - Minor little fixes or tweaks from me.
        - A double free error fix from Tony Battersby.

   - The partition 4k issue fixes from Matthew and Boaz.

   - Add support for zero+unprovision for blkdev_issue_zeroout() from
     Martin"

* 'for-3.20/core' of git://git.kernel.dk/linux-block: (27 commits)
  block: remove unused function blk_bio_map_sg
  block: handle the null_mapped flag correctly in blk_rq_map_user_iov
  blk-mq: fix double-free in error path
  block: prevent request-to-request merging with gaps if not allowed
  blk-mq: make blk_mq_run_queues() static
  dm: fix multipath regression due to initializing wrong request
  cfq-iosched: handle failure of cfq group allocation
  block: Quiesce zeroout wrapper
  block: rewrite and split __bio_copy_iov()
  block: merge __bio_map_user_iov into bio_map_user_iov
  block: merge __bio_map_kern into bio_map_kern
  block: pass iov_iter to the BLOCK_PC mapping functions
  block: add a helper to free bio bounce buffer pages
  block: use blk_rq_map_user_iov to implement blk_rq_map_user
  block: simplify bio_map_kern
  block: mark blk-mq devices as stackable
  block: keep established cmd_flags when cloning into a blk-mq request
  block: add blk-mq support to blk_insert_cloned_request()
  block: require blk_rq_prep_clone() be given an initialized clone request
  blk-mq: add tag allocation policy
  ...
2015-02-12 14:13:23 -08:00
Sebastian Ott 6765cc2ac6 s390/dasd: cleanup profiling
The dasd driver has a lot of duplicated code to handle
dasd_global_profile. With this patch we use the same code for the
global and the per device profiling data. Note that dasd_stats_write
had to change slightly to maintain some odd differences between
A) per device and global profile and B) proc and sysfs interface
usage.

Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Reviewed-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-01-29 09:19:29 +01:00
Sebastian Ott 8ea55c95c3 s390/dasd: add locking for global_profile access
Access to DASDs global statistics is done without locking which
can lead to inconsistent data. Add locking to fix this. Also move
the relevant structs in a global dasd_profile struct.

Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Reviewed-by: Stefan Haberland <stefan.haberland@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
2015-01-29 09:19:27 +01:00