Enable the display of supported and active fc4s for zfcp in the FC
transport class. zfcp only supports FCP, so simply hard-code this
information. The zfcp hbaapi already has this information hardcoded,
but this would allow to switch from the coding in the zfcp hbaapi to
the common FC transport attributes in the future.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
In case the SCSI error recovery starts because of a SCSI command
timeout, but then something else triggers the rport to be deleted, the
SCSI error recovery will run to the end and set the SCSI device
offline. To prevent this, call the FC transport function
fc_block_scsi_eh which waits until the rport leaves the BLOCKED state.
This guarantees that communication is possible if the rport is ONLINE,
or the SCSI devices will be removed if the rport state switches to
NOT_PRESENT.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The SCSI midlayer retries commands based on the remote port state and
the command status reported by the driver. Returning
DID_TRANSPORT_DISRUPTED is a better approach, use this for reporting
FSF errors back to the SCSI midlayer. See
http://marc.info/?l=linux-scsi&m=125668044215051&w=2 as reference.
There is also no need in special treatment of ABORTED commands, so
remove the ZFCP_STATUS_FSFREQ_ABORTED, the commands are then returned
with DID_TRANSPORT_DISRUPTED.
Also remove the ZFCP_STATUS_FSFREQ_RETRY: It is useless, no retry is
happening in the FSF layer and nobody checks the state of this flag.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Introduce kmem_cache for ELS ADISC data to guarantee the required
hardware alignment and free the allocated memory in case the send
failes.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Remove some redundancies in FC related code and trace:
- drop redundant data from SAN trace (local s_id that only changes
during link down, ls_code that is already part of payload, d_id in
ct response trace that is always the same as in ct request trace)
- use one common fsf struct to hold zfcp data for ct and els requests
- leverage common fsf struct for FC passthrough job data, allocate it
with dd_bsg_data for passthrough requests and unify common code for
ct and els passthrough request
- simplify callback handling in zfcp_fc
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Instead of assigning 4 bytes with the highest byte masked out, use a 3
byte array with the ntoh24 and h24ton helper functions, thus
eliminating the need for the ZFCP_DID_MASK.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The well-known-address (WKA) port handling code is part of the FC code
in zfcp. Move everything WKA related to the zfcp_fc files and use the
common zfcp_fc prefix for structs and functions. Drop the unused key
management service while renaming the struct, no request could ever
reach this service in zfcp and it is obsolete anyway.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Use common code definitions for FC GPN_FT and GID_PN
instead of inventing private ones. Move the private structs still
required inside zfcp to zfcp_fc header file.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Use common code definitions for FC plogi, logo, rscn and adisc structs
instead of inventing private ones. Move the private struct for issuing
ELS ADISC inside zfcp to zfcp_fc header file.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Use common data structures for FCP CMND, FCP RSP and related
definitions and remove zfcp private definitions. Split the FCP CMND
setup and FCP RSP evaluation code in seperate functions. Use inline
functions to not negatively impact the I/O path.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
If an error occurs that triggers the call to fc_remote_port_delete,
ideally this call would happen before any I/O is passed back to the
SCSI midlayer through scsi_done. The SCSI midlayer will retry the
commands and fc_remote_port_chkready will return the correct status
code. But with the delay between calling scsi_done in softirq context
and the call to fc_remote_port_delete from the workqueue, there is a
window where zfcp returns DID_ERROR. This leads to SCSI error recovery
which then leads to offline SCSI devices since all recovery actions
will fail with the rport now being blocked.
In this window, zfcp has to return DID_IMM_RETRY just as the FC
transport class would do in fc_remote_port_chkready for the blocked
fc_rport. As soon as the fc_rport is BLOCKED, fc_remote_port_chkready
will do the right thing.
Additionally, there are two more cases to catch in zfcp_scsi_queuecommand:
- After the port has been opened, the unit has to be opened. During
this period I/O has to be retried. This can also be handled with
DID_IMM_RETRY.
- If the access to the unit fails, but the port is good, then
this single unit cannot be accessed and I/O to this unit has to fail
without involving the FC transport class.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The port_scan work was scheduled to the work_queue provided by the
kernel. This resulted on SMP systems to a likely situation that more
than one scan_work were processed in parallel. This is not required
and openes the possibility of race conditions between the removal of
invalid ports and the enqueue of just scanned ports. This patch
synchronizes the scan_work tasks by scheduling them to adapter local
work_queue.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The flag ZFCP_STATUS_COMMON_REMOVE was used to indicate that a
resource is not ready to be used or about to be removed from the
system. This is now better done by an improved list handling
and therefore the additional indicator is not required anymore.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
With the reference counting for zfcp data structures, it is now
possible to implement module unloading again. Module unloading
requires to free all data structures in the module exit function. This
is done by unregistering zfcp from s390 cio and the SCSI midlayer
first in the module exit function.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The latencies traced per fsf request are traced for sysfs output and
for blktrace, each in one function. Simplify the tracing code by
merging both tracing functions into one.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
When accessing port and unit attributes, use container_of instead of
dev_get_drvdata. This eliminates some code checker warnings about
aliased access of data structures.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The callback for suspend is not required because it contains exactly
the same functionality as the _set_offline routine does.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The global config_mutex was required for the serialization of a
configuration change within the zfcp driver. This global locking is
now obsolete and can be removed. The requirement of serializing the
access to a zfcp_adapter reference via a ccw_device is realized wth a
static spinlock.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Replace the local reference counting by already available mechanisms
offered by kref. Where possible existing device structures were used,
including the same functionality.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The global config_lock was used to protect the configuration organized
in independent lists. It is not necessary to have a lock on driver
level for this purpose. This patch replaces the global config_lock
with a set of local list locks.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adapt the change_queue_depth callback in zfcp for the new reason
parameter. Simply pass each call back to the SCSI midlayer, there are
no resource adjustments necessary for zfcp.
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Removes check for (depth <= default_depth) in case of
SCSI_QDEPTH_RAMP_UP call back, not needed after added
max_queue_depth per sdev.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch modifies scsi_host_template->change_queue_depth so that
it takes an argument indicating why it is being called. This will be
used so that if a LLD needs to do some extra processing when
handling queue fulls or later ramp ups, it can do so.
This is a simple port of the drivers setting a change_queue_depth
callback. In the patch I just have these LLDs adjust the queue depth
if the user was requesting it.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
[Vasu.Dev: v2
Also converted pmcraid_change_queue_depth and then verified
all modules compile using "make allmodconfig" for any new build
warnings on X86_64.
Updated original description after combing two original
patches from Mike to make this patch git bisectable.]
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
[jejb: fixed up 53c700]
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
When configuring a LUN for use in zfcp, flush the SCSI work to ensure
the SCSI device has been created before returning. This means that a
configuration procedure can run these commands in a script and the
SCSI device is available immediately after the unit_add:
echo 1 > /sys/bus/ccw/drivers/zfcp/0.0.181d/online
echo 0x401040C300000000 > \
/sys/bus/ccw/drivers/zfcp/0.0.181d/0x500507630313c562/unit_add
lsscsi
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Add HZ since the start_timer function expects jiffies, not seconds.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
After opening a remote port zfcp checks if the WWPN returned in the
PLOGI maches the WWPN of the port that should have been opened. On a
mismatch zfcp assumes that the DID just changed, queries the FC
nameserver and tries again. If the situation persists the erp will
give up.
With this strategy, if the remote port always returns the wrong PLOGI
data, the remote port will not be opened. Introduce a warning, so that
the system administrator knows why the remote port is not being opened
and to have a pointer to investigate the problem on the storage
system.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
For ports, zfcp gets the DID from the FC nameserver and tries to open
the port. If the open succeeds, zfcp compares the WWPN from the
nameserver with the WWPN in the PLOGI payload. In case of a mismatch,
zfcp assumes that the DID of the port just changed and we opened the
wrong port. This means that zfcp has to forget the DID, lookup the DID
again and retry.
This error case had a problem that zfcp forgets the DID, but never
looks up a new one, stalling the ERP in this case. Fix this by
triggering the DID lookup and properly exit from the ERP. The DID
lookup will trigger a new ERP action.
Also ensure when trying to open the port again with the new DID, first
close the open port, even in the NOESC case.
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The pointer that is allocated with kmalloc() is passed to strsep()
which modifies it. Later on the modified pointer value will be passed
to kfree. Save the original pointer and pass that one to kfree
instead.
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Running chchp --vary 0 and chccwdev -d on a FCP device with scsi
devices attached can lead to this thread hanging:
================================================================
STACK TRACE FOR TASK: 0x2fbfcc00 (kslowcrw)
STACK:
0 schedule+1136 [0x45f99c]
1 schedule_timeout+534 [0x46054e]
2 wait_for_common+374 [0x45f442]
3 blk_execute_rq+160 [0x217a2c]
4 scsi_execute+278 [0x26daf2]
5 scsi_execute_req+150 [0x26dc86]
6 sd_sync_cache+138 [0x28460a]
7 sd_shutdown+130 [0x28486a]
8 sd_remove+104 [0x284c84]
9 __device_release_driver+152 [0x257430]
10 device_release_driver+56 [0x2575c8]
11 bus_remove_device+214 [0x25672a]
12 device_del+352 [0x25456c]
13 __scsi_remove_device+108 [0x272630]
14 scsi_remove_device+66 [0x2726ba]
15 zfcp_ccw_remove+824 [0x335558]
16 ccw_device_remove+62 [0x2b3f2a]
17 __device_release_driver+152 [0x257430]
18 device_release_driver+56 [0x2575c8]
19 bus_remove_device+214 [0x25672a]
20 device_del+352 [0x25456c]
21 ccw_device_unregister+92 [0x2b48c4]
22 io_subchannel_remove+108 [0x2b4950]
23 css_remove+62 [0x2af7ee]
24 __device_release_driver+152 [0x257430]
25 device_release_driver+56 [0x2575c8]
26 bus_remove_device+214 [0x25672a]
27 device_del+352 [0x25456c]
28 device_unregister+38 [0x25464a]
29 css_sch_device_unregister+68 [0x2af97c]
30 ccw_device_call_sch_unregister+78 [0x2b581e]
31 worker_thread+604 [0x69eb0]
32 kthread+154 [0x6ff42]
33 kernel_thread_starter+6 [0x1c952]
================================================================
The problem is that the chchp --vary 0 leads to zfcp first calling
fc_remote_port_delete which blocks all scsi devices on the remote
port. Calling scsi_remove_device later lets the sd driver issue a
SYNCHRONIZE_CACHE command. This command stays on the "stopped" request
requeue because the SCSI device is blocked. Fix this by first removing
the scsi and fc hosts which removes all scsi devices and do not use
scsi_remove_device.
Reviewed-by: Felix Beck <felix.beck@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.31-39.x.20090917-s390xdefault #1
-------------------------------------------------------
kslowcrw/83 is trying to acquire lock:
(&adapter->scan_work){+.+.+.}, at: [<0000000000169c5c>] __cancel_work_timer+0x64/0x3d4
but task is already holding lock:
(&zfcp_data.config_mutex){+.+.+.}, at: [<00000000004671ea>] zfcp_ccw_remove+0x66/0x384
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&zfcp_data.config_mutex){+.+.+.}:
[<0000000000189962>] __lock_acquire+0xe26/0x1834
[<000000000018a4b6>] lock_acquire+0x146/0x178
[<000000000058cb5a>] mutex_lock_nested+0x82/0x3ec
[<0000000000477170>] zfcp_fc_scan_ports+0x3ec/0x728
[<0000000000168e34>] worker_thread+0x278/0x3a8
[<000000000016ff08>] kthread+0x9c/0xa4
[<0000000000109ebe>] kernel_thread_starter+0x6/0xc
[<0000000000109eb8>] kernel_thread_starter+0x0/0xc
-> #0 (&adapter->scan_work){+.+.+.}:
[<0000000000189e60>] __lock_acquire+0x1324/0x1834
[<000000000018a4b6>] lock_acquire+0x146/0x178
[<0000000000169c9a>] __cancel_work_timer+0xa2/0x3d4
[<0000000000465cb2>] zfcp_adapter_dequeue+0x32/0x14c
[<00000000004673e4>] zfcp_ccw_remove+0x260/0x384
[<00000000004250f6>] ccw_device_remove+0x42/0x1ac
[<00000000003cb6be>] __device_release_driver+0x9a/0x10c
[<00000000003cb856>] device_release_driver+0x3a/0x4c
[<00000000003ca94c>] bus_remove_device+0xcc/0x114
[<00000000003c8506>] device_del+0x162/0x21c
[<0000000000425ff2>] ccw_device_unregister+0x5e/0x7c
[<000000000042607e>] io_subchannel_remove+0x6e/0x9c
[<000000000041ff9a>] css_remove+0x3e/0x7c
[<00000000003cb6be>] __device_release_driver+0x9a/0x10c
[<00000000003cb856>] device_release_driver+0x3a/0x4c
[<00000000003ca94c>] bus_remove_device+0xcc/0x114
[<00000000003c8506>] device_del+0x162/0x21c
[<00000000003c85e8>] device_unregister+0x28/0x38
[<0000000000420152>] css_sch_device_unregister+0x46/0x58
[<00000000004276a6>] io_subchannel_sch_event+0x28e/0x794
[<0000000000420442>] css_evaluate_known_subchannel+0x46/0xd0
[<0000000000420ebc>] slow_eval_known_fn+0x88/0xa0
[<00000000003caffa>] bus_for_each_dev+0x7e/0xd0
[<000000000042188c>] for_each_subchannel_staged+0x6c/0xd4
[<0000000000421a00>] css_slow_path_func+0x54/0xd8
[<0000000000168e34>] worker_thread+0x278/0x3a8
[<000000000016ff08>] kthread+0x9c/0xa4
[<0000000000109ebe>] kernel_thread_starter+0x6/0xc
[<0000000000109eb8>] kernel_thread_starter+0x0/0xc
cancel_work_sync is called while holding the config_mutex. But the
work that is being cancelled or flushed also uses the config_mutex.
Fix the resulting deadlock possibility by calling cancel_work_sync
earlier without holding the mutex. The best place to do is is after
offlining the device. No new port scan work will be scheduled for the
offline device, so this is a safe place to call cancel_work_sync.
Reviewed-by: Felix Beck <felix.beck@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
With the change that the zfcp_adapter struct is only allocated when
the device is set online, the shutdown handler has to check for a
non-existing zfcp_adapter struct. On the other hand, this check is not
necessary in the offline callback, since an online device has the
zfcp_adapter allocated and we go through the offline callback before
removing the ccw device.
Reviewed-by: Felix Beck <felix.beck@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
With the change for delaying the allocation of zfcp_adapter, the
initial device parameter function has to first call
ccw_device_set_online which allocates the zfcp_adapter structure.
Change this and adapt the cfdc part accordingly.
Reviewed-by: Felix Beck <felix.beck@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The common initialization of ct/gs and els requests missed the
initialization of unchained requests. Fix this by moving the common
parts to a place that is called for all ct/gs and els requests.
Reviewed-by: Felix Beck <felix.beck@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Remove expensive ktime_get()/ktime_us_delta() functions from the hot
path and use get_clock_monotonic() instead. This elimates seven
function calls and avoids a lot of unnecessary calculations.
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The timestamp calculation used for s390dbf output is the same in a
private zfcp function and in debug.c. Replace both with a common
inline function.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
dev_set_name tries to allocate memory, so check the return value for
allocation failures. After dev_set_name succeeds, call device_register
as next step to be able to use put_device during error handling.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Don't use kfree directly after device registration started.
Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The config semaphore is only used as a mutex, so replace it with a
simple mutex.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
So far, zfcp allocated all resources required for FCP
adapters/subchannels when the device was discovered in the ccw_probe
callback. If there are lots of unused FCP subchannels attached to a
system, this is a waste of resources. To alleviate this, defer the
resource allocation to the first call to ccw_set_online. To avoid
disruptions during possible following calls to ccw_set_offline and
then ccw_set_online, keep the adapter resources until the device is
finally being removed via ccw_remove. While doing this, also manage
the zfcp erp thread together with all other adapter resources in
zfcp_adapter_enqueue/dequeue.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The recommendation for a timeout of 2 * R_A_TOV is the same for ct/gs
and els requests, so set it in the common function used for
initializing both request types. Besides, the timer inside zfcp should
only run longer than the timeout set for the channel, so 10 seconds
more should be enough (instead of 60 seconds).
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Switch the creation of the zfcp erp thread from the deprecated
kernel_thread API to the kthread API. This allows also the removal of
some flags in zfcp since the kthread API handles thread creation and
shutdown internally. To allow the usage of the kthread_stop function,
replace the erp ready semaphore with a waitqueue for waiting until erp
actions arrive on the ready queue.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The fc_rport structure reserves a reference where a LLD can put
information required in a situation where the fc transport class is
triggering LLD callbacks. The zfcp driver was using this variable
directly which is discouraged. This patch solves this issue by making
this reference unnecessary. In addition the dev_loss_tmo callback is
removed, it is not required: zfcp does not access the fc_rport after
calling fc_remote_port_delete.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Update the Fibre Channel related code to use the zfcp_fc prefix.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Extract independent data structures and introduce common _setup and
_destroy routines for QDIO and Fibre Channel related data structures
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Change the dbf data and functions to use the zfcp_dbf prefix
throughout the code. Also change the calls to dbf to use zfcp_dbf
instead of zfcp_adapter.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Don't let the erp wait for gid_pn requests to complete. Instead, queue
the gid_pn work, exit erp and let the finished gid_pn work trigger a
new port reopen.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The zfcp_adapter structure was growing over time to a size of almost
one memory page. To reduce the size of the data structure and to
seperate different layers, put all qdio related data in the new
zfcp_qdio data structure.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Split all qdio related attributes out of zfcp_fsf_req and put it in
new structure.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Remove the global driver work queue and replace it with a workqueue
local to the adapter. The usage of this workqueue makes this the
correct place for the structure. In addition multiple adapters won't
block each other due to the serialization of the queued work.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The flag ZFCP_REQ_AUTO_CLEANUP was useless as the
ZFCP_STATUS_FSFREQ_CLEANUP flag is there for exactly the same purpose.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Remove the special case for NO_QTCB requests and optimize the
mempool and cache processing for fsfreqs. Especially use seperate
mempools for the zfcp_fsf_req and zfcp_qtcb structs.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The combination wait_queue/wakeup in conjunction with the flag
ZFCP_STATUS_FSFREQ_COMPLETED to signal the completion of an fsfreq
was not race-safe and can be better solved by a completion.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
There is no need for the QDIO layer to have knowledge or do things
wich are done better by the FSF layer and vice versa. Straighten a
few things to improve vividness.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
An adapter shutdown implicitly closes all open ports. Make sure to
mark all WKA ports as offline, not only the directory server. Also
make sure that no pending wka port work is running when the adapter is
being removed.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
When the FCP channel returns a series of commands with the error
status "test link", zfcp will send a series of ELS ADISC commands.
This is technically no problem, but it is enough to only issue one
test command per remote port. So, track whether a ELS ADISC command is
already pending, and do not send a new one if there is already a
pending command.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Using a bitwise OR to not set anything at all is pointless so remove
the useless statement.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The default trace level is to only trace failed FSF commands. Thus it
is not necessary to collect trace data for most FSF commands, since
it will be thrown away later. Restructure the FSF/HBA trace
infrastructure to first check the trace level in a inline function and
only do the expensive data collection for matching trace levels.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The default trace level is to only trace failed SCSI commands. Thus it
is not necessary to collect trace data for most SCSI commands since it
will be thrown away later. Restructure the SCSI trace infrastructure
to first check the trace level in a inline function and only do the
expensive data collection for matching trace levels.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The struct zfcp_adapter includes everything related to the debug
traces. This introduces dependences between the definitions in
zfcp_def.h and zfcp_dbf.h. Move all debug related data structures to a
new data structure to break those dependencies and manage the debug
data in zfcp_dbf.[hc].
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
In certain error scenarios ports, rports are getting attached,
validated and removed from the systems environment. Depending on the
layer this occurs asynchronously. This patch fixes the few races
which existed and ensures all references and cross references are
cleared at the time they're invalid. In addition fc transports
actions are only scheduled when required.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The trace record for SCSI abort requests has a field for the request
id of the request to be aborted. Put the real request id instead of
zero.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Under certain conditions it is possible that a WKA port ist not opened
within the expected timeframe of half a second. In this situation
the WKA port remains in the state OPENING preventing any succeding
request to open the port. This led to unrecoverable remote ports.
Fixing this by always setting an appropriate WKA port status before
leaving the function and removing the timeout value here since it's
not needed here because the general timeout processing would deal
with it if required.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
In a LOWMEM condition an ERP notification would have been sent twice
causing an unpredictable behaviour of the ERP.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When calling fc_remote_port_add make sure to not call it again before
fc_remote_port_delete has been called. In other words, ensure to
create a new fc_rport, then delete it, then create a new one again.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Depending on interruptions on some storage systems, the complete
channel can stall which looks like an outbound queue stall to Linux.
When trying to acquire a free SBAL for a non-SCSI command, zfcp waits
for 5 seconds for a free slot to appear. This is the right place to
detect a queue stall: If the wait times out, we assume a stalled queue
and try to recover this.
The overall strategy should be to trigger the erp from specific
events, and not try an overall escalation from one failed port to a
full-blown queue recovery. If we manage to send a command, the status
codes for this command or a timeout will trigger the right follow-on
actions.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
If an action fails, retry it until the erp count exceeds the
threshold. If there is something fundamentally wrong, the FSF layer
will trigger a more appropriate action depending on the FSF status
codes.
The followup for successful actions is a different followup than
retrying failed actions, so split the code two functions to make this
clear.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
After closing the port, we want it to be "not open" to consider the
action to be successful.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The ELS ADISC and the GID_PN requests sent from zfcp fit into
unchained FSF requests. Change the FSF allocation logic to use
unchained requests whenever possible where everything fits in one
SBAL. This avoids acquiring more SBALs than necessary, especially
during zfcp recovery when things might be stalled.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
zfcp_erp_notify uses the ZFCP_ERP_STATUS_* flags, so it is
ZFCP_STATUS_ERP_LOWMEM instead of ZFCP_ERP_NOMEM. Signalling
ZFCP_ERP_FAILED is not necessary, the missing d_id will show that the
nameserver did not return the d_id.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When a fsf_req or a qtcb cannot be allocated return -ENOMEM instead of
-EIO.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
req_q_util is not atomic, so the qdio_stat_lock must be held when
reading this variable.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
We should not modify the port status after triggering an ERP action
for the port. It is not guaranteed which status is finally active
when the ERP action is performed. This can lead to situations which
are unwanted and hard to debug in case of a failure.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
* git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi-misc-2.6:
[SCSI] aic79xx: make driver respect nvram for IU and QAS settings
[SCSI] don't attach ULD to Dell Universal Xport
[SCSI] lpfc 8.3.3 : Update driver version to 8.3.3
[SCSI] lpfc 8.3.3 : Add support for Target Reset handler entrypoint
[SCSI] lpfc 8.3.3 : Fix a couple of spin_lock and memory issues and a crash
[SCSI] lpfc 8.3.3 : FC/FCOE discovery fixes
[SCSI] lpfc 8.3.3 : Fix various SLI-3 vs SLI-4 differences
[SCSI] qla2xxx: Resolve a performance issue in interrupt
[SCSI] cnic, bnx2i: Fix build failure when CONFIG_PCI is not set.
[SCSI] nsp_cs: time_out reaches -1
[SCSI] qla2xxx: fix printk format warnings
[SCSI] ncr53c8xx: div reaches -1
[SCSI] compat: don't perform unneeded copy in sg_io code
[SCSI] zfcp: Update FC pass-through support
[SCSI] zfcp: Add FC pass-through support
[SCSI] FC Pass Thru support
Don't access the block layer request, get the payload length instead
from the FC job. Simplify access to the zfcp_port, only the d_id is
required, if the port is no longer accessed later. This is possible
when the els_handler does not access the port pointer from the ELS
request.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Provide the ability to do fibre channel requests from the userspace to
our zfcp driver. Patch builds upon extension to the fibre channel
tranport class by James Smart and Seokmann Ju. See here
http://marc.info/?l=linux-scsi&m=123808882309133&w=2
Signed-off-by: Sven Schuetz <sven@linux.vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Keep the information about the device and model id in zfcp_ccw. This
requires an additional helper function to check for the privileged
cfdc subchannel, but it allows the removal of the redundant defines
from the zfcp_def header file.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
If the destination ID (D_ID) of a remote storage port changed, e.g.
re-plugged cable on the switch in a different switch port, the port
was never (re-)attached within Linux. This patch fixes the broken
mapping between the WWPN and the D_ID.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
In rare cases, open port request might timeout, erp calls
zfcp_port_put, port gets dequeued. Now, the late returning (or
dismissed) fsf-port-open calls the fsf_port_open_handler that tries to
reference the port data structure leading to a kernel oops.
Signed-off-by: Martin Petermann <martin.petermann@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Add comments where there is a deliberate fall through in switch/case
statements. This makes some code checkers happy and makes it clear
that there is no missing break statement.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
enum dma_data_direction only has the 4 values DMA_BIDIRECTIONAL,
DMA_TO_DEVICE, DMA_FROM_DEVICE and DMA_NONE. No need to have the
default case. While changing this, setup sbtype in one place to make
sparse happy.
The default value of retval is already -EIO, so remove the
additional assignment for these two cases.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
zfcp did always set the queue_depth for SCSI devices to 32, not
allowing to change this. Introduce a kernel parameter zfcp.queue_depth
and the change_queue_depth callback to allow changing the queue_depth
when it is required.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Update the newly introduced message for the boxed status to conform to
match the style of s390 and zfcp messages.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The zfcp traces used the fsf_req address in place of the req_id.
Change this to save the correct req_id.
Reviewed-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The zfcp_port might have been removed, while the FC fast_io_fail timer
is still running and could trigger the terminate_rport_io callback.
Set the pointer to the zfcp_port to NULL and check accordingly
before using it.
Reviewed-by: Martin Petermann <martin@linux.vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Before dropping the reference count with zfcp_adapter_put, increase it
with zfcp_adapter_get when issuing cfdc requests.
Reviewed-by: Martin Petermann <martin@linux.vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
If this problem appears zfcp ports cannot be de-queued since it is
checked for a zero refcount. The port reference counting is wrong for
existing zfcp ports when e.g. an adapter gets on-line again. During
port scanning the reference counting for existing ports should not be
changed.
Signed-off-by: Martin Petermann <martin.petermann@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The current sbal counting can be wrong if a fsf request is
waiting for free sbals and at the same time qdio request queue
is shutdown and re-opened. Revering a previous patch fixes this
issue.
Signed-off-by: Martin Petermann <martin.petermann@de.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When the abort handler cannot find a pending FSF request, the request
completion could just be running. This means we cannot return SUCCESS,
since this would lead to call to scsi_done after exiting the SCSI
error handler which is not allowed.
Reviewed-by: Martin Petermann <martin@linux.vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
A remote port remains in error state even if we receive a RSCN
stating that the connection is re-established. The port recovery
is not started due to a flag which is not reset.
The solution is to clear the flag in question before we trigger a ERP.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Error codes specific to the control file requests are evaluated by the
actcli tool, so don't report -ENXIO for those. Generic problems are
still checked for outside the command specific handler.
Reviewed-by: Martin Petermann <martin@linux.vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
On some hardware it can take some time to add a unit. If
some remove this unit during this process the remove will
fail.
Signed-off-by: Martin Petermann <martin@linux.vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The remote port remains in error state even if the connection
is re-established. A wrong precondition check was performed on
the port status leading to a cancellation of the port reopen.
Remove the pre-req check because it's not required and better
handled within the ERP.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The ERP thread is performing a task before it is executing the
corresponding down on the semaphore. The response handler of the
just started exchange config should wait for the completion by
performing a down on this semaphore. Since this semaphore is still
positive from the ERP enqueue the handler won't wait and therefore
the exchange config will always fail leaving the adapter in error.
The problem can be solved by performing the down on the semaphore
before starting an ERP task. This is the logically correct order.
Only walk the ERP loop if there is a task to perform.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The nameserver port might be in state online when the adapter is
offlined. On adapter reactivation the nameserver port is not
re-opened due to the PORT_ONLINE status. This results in an
unsuccessful recovery. In forcing the nameserver port status
to offline on all adapter offline events this issue is prevented.
Waiting for the reference count to drop to zero in
zfcp_wka_port_offline is not required, so remove it.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When running the scsi_scan from the zfcp workqueue and the target
device does not respond, the zfcp workqueue can block until the
scsi_scan hits a timeout. Move the work to the scsi host workqueue,
since this one is also used for the scan from the SCSI midlayer.
Signed-off-by: Swen Schillig <swen@vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Fix problem that zfcp_fsf_exchange_config_data_sync and
zfcp_fsf_exchange_config_data_sync could try to call zfcp_fsf_req_free
with a NULL pointer.
Reviewed-by: Martin Petermann <martin@linux.vnet.ibm.com>
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Christof Schmitt