If we take an EEH error early enough, we oops:
Call Trace:
[c000000010483770] [c000000000013ee4] .show_stack+0xd8/0x218 (unreliable)
[c000000010483850] [c000000000658940] .dump_stack+0x28/0x3c
[c0000000104838d0] [c000000000057a68] .eeh_dn_check_failure+0x2b8/0x304
[c000000010483990] [c0000000000259c8] .rtas_read_config+0x120/0x168
[c000000010483a40] [c000000000025af4] .rtas_pci_read_config+0xe4/0x124
[c000000010483af0] [c00000000037af18] .pci_bus_read_config_word+0xac/0x104
[c000000010483bc0] [c0000000008fec98] .pcibios_allocate_resources+0x7c/0x220
[c000000010483c90] [c0000000008feed8] .pcibios_resource_survey+0x9c/0x418
[c000000010483d80] [c0000000008fea10] .pcibios_init+0xbc/0xf4
[c000000010483e20] [c000000000009844] .do_one_initcall+0x98/0x1d8
[c000000010483ed0] [c0000000008f0560] .kernel_init+0x228/0x2e8
[c000000010483f90] [c000000000031a08] .kernel_thread+0x54/0x70
EEH: Detected PCI bus error on device <null>
EEH: This PCI device has failed 1 times in the last hour:
EEH: location=U78A5.001.WIH8464-P1 driver= pci addr=0001:00:01.0
EEH: of node=/pci@800000020000209/usb@1
EEH: PCI device/vendor: 00351033
EEH: PCI cmd/status register: 12100146
Unable to handle kernel paging request for data at address 0x00000468
Oops: Kernel access of bad area, sig: 11 [#1]
....
NIP [c000000000057610] .rtas_set_slot_reset+0x38/0x10c
LR [c000000000058724] .eeh_reset_device+0x5c/0x124
Call Trace:
[c00000000bc6bd00] [c00000000005a0e0] .pcibios_remove_pci_devices+0x7c/0xb0 (unreliable)
[c00000000bc6bd90] [c000000000058724] .eeh_reset_device+0x5c/0x124
[c00000000bc6be40] [c0000000000589c0] .handle_eeh_events+0x1d4/0x39c
[c00000000bc6bf00] [c000000000059124] .eeh_event_handler+0xf0/0x188
[c00000000bc6bf90] [c000000000031a08] .kernel_thread+0x54/0x70
We called rtas_set_slot_reset while scanning the bus and before the pci_dn
to pcidev mapping has been created. Since we only need the pcidev to work
out the type of reset and that only gets set after the module for the
device loads, lets just do a hot reset if the pcidev is NULL.
Signed-off-by: Anton Blanchard <anton@samba.org>
Acked-by: Linas Vepstas <linasvepstas@gmail.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
confirm_error_lock needs to be a real spinlock in RT. Convert it to
raw_spinlock.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
During a EEH recover, the pci_dev structure can be null, mainly if an
eeh event is detected during cpi config operation. In this case, the
pci_dev will not be known (and will be null) the kernel will crash
with the following message:
Unable to handle kernel paging request for data at address 0x000000a0
Faulting instruction address: 0xc00000000006b8b4
Oops: Kernel access of bad area, sig: 11 [#1]
NIP [c00000000006b8b4] .eeh_event_handler+0x10c/0x1a0
LR [c00000000006b8a8] .eeh_event_handler+0x100/0x1a0
Call Trace:
[c0000003a80dff00] [c00000000006b8a8] .eeh_event_handler+0x100/0x1a0
[c0000003a80dff90] [c000000000031f1c] .kernel_thread+0x54/0x70
The bug occurs because pci_name() tries to access a null pointer.
This patch just guarantee that pci_name() is not called on Null pointers.
Signed-off-by: Breno Leitao <leitao@linux.vnet.ibm.com>
Signed-off-by: Linas Vepstas <linasvepstas@gmail.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
By default, the EEH framework on powerpc does what's known as a "hot
reset" during recovery of a PCI Express device. We've found a case
where the device needs a "fundamental reset" to recover properly. The
current PCI error recovery and EEH frameworks do not support this
distinction.
The attached patch makes changes to EEH to utilize the new bit field.
Signed-off-by: Mike Mason <mmlnx@us.ibm.com>
Signed-off-by: Richard Lary <rlary@us.ibm.com>
Signed-off-by: Jesse Barnes <jbarnes@virtuousgeek.org>
To properly fix PCI hotplug, it's useful to be able to make the fixup
passes on all devices whether they were just hot plugged or already
there.
The EEH code however used to not be very friendly with calling
eeh_add_device_late() multiple time, and not very rebust in the way it
generally tests whether a device is in the expected state vs. the EEH
code.
This improves it, along with cleaning up a couple of debug printk's.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Paul Mackerras <paulus@samba.org>
This patch changes the EEH_MAX_FAILS action from panic to printing an
error message. Panicking under under this condition is too harsh.
Although performance will be affected and the device may not recover,
the system is still running, which at the very least should allow for a
more graceful shutdown. The patch also removes the msleep() within a
spinlock, which can lead to a deadlock and is not recommended.
Signed-off-by: Mike Mason <mmlnx@us.ibm.com>
Acked-by: Linas Vepstas <linasvepstas@gmail.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
The following patch restores the PERR and SERR bits in the PCI
command register during an EEH device recovery. We have found
at least one case (an Agilent test card) where the PERR/SERR
bits are set to 1 by firmware at boot time, but are not restored
to 1 during EEH recovery. The patch fixes the Agilent card
problem. It has been tested on several other EEH-enabled cards
with no regressions.
Signed-off-by: Mike Mason <mmlnx@us.ibm.com>
Acked-by: Linas Vepstas <linasvepstas@gmail.com>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Use proc_create()/proc_create_data() to make sure that ->proc_fops and ->data
be setup before gluing PDE to main tree.
Add correct ->owner to proc_fops to fix reading/module unloading race.
Signed-off-by: Denis V. Lunev <den@openvz.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Add a DEBUG config setting which turns on all (most) of the debugging
under platforms/pseries.
To have this take effect we need to remove all the #undef DEBUG's, in
various files. We leave the #undef DEBUG in platforms/pseries/lpar.c,
as this enables debugging printks from the low-level hash table routines,
and tends to make your system unusable. If you want those enabled you
still have to turn them on by hand.
Also some of the RAS code has a DEBUG block which causes a functional
change, so I've keyed this off a different (non-existant) debug #define.
This is only enabled if you have PPC_EARLY_DEBUG enabled also.
Signed-off-by: Michael Ellerman <michael@ellerman.id.au>
Signed-off-by: Paul Mackerras <paulus@samba.org>
A couple of places are duplicating the function of
of_device_is_available; convert them to use it.
Signed-off-by: Nathan Lynch <ntl@pobox.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
If an "empty" slot is failing, make sure its a permanent failure;
else process the error normally.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Perform all error checking at the "partitonable endpoint"
of the device.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Fix old buglet; a warning message should have been printed
when a hardware reset takes too long.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Bugfix: avoid crash if there's no PCI device for a given
openfirmware node.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
It seems that some versions of firmware will report a device
node status as the string "okay". As we are not expecting this
string, the device node will be ignored by the EEH subsystem.
Which means EEH will not be enabled.
When EEH is not enabled, PCI errors will be converted into
Machine Check exceptions, and we'll have a very unhappy system.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Remove dead code, and a misleading comment about EEH checking
for video devices. The removed code is a left-over from the
olden days where there was concern over how video devices
worked in Linux. We are never going to go that way again,
so kill this.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
----
arch/powerpc/platforms/pseries/eeh.c | 17 -----------------
1 file changed, 17 deletions(-)
Signed-off-by: Paul Mackerras <paulus@samba.org>
Twiddle the copyright notices. Per current guidelines, the use
of the (C) or (c) in source code is deprecated.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
----
arch/powerpc/platforms/pseries/eeh.c | 6 +++++-
arch/powerpc/platforms/pseries/eeh_cache.c | 3 ++-
arch/powerpc/platforms/pseries/eeh_driver.c | 6 +++---
3 files changed, 10 insertions(+), 5 deletions(-)
Signed-off-by: Paul Mackerras <paulus@samba.org>
Track and report the number of times we read an all-1s value (0xff,
0xffff or 0xffffffff) from each device which is valid data, not
indicating EEH isolation.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
----
arch/powerpc/platforms/pseries/eeh.c | 5 +++++
arch/powerpc/platforms/pseries/eeh_sysfs.c | 3 +++
include/asm-powerpc/pci-bridge.h | 1 +
3 files changed, 9 insertions(+)
Signed-off-by: Paul Mackerras <paulus@samba.org>
When an EEH event is detected, and after the device driver
has been notified, but before the device is reset, enable
MMIO to the adapter, and grab the contents of the PCI status
and command registers, the PCI-X status and command, and the
PCI-E capability 10 and AER registers. Pass these up to the
RTAS error log, and also printk them.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
If an EEH event is observed, capture PCI config space info about
the device, wrap it up and pass it to the event logger. This
pach just slots in the basic logging function. A later patch
will provide for more through data gathering.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Adds the pSeries platform implementation for a new PCI API
which can be used to issue various types of PCI-E reset,
including PCI-E warm reset and PCI-E hot reset. This is needed
for an ipr PCI-E adapter which does not properly implement BIST.
Running BIST on this adapter results in PCI-E errors. The only
reliable reset mechanism that exists on this hardware is PCI
Fundamental reset (warm reset).
Acked-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Rework how multi-function PCI devices are identified and traversed.
This fixes a bug with multi-function recovery on Power4 that was
introduced by a recent Power4 EEH patch.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
After requesting a state change, verify that the state change
actually ocurred, and the system ends up in the expected state.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
The EEH event notification system passes around data that is
not needed or at least, not used properly. Stop passing this
data; get it in a more reliable fashion.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Modify routine that returns PCI slot status to wait for slot status
to become available. This is needed, as slots that are in some remote
card cage may go offline for extended periods of time. New users for
this routine in following patches.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Some firmware versions will return a slot reset state of "1"
when a slot is EEH frozen. Recognize this as a state that can be
handled.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Provide support for the new ibm,get-config-addr-info2 RTAS token,
whenever it is actually available.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Some drivers will attempt to perform a lot of mmio even after
an EEH event was detected. This is especially the case for fast cpu's
and PCI-E slots. Be a bit more lenient in allowing this.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Change the order in which pci error state is examined;
the "capabilites" is not valid if "reset state" is 5.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Many struct file_operations in the kernel can be "const". Marking them const
moves these to the .rodata section, which avoids false sharing with potential
dirty data. In addition it'll catch accidental writes at compile time to
these shared resources.
[akpm@osdl.org: sparc64 fix]
Signed-off-by: Arjan van de Ven <arjan@linux.intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
It appears that EEH is improperly enabled for some Power4 systems.
On these systems, the ibm,set-eeh-option returns a value of success
even when EEH is not supported on the given node. Thus, an explicit
check for support is required.
During boot, on power4, without this patch, one sees messages
similar to:
EEH: event on unsupported device, rc=0 dn=/pci@400000000110/IBM,sp@1
EEH: event on unsupported device, rc=0 dn=/pci@400000000110/pci@2
EEH: event on unsupported device, rc=0 dn=/pci@400000000110/pci@2,2
etc.
The patch makes these go away.
Without this patch, EEH recovery does seem to work correctly for
at least some devices (I tested ethernet e1000), but fails to
recover others (the Emulex LightPulse LPFC, most notably).
Off the top of my head, I don't remember why some devices are
affected, but not others.
The PAPR indicates that the correct way to test for EEH is as
done in this patch; its not clear to me if this was in the PAPR
all along, or recently added; if it was there all along, its not
clear to me why this hadn't been fixed long ago. I suspect only
certain firmware levels are affected.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
If one attempts to create a device driver recovery sequence that
does not depend on a hard reset of the device, but simply just
attempts to resume processing, then one discovers that the
recovery sequence implemented on powerpc is not quite right.
This patch fixes this up.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Bug fix: when marking a slot as frozen, we forgot to mark
pci device itself as frozen. (we did manage to mark the
pci children, but forget the parent itself.)
This is needed so that some device drivers can check the
pci status in critical sections (e.g. in spin loops with
interrupts disabled).
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
On detection of an EEH error, some Power4 systems seem to occasionally
want to be reset twice before they report themselves as fully recovered.
This patch re-arranges the code to attempt additional resets if the first
one doesn't take.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Add wrapper around the rtas call to enable MMIO or DMA on a frozen pci
slot.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Clean up subroutine documentation; mostly formatting changes, with
some new content.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Now that get_property() returns a void *, there's no need to cast its
return value. Also, treat the return value as const, so we can
constify get_property later.
pseries platform changes.
Built for pseries_defconfig
Signed-off-by: Jeremy Kerr <jk@ozlabs.org>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Repeated calls to eeh_remove_device() can result in multiple
(and thus unbalanced) calls to pci_dev_put(). Make sure the
pci_device_put() is called only once (since there was only
one call to the matching pci_device_get()).
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
This patch removes unnecessary exports, marks functions as static when
possible, and simplifies some list-related code.
Signed-off-by: Nathan Fontenot <nfont@austin.ibm.com>
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
This removes statically assigned platform numbers and reworks the
powerpc platform probe code to use a better mechanism. With this,
board support files can simply declare a new machine type with a
macro, and implement a probe() function that uses the flattened
device-tree to detect if they apply for a given machine.
We now have a machine_is() macro that replaces the comparisons of
_machine with the various PLATFORM_* constants. This commit also
changes various drivers to use the new macro instead of looking at
_machine.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Some hotplug driver functions were migrated to the kernel for use by EEH
in commit 2bf6a8fa21.
Previously, the PCI Hotplug module had been changed to use the new
OFDT-based PCI probe when appropriate:
5fa80fcdca
When rpaphp_pci_config_slot() was moved from the rpaphp driver to the
new kernel function pcibios_add_pci_devices(), the OFDT-based probe
stuff was dropped. This patch restores it.
Signed-off-by: John Rose <johnrose@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
Remove warning in eeh code about mixed variables and code.
Signed-off-by: Olof Johansson <olof@lixom.net>
Acked-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
This fixes a crash on null-pointer deref during dlpar slot addition.
Signed-off-by: Linas Vepstas <linas@austin.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
(cherry picked from 1c87c0f84943fbbc91826967ff4fea1b059a526f commit)
Anton Blanchard