In order to be compatible with aRFS rules, this patch adds
spin_lock for flow director rule adding, deleting, querying,
and packages the rule configuration.
Signed-off-by: Jian Shen <shenjian15@huawei.com>
Signed-off-by: Huazhong Tan <tanhuazhong@huawei.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Allocate CPU rmap and add entry for each irq. CPU rmap is
used in aRFS to get the queue number of the rx completion
interrupts.
In additional, remove the calling of
irq_set_affinity_notifier() in hns3_nic_init_irq(), because
we have registered notifier in irq_cpu_rmap_add() for each
vector, otherwise it may cause use-after-free issue.
Signed-off-by: Jian Shen <shenjian15@huawei.com>
Signed-off-by: Huazhong Tan <tanhuazhong@huawei.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Oleksij Rempel says:
====================
MIPS: ath79: add ag71xx support
2019.05.24 v6:
- ag71xx: remove double union
- ag71xx: reverse Christmas tree for all functions
- ag71xx: add Reviewed-by: Andrew Lunn <andrew@lunn.ch>
2019.05.20 v5:
- ag71xx: remove MII_CMD_WRITE, the name is confusing. It is
actually disables MII_CMD_READ.
- ag71xx: rework ag71xx_mdio_mii_read/write
- ag71xx: set proper mask for the addr in ag71xx_mdio_mii_read/write
- Kconfig: remove MDIO_BITBANG
- ag71xx: ./scripts/checkpatch.pl it.
2019.05.19 v4:
- DT: define eth and mdio clocks
- ag71xx: remove module parameters
- ag71xx: return proper error value on mdio_read/write
- ag71xx: use proper mdio clock registration
- ag71xx: add ag71xx_dma_wait_stop() for ag71xx_dma_reset()
- ag71xx: remove ag71xx_speed_str()
- ag71xx: use phydev->link/sped/duplex instead of ag-> variants
- ag71xx: use WARN() instead of BUG()
- ag71xx: drop big part of ag71xx_phy_link_adjust()
- ag71xx: drop most of ag71xx_do_ioctl()
- ag71xx: register eth clock
- ag71xx: remove AG71XX_ETH0_NO_MDIO quirk.
2019.04.22 v3:
- ag71xx: use phy_modes() instead of ag71xx_get_phy_if_mode_name()
- ag71xx: remove .ndo_poll_controller support
- ag71xx: unregister_netdev before disconnecting phy.
2019.04.18 v2:
- ag71xx: add list of openwrt authors
- ag71xx: remove redundant PHY_POLL assignment
- ag71xx: use phy_attached_info instead of netif_info
- ag71xx: remove redundant netif_carrier_off() on .stop.
- DT: use "ethernet" instead of "eth"
This patch series provide ethernet support for many Atheros/QCA
MIPS based SoCs.
I reworked ag71xx driver which was previously maintained within OpenWRT
repository. So far, following changes was made to make upstreaming
easier:
- everything what can be some how used in user space was removed. Most
of it was debug functionality.
- most of deficetree bindings was removed. Not every thing made sense
and most of it is SoC specific, so it is possible to detect it by
compatible.
- mac and mdio parts are merged in to one driver. It makes easier to
maintaine SoC specific quirks.
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
Add support for Atheros/QCA AR7XXX/AR9XXX/QCA95XX built-in ethernet mac support
Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: Oleksij Rempel <o.rempel@pengutronix.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
Add ethernet nodes supported by ag71xx driver.
Signed-off-by: Oleksij Rempel <o.rempel@pengutronix.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
Add binding documentation for Atheros/QCA networking IP core used
in many routers.
Signed-off-by: Oleksij Rempel <o.rempel@pengutronix.de>
Reviewed-by: Rob Herring <robh@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
Michal Kalderon says:
====================
qed*: Improve performance on 100G link for offload protocols
This patch series modifies the current implementation of PF selection.
The refactoring of the llh code enables setting additional filters
(mac / protocol) per PF, and improves performance for offload protocols
(RoCE, iWARP, iSCSI, fcoe) on 100G link (was capped at 90G per single
PF).
Improved performance on 100G link is achieved by configuring engine
affinty to each PF.
The engine affinity is read from the Management FW and hw is configured accordingly.
A new hw resource called PPFID is exposed and an API is introduced to utilize
it. This additional resource enables setting the affinity of a PF and providing
more classification rules per PF.
qedr,qedi,qedf are also modified as part of the series. Without the
changes functionality is broken.
v1 --> v2
---------
- Remove iWARP module parameter. Instead use devlink param infrastructure
for setting the iwarp_cmt mode. Additional patch added to the series for
adding the devlink support.
- Fix kbuild test robot warning on qed_llh_filter initialization.
- Remove comments inside function calls
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
MSI-X vector index is determined using qed device information and
affinity to use.
Signed-off-by: Chad Dupuis <cdupuis@marvell.com>
Signed-off-by: Saurav Kashyap <skashyap@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
MSI-X vector index is determined using qed device information and
affinity to use.
Signed-off-by: Manish Rangankar <mrangankar@marvell.com>
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Always request for number of irqs equals to number of queues.
This reverts commit 1a291bce5e.
Signed-off-by: Manish Rangankar <mrangankar@marvell.com>
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Add iWARP engine affinity setting for supporting iWARP over 100g.
iWARP cannot be distinguished by the LLH from L2, hence the
engine division will affect L2 as well. For this reason we add
a parameter to devlink to determine the engine division.
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The table currently contains a single parameter for
configuring whether iWARP should be enabled on a 100g
device. Enabling iWARP on a 100g device impacts L2
performance and is therefore not enabled by default.
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
In 100g mode the doorbell bar is united for both engines. Set
the correct offset in the hwfn so that the doorbell returned
for RoCE is in the affined hwfn.
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Denis Bolotin <denis.bolotin@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Use the msix vectors of the affined hwfn and not the
leading one.
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
To enable 100g support for offload protocols each PF gets
a dedicated engine to work on from the MFW.
This patch modifies the code to use the affined hwfn instead
of the leading one.
The offload protocols require the ll2 to be opened on both
engines, and not just the affined hwfn.
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
When initializing status blocks use the affined hwfn
instead of the leading one for RDMA / Storage
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
This patch refactors the current llh implementation. It exposes a hw
resource called ppfid (port-pfid) and implements an API for configuring
the resource. Default configuration which was used until now limited
the number of filters per PF and did not support engine affinity per
protocol. The new API enables allocating more filter rules per PF and
enables affinitizing protocol packets to a certain engine which
enables full 100g protocol offload support.
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
This patch modifies the dmae API to enable performing a dmae operation
to another PF. This enables sharing between the llh entries between PFs
and thus increasing the amount of filters per PF under certain
configurations.
The llh entries require using the dmae since the memory is widebus,
which requires atomicity in access.
Signed-off-by: Ariel Elior <ariel.elior@marvell.com>
Signed-off-by: Michal Kalderon <michal.kalderon@marvell.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEK2m5VNv+CHkogTfJ8vlZVpUNgaMFAlzqGSoACgkQ8vlZVpUN
gaNhqQgAiUHwKalYrZ82NwBQGnHcKcWv3JEE9vt8Bsu4fPUzirrEqYiSudvj6nHv
8uYFKHmGx7+GEWxLfwlVZzRjLlgZqa0kpyfNFEL01KFdbFsKQN4gTYvvky+OVftr
nRZ7tp66Y5hErwn/Y0wWn9WHFOykhxGi+kv5m5CFZ7MNec/b+1H2U1hXkhSt6oug
IO2wLZYLFSPXlrqfJLV7HYJ/OX1mO7g1viCNGvpRmrvLmjmO09q0/6DF3QNAvGmj
sXXu0eV+N/Ir0so0RbeN60ZeDXaoyeOZbXFlH9zfJEgkoFv+adZjT65bQEvSUWQ2
J/v4rLXd8gmCiVwOuEbCoLKebT/nbg==
=Wf1M
-----END PGP SIGNATURE-----
Merge tag 'random_for_linus_stable' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/random
Pull /dev/random fix from Ted Ts'o:
"Fix a soft lockup regression when reading from /dev/random in early
boot"
* tag 'random_for_linus_stable' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/random:
random: fix soft lockup when trying to read from an uninitialized blocking pool
Fixes: eb9d1bf079bb: "random: only read from /dev/random after its pool has received 128 bits"
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Starting with GCC 9, -Warray-bounds detects cases when memset is called
starting on a member of a struct but the size to be cleared ends up
writing over further members.
Such a call happens in the trace code to clear, at once, all members
after and including `seq` on struct trace_iterator:
In function 'memset',
inlined from 'ftrace_dump' at kernel/trace/trace.c:8914:3:
./include/linux/string.h:344:9: warning: '__builtin_memset' offset
[8505, 8560] from the object at 'iter' is out of the bounds of
referenced subobject 'seq' with type 'struct trace_seq' at offset
4368 [-Warray-bounds]
344 | return __builtin_memset(p, c, size);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to avoid GCC complaining about it, we compute the address
ourselves by adding the offsetof distance instead of referring
directly to the member.
Since there are two places doing this clear (trace.c and trace_kdb.c),
take the chance to move the workaround into a single place in
the internal header.
Link: http://lkml.kernel.org/r/20190523124535.GA12931@gmail.com
Signed-off-by: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
[ Removed unnecessary parenthesis around "iter" ]
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
Maxime Chevallier says:
====================
net: mvpp2: Classifier updates, RSS
Here is a set of updates for the PPv2 classifier, the main feature being
the support for steering to RSS contexts, to leverage all the available
RSS tables in the controller.
The first two patches are non-critical fixes for the classifier, the
first one prevents us from allocating too much room to store the
classification rules, the second one configuring the C2 engine as
suggested by the PPv2 functionnal specs.
Patches 3 to 5 introduce support for RSS contexts in mvpp2, allowing us
to steer traffic to dedicated RSS tables.
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
When steering to an RXQ, we can perform an extra RSS step to assign a
queue from an RSS table.
This is done by setting the RSS_EN attribute in the C2 engine. In that
case, the RXQ that is assigned is the global RSS context id, that is
then translated to an RSS table using the RXQ2RSS table.
An example using ethtool to steer to RXQ 2 and 3 would be :
ethtool -X eth0 weight 0 0 1 1 context new
(This would print the allocated context id, let's say it's 1)
ethtool -N eth0 flow-type udp4 dst-port 1234 context 1 loc 0
The hash parameters are the ones that are globally configured for RSS :
ethtool -N eth0 rx-flow-hash udp4 sdfn
When an RSS context is removed while there are active classification
rules using this context, these rules are removed.
Signed-off-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
ethtool_rx_flow_rule_create takes into parameter the ethtool flow spec,
which doesn't contain the rss context id. We therefore need to extract
it ourself before parsing the ethtool rule.
The FLOW_RSS flag is only set in info->fs.flow_type, and not
info->flow_type.
Signed-off-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The PPv2 controller has 8 RSS tables that are shared across all ports on
a given PPv2 instance. The previous implementation allocated one table
per port, leaving others unused.
By using RSS contexts, we can make use of multiple RSS tables per
port, one being the default table (always id 0), the other ones being
used as destinations for flow steering, in the same way as rx rings.
This commit introduces RSS contexts management in the PPv2 driver. We
always reserve one table per port, allocated when the port is probed.
The global table list is stored in the struct mvpp2, as it's a global
resource. Each port then maintains a list of indices in that global
table, that way each port can have it's own numbering scheme starting
from 0.
One limitation that seems unavoidable is that the hashing parameters are
shared across all RSS contexts for a given port. Hashing parameters for
ctx 0 will be applied to all contexts.
Signed-off-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The C2 TCAM has internal FIFOs that are only useful for the built-in
self-tests. Disable these FIFOS at init, as recommended in the
functionnal specs.
Suggested-by: Alan Winkowski <walan@marvell.com>
Signed-off-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
As of today, the classification offload implementation only supports 4
different rules to be offloaded. This number has been hardcoded in the
rule insertion function, and the wrong define is being used elsewhere.
Use the correct #define everywhere to make sure we always check for the
correct number of rules.
Signed-off-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
ethtool ops get_rxfh_context and set_rxfh_context are used to create,
remove and access parameters associated to RSS contexts, in a similar
fashion to get_rxfh and set_rxfh.
Add a small descritopn of these callbacks in the struct ethtool_ops doc.
Signed-off-by: Maxime Chevallier <maxime.chevallier@bootlin.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEK2m5VNv+CHkogTfJ8vlZVpUNgaMFAlzppnIACgkQ8vlZVpUN
gaOWcwf/YmIeCi7HHuOJG5STYhMZjbAoK7eCNSjmP0HBIpyZSBaSZg1/ZEmtTVA6
SyGWxYD2xymphkEcRQ20pF8h2CYurHsjYl9RH+Im2iaCzdeFKvgfYxSSsqsaZixM
ejQK22W6mVULd1RqFGNPeo+5v7Fxn6fK0zw2k5JrLjFnIRq/XIA7qMdjblPOcfi+
QT/K9a2DZ5vHBGDKjEiVA+a0HX6bxdGTiiT4LW+uiHUJUESBWNQJqOHJqno9VdFh
J97/3XJHMGPAbjD4AiINAL0x8IZ2FXx1H+QgVDnrxy8lVrYaMVvWMEokMQ7HvkFr
SmYddgBPUHO+kk4u34nznZNuesvOqQ==
=dFk1
-----END PGP SIGNATURE-----
Merge tag 'ext4_for_linus_stable' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
Pull ext4 fixes from Ted Ts'o:
"Bug fixes (including a regression fix) for ext4"
* tag 'ext4_for_linus_stable' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4:
ext4: fix dcache lookup of !casefolded directories
ext4: do not delete unlinked inode from orphan list on failed truncate
ext4: wait for outstanding dio during truncate in nojournal mode
ext4: don't perform block validity checks on the journal inode
Biao Huang says:
====================
fix some bugs in stmmac
changes in v4:
since MTL_OPERATION_MODE write back issue has be fixed in the latest driver,
remove original patch#3
changes in v3:
add a Fixes:tag for each patch
changes in v2:
1. update rx_tail_addr as Jose's comment
2. changes clk_csr condition as Alex's proposition
3. remove init lines in dwmac-mediatek, get clk_csr from dts instead.
v1:
This series fix some bugs in stmmac driver
3 patches are for common stmmac or dwmac4:
1. update rx tail pointer to fix rx dma hang issue.
2. change condition for mdc clock to fix csr_clk can't be zero issue.
3. write the modified value back to MTL_OPERATION_MODE.
1 patch is for dwmac-mediatek:
modify csr_clk value to fix mdio read/write fail issue for dwmac-mediatek
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
1. the frequency of csr clock is 66.5MHz, so the csr_clk value should
be 0 other than 5.
2. the csr_clk can be got from device tree, so remove initialization here.
Fixes: 9992f37e34 ("stmmac: dwmac-mediatek: add support for mt2712")
Signed-off-by: Biao Huang <biao.huang@mediatek.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The specific clk_csr value can be zero, and
stmmac_clk is necessary for MDC clock which can be set dynamically.
So, change the condition from plat->clk_csr to plat->stmmac_clk to
fix clk_csr can't be zero issue.
Fixes: cd7201f477 ("stmmac: MDC clock dynamically based on the csr clock input")
Signed-off-by: Biao Huang <biao.huang@mediatek.com>
Acked-by: Alexandre TORGUE <alexandre.torgue@st.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Currently we will not update the receive descriptor tail pointer in
stmmac_rx_refill. Rx dma will think no available descriptors and stop
once received packets exceed DMA_RX_SIZE, so that the rx only test will fail.
Update the receive tail pointer in stmmac_rx_refill to add more descriptors
to the rx channel, so packets can be received continually
Fixes: 54139cf3bb ("net: stmmac: adding multiple buffers for rx")
Signed-off-by: Biao Huang <biao.huang@mediatek.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
In function ip_ra_control(), the pointer new_ra is allocated a memory
space via kmalloc(). And it is used in the following codes. However,
when there is a memory allocation error, kmalloc() fails. Thus null
pointer dereference may happen. And it will cause the kernel to crash.
Therefore, we should check the return value and handle the error.
Signed-off-by: Gen Zhang <blackgod016574@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
In function ip6_ra_control(), the pointer new_ra is allocated a memory
space via kmalloc(). And it is used in the following codes. However,
when there is a memory allocation error, kmalloc() fails. Thus null
pointer dereference may happen. And it will cause the kernel to crash.
Therefore, we should check the return value and handle the error.
Signed-off-by: Gen Zhang <blackgod016574@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
One of the more common cases of allocation size calculations is finding
the size of a structure that has a zero-sized array at the end, along
with memory for some number of elements for that array. For example:
struct foo {
int stuff;
struct boo entry[];
};
instance = kzalloc(sizeof(struct foo) + count * sizeof(struct boo), GFP_KERNEL);
Instead of leaving these open-coded and prone to type mistakes, we can
now use the new struct_size() helper:
instance = kzalloc(struct_size(instance, entry, count), GFP_KERNEL);
This code was detected with the help of Coccinelle.
Signed-off-by: Gustavo A. R. Silva <gustavo@embeddedor.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
code and I forgot to incorporate them.
I also added a small clean up patch that was sent to me a while ago
and I just noticed it.
-----BEGIN PGP SIGNATURE-----
iIoEABYIADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCXOixqRQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qlPsAQCmNzno+SJMXLLIojZ80pqs9PqLqIrW
iBopFNihRBAkxgEAle8Pvr53S4R/Hjn6j++kxBm/uWaEfICOzyqcSyqxlQA=
=Sin+
-----END PGP SIGNATURE-----
Merge tag 'trace-v5.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace
Pull tracing fixes from Steven Rostedt:
"Tom Zanussi sent me some small fixes and cleanups to the histogram
code and I forgot to incorporate them.
I also added a small clean up patch that was sent to me a while ago
and I just noticed it"
* tag 'trace-v5.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace:
kernel/trace/trace.h: Remove duplicate header of trace_seq.h
tracing: Add a check_val() check before updating cond_snapshot() track_val
tracing: Check keys for variable references in expressions too
tracing: Prevent hist_field_var_ref() from accessing NULL tracing_map_elts
Found by visual inspection, this wasn't caught by my xfstest, since it's
effect is ignoring positive dentries in the cache the fallback just goes
to the disk. it was introduced in the last iteration of the
case-insensitive patch.
d_compare should return 0 when the entries match, so make sure we are
correctly comparing the entire string if the encoding feature is set and
we are on a case-INsensitive directory.
Fixes: b886ee3e77 ("ext4: Support case-insensitive file name lookups")
Signed-off-by: Gabriel Krisman Bertazi <krisman@collabora.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
This commit adds ibumad to .gitignore which is
currently ommited from the ignore file.
Signed-off-by: Matteo Croce <mcroce@redhat.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Jiong Wang says:
====================
v9:
- Split patch 5 in v8.
make bpf uapi header file sync a separate patch. (Alexei)
v8:
- For stack slot read, mark them as REG_LIVE_READ64. (Alexei)
- Change DEF_NOT_SUBREG from -1 to 0. (Alexei)
- Rebased on top of latest bpf-next.
v7:
- Drop the first patch in v6, the one adding 32-bit return value and
argument type. (Alexei)
- Rename bpf_jit_hardware_zext to bpf_jit_needs_zext. (Alexei)
- Use mov32 with imm == 1 to indicate it is zext. (Alexei)
- JIT back-ends peephole next insn to optimize out unnecessary zext
inserted by verifier. (Alexei)
- Testing:
+ patch set tested (bpf selftest) on x64 host with llvm 9.0
no regression observed no both JIT and interpreter modes.
+ patch set tested (bpf selftest) on x32 host.
By Yanqing Wang, thanks!
no regression observed on both JIT and interpreter modes.
+ patch set tested (bpf selftest) on RV64 host with llvm 9.0,
By Björn Töpel, thanks!
no regression observed before and after this set with JIT_ALWAYS_ON.
test_progs_32 also enabled as LLVM 9.0 is used by Björn.
+ cross compiled the other affected targets, arm, PowerPC, SPARC, S390.
v6:
- Fixed s390 kbuild test robot error. (kbuild)
- Make comment style in backends patches more consistent.
v5:
- Adjusted several test_verifier helpers to make them works on hosts
w and w/o hardware zext. (Naveen)
- Make sure zext flag not set when verifier by-passed, for example,
libtest_bpf.ko. (Naveen)
- Conservatively mark bpf main return value as 64-bit. (Alexei)
- Make sure read flag is either READ64 or READ32, not the mix of both.
(Alexei)
- Merged patch 1 and 2 in v4. (Alexei)
- Fixed kbuild test robot warning on NFP. (kbuild)
- Proposed new BPF_ZEXT insn to have optimal code-gen for various JIT
back-ends.
- Conservately set zext flags for patched-insn.
- Fixed return value zext for helper function calls.
- Also adjusted test_verifier scalability unit test to avoid triggerring
too many insn patch which will hang computer.
- re-tested on x86 host with llvm 9.0, no regression on test_verifier,
test_progs, test_progs_32.
- re-tested offload target (nfp), no regression on local testsuite.
v4:
- added the two missing fixes which addresses two Jakub's reviewes in v3.
- rebase on top of bpf-next.
v3:
- remove redundant check in "propagate_liveness_reg". (Jakub)
- add extra check in "mark_reg_read" to prune more search. (Jakub)
- re-implemented "prog_flags" passing mechanism, removed use of
global switch inside libbpf.
- enabled high 32-bit randomization beyond "test_verifier" and
"test_progs". Now it should have been enabled for all possible
tests. Re-run all tests, haven't noticed regression.
- remove RFC tag.
v2:
- rebased on top of bpf-next master.
- added comments for what is sub-register def index. (Edward, Alexei)
- removed patch 1 which turns bit mask from enum to macro. (Alexei)
- removed sysctl/bpf_jit_32bit_opt. (Alexei)
- merged sub-register def insn index into reg state. (Alexei)
- change test methodology (Alexei):
+ instead of simple unit tests on x86_64 for which this optimization
doesn't enabled due to there is hardware support, poison high
32-bit for whose def identified as safe to do so. this could let
the correctness of this patch set checked when daily bpf selftest
ran which delivers very stressful test on host machine like x86_64.
+ hi32 poisoning is gated by a new BPF_F_TEST_RND_HI32 prog flags.
+ BPF_F_TEST_RND_HI32 is enabled for all tests of "test_progs" and
"test_verifier", the latter needs minor tweak on two unit tests,
please see the patch for the change.
+ introduced a new global variable "libbpf_test_mode" into libbpf.
once it is set to true, it will set BPF_F_TEST_RND_HI32 for all the
later PROG_LOAD syscall, the goal is to easy the enable of hi32
poison on exsiting testsuite.
we could also introduce new APIs, for example "bpf_prog_test_load",
then use -Dbpf_prog_load=bpf_prog_test_load to migrate tests under
test_progs, but there are several load APIs, and such new API need
some change on struture like "struct bpf_prog_load_attr".
+ removed old unit tests. it is based on insn scan and requires quite
a few test_verifier generic code change. given hi32 randomization
could offer good test coverage, the unit tests doesn't add much
extra test value.
- enhanced register width check ("is_reg64") when record sub-register
write, now, it returns more accurate width.
- Re-run all tests under "test_progs" and "test_verifier" on x86_64, no
regression. Fixed a couple of bugs exposed:
1. ctx field size transformation was not taken into account.
2. insn patch could cause lost of original aux data which is
important for ctx field conversion.
3. return value for propagate_liveness was wrong and caused
regression on processed insn number.
4. helper call arg wasn't handled properly that path prune may cause
64-bit read info in pruned path lost.
- Re-run Cilium bpf prog for processed-insn-number benchmarking, no
regression.
v1:
- Fixed the missing handling on callee-saved for bpf-to-bpf call,
sub-register defs therefore moved to frame state. (Jakub Kicinski)
- Removed redundant "cross_reg". (Jakub Kicinski)
- Various coding styles & grammar fixes. (Jakub Kicinski, Quentin Monnet)
eBPF ISA specification requires high 32-bit cleared when low 32-bit
sub-register is written. This applies to destination register of ALU32 etc.
JIT back-ends must guarantee this semantic when doing code-gen. x86_64 and
AArch64 ISA has the same semantics, so the corresponding JIT back-end
doesn't need to do extra work.
However, 32-bit arches (arm, x86, nfp etc.) and some other 64-bit arches
(PowerPC, SPARC etc) need to do explicit zero extension to meet this
requirement, otherwise code like the following will fail.
u64_value = (u64) u32_value
... other uses of u64_value
This is because compiler could exploit the semantic described above and
save those zero extensions for extending u32_value to u64_value, these JIT
back-ends are expected to guarantee this through inserting extra zero
extensions which however could be a significant increase on the code size.
Some benchmarks show there could be ~40% sub-register writes out of total
insns, meaning at least ~40% extra code-gen.
One observation is these extra zero extensions are not always necessary.
Take above code snippet for example, it is possible u32_value will never be
casted into a u64, the value of high 32-bit of u32_value then could be
ignored and extra zero extension could be eliminated.
This patch implements this idea, insns defining sub-registers will be
marked when the high 32-bit of the defined sub-register matters. For
those unmarked insns, it is safe to eliminate high 32-bit clearnace for
them.
Algo
====
We could use insn scan based static analysis to tell whether one
sub-register def doesn't need zero extension. However, using such static
analysis, we must do conservative assumption at branching point where
multiple uses could be introduced. So, for any sub-register def that is
active at branching point, we need to mark it as needing zero extension.
This could introducing quite a few false alarms, for example ~25% on
Cilium bpf_lxc.
It will be far better to use dynamic data-flow tracing which verifier
fortunately already has and could be easily extend to serve the purpose of
this patch set.
- Split read flags into READ32 and READ64.
- Record index of insn that does sub-register write. Keep the index inside
reg state and update it during verifier insn walking.
- A full register read on a sub-register marks its definition insn as
needing zero extension on dst register.
A new sub-register write overrides the old one.
- When propagating read64 during path pruning, also mark any insn defining
a sub-register that is read in the pruned path as full-register.
Benchmark
=========
- I estimate the JITed image could be 10% ~ 30% smaller on these affected
arches (nfp, arm, x32, risv, ppc, sparc, s390), depending on the prog.
- For Cilium bpf_lxc, there is ~11500 insns in the compiled binary (use
latest LLVM snapshot, and with -mcpu=v3 -mattr=+alu32 enabled), 4460 of
them has sub-register writes (~40%). Calculated by:
cat dump | grep -P "\tw" | wc -l (ALU32)
cat dump | grep -P "r.*=.*u32" | wc -l (READ_W)
cat dump | grep -P "r.*=.*u16" | wc -l (READ_H)
cat dump | grep -P "r.*=.*u8" | wc -l (READ_B)
After this patch set enabled, > 25% of those 4460 could be identified as
doesn't needing zero extension on the destination, and the percentage
could go further up to more than 50% with some follow up optimizations
based on the infrastructure offered by this set. This leads to
significant save on JITed image.
====================
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
This patch eliminate zero extension code-gen for instructions including
both alu and load/store. The only exception is for ctx load, because
offload target doesn't go through host ctx convert logic so we do
customized load and ignores zext flag set by verifier.
Cc: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Cc: Björn Töpel <bjorn.topel@gmail.com>
Acked-by: Björn Töpel <bjorn.topel@gmail.com>
Tested-by: Björn Töpel <bjorn.topel@gmail.com>
Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
The previous libbpf patch allows user to specify "prog_flags" to bpf
program load APIs. To enable high 32-bit randomization for a test, we need
to set BPF_F_TEST_RND_HI32 in "prog_flags".
To enable such randomization for all tests, we need to make sure all places
are passing BPF_F_TEST_RND_HI32. Changing them one by one is not
convenient, also, it would be better if a test could be switched to
"normal" running mode without code change.
Given the program load APIs used across bpf selftests are mostly:
bpf_prog_load: load from file
bpf_load_program: load from raw insns
A test_stub.c is implemented for bpf seltests, it offers two functions for
testing purpose:
bpf_prog_test_load
bpf_test_load_program
The are the same as "bpf_prog_load" and "bpf_load_program", except they
also set BPF_F_TEST_RND_HI32. Given *_xattr functions are the APIs to
customize any "prog_flags", it makes little sense to put these two
functions into libbpf.
Then, the following CFLAGS are passed to compilations for host programs:
-Dbpf_prog_load=bpf_prog_test_load
-Dbpf_load_program=bpf_test_load_program
They migrate the used load APIs to the test version, hence enable high
32-bit randomization for these tests without changing source code.
Besides all these, there are several testcases are using
"bpf_prog_load_attr" directly, their call sites are updated to pass
BPF_F_TEST_RND_HI32.
Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
- bpf_fill_ld_abs_vlan_push_pop:
Prevent zext happens inside PUSH_CNT loop. This could happen because
of BPF_LD_ABS (32-bit def) + BPF_JMP (64-bit use), or BPF_LD_ABS +
EXIT (64-bit use of R0). So, change BPF_JMP to BPF_JMP32 and redefine
R0 at exit path to cut off the data-flow from inside the loop.
- bpf_fill_jump_around_ld_abs:
Jump range is limited to 16 bit. every ld_abs is replaced by 6 insns,
but on arches like arm, ppc etc, there will be one BPF_ZEXT inserted
to extend the error value of the inlined ld_abs sequence which then
contains 7 insns. so, set the dividend to 7 so the testcase could
work on all arches.
- bpf_fill_scale1/bpf_fill_scale2:
Both contains ~1M BPF_ALU32_IMM which will trigger ~1M insn patcher
call because of hi32 randomization later when BPF_F_TEST_RND_HI32 is
set for bpf selftests. Insn patcher is not efficient that 1M call to
it will hang computer. So , change to BPF_ALU64_IMM to avoid hi32
randomization.
Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Jian Shen