2017-04-04 18:31:42 +08:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2016, Mellanox Technologies inc. All rights reserved.
|
|
|
|
*
|
|
|
|
* This software is available to you under a choice of one of two
|
|
|
|
* licenses. You may choose to be licensed under the terms of the GNU
|
|
|
|
* General Public License (GPL) Version 2, available from the file
|
|
|
|
* COPYING in the main directory of this source tree, or the
|
|
|
|
* OpenIB.org BSD license below:
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or
|
|
|
|
* without modification, are permitted provided that the following
|
|
|
|
* conditions are met:
|
|
|
|
*
|
|
|
|
* - Redistributions of source code must retain the above
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
* disclaimer.
|
|
|
|
*
|
|
|
|
* - Redistributions in binary form must reproduce the above
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
* disclaimer in the documentation and/or other materials
|
|
|
|
* provided with the distribution.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
|
|
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
|
|
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
|
|
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
|
|
|
|
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
|
|
|
|
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
|
|
|
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
|
|
* SOFTWARE.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/file.h>
|
|
|
|
#include <linux/anon_inodes.h>
|
2018-07-11 10:55:19 +08:00
|
|
|
#include <linux/sched/mm.h>
|
2017-04-04 18:31:42 +08:00
|
|
|
#include <rdma/ib_verbs.h>
|
|
|
|
#include <rdma/uverbs_types.h>
|
|
|
|
#include <linux/rcupdate.h>
|
2017-08-03 21:06:55 +08:00
|
|
|
#include <rdma/uverbs_ioctl.h>
|
IB/core: Add new ioctl interface
In this ioctl interface, processing the command starts from
properties of the command and fetching the appropriate user objects
before calling the handler.
Parsing and validation is done according to a specifier declared by
the driver's code. In the driver, all supported objects are declared.
These objects are separated to different object namepsaces. Dividing
objects to namespaces is done at initialization by using the higher
bits of the object ids. This initialization can mix objects declared
in different places to one parsing tree using in this ioctl interface.
For each object we list all supported methods. Similarly to objects,
methods are separated to method namespaces too. Namespacing is done
similarly to the objects case. This could be used in order to add
methods to an existing object.
Each method has a specific handler, which could be either a default
handler or a driver specific handler.
Along with the handler, a bunch of attributes are specified as well.
Similarly to objects and method, attributes are namespaced and hashed
by their ids at initialization too. All supported attributes are
subject to automatic fetching and validation. These attributes include
the command, response and the method's related objects' ids.
When these entities (objects, methods and attributes) are used, the
high bits of the entities ids are used in order to calculate the hash
bucket index. Then, these high bits are masked out in order to have a
zero based index. Since we use these high bits for both bucketing and
namespacing, we get a compact representation and O(1) array access.
This is mandatory for efficient dispatching.
Each attribute has a type (PTR_IN, PTR_OUT, IDR and FD) and a length.
Attributes could be validated through some attributes, like:
(*) Minimum size / Exact size
(*) Fops for FD
(*) Object type for IDR
If an IDR/fd attribute is specified, the kernel also states the object
type and the required access (NEW, WRITE, READ or DESTROY).
All uobject/fd management is done automatically by the infrastructure,
meaning - the infrastructure will fail concurrent commands that at
least one of them requires concurrent access (WRITE/DESTROY),
synchronize actions with device removals (dissociate context events)
and take care of reference counting (increase/decrease) for concurrent
actions invocation. The reference counts on the actual kernel objects
shall be handled by the handlers.
objects
+--------+
| |
| | methods +--------+
| | ns method method_spec +-----+ |len |
+--------+ +------+[d]+-------+ +----------------+[d]+------------+ |attr1+-> |type |
| object +> |method+-> | spec +-> + attr_buckets +-> |default_chain+--> +-----+ |idr_type|
+--------+ +------+ |handler| | | +------------+ |attr2| |access |
| | | | +-------+ +----------------+ |driver chain| +-----+ +--------+
| | | | +------------+
| | +------+
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
+--------+
[d] = Hash ids to groups using the high order bits
The right types table is also chosen by using the high bits from
the ids. Currently we have either default or driver specific groups.
Once validation and object fetching (or creation) completed, we call
the handler:
int (*handler)(struct ib_device *ib_dev, struct ib_uverbs_file *ufile,
struct uverbs_attr_bundle *ctx);
ctx bundles attributes of different namespaces. Each element there
is an array of attributes which corresponds to one namespaces of
attributes. For example, in the usually used case:
ctx core
+----------------------------+ +------------+
| core: +---> | valid |
+----------------------------+ | cmd_attr |
| driver: | +------------+
|----------------------------+--+ | valid |
| | cmd_attr |
| +------------+
| | valid |
| | obj_attr |
| +------------+
|
| drivers
| +------------+
+> | valid |
| cmd_attr |
+------------+
| valid |
| cmd_attr |
+------------+
| valid |
| obj_attr |
+------------+
Signed-off-by: Matan Barak <matanb@mellanox.com>
Reviewed-by: Yishai Hadas <yishaih@mellanox.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>
2017-08-03 21:06:57 +08:00
|
|
|
#include <rdma/rdma_user_ioctl.h>
|
2017-04-04 18:31:42 +08:00
|
|
|
#include "uverbs.h"
|
|
|
|
#include "core_priv.h"
|
|
|
|
#include "rdma_core.h"
|
|
|
|
|
|
|
|
void uverbs_uobject_get(struct ib_uobject *uobject)
|
|
|
|
{
|
|
|
|
kref_get(&uobject->ref);
|
|
|
|
}
|
|
|
|
|
2017-04-18 17:03:37 +08:00
|
|
|
static void uverbs_uobject_free(struct kref *ref)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
|
|
|
struct ib_uobject *uobj =
|
|
|
|
container_of(ref, struct ib_uobject, ref);
|
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
if (uobj->uapi_object->type_class->needs_kfree_rcu)
|
2017-04-04 18:31:42 +08:00
|
|
|
kfree_rcu(uobj, rcu);
|
|
|
|
else
|
|
|
|
kfree(uobj);
|
|
|
|
}
|
|
|
|
|
|
|
|
void uverbs_uobject_put(struct ib_uobject *uobject)
|
|
|
|
{
|
2017-04-18 17:03:37 +08:00
|
|
|
kref_put(&uobject->ref, uverbs_uobject_free);
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
static int uverbs_try_lock_object(struct ib_uobject *uobj,
|
|
|
|
enum rdma_lookup_mode mode)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
|
|
|
/*
|
2017-04-18 17:03:37 +08:00
|
|
|
* When a shared access is required, we use a positive counter. Each
|
|
|
|
* shared access request checks that the value != -1 and increment it.
|
|
|
|
* Exclusive access is required for operations like write or destroy.
|
|
|
|
* In exclusive access mode, we check that the counter is zero (nobody
|
|
|
|
* claimed this object) and we set it to -1. Releasing a shared access
|
|
|
|
* lock is done simply by decreasing the counter. As for exclusive
|
|
|
|
* access locks, since only a single one of them is is allowed
|
|
|
|
* concurrently, setting the counter to zero is enough for releasing
|
|
|
|
* this lock.
|
2017-04-04 18:31:42 +08:00
|
|
|
*/
|
2018-07-26 11:40:13 +08:00
|
|
|
switch (mode) {
|
|
|
|
case UVERBS_LOOKUP_READ:
|
atomics/treewide: Rename __atomic_add_unless() => atomic_fetch_add_unless()
While __atomic_add_unless() was originally intended as a building-block
for atomic_add_unless(), it's now used in a number of places around the
kernel. It's the only common atomic operation named __atomic*(), rather
than atomic_*(), and for consistency it would be better named
atomic_fetch_add_unless().
This lack of consistency is slightly confusing, and gets in the way of
scripting atomics. Given that, let's clean things up and promote it to
an official part of the atomics API, in the form of
atomic_fetch_add_unless().
This patch converts definitions and invocations over to the new name,
including the instrumented version, using the following script:
----
git grep -w __atomic_add_unless | while read line; do
sed -i '{s/\<__atomic_add_unless\>/atomic_fetch_add_unless/}' "${line%%:*}";
done
git grep -w __arch_atomic_add_unless | while read line; do
sed -i '{s/\<__arch_atomic_add_unless\>/arch_atomic_fetch_add_unless/}' "${line%%:*}";
done
----
Note that we do not have atomic{64,_long}_fetch_add_unless(), which will
be introduced by later patches.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Will Deacon <will.deacon@arm.com>
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Palmer Dabbelt <palmer@sifive.com>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/lkml/20180621121321.4761-2-mark.rutland@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:04 +08:00
|
|
|
return atomic_fetch_add_unless(&uobj->usecnt, 1, -1) == -1 ?
|
2017-04-04 18:31:42 +08:00
|
|
|
-EBUSY : 0;
|
2018-07-26 11:40:13 +08:00
|
|
|
case UVERBS_LOOKUP_WRITE:
|
2018-07-26 11:40:14 +08:00
|
|
|
/* lock is exclusive */
|
2018-07-26 11:40:13 +08:00
|
|
|
return atomic_cmpxchg(&uobj->usecnt, 0, -1) == 0 ? 0 : -EBUSY;
|
2018-07-26 11:40:14 +08:00
|
|
|
case UVERBS_LOOKUP_DESTROY:
|
|
|
|
return 0;
|
2018-07-26 11:40:13 +08:00
|
|
|
}
|
|
|
|
return 0;
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
static void assert_uverbs_usecnt(struct ib_uobject *uobj,
|
|
|
|
enum rdma_lookup_mode mode)
|
2018-07-26 11:40:12 +08:00
|
|
|
{
|
|
|
|
#ifdef CONFIG_LOCKDEP
|
2018-07-26 11:40:13 +08:00
|
|
|
switch (mode) {
|
|
|
|
case UVERBS_LOOKUP_READ:
|
2018-07-26 11:40:12 +08:00
|
|
|
WARN_ON(atomic_read(&uobj->usecnt) <= 0);
|
2018-07-26 11:40:13 +08:00
|
|
|
break;
|
|
|
|
case UVERBS_LOOKUP_WRITE:
|
|
|
|
WARN_ON(atomic_read(&uobj->usecnt) != -1);
|
|
|
|
break;
|
2018-07-26 11:40:14 +08:00
|
|
|
case UVERBS_LOOKUP_DESTROY:
|
|
|
|
break;
|
2018-07-26 11:40:13 +08:00
|
|
|
}
|
2018-07-26 11:40:12 +08:00
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2018-07-26 11:40:15 +08:00
|
|
|
* This must be called with the hw_destroy_rwsem locked for read or write,
|
|
|
|
* also the uobject itself must be locked for write.
|
2018-07-26 11:40:12 +08:00
|
|
|
*
|
|
|
|
* Upon return the HW object is guaranteed to be destroyed.
|
|
|
|
*
|
|
|
|
* For RDMA_REMOVE_ABORT, the hw_destroy_rwsem is not required to be held,
|
|
|
|
* however the type's allocat_commit function cannot have been called and the
|
|
|
|
* uobject cannot be on the uobjects_lists
|
|
|
|
*
|
|
|
|
* For RDMA_REMOVE_DESTROY the caller shold be holding a kref (eg via
|
|
|
|
* rdma_lookup_get_uobject) and the object is left in a state where the caller
|
|
|
|
* needs to call rdma_lookup_put_uobject.
|
|
|
|
*
|
|
|
|
* For all other destroy modes this function internally unlocks the uobject
|
|
|
|
* and consumes the kref on the uobj.
|
|
|
|
*/
|
|
|
|
static int uverbs_destroy_uobject(struct ib_uobject *uobj,
|
|
|
|
enum rdma_remove_reason reason)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_file *ufile = uobj->ufile;
|
|
|
|
unsigned long flags;
|
|
|
|
int ret;
|
|
|
|
|
2018-07-26 11:40:15 +08:00
|
|
|
lockdep_assert_held(&ufile->hw_destroy_rwsem);
|
2018-07-26 11:40:13 +08:00
|
|
|
assert_uverbs_usecnt(uobj, UVERBS_LOOKUP_WRITE);
|
2018-07-26 11:40:12 +08:00
|
|
|
|
|
|
|
if (uobj->object) {
|
2018-08-10 10:14:37 +08:00
|
|
|
ret = uobj->uapi_object->type_class->destroy_hw(uobj, reason);
|
2018-07-26 11:40:12 +08:00
|
|
|
if (ret) {
|
|
|
|
if (ib_is_destroy_retryable(ret, reason, uobj))
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
/* Nothing to be done, dangle the memory and move on */
|
|
|
|
WARN(true,
|
|
|
|
"ib_uverbs: failed to remove uobject id %d, driver err=%d",
|
|
|
|
uobj->id, ret);
|
|
|
|
}
|
|
|
|
|
|
|
|
uobj->object = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (reason == RDMA_REMOVE_ABORT) {
|
|
|
|
WARN_ON(!list_empty(&uobj->list));
|
|
|
|
WARN_ON(!uobj->context);
|
2018-08-10 10:14:37 +08:00
|
|
|
uobj->uapi_object->type_class->alloc_abort(uobj);
|
2018-07-26 11:40:12 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
uobj->context = NULL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* For DESTROY the usecnt is held write locked, the caller is expected
|
2018-07-26 11:40:20 +08:00
|
|
|
* to put it unlock and put the object when done with it. Only DESTROY
|
|
|
|
* can remove the IDR handle.
|
2018-07-26 11:40:12 +08:00
|
|
|
*/
|
|
|
|
if (reason != RDMA_REMOVE_DESTROY)
|
|
|
|
atomic_set(&uobj->usecnt, 0);
|
2018-07-26 11:40:20 +08:00
|
|
|
else
|
2018-08-10 10:14:37 +08:00
|
|
|
uobj->uapi_object->type_class->remove_handle(uobj);
|
2018-07-26 11:40:12 +08:00
|
|
|
|
|
|
|
if (!list_empty(&uobj->list)) {
|
|
|
|
spin_lock_irqsave(&ufile->uobjects_lock, flags);
|
|
|
|
list_del_init(&uobj->list);
|
|
|
|
spin_unlock_irqrestore(&ufile->uobjects_lock, flags);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Pairs with the get in rdma_alloc_commit_uobject(), could
|
|
|
|
* destroy uobj.
|
|
|
|
*/
|
|
|
|
uverbs_uobject_put(uobj);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* When aborting the stack kref remains owned by the core code, and is
|
|
|
|
* not transferred into the type. Pairs with the get in alloc_uobj
|
|
|
|
*/
|
|
|
|
if (reason == RDMA_REMOVE_ABORT)
|
|
|
|
uverbs_uobject_put(uobj);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:14 +08:00
|
|
|
/*
|
|
|
|
* This calls uverbs_destroy_uobject() using the RDMA_REMOVE_DESTROY
|
|
|
|
* sequence. It should only be used from command callbacks. On success the
|
|
|
|
* caller must pair this with rdma_lookup_put_uobject(LOOKUP_WRITE). This
|
|
|
|
* version requires the caller to have already obtained an
|
|
|
|
* LOOKUP_DESTROY uobject kref.
|
|
|
|
*/
|
|
|
|
int uobj_destroy(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
struct ib_uverbs_file *ufile = uobj->ufile;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
down_read(&ufile->hw_destroy_rwsem);
|
|
|
|
|
|
|
|
ret = uverbs_try_lock_object(uobj, UVERBS_LOOKUP_WRITE);
|
|
|
|
if (ret)
|
|
|
|
goto out_unlock;
|
|
|
|
|
|
|
|
ret = uverbs_destroy_uobject(uobj, RDMA_REMOVE_DESTROY);
|
|
|
|
if (ret) {
|
|
|
|
atomic_set(&uobj->usecnt, 0);
|
|
|
|
goto out_unlock;
|
|
|
|
}
|
|
|
|
|
|
|
|
out_unlock:
|
|
|
|
up_read(&ufile->hw_destroy_rwsem);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2018-07-04 16:32:06 +08:00
|
|
|
/*
|
2018-07-26 11:40:11 +08:00
|
|
|
* uobj_get_destroy destroys the HW object and returns a handle to the uobj
|
|
|
|
* with a NULL object pointer. The caller must pair this with
|
|
|
|
* uverbs_put_destroy.
|
2018-07-04 16:32:06 +08:00
|
|
|
*/
|
2018-08-10 10:14:37 +08:00
|
|
|
struct ib_uobject *__uobj_get_destroy(const struct uverbs_api_object *obj,
|
2018-11-26 02:51:13 +08:00
|
|
|
u32 id,
|
|
|
|
const struct uverbs_attr_bundle *attrs)
|
2018-07-04 16:32:06 +08:00
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
int ret;
|
|
|
|
|
2018-11-26 02:51:13 +08:00
|
|
|
uobj = rdma_lookup_get_uobject(obj, attrs->ufile, id,
|
|
|
|
UVERBS_LOOKUP_DESTROY);
|
2018-07-04 16:32:06 +08:00
|
|
|
if (IS_ERR(uobj))
|
2018-07-26 11:40:11 +08:00
|
|
|
return uobj;
|
2018-07-04 16:32:06 +08:00
|
|
|
|
2018-07-26 11:40:14 +08:00
|
|
|
ret = uobj_destroy(uobj);
|
2018-07-26 11:40:11 +08:00
|
|
|
if (ret) {
|
2018-07-26 11:40:14 +08:00
|
|
|
rdma_lookup_put_uobject(uobj, UVERBS_LOOKUP_DESTROY);
|
2018-07-26 11:40:11 +08:00
|
|
|
return ERR_PTR(ret);
|
|
|
|
}
|
2018-07-04 16:32:06 +08:00
|
|
|
|
2018-07-26 11:40:11 +08:00
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2018-11-26 02:51:14 +08:00
|
|
|
* Does both uobj_get_destroy() and uobj_put_destroy(). Returns 0 on success
|
|
|
|
* (negative errno on failure). For use by callers that do not need the uobj.
|
2018-07-26 11:40:11 +08:00
|
|
|
*/
|
2018-08-10 10:14:37 +08:00
|
|
|
int __uobj_perform_destroy(const struct uverbs_api_object *obj, u32 id,
|
2018-11-26 02:51:14 +08:00
|
|
|
const struct uverbs_attr_bundle *attrs)
|
2018-07-26 11:40:11 +08:00
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
2018-11-26 02:51:13 +08:00
|
|
|
uobj = __uobj_get_destroy(obj, id, attrs);
|
2018-07-26 11:40:11 +08:00
|
|
|
if (IS_ERR(uobj))
|
|
|
|
return PTR_ERR(uobj);
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
rdma_lookup_put_uobject(uobj, UVERBS_LOOKUP_WRITE);
|
2018-11-26 02:51:14 +08:00
|
|
|
return 0;
|
2018-07-04 16:32:06 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:12 +08:00
|
|
|
/* alloc_uobj must be undone by uverbs_destroy_uobject() */
|
2018-07-04 16:32:08 +08:00
|
|
|
static struct ib_uobject *alloc_uobj(struct ib_uverbs_file *ufile,
|
2018-08-10 10:14:37 +08:00
|
|
|
const struct uverbs_api_object *obj)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
2018-07-11 03:43:06 +08:00
|
|
|
struct ib_uobject *uobj;
|
|
|
|
struct ib_ucontext *ucontext;
|
|
|
|
|
2018-11-26 02:51:13 +08:00
|
|
|
ucontext = ib_uverbs_get_ucontext_file(ufile);
|
2018-07-11 03:43:06 +08:00
|
|
|
if (IS_ERR(ucontext))
|
|
|
|
return ERR_CAST(ucontext);
|
2017-04-04 18:31:42 +08:00
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
uobj = kzalloc(obj->type_attrs->obj_size, GFP_KERNEL);
|
2017-04-04 18:31:42 +08:00
|
|
|
if (!uobj)
|
|
|
|
return ERR_PTR(-ENOMEM);
|
|
|
|
/*
|
|
|
|
* user_handle should be filled by the handler,
|
|
|
|
* The object is added to the list in the commit stage.
|
|
|
|
*/
|
2018-07-04 16:32:08 +08:00
|
|
|
uobj->ufile = ufile;
|
2018-07-11 03:43:06 +08:00
|
|
|
uobj->context = ucontext;
|
2018-07-11 10:55:16 +08:00
|
|
|
INIT_LIST_HEAD(&uobj->list);
|
2018-08-10 10:14:37 +08:00
|
|
|
uobj->uapi_object = obj;
|
2018-02-13 18:18:36 +08:00
|
|
|
/*
|
|
|
|
* Allocated objects start out as write locked to deny any other
|
|
|
|
* syscalls from accessing them until they are committed. See
|
|
|
|
* rdma_alloc_commit_uobject
|
|
|
|
*/
|
|
|
|
atomic_set(&uobj->usecnt, -1);
|
2017-04-04 18:31:42 +08:00
|
|
|
kref_init(&uobj->ref);
|
|
|
|
|
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int idr_add_uobj(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
idr_preload(GFP_KERNEL);
|
2018-07-04 16:32:10 +08:00
|
|
|
spin_lock(&uobj->ufile->idr_lock);
|
2017-04-04 18:31:42 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We start with allocating an idr pointing to NULL. This represents an
|
|
|
|
* object which isn't initialized yet. We'll replace it later on with
|
|
|
|
* the real object once we commit.
|
|
|
|
*/
|
2018-07-04 16:32:10 +08:00
|
|
|
ret = idr_alloc(&uobj->ufile->idr, NULL, 0,
|
2017-04-04 18:31:42 +08:00
|
|
|
min_t(unsigned long, U32_MAX - 1, INT_MAX), GFP_NOWAIT);
|
|
|
|
if (ret >= 0)
|
|
|
|
uobj->id = ret;
|
|
|
|
|
2018-07-04 16:32:10 +08:00
|
|
|
spin_unlock(&uobj->ufile->idr_lock);
|
2017-04-04 18:31:42 +08:00
|
|
|
idr_preload_end();
|
|
|
|
|
|
|
|
return ret < 0 ? ret : 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Returns the ib_uobject or an error. The caller should check for IS_ERR. */
|
2018-07-04 16:32:08 +08:00
|
|
|
static struct ib_uobject *
|
2018-08-10 10:14:37 +08:00
|
|
|
lookup_get_idr_uobject(const struct uverbs_api_object *obj,
|
2018-07-26 11:40:13 +08:00
|
|
|
struct ib_uverbs_file *ufile, s64 id,
|
|
|
|
enum rdma_lookup_mode mode)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
2018-07-11 10:55:14 +08:00
|
|
|
unsigned long idrno = id;
|
|
|
|
|
|
|
|
if (id < 0 || id > ULONG_MAX)
|
|
|
|
return ERR_PTR(-EINVAL);
|
2017-04-04 18:31:42 +08:00
|
|
|
|
|
|
|
rcu_read_lock();
|
|
|
|
/* object won't be released as we're protected in rcu */
|
2018-07-11 10:55:14 +08:00
|
|
|
uobj = idr_find(&ufile->idr, idrno);
|
2017-04-04 18:31:42 +08:00
|
|
|
if (!uobj) {
|
|
|
|
uobj = ERR_PTR(-ENOENT);
|
|
|
|
goto free;
|
|
|
|
}
|
|
|
|
|
2018-02-13 18:18:37 +08:00
|
|
|
/*
|
|
|
|
* The idr_find is guaranteed to return a pointer to something that
|
|
|
|
* isn't freed yet, or NULL, as the free after idr_remove goes through
|
|
|
|
* kfree_rcu(). However the object may still have been released and
|
|
|
|
* kfree() could be called at any time.
|
|
|
|
*/
|
|
|
|
if (!kref_get_unless_zero(&uobj->ref))
|
|
|
|
uobj = ERR_PTR(-ENOENT);
|
|
|
|
|
2017-04-04 18:31:42 +08:00
|
|
|
free:
|
|
|
|
rcu_read_unlock();
|
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
static struct ib_uobject *
|
2018-08-10 10:14:37 +08:00
|
|
|
lookup_get_fd_uobject(const struct uverbs_api_object *obj,
|
2018-07-26 11:40:13 +08:00
|
|
|
struct ib_uverbs_file *ufile, s64 id,
|
|
|
|
enum rdma_lookup_mode mode)
|
2017-04-04 18:31:46 +08:00
|
|
|
{
|
2018-08-10 10:14:37 +08:00
|
|
|
const struct uverbs_obj_fd_type *fd_type;
|
2017-04-04 18:31:46 +08:00
|
|
|
struct file *f;
|
|
|
|
struct ib_uobject *uobject;
|
2018-07-11 10:55:14 +08:00
|
|
|
int fdno = id;
|
2017-04-04 18:31:46 +08:00
|
|
|
|
2018-07-11 10:55:14 +08:00
|
|
|
if (fdno != id)
|
|
|
|
return ERR_PTR(-EINVAL);
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
if (mode != UVERBS_LOOKUP_READ)
|
2017-04-04 18:31:46 +08:00
|
|
|
return ERR_PTR(-EOPNOTSUPP);
|
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
if (!obj->type_attrs)
|
|
|
|
return ERR_PTR(-EIO);
|
|
|
|
fd_type =
|
|
|
|
container_of(obj->type_attrs, struct uverbs_obj_fd_type, type);
|
|
|
|
|
2018-07-11 10:55:14 +08:00
|
|
|
f = fget(fdno);
|
2017-04-04 18:31:46 +08:00
|
|
|
if (!f)
|
|
|
|
return ERR_PTR(-EBADF);
|
|
|
|
|
|
|
|
uobject = f->private_data;
|
|
|
|
/*
|
|
|
|
* fget(id) ensures we are not currently running uverbs_close_fd,
|
|
|
|
* and the caller is expected to ensure that uverbs_close_fd is never
|
|
|
|
* done while a call top lookup is possible.
|
|
|
|
*/
|
|
|
|
if (f->f_op != fd_type->fops) {
|
|
|
|
fput(f);
|
|
|
|
return ERR_PTR(-EBADF);
|
|
|
|
}
|
|
|
|
|
|
|
|
uverbs_uobject_get(uobject);
|
|
|
|
return uobject;
|
|
|
|
}
|
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
struct ib_uobject *rdma_lookup_get_uobject(const struct uverbs_api_object *obj,
|
2018-07-11 10:55:14 +08:00
|
|
|
struct ib_uverbs_file *ufile, s64 id,
|
2018-07-26 11:40:13 +08:00
|
|
|
enum rdma_lookup_mode mode)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
int ret;
|
|
|
|
|
2018-11-26 14:28:33 +08:00
|
|
|
if (IS_ERR(obj) && PTR_ERR(obj) == -ENOMSG) {
|
|
|
|
/* must be UVERBS_IDR_ANY_OBJECT, see uapi_get_object() */
|
|
|
|
uobj = lookup_get_idr_uobject(NULL, ufile, id, mode);
|
|
|
|
if (IS_ERR(uobj))
|
|
|
|
return uobj;
|
|
|
|
} else {
|
|
|
|
if (IS_ERR(obj))
|
|
|
|
return ERR_PTR(-EINVAL);
|
|
|
|
|
|
|
|
uobj = obj->type_class->lookup_get(obj, ufile, id, mode);
|
|
|
|
if (IS_ERR(uobj))
|
|
|
|
return uobj;
|
|
|
|
|
|
|
|
if (uobj->uapi_object != obj) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto free;
|
|
|
|
}
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:16 +08:00
|
|
|
/*
|
|
|
|
* If we have been disassociated block every command except for
|
|
|
|
* DESTROY based commands.
|
|
|
|
*/
|
|
|
|
if (mode != UVERBS_LOOKUP_DESTROY &&
|
|
|
|
!srcu_dereference(ufile->device->ib_dev,
|
|
|
|
&ufile->device->disassociate_srcu)) {
|
|
|
|
ret = -EIO;
|
|
|
|
goto free;
|
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
ret = uverbs_try_lock_object(uobj, mode);
|
2018-07-11 10:55:19 +08:00
|
|
|
if (ret)
|
2017-04-04 18:31:42 +08:00
|
|
|
goto free;
|
|
|
|
|
|
|
|
return uobj;
|
|
|
|
free:
|
2018-11-26 14:28:33 +08:00
|
|
|
uobj->uapi_object->type_class->lookup_put(uobj, mode);
|
2017-04-04 18:31:42 +08:00
|
|
|
uverbs_uobject_put(uobj);
|
|
|
|
return ERR_PTR(ret);
|
|
|
|
}
|
2019-02-08 00:44:47 +08:00
|
|
|
struct ib_uobject *_uobj_get_read(enum uverbs_default_objects type,
|
|
|
|
u32 object_id,
|
|
|
|
struct uverbs_attr_bundle *attrs)
|
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
|
|
|
uobj = rdma_lookup_get_uobject(uobj_get_type(attrs, type), attrs->ufile,
|
|
|
|
object_id, UVERBS_LOOKUP_READ);
|
|
|
|
if (IS_ERR(uobj))
|
|
|
|
return uobj;
|
|
|
|
|
|
|
|
attrs->context = uobj->context;
|
|
|
|
|
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct ib_uobject *_uobj_get_write(enum uverbs_default_objects type,
|
|
|
|
u32 object_id,
|
|
|
|
struct uverbs_attr_bundle *attrs)
|
|
|
|
{
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
|
|
|
uobj = rdma_lookup_get_uobject(uobj_get_type(attrs, type), attrs->ufile,
|
|
|
|
object_id, UVERBS_LOOKUP_WRITE);
|
|
|
|
|
|
|
|
if (IS_ERR(uobj))
|
|
|
|
return uobj;
|
|
|
|
|
|
|
|
attrs->context = uobj->context;
|
|
|
|
|
|
|
|
return uobj;
|
|
|
|
}
|
2017-04-04 18:31:42 +08:00
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
static struct ib_uobject *
|
|
|
|
alloc_begin_idr_uobject(const struct uverbs_api_object *obj,
|
|
|
|
struct ib_uverbs_file *ufile)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
uobj = alloc_uobj(ufile, obj);
|
2017-04-04 18:31:42 +08:00
|
|
|
if (IS_ERR(uobj))
|
|
|
|
return uobj;
|
|
|
|
|
|
|
|
ret = idr_add_uobj(uobj);
|
|
|
|
if (ret)
|
|
|
|
goto uobj_put;
|
|
|
|
|
2018-07-11 03:43:06 +08:00
|
|
|
ret = ib_rdmacg_try_charge(&uobj->cg_obj, uobj->context->device,
|
2017-04-04 18:31:42 +08:00
|
|
|
RDMACG_RESOURCE_HCA_OBJECT);
|
|
|
|
if (ret)
|
|
|
|
goto idr_remove;
|
|
|
|
|
|
|
|
return uobj;
|
|
|
|
|
|
|
|
idr_remove:
|
2018-07-11 10:55:16 +08:00
|
|
|
spin_lock(&ufile->idr_lock);
|
|
|
|
idr_remove(&ufile->idr, uobj->id);
|
|
|
|
spin_unlock(&ufile->idr_lock);
|
2017-04-04 18:31:42 +08:00
|
|
|
uobj_put:
|
|
|
|
uverbs_uobject_put(uobj);
|
|
|
|
return ERR_PTR(ret);
|
|
|
|
}
|
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
static struct ib_uobject *
|
|
|
|
alloc_begin_fd_uobject(const struct uverbs_api_object *obj,
|
|
|
|
struct ib_uverbs_file *ufile)
|
2017-04-04 18:31:46 +08:00
|
|
|
{
|
|
|
|
int new_fd;
|
|
|
|
struct ib_uobject *uobj;
|
|
|
|
|
|
|
|
new_fd = get_unused_fd_flags(O_CLOEXEC);
|
|
|
|
if (new_fd < 0)
|
|
|
|
return ERR_PTR(new_fd);
|
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
uobj = alloc_uobj(ufile, obj);
|
2017-04-04 18:31:46 +08:00
|
|
|
if (IS_ERR(uobj)) {
|
|
|
|
put_unused_fd(new_fd);
|
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
2018-07-04 16:32:11 +08:00
|
|
|
uobj->id = new_fd;
|
|
|
|
uobj->ufile = ufile;
|
2017-04-04 18:31:46 +08:00
|
|
|
|
|
|
|
return uobj;
|
|
|
|
}
|
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
struct ib_uobject *rdma_alloc_begin_uobject(const struct uverbs_api_object *obj,
|
2018-07-04 16:32:08 +08:00
|
|
|
struct ib_uverbs_file *ufile)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
2018-07-26 11:40:15 +08:00
|
|
|
struct ib_uobject *ret;
|
|
|
|
|
2018-11-26 14:28:33 +08:00
|
|
|
if (IS_ERR(obj))
|
2018-08-10 10:14:37 +08:00
|
|
|
return ERR_PTR(-EINVAL);
|
|
|
|
|
2018-07-26 11:40:15 +08:00
|
|
|
/*
|
|
|
|
* The hw_destroy_rwsem is held across the entire object creation and
|
|
|
|
* released during rdma_alloc_commit_uobject or
|
|
|
|
* rdma_alloc_abort_uobject
|
|
|
|
*/
|
|
|
|
if (!down_read_trylock(&ufile->hw_destroy_rwsem))
|
|
|
|
return ERR_PTR(-EIO);
|
|
|
|
|
2018-08-10 10:14:37 +08:00
|
|
|
ret = obj->type_class->alloc_begin(obj, ufile);
|
2018-07-26 11:40:15 +08:00
|
|
|
if (IS_ERR(ret)) {
|
|
|
|
up_read(&ufile->hw_destroy_rwsem);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
return ret;
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:12 +08:00
|
|
|
static void alloc_abort_idr_uobject(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
ib_rdmacg_uncharge(&uobj->cg_obj, uobj->context->device,
|
|
|
|
RDMACG_RESOURCE_HCA_OBJECT);
|
|
|
|
|
|
|
|
spin_lock(&uobj->ufile->idr_lock);
|
|
|
|
idr_remove(&uobj->ufile->idr, uobj->id);
|
|
|
|
spin_unlock(&uobj->ufile->idr_lock);
|
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:20 +08:00
|
|
|
static int __must_check destroy_hw_idr_uobject(struct ib_uobject *uobj,
|
|
|
|
enum rdma_remove_reason why)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
|
|
|
const struct uverbs_obj_idr_type *idr_type =
|
2018-08-10 10:14:37 +08:00
|
|
|
container_of(uobj->uapi_object->type_attrs,
|
|
|
|
struct uverbs_obj_idr_type, type);
|
2017-04-04 18:31:42 +08:00
|
|
|
int ret = idr_type->destroy_object(uobj, why);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We can only fail gracefully if the user requested to destroy the
|
2018-06-20 22:11:39 +08:00
|
|
|
* object or when a retry may be called upon an error.
|
|
|
|
* In the rest of the cases, just remove whatever you can.
|
2017-04-04 18:31:42 +08:00
|
|
|
*/
|
2018-06-20 22:11:39 +08:00
|
|
|
if (ib_is_destroy_retryable(ret, why, uobj))
|
2017-04-04 18:31:42 +08:00
|
|
|
return ret;
|
|
|
|
|
2018-07-26 11:40:12 +08:00
|
|
|
if (why == RDMA_REMOVE_ABORT)
|
|
|
|
return 0;
|
2018-07-11 10:55:16 +08:00
|
|
|
|
2018-07-26 11:40:20 +08:00
|
|
|
ib_rdmacg_uncharge(&uobj->cg_obj, uobj->context->device,
|
|
|
|
RDMACG_RESOURCE_HCA_OBJECT);
|
2017-04-04 18:31:42 +08:00
|
|
|
|
2018-07-26 11:40:12 +08:00
|
|
|
return 0;
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:20 +08:00
|
|
|
static void remove_handle_idr_uobject(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
spin_lock(&uobj->ufile->idr_lock);
|
|
|
|
idr_remove(&uobj->ufile->idr, uobj->id);
|
|
|
|
spin_unlock(&uobj->ufile->idr_lock);
|
|
|
|
/* Matches the kref in alloc_commit_idr_uobject */
|
|
|
|
uverbs_uobject_put(uobj);
|
|
|
|
}
|
|
|
|
|
2017-04-04 18:31:46 +08:00
|
|
|
static void alloc_abort_fd_uobject(struct ib_uobject *uobj)
|
|
|
|
{
|
2018-07-11 10:55:21 +08:00
|
|
|
put_unused_fd(uobj->id);
|
2017-04-04 18:31:46 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:20 +08:00
|
|
|
static int __must_check destroy_hw_fd_uobject(struct ib_uobject *uobj,
|
|
|
|
enum rdma_remove_reason why)
|
2017-04-04 18:31:46 +08:00
|
|
|
{
|
2018-08-10 10:14:37 +08:00
|
|
|
const struct uverbs_obj_fd_type *fd_type = container_of(
|
|
|
|
uobj->uapi_object->type_attrs, struct uverbs_obj_fd_type, type);
|
2018-07-04 16:32:11 +08:00
|
|
|
int ret = fd_type->context_closed(uobj, why);
|
2017-04-04 18:31:46 +08:00
|
|
|
|
2018-06-20 22:11:39 +08:00
|
|
|
if (ib_is_destroy_retryable(ret, why, uobj))
|
2017-04-04 18:31:46 +08:00
|
|
|
return ret;
|
|
|
|
|
2018-07-26 11:40:12 +08:00
|
|
|
return 0;
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:20 +08:00
|
|
|
static void remove_handle_fd_uobject(struct ib_uobject *uobj)
|
|
|
|
{
|
|
|
|
}
|
|
|
|
|
2018-07-11 10:55:21 +08:00
|
|
|
static int alloc_commit_idr_uobject(struct ib_uobject *uobj)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
2018-07-11 10:55:15 +08:00
|
|
|
struct ib_uverbs_file *ufile = uobj->ufile;
|
|
|
|
|
|
|
|
spin_lock(&ufile->idr_lock);
|
2017-04-04 18:31:42 +08:00
|
|
|
/*
|
|
|
|
* We already allocated this IDR with a NULL object, so
|
|
|
|
* this shouldn't fail.
|
2018-07-11 10:55:15 +08:00
|
|
|
*
|
|
|
|
* NOTE: Once we set the IDR we loose ownership of our kref on uobj.
|
|
|
|
* It will be put by remove_commit_idr_uobject()
|
2017-04-04 18:31:42 +08:00
|
|
|
*/
|
2018-07-11 10:55:15 +08:00
|
|
|
WARN_ON(idr_replace(&ufile->idr, uobj, uobj->id));
|
|
|
|
spin_unlock(&ufile->idr_lock);
|
2018-07-11 10:55:21 +08:00
|
|
|
|
|
|
|
return 0;
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2018-07-11 10:55:21 +08:00
|
|
|
static int alloc_commit_fd_uobject(struct ib_uobject *uobj)
|
2017-04-04 18:31:46 +08:00
|
|
|
{
|
2018-08-10 10:14:37 +08:00
|
|
|
const struct uverbs_obj_fd_type *fd_type = container_of(
|
|
|
|
uobj->uapi_object->type_attrs, struct uverbs_obj_fd_type, type);
|
2018-07-11 10:55:15 +08:00
|
|
|
int fd = uobj->id;
|
2018-07-11 10:55:21 +08:00
|
|
|
struct file *filp;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The kref for uobj is moved into filp->private data and put in
|
|
|
|
* uverbs_close_fd(). Once alloc_commit() succeeds uverbs_close_fd()
|
|
|
|
* must be guaranteed to be called from the provided fops release
|
|
|
|
* callback.
|
|
|
|
*/
|
|
|
|
filp = anon_inode_getfile(fd_type->name,
|
|
|
|
fd_type->fops,
|
|
|
|
uobj,
|
|
|
|
fd_type->flags);
|
|
|
|
if (IS_ERR(filp))
|
|
|
|
return PTR_ERR(filp);
|
|
|
|
|
|
|
|
uobj->object = filp;
|
|
|
|
|
|
|
|
/* Matching put will be done in uverbs_close_fd() */
|
|
|
|
kref_get(&uobj->ufile->ref);
|
2018-07-11 10:55:15 +08:00
|
|
|
|
2017-04-04 18:31:46 +08:00
|
|
|
/* This shouldn't be used anymore. Use the file object instead */
|
2018-07-04 16:32:11 +08:00
|
|
|
uobj->id = 0;
|
2018-07-11 10:55:15 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* NOTE: Once we install the file we loose ownership of our kref on
|
|
|
|
* uobj. It will be put by uverbs_close_fd()
|
|
|
|
*/
|
2018-07-11 10:55:21 +08:00
|
|
|
fd_install(fd, filp);
|
|
|
|
|
|
|
|
return 0;
|
2017-04-04 18:31:46 +08:00
|
|
|
}
|
|
|
|
|
2018-07-11 10:55:15 +08:00
|
|
|
/*
|
|
|
|
* In all cases rdma_alloc_commit_uobject() consumes the kref to uobj and the
|
2018-07-11 10:55:20 +08:00
|
|
|
* caller can no longer assume uobj is valid. If this function fails it
|
|
|
|
* destroys the uboject, including the attached HW object.
|
2018-07-11 10:55:15 +08:00
|
|
|
*/
|
2018-07-11 10:55:20 +08:00
|
|
|
int __must_check rdma_alloc_commit_uobject(struct ib_uobject *uobj)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
2018-07-04 16:32:07 +08:00
|
|
|
struct ib_uverbs_file *ufile = uobj->ufile;
|
2018-07-11 10:55:21 +08:00
|
|
|
int ret;
|
2018-07-04 16:32:07 +08:00
|
|
|
|
2018-07-11 10:55:21 +08:00
|
|
|
/* alloc_commit consumes the uobj kref */
|
2018-08-10 10:14:37 +08:00
|
|
|
ret = uobj->uapi_object->type_class->alloc_commit(uobj);
|
2018-07-11 10:55:21 +08:00
|
|
|
if (ret) {
|
2018-07-26 11:40:12 +08:00
|
|
|
uverbs_destroy_uobject(uobj, RDMA_REMOVE_ABORT);
|
2018-07-26 11:40:15 +08:00
|
|
|
up_read(&ufile->hw_destroy_rwsem);
|
2018-07-11 10:55:21 +08:00
|
|
|
return ret;
|
|
|
|
}
|
2018-02-13 18:18:36 +08:00
|
|
|
|
2018-07-11 10:55:16 +08:00
|
|
|
/* kref is held so long as the uobj is on the uobj list. */
|
|
|
|
uverbs_uobject_get(uobj);
|
2018-07-11 10:55:18 +08:00
|
|
|
spin_lock_irq(&ufile->uobjects_lock);
|
2018-07-04 16:32:07 +08:00
|
|
|
list_add(&uobj->list, &ufile->uobjects);
|
2018-07-11 10:55:18 +08:00
|
|
|
spin_unlock_irq(&ufile->uobjects_lock);
|
2018-02-13 18:18:39 +08:00
|
|
|
|
2018-07-11 10:55:21 +08:00
|
|
|
/* matches atomic_set(-1) in alloc_uobj */
|
|
|
|
atomic_set(&uobj->usecnt, 0);
|
|
|
|
|
2018-07-26 11:40:15 +08:00
|
|
|
/* Matches the down_read in rdma_alloc_begin_uobject */
|
2018-07-11 10:55:18 +08:00
|
|
|
up_read(&ufile->hw_destroy_rwsem);
|
2017-04-04 18:31:42 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-07-11 10:55:16 +08:00
|
|
|
/*
|
|
|
|
* This consumes the kref for uobj. It is up to the caller to unwind the HW
|
|
|
|
* object and anything else connected to uobj before calling this.
|
|
|
|
*/
|
2017-04-04 18:31:42 +08:00
|
|
|
void rdma_alloc_abort_uobject(struct ib_uobject *uobj)
|
|
|
|
{
|
2018-07-26 11:40:15 +08:00
|
|
|
struct ib_uverbs_file *ufile = uobj->ufile;
|
|
|
|
|
2018-07-26 11:40:12 +08:00
|
|
|
uobj->object = NULL;
|
|
|
|
uverbs_destroy_uobject(uobj, RDMA_REMOVE_ABORT);
|
2018-07-26 11:40:15 +08:00
|
|
|
|
|
|
|
/* Matches the down_read in rdma_alloc_begin_uobject */
|
|
|
|
up_read(&ufile->hw_destroy_rwsem);
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
static void lookup_put_idr_uobject(struct ib_uobject *uobj,
|
|
|
|
enum rdma_lookup_mode mode)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
static void lookup_put_fd_uobject(struct ib_uobject *uobj,
|
|
|
|
enum rdma_lookup_mode mode)
|
2017-04-04 18:31:46 +08:00
|
|
|
{
|
|
|
|
struct file *filp = uobj->object;
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
WARN_ON(mode != UVERBS_LOOKUP_READ);
|
2017-04-04 18:31:46 +08:00
|
|
|
/* This indirectly calls uverbs_close_fd and free the object */
|
|
|
|
fput(filp);
|
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:13 +08:00
|
|
|
void rdma_lookup_put_uobject(struct ib_uobject *uobj,
|
|
|
|
enum rdma_lookup_mode mode)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
2018-07-26 11:40:13 +08:00
|
|
|
assert_uverbs_usecnt(uobj, mode);
|
2018-08-10 10:14:37 +08:00
|
|
|
uobj->uapi_object->type_class->lookup_put(uobj, mode);
|
2017-04-04 18:31:42 +08:00
|
|
|
/*
|
|
|
|
* In order to unlock an object, either decrease its usecnt for
|
2017-04-18 17:03:37 +08:00
|
|
|
* read access or zero it in case of exclusive access. See
|
2017-04-04 18:31:42 +08:00
|
|
|
* uverbs_try_lock_object for locking schema information.
|
|
|
|
*/
|
2018-07-26 11:40:13 +08:00
|
|
|
switch (mode) {
|
|
|
|
case UVERBS_LOOKUP_READ:
|
2017-04-04 18:31:42 +08:00
|
|
|
atomic_dec(&uobj->usecnt);
|
2018-07-26 11:40:13 +08:00
|
|
|
break;
|
|
|
|
case UVERBS_LOOKUP_WRITE:
|
2017-04-04 18:31:42 +08:00
|
|
|
atomic_set(&uobj->usecnt, 0);
|
2018-07-26 11:40:13 +08:00
|
|
|
break;
|
2018-07-26 11:40:14 +08:00
|
|
|
case UVERBS_LOOKUP_DESTROY:
|
|
|
|
break;
|
2018-07-26 11:40:13 +08:00
|
|
|
}
|
2017-04-04 18:31:42 +08:00
|
|
|
|
2018-07-11 10:55:16 +08:00
|
|
|
/* Pairs with the kref obtained by type->lookup_get */
|
2017-04-04 18:31:42 +08:00
|
|
|
uverbs_uobject_put(uobj);
|
|
|
|
}
|
|
|
|
|
2018-07-26 11:40:20 +08:00
|
|
|
void setup_ufile_idr_uobject(struct ib_uverbs_file *ufile)
|
|
|
|
{
|
|
|
|
spin_lock_init(&ufile->idr_lock);
|
|
|
|
idr_init(&ufile->idr);
|
|
|
|
}
|
|
|
|
|
|
|
|
void release_ufile_idr_uobject(struct ib_uverbs_file *ufile)
|
|
|
|
{
|
|
|
|
struct ib_uobject *entry;
|
|
|
|
int id;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* At this point uverbs_cleanup_ufile() is guaranteed to have run, and
|
|
|
|
* there are no HW objects left, however the IDR is still populated
|
|
|
|
* with anything that has not been cleaned up by userspace. Since the
|
|
|
|
* kref on ufile is 0, nothing is allowed to call lookup_get.
|
|
|
|
*
|
|
|
|
* This is an optimized equivalent to remove_handle_idr_uobject
|
|
|
|
*/
|
|
|
|
idr_for_each_entry(&ufile->idr, entry, id) {
|
|
|
|
WARN_ON(entry->object);
|
|
|
|
uverbs_uobject_put(entry);
|
|
|
|
}
|
|
|
|
|
|
|
|
idr_destroy(&ufile->idr);
|
|
|
|
}
|
|
|
|
|
2017-04-04 18:31:42 +08:00
|
|
|
const struct uverbs_obj_type_class uverbs_idr_class = {
|
|
|
|
.alloc_begin = alloc_begin_idr_uobject,
|
|
|
|
.lookup_get = lookup_get_idr_uobject,
|
|
|
|
.alloc_commit = alloc_commit_idr_uobject,
|
|
|
|
.alloc_abort = alloc_abort_idr_uobject,
|
|
|
|
.lookup_put = lookup_put_idr_uobject,
|
2018-07-26 11:40:20 +08:00
|
|
|
.destroy_hw = destroy_hw_idr_uobject,
|
|
|
|
.remove_handle = remove_handle_idr_uobject,
|
2017-04-04 18:31:42 +08:00
|
|
|
/*
|
|
|
|
* When we destroy an object, we first just lock it for WRITE and
|
|
|
|
* actually DESTROY it in the finalize stage. So, the problematic
|
|
|
|
* scenario is when we just started the finalize stage of the
|
|
|
|
* destruction (nothing was executed yet). Now, the other thread
|
|
|
|
* fetched the object for READ access, but it didn't lock it yet.
|
|
|
|
* The DESTROY thread continues and starts destroying the object.
|
|
|
|
* When the other thread continue - without the RCU, it would
|
|
|
|
* access freed memory. However, the rcu_read_lock delays the free
|
|
|
|
* until the rcu_read_lock of the READ operation quits. Since the
|
2017-04-18 17:03:37 +08:00
|
|
|
* exclusive lock of the object is still taken by the DESTROY flow, the
|
2017-04-04 18:31:42 +08:00
|
|
|
* READ operation will get -EBUSY and it'll just bail out.
|
|
|
|
*/
|
|
|
|
.needs_kfree_rcu = true,
|
|
|
|
};
|
2018-06-17 17:59:50 +08:00
|
|
|
EXPORT_SYMBOL(uverbs_idr_class);
|
2017-04-04 18:31:42 +08:00
|
|
|
|
2017-04-04 18:31:46 +08:00
|
|
|
void uverbs_close_fd(struct file *f)
|
|
|
|
{
|
2018-07-04 16:32:11 +08:00
|
|
|
struct ib_uobject *uobj = f->private_data;
|
2018-07-11 10:55:17 +08:00
|
|
|
struct ib_uverbs_file *ufile = uobj->ufile;
|
|
|
|
|
2018-07-11 10:55:18 +08:00
|
|
|
if (down_read_trylock(&ufile->hw_destroy_rwsem)) {
|
2018-07-26 11:40:12 +08:00
|
|
|
/*
|
|
|
|
* lookup_get_fd_uobject holds the kref on the struct file any
|
|
|
|
* time a FD uobj is locked, which prevents this release
|
|
|
|
* method from being invoked. Meaning we can always get the
|
|
|
|
* write lock here, or we have a kernel bug.
|
|
|
|
*/
|
2018-07-26 11:40:13 +08:00
|
|
|
WARN_ON(uverbs_try_lock_object(uobj, UVERBS_LOOKUP_WRITE));
|
2018-07-26 11:40:12 +08:00
|
|
|
uverbs_destroy_uobject(uobj, RDMA_REMOVE_CLOSE);
|
2018-07-11 10:55:18 +08:00
|
|
|
up_read(&ufile->hw_destroy_rwsem);
|
2018-07-11 10:55:17 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Matches the get in alloc_begin_fd_uobject */
|
|
|
|
kref_put(&ufile->ref, ib_uverbs_release_file);
|
2017-04-04 18:31:46 +08:00
|
|
|
|
2018-07-11 10:55:16 +08:00
|
|
|
/* Pairs with filp->private_data in alloc_begin_fd_uobject */
|
2018-07-04 16:32:11 +08:00
|
|
|
uverbs_uobject_put(uobj);
|
2017-04-04 18:31:46 +08:00
|
|
|
}
|
2019-01-22 14:29:56 +08:00
|
|
|
EXPORT_SYMBOL(uverbs_close_fd);
|
2017-04-04 18:31:46 +08:00
|
|
|
|
2018-07-11 10:55:19 +08:00
|
|
|
/*
|
|
|
|
* Drop the ucontext off the ufile and completely disconnect it from the
|
|
|
|
* ib_device
|
|
|
|
*/
|
|
|
|
static void ufile_destroy_ucontext(struct ib_uverbs_file *ufile,
|
|
|
|
enum rdma_remove_reason reason)
|
|
|
|
{
|
|
|
|
struct ib_ucontext *ucontext = ufile->ucontext;
|
2018-09-17 01:43:12 +08:00
|
|
|
struct ib_device *ib_dev = ucontext->device;
|
2018-07-11 10:55:19 +08:00
|
|
|
int ret;
|
|
|
|
|
2018-09-17 01:43:12 +08:00
|
|
|
/*
|
|
|
|
* If we are closing the FD then the user mmap VMAs must have
|
|
|
|
* already been destroyed as they hold on to the filep, otherwise
|
|
|
|
* they need to be zap'd.
|
|
|
|
*/
|
2018-09-17 01:43:08 +08:00
|
|
|
if (reason == RDMA_REMOVE_DRIVER_REMOVE) {
|
|
|
|
uverbs_user_mmap_disassociate(ufile);
|
2018-12-11 03:09:48 +08:00
|
|
|
if (ib_dev->ops.disassociate_ucontext)
|
|
|
|
ib_dev->ops.disassociate_ucontext(ucontext);
|
2018-09-17 01:43:08 +08:00
|
|
|
}
|
2018-07-11 10:55:19 +08:00
|
|
|
|
2018-09-17 01:43:12 +08:00
|
|
|
ib_rdmacg_uncharge(&ucontext->cg_obj, ib_dev,
|
2018-07-11 10:55:19 +08:00
|
|
|
RDMACG_RESOURCE_HCA_HANDLE);
|
|
|
|
|
2018-11-28 19:16:44 +08:00
|
|
|
rdma_restrack_del(&ucontext->res);
|
|
|
|
|
2018-07-11 10:55:19 +08:00
|
|
|
/*
|
|
|
|
* FIXME: Drivers are not permitted to fail dealloc_ucontext, remove
|
|
|
|
* the error return.
|
|
|
|
*/
|
2018-12-11 03:09:48 +08:00
|
|
|
ret = ib_dev->ops.dealloc_ucontext(ucontext);
|
2018-07-11 10:55:19 +08:00
|
|
|
WARN_ON(ret);
|
|
|
|
|
|
|
|
ufile->ucontext = NULL;
|
|
|
|
}
|
|
|
|
|
2018-07-04 16:32:08 +08:00
|
|
|
static int __uverbs_cleanup_ufile(struct ib_uverbs_file *ufile,
|
|
|
|
enum rdma_remove_reason reason)
|
2017-04-04 18:31:42 +08:00
|
|
|
{
|
2018-06-20 22:11:39 +08:00
|
|
|
struct ib_uobject *obj, *next_obj;
|
|
|
|
int ret = -EINVAL;
|
2017-04-04 18:31:42 +08:00
|
|
|
|
2018-06-20 22:11:39 +08:00
|
|
|
/*
|
|
|
|
* This shouldn't run while executing other commands on this
|
|
|
|
* context. Thus, the only thing we should take care of is
|
|
|
|
* releasing a FD while traversing this list. The FD could be
|
|
|
|
* closed and released from the _release fop of this FD.
|
|
|
|
* In order to mitigate this, we add a lock.
|
|
|
|
* We take and release the lock per traversal in order to let
|
|
|
|
* other threads (which might still use the FDs) chance to run.
|
|
|
|
*/
|
2018-07-04 16:32:07 +08:00
|
|
|
list_for_each_entry_safe(obj, next_obj, &ufile->uobjects, list) {
|
2018-06-20 22:11:39 +08:00
|
|
|
/*
|
|
|
|
* if we hit this WARN_ON, that means we are
|
|
|
|
* racing with a lookup_get.
|
|
|
|
*/
|
2018-07-26 11:40:13 +08:00
|
|
|
WARN_ON(uverbs_try_lock_object(obj, UVERBS_LOOKUP_WRITE));
|
2018-07-26 11:40:12 +08:00
|
|
|
if (!uverbs_destroy_uobject(obj, reason))
|
|
|
|
ret = 0;
|
2018-08-28 19:40:32 +08:00
|
|
|
else
|
|
|
|
atomic_set(&obj->usecnt, 0);
|
2018-06-20 22:11:39 +08:00
|
|
|
}
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2018-07-11 10:55:19 +08:00
|
|
|
/*
|
|
|
|
* Destroy the uncontext and every uobject associated with it. If called with
|
|
|
|
* reason != RDMA_REMOVE_CLOSE this will not return until the destruction has
|
|
|
|
* been completed and ufile->ucontext is NULL.
|
|
|
|
*
|
|
|
|
* This is internally locked and can be called in parallel from multiple
|
|
|
|
* contexts.
|
|
|
|
*/
|
|
|
|
void uverbs_destroy_ufile_hw(struct ib_uverbs_file *ufile,
|
|
|
|
enum rdma_remove_reason reason)
|
2018-06-20 22:11:39 +08:00
|
|
|
{
|
2018-07-11 10:55:19 +08:00
|
|
|
if (reason == RDMA_REMOVE_CLOSE) {
|
|
|
|
/*
|
|
|
|
* During destruction we might trigger something that
|
|
|
|
* synchronously calls release on any file descriptor. For
|
|
|
|
* this reason all paths that come from file_operations
|
|
|
|
* release must use try_lock. They can progress knowing that
|
|
|
|
* there is an ongoing uverbs_destroy_ufile_hw that will clean
|
|
|
|
* up the driver resources.
|
|
|
|
*/
|
|
|
|
if (!mutex_trylock(&ufile->ucontext_lock))
|
|
|
|
return;
|
|
|
|
|
|
|
|
} else {
|
|
|
|
mutex_lock(&ufile->ucontext_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
down_write(&ufile->hw_destroy_rwsem);
|
2018-07-04 16:32:07 +08:00
|
|
|
|
2017-04-04 18:31:42 +08:00
|
|
|
/*
|
2018-07-11 10:55:19 +08:00
|
|
|
* If a ucontext was never created then we can't have any uobjects to
|
|
|
|
* cleanup, nothing to do.
|
2017-04-04 18:31:42 +08:00
|
|
|
*/
|
2018-07-11 10:55:19 +08:00
|
|
|
if (!ufile->ucontext)
|
|
|
|
goto done;
|
|
|
|
|
|
|
|
ufile->ucontext->closing = true;
|
2018-07-04 16:32:07 +08:00
|
|
|
ufile->ucontext->cleanup_retryable = true;
|
|
|
|
while (!list_empty(&ufile->uobjects))
|
2018-07-04 16:32:08 +08:00
|
|
|
if (__uverbs_cleanup_ufile(ufile, reason)) {
|
2018-06-20 22:11:39 +08:00
|
|
|
/*
|
|
|
|
* No entry was cleaned-up successfully during this
|
|
|
|
* iteration
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
}
|
2017-04-04 18:31:42 +08:00
|
|
|
|
2018-07-04 16:32:07 +08:00
|
|
|
ufile->ucontext->cleanup_retryable = false;
|
|
|
|
if (!list_empty(&ufile->uobjects))
|
2018-07-04 16:32:08 +08:00
|
|
|
__uverbs_cleanup_ufile(ufile, reason);
|
2017-04-04 18:31:42 +08:00
|
|
|
|
2018-07-11 10:55:19 +08:00
|
|
|
ufile_destroy_ucontext(ufile, reason);
|
|
|
|
|
|
|
|
done:
|
2018-07-11 10:55:18 +08:00
|
|
|
up_write(&ufile->hw_destroy_rwsem);
|
2018-07-11 10:55:19 +08:00
|
|
|
mutex_unlock(&ufile->ucontext_lock);
|
2017-04-04 18:31:42 +08:00
|
|
|
}
|
|
|
|
|
2017-04-04 18:31:46 +08:00
|
|
|
const struct uverbs_obj_type_class uverbs_fd_class = {
|
|
|
|
.alloc_begin = alloc_begin_fd_uobject,
|
|
|
|
.lookup_get = lookup_get_fd_uobject,
|
|
|
|
.alloc_commit = alloc_commit_fd_uobject,
|
|
|
|
.alloc_abort = alloc_abort_fd_uobject,
|
|
|
|
.lookup_put = lookup_put_fd_uobject,
|
2018-07-26 11:40:20 +08:00
|
|
|
.destroy_hw = destroy_hw_fd_uobject,
|
|
|
|
.remove_handle = remove_handle_fd_uobject,
|
2017-04-04 18:31:46 +08:00
|
|
|
.needs_kfree_rcu = false,
|
|
|
|
};
|
2018-06-17 17:59:50 +08:00
|
|
|
EXPORT_SYMBOL(uverbs_fd_class);
|
2017-04-04 18:31:46 +08:00
|
|
|
|
2018-07-04 16:32:08 +08:00
|
|
|
struct ib_uobject *
|
2018-08-10 10:14:37 +08:00
|
|
|
uverbs_get_uobject_from_file(u16 object_id,
|
2018-07-04 16:32:08 +08:00
|
|
|
struct ib_uverbs_file *ufile,
|
2018-07-11 10:55:14 +08:00
|
|
|
enum uverbs_obj_access access, s64 id)
|
2017-08-03 21:06:55 +08:00
|
|
|
{
|
2018-08-10 10:14:37 +08:00
|
|
|
const struct uverbs_api_object *obj =
|
|
|
|
uapi_get_object(ufile->device->uapi, object_id);
|
|
|
|
|
2017-08-03 21:06:55 +08:00
|
|
|
switch (access) {
|
|
|
|
case UVERBS_ACCESS_READ:
|
2018-08-10 10:14:37 +08:00
|
|
|
return rdma_lookup_get_uobject(obj, ufile, id,
|
2018-07-26 11:40:13 +08:00
|
|
|
UVERBS_LOOKUP_READ);
|
2017-08-03 21:06:55 +08:00
|
|
|
case UVERBS_ACCESS_DESTROY:
|
2018-07-26 11:40:14 +08:00
|
|
|
/* Actual destruction is done inside uverbs_handle_method */
|
2018-08-10 10:14:37 +08:00
|
|
|
return rdma_lookup_get_uobject(obj, ufile, id,
|
2018-07-26 11:40:14 +08:00
|
|
|
UVERBS_LOOKUP_DESTROY);
|
2017-08-03 21:06:55 +08:00
|
|
|
case UVERBS_ACCESS_WRITE:
|
2018-08-10 10:14:37 +08:00
|
|
|
return rdma_lookup_get_uobject(obj, ufile, id,
|
2018-07-26 11:40:13 +08:00
|
|
|
UVERBS_LOOKUP_WRITE);
|
2017-08-03 21:06:55 +08:00
|
|
|
case UVERBS_ACCESS_NEW:
|
2018-08-10 10:14:37 +08:00
|
|
|
return rdma_alloc_begin_uobject(obj, ufile);
|
2017-08-03 21:06:55 +08:00
|
|
|
default:
|
|
|
|
WARN_ON(true);
|
|
|
|
return ERR_PTR(-EOPNOTSUPP);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
int uverbs_finalize_object(struct ib_uobject *uobj,
|
|
|
|
enum uverbs_obj_access access,
|
|
|
|
bool commit)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* refcounts should be handled at the object level and not at the
|
|
|
|
* uobject level. Refcounts of the objects themselves are done in
|
|
|
|
* handlers.
|
|
|
|
*/
|
|
|
|
|
|
|
|
switch (access) {
|
|
|
|
case UVERBS_ACCESS_READ:
|
2018-07-26 11:40:13 +08:00
|
|
|
rdma_lookup_put_uobject(uobj, UVERBS_LOOKUP_READ);
|
2017-08-03 21:06:55 +08:00
|
|
|
break;
|
|
|
|
case UVERBS_ACCESS_WRITE:
|
2018-07-26 11:40:13 +08:00
|
|
|
rdma_lookup_put_uobject(uobj, UVERBS_LOOKUP_WRITE);
|
2017-08-03 21:06:55 +08:00
|
|
|
break;
|
|
|
|
case UVERBS_ACCESS_DESTROY:
|
2018-07-26 11:40:14 +08:00
|
|
|
if (uobj)
|
|
|
|
rdma_lookup_put_uobject(uobj, UVERBS_LOOKUP_DESTROY);
|
2017-08-03 21:06:55 +08:00
|
|
|
break;
|
|
|
|
case UVERBS_ACCESS_NEW:
|
|
|
|
if (commit)
|
|
|
|
ret = rdma_alloc_commit_uobject(uobj);
|
|
|
|
else
|
|
|
|
rdma_alloc_abort_uobject(uobj);
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
WARN_ON(true);
|
|
|
|
ret = -EOPNOTSUPP;
|
|
|
|
}
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|