funcX Client¶
The funcX Client is the programmatic interface to FuncX from Python. The client provides a simple and intuitive interface to:
- Register functions
- Register containers and execution environments
- Launch registered function against endpoints
- Check the status of launched functions and
- Retrieve outputs from functions
The following shows an example of creating a client.
from funcx.sdk.client import FuncXClient
fxc = FuncXClient()
Instantiating a client will start an authentication process where you will be asked to authenticate with Globus Auth. We require every interaction with the funcX Web Service to be authenticated using a Bearer token, this allows funcX to enforce access control on both functions and endpoints. As part of the authentication process we request access to your identity information (to retrieve your email address) and Globus Groups management access. We require Groups access in order to facilitate sharing. Users can share functions with others by associating a Globus Group with the function. If a group is set, the funcX Web Service uses Groups Management access to determine whether a user is a member of the specified group.
Note
The funcX Web Service internally caches function, endpoint, and access control lookups. Caches are based on user authentication tokens. To refresh the caches you can re-authenticate your client with force_login=True.
Registering Functions¶
You can register a Python function with funcX via register_function(). Function registration will serialize the function body and transmit it to the funcX Web Service. Once a function is registered with the service it will return a UUID that can be used to invoke the function.
Note
You must import any dependencies required by the function inside the function body.
You can associate a Globus Group with a function to enable sharing. If set, the Web Service will check if the invoking user is a member of the group. This is achieved by setting group=<globus_group_id> when registering a function.
You can also set a function to be publicly accssible by setting public=True when registering the function.
Running Functions¶
You can invoke a function using the UUID returned when registering the function. The client’s run() function requires you to specify the function_id and endpoint_id. In addition, you can pass *args and **kwargs to the run function and they will be used when invoking your function. We serialize all inputs and outputs when running a function.
The result of your function’s invocation can be retrieved using the client’s get_result() function. This will either raise an exception if the task is still pending, or give you back the deserialized result of your invocation.
Note
If your function’s execution raises an exception, get_result() will reraise it.
To minimize the overhead of communicating with the funcX Web Service we provide batch request and status capabilities. Documentation on batch requests can be found below.
Client Throttling¶
In order to avoid accidentally DoS’ing the funcX Web Service we place soft throttling restrictions on the funcX client. There are two key throttling measures: firstly, we limit the number of requests a client can make to the Web Service to 5 every 5 seconds, and secondly, we limit the size of inputs and outputs transmitted through the service to 2MB.
Batching requests and status can help reduce the number of requests made to the Web Service. In addition, the limit on the number of requests made to the Web Service can be removed by setting throttling_enabled to False.
fxc = FuncXClient()
fxc.throttling_enabled = False
FuncXClient Reference:¶
-
class
funcx.FuncXClient(http_timeout=None, funcx_home='~/.funcx', force_login=False, fx_authorizer=None, funcx_service_address='https://dev.funcx.org/api/v1', **kwargs)¶ Main class for interacting with the funcX service
Holds helper operations for performing common tasks with the funcX service.
-
add_to_whitelist(endpoint_id, function_ids)¶ Adds the function to the endpoint’s whitelist
Parameters: Returns: The response of the request
Return type: json
-
batch_run(batch)¶ Initiate a batch of tasks to funcX
Parameters: batch (a Batch object) – Returns: task_ids Return type: a list of UUID strings that identify the tasks
-
create_batch()¶ Create a Batch instance to handle batch submission in funcX
Returns: Status block containing “status” key. Return type: Batch instance
-
delete_from_whitelist(endpoint_id, function_ids)¶ List the endpoint’s whitelist
Parameters: Returns: The response of the request
Return type: json
-
get_batch_result(task_id_list)¶ Request results for a batch of task_ids
-
get_batch_status(task_id_list)¶ Request status for a batch of task_ids
-
get_container(container_uuid, container_type)¶ Get the details of a container for staging it locally.
Parameters: Returns: The details of the containers to deploy
Return type:
-
get_containers(name, description=None)¶ Register a DLHub endpoint with the funcX service and get the containers to launch.
Parameters: Returns: The port to connect to and a list of containers
Return type:
-
get_endpoint_status(endpoint_uuid)¶ Get the status reports for an endpoint.
Parameters: endpoint_uuid (str) – UUID of the endpoint in question Returns: The details of the endpoint’s stats Return type: dict
-
get_result(task_id)¶ Get the result of a funcX task
Parameters: task_id (str) – UUID of the task Returns: Result obj Return type: If task completed Raises: Exception obj: Exception due to which the task failed
-
get_task_status(task_id)¶ Get the status of a funcX task.
Parameters: task_id (str) – UUID of the task Returns: Status block containing “status” key. Return type: dict
-
get_whitelist(endpoint_id)¶ List the endpoint’s whitelist
Parameters: endpoint_id (str) – The uuid of the endpoint Returns: The response of the request Return type: json
-
logout()¶ Remove credentials from your local system
-
map_run(*args, endpoint_id=None, function_id=None, asynchronous=False, **kwargs)¶ Initiate an invocation
Parameters: - *args (Any) – Args as specified by the function signature
- endpoint_id (uuid str) – Endpoint UUID string. Required
- function_id (uuid str) – Function UUID string. Required
- asynchronous (bool) – Whether or not to run the function asynchronously
Returns: - task_id (str)
- UUID string that identifies the task
-
register_container(location, container_type, name='', description='')¶ Register a container with the funcX service.
Parameters: - location (str) – The location of the container (e.g., its docker url). Required
- container_type (str) – The type of containers that will be used (Singularity, Shifter, Docker). Required
- name (str) – A name for the container. Default = ‘’
- description (str) – A description to associate with the container. Default = ‘’
Returns: The id of the container
Return type:
-
register_endpoint(name, endpoint_uuid, description=None)¶ Register an endpoint with the funcX service.
Parameters: Returns: - {‘endopoint_id’ : <>,
’address’ : <>, ‘client_ports’: <>}
Return type: A dict
-
register_function(function, function_name=None, container_uuid=None, description=None, public=False, group=None)¶ Register a function code with the funcX service.
Parameters: - function (Python Function) – The function to be registered for remote execution
- function_name (str) – The entry point (function name) of the function. Default: None
- container_uuid (str) – Container UUID from registration with funcX
- description (str) – Description of the file
- public (bool) – Whether or not the function is publicly accessible. Default = False
- group (str) – A globus group uuid to share this function with
Returns: function uuid – UUID identifier for the registered function
Return type:
-
run(*args, endpoint_id=None, function_id=None, **kwargs)¶ Initiate an invocation
Parameters: - *args (Any) – Args as specified by the function signature
- endpoint_id (uuid str) – Endpoint UUID string. Required
- function_id (uuid str) – Function UUID string. Required
- asynchronous (bool) – Whether or not to run the function asynchronously
Returns: - task_id (str)
- UUID string that identifies the task
-