funcX Client

The funcX Client is the programmatic interface to FuncX from Python. The client provides a simple and intuitive interface to:

  1. Register functions
  2. Register containers and execution environments
  3. Launch registered function against endpoints
  4. Check the status of launched functions and
  5. Retrieve outputs from functions

The following shows an example of creating a client.

from funcx.sdk.client import FuncXClient
fxc = FuncXClient()

Instantiating a client will start an authentication process where you will be asked to authenticate with Globus Auth. We require every interaction with the funcX Web Service to be authenticated using a Bearer token, this allows funcX to enforce access control on both functions and endpoints. As part of the authentication process we request access to your identity information (to retrieve your email address) and Globus Groups management access. We require Groups access in order to facilitate sharing. Users can share functions with others by associating a Globus Group with the function. If a group is set, the funcX Web Service uses Groups Management access to determine whether a user is a member of the specified group.

Note

The funcX Web Service internally caches function, endpoint, and access control lookups. Caches are based on user authentication tokens. To refresh the caches you can re-authenticate your client with force_login=True.

Registering Functions

You can register a Python function with funcX via register_function(). Function registration will serialize the function body and transmit it to the funcX Web Service. Once a function is registered with the service it will return a UUID that can be used to invoke the function.

Note

You must import any dependencies required by the function inside the function body.

You can associate a Globus Group with a function to enable sharing. If set, the Web Service will check if the invoking user is a member of the group. This is achieved by setting group=<globus_group_id> when registering a function.

You can also set a function to be publicly accssible by setting public=True when registering the function.

Running Functions

You can invoke a function using the UUID returned when registering the function. The client’s run() function requires you to specify the function_id and endpoint_id. In addition, you can pass *args and **kwargs to the run function and they will be used when invoking your function. We serialize all inputs and outputs when running a function.

The result of your function’s invocation can be retrieved using the client’s get_result() function. This will either raise an exception if the task is still pending, or give you back the deserialized result of your invocation.

Note

If your function’s execution raises an exception, get_result() will reraise it.

To minimize the overhead of communicating with the funcX Web Service we provide batch request and status capabilities. Documentation on batch requests can be found below.

Client Throttling

In order to avoid accidentally DoS’ing the funcX Web Service we place soft throttling restrictions on the funcX client. There are two key throttling measures: firstly, we limit the number of requests a client can make to the Web Service to 5 every 5 seconds, and secondly, we limit the size of inputs and outputs transmitted through the service to 2MB.

Batching requests and status can help reduce the number of requests made to the Web Service. In addition, the limit on the number of requests made to the Web Service can be removed by setting throttling_enabled to False.

fxc = FuncXClient()
fxc.throttling_enabled = False

FuncXClient Reference:

class funcx.FuncXClient(http_timeout=None, funcx_home='~/.funcx', force_login=False, fx_authorizer=None, search_authorizer=None, openid_authorizer=None, funcx_service_address='https://api2.funcx.org/v2', check_endpoint_version=False, asynchronous=False, loop=None, results_ws_uri='wss://api.funcx.org/ws/v2/', use_offprocess_checker=True, **kwargs)

Main class for interacting with the funcX service

Holds helper operations for performing common tasks with the funcX service.

add_to_whitelist(endpoint_id, function_ids)

Adds the function to the endpoint’s whitelist

Parameters:
  • endpoint_id (str) – The uuid of the endpoint
  • function_ids (list) – A list of function id’s to be whitelisted
Returns:

The response of the request
Return type:

json
batch_run(batch)

Initiate a batch of tasks to funcX

Parameters:batch (a Batch object) –
Returns:task_ids
Return type:a list of UUID strings that identify the tasks
create_batch(task_group_id=None)

Create a Batch instance to handle batch submission in funcX

Parameters:task_group_id (str) – Override the session wide session_task_group_id with a different task_group_id for this batch. If task_group_id is not specified, it will default to using the client’s session_task_group_id
Returns:Status block containing “status” key.
Return type:Batch instance
delete_from_whitelist(endpoint_id, function_ids)

List the endpoint’s whitelist

Parameters:
  • endpoint_id (str) – The uuid of the endpoint
  • function_ids (list) – A list of function id’s to be whitelisted
Returns:

The response of the request
Return type:

json
get_batch_result(task_id_list)

Request status for a batch of task_ids

get_container(container_uuid, container_type)

Get the details of a container for staging it locally.

Parameters:
  • container_uuid (str) – UUID of the container in question
  • container_type (str) – The type of containers that will be used (Singularity, Shifter, Docker)
Returns:

The details of the containers to deploy
Return type:

dict
get_containers(name, description=None)

Register a DLHub endpoint with the funcX service and get the containers to launch.

Parameters:
  • name (str) – Name of the endpoint
  • description (str) – Description of the endpoint
Returns:

The port to connect to and a list of containers
Return type:

int
get_endpoint_status(endpoint_uuid)

Get the status reports for an endpoint.

Parameters:endpoint_uuid (str) – UUID of the endpoint in question
Returns:The details of the endpoint’s stats
Return type:dict
get_result(task_id)

Get the result of a funcX task

Parameters:task_id (str) – UUID of the task
Returns:Result obj
Return type:If task completed
Raises:Exception obj: Exception due to which the task failed
get_task(task_id)

Get a funcX task.

Parameters:task_id (str) – UUID of the task
Returns:Task block containing “status” key.
Return type:dict
get_whitelist(endpoint_id)

List the endpoint’s whitelist

Parameters:endpoint_id (str) – The uuid of the endpoint
Returns:The response of the request
Return type:json
logout()

Remove credentials from your local system

map_run(*args, endpoint_id=None, function_id=None, asynchronous=False, **kwargs)

Initiate an invocation

Parameters:
  • *args (Any) – Args as specified by the function signature
  • endpoint_id (uuid str) – Endpoint UUID string. Required
  • function_id (uuid str) – Function UUID string. Required
  • asynchronous (bool) – Whether or not to run the function asynchronously
Returns:

  • task_id (str)
  • UUID string that identifies the task
register_container(location, container_type, name='', description='')

Register a container with the funcX service.

Parameters:
  • location (str) – The location of the container (e.g., its docker url). Required
  • container_type (str) – The type of containers that will be used (Singularity, Shifter, Docker). Required
  • name (str) – A name for the container. Default = ‘’
  • description (str) – A description to associate with the container. Default = ‘’
Returns:

The id of the container
Return type:

str
register_endpoint(name, endpoint_uuid, metadata=None, endpoint_version=None)

Register an endpoint with the funcX service.

Parameters:
  • name (str) – Name of the endpoint
  • endpoint_uuid (str) – The uuid of the endpoint
  • metadata (dict) – endpoint metadata, see default_config example
  • endpoint_version (str) – Version string to be passed to the webService as a compatibility check
Returns:

{‘endpoint_id’ : <>,

’address’ : <>, ‘client_ports’: <>}
Return type:

A dict
register_function(function, function_name=None, container_uuid=None, description=None, public=False, group=None, searchable=True)

Register a function code with the funcX service.

Parameters:
  • function (Python Function) – The function to be registered for remote execution
  • function_name (str) – The entry point (function name) of the function. Default: None
  • container_uuid (str) – Container UUID from registration with funcX
  • description (str) – Description of the file
  • public (bool) – Whether or not the function is publicly accessible. Default = False
  • group (str) – A globus group uuid to share this function with
  • searchable (bool) – If true, the function will be indexed into globus search with the appropriate permissions
Returns:

function uuid – UUID identifier for the registered function
Return type:

str
run(*args, endpoint_id=None, function_id=None, **kwargs)

Initiate an invocation

Parameters:
  • *args (Any) – Args as specified by the function signature
  • endpoint_id (uuid str) – Endpoint UUID string. Required
  • function_id (uuid str) – Function UUID string. Required
  • asynchronous (bool) – Whether or not to run the function asynchronously
Returns:

  • task_id (str)
  • UUID string that identifies the task if asynchronous is False
  • funcX Task (asyncio.Task)
  • A future that will eventually resolve into the function’s result if
  • asynchronous is True
search_endpoint(q, scope='all', owner_id=None)
Parameters:
  • q –
  • scope (str) – Can be one of {‘all’, ‘my-endpoints’, ‘shared-with-me’}
  • owner_id – should be urn like f”urn:globus:auth:identity:{owner_uuid}”
search_function(q, offset=0, limit=10, advanced=False)

Search for function via the funcX service

Parameters:
  • q (str) – free-form query string
  • offset (int) – offset into total results
  • limit (int) – max number of results to return
  • advanced (bool) – allows elastic-search like syntax in query string
Returns:
Return type:

FunctionSearchResults
update_table(return_msg, task_id)

Parses the return message from the service and updates the internal func_table

Parameters:
  • return_msg (str) – Return message received from the funcx service
  • task_id (str) – task id string
version_check()

Check this client version meets the service’s minimum supported version.