2019-12-09 15:58:26 -08:00
# Unity ML-Agents Python Low Level API
2018-09-04 21:16:20 -07:00
The `mlagents` Python package contains two components: a low level API which
2019-12-14 14:56:46 -08:00
allows you to interact directly with a Unity Environment (`mlagents_envs` ) and
2018-09-04 21:16:20 -07:00
an entry point to train (`mlagents-learn` ) which allows you to train agents in
Unity Environments using our implementations of reinforcement learning or
2020-03-09 12:04:24 -07:00
imitation learning. This document describes how to use the `mlagents_envs` API.
For information on using `mlagents-learn` , see [here ](Training-ML-Agents.md ).
2022-02-02 19:32:23 -05:00
For Python Low Level API documentation, see [here ](Python-LLAPI-Documentation.md ).
2018-09-04 21:16:20 -07:00
2020-04-30 09:58:28 -07:00
The Python Low Level API can be used to interact directly with your Unity
learning environment. As such, it can serve as the basis for developing and
evaluating new learning algorithms.
2019-12-09 15:58:26 -08:00
2019-12-14 14:56:46 -08:00
## mlagents_envs
2018-09-04 21:16:20 -07:00
2020-04-30 09:58:28 -07:00
The ML-Agents Toolkit Low Level API is a Python API for controlling the
simulation loop of an environment or game built with Unity. This API is used by
the training algorithms inside the ML-Agent Toolkit, but you can also write your
own Python programs using this API.
2018-09-04 21:16:20 -07:00
The key objects in the Python API include:
- **UnityEnvironment** — the main interface between the Unity application and
your code. Use UnityEnvironment to start and control a simulation or training
session.
2020-04-06 12:50:32 -07:00
- **BehaviorName** - is a string that identifies a behavior in the simulation.
- **AgentId** - is an `int` that serves as unique identifier for Agents in the
simulation.
- **DecisionSteps** — contains the data from Agents belonging to the same
"Behavior" in the simulation, such as observations and rewards. Only Agents
that requested a decision since the last call to `env.step()` are in the
DecisionSteps object.
- **TerminalSteps** — contains the data from Agents belonging to the same
"Behavior" in the simulation, such as observations and rewards. Only Agents
whose episode ended since the last call to `env.step()` are in the
TerminalSteps object.
- **BehaviorSpec** — describes the shape of the observation data inside
DecisionSteps and TerminalSteps as well as the expected action shapes.
2020-04-30 09:58:28 -07:00
These classes are all defined in the
[base_env ](../ml-agents-envs/mlagents_envs/base_env.py ) script.
2018-09-04 21:16:20 -07:00
2020-04-30 09:58:28 -07:00
An Agent "Behavior" is a group of Agents identified by a `BehaviorName` that
share the same observations and action types (described in their
`BehaviorSpec` ). You can think about Agent Behavior as a group of agents that
will share the same policy. All Agents with the same behavior have the same goal
and reward signals.
2018-09-04 21:16:20 -07:00
To communicate with an Agent in a Unity environment from a Python program, the
2019-12-09 15:58:26 -08:00
Agent in the simulation must have `Behavior Parameters` set to communicate. You
must set the `Behavior Type` to `Default` and give it a `Behavior Name` .
2018-09-04 21:16:20 -07:00
_Notice: Currently communication between Unity and Python takes place over an
open socket without authentication. As such, please make sure that the network
where training takes place is secure. This will be addressed in a future
release._
2019-12-09 15:58:26 -08:00
## Loading a Unity Environment
2018-09-04 21:16:20 -07:00
Python-side communication happens through `UnityEnvironment` which is located in
2020-04-30 09:58:28 -07:00
[`environment.py` ](../ml-agents-envs/mlagents_envs/environment.py ). To load a
Unity environment from a built binary file, put the file in the same directory
as `envs` . For example, if the filename of your Unity environment is `3DBall` ,
in python, run:
2018-09-04 21:16:20 -07:00
``` python
2019-12-14 14:56:46 -08:00
from mlagents_envs . environment import UnityEnvironment
2020-05-08 10:42:56 -07:00
# This is a non-blocking call that only loads the environment.
2020-03-23 20:48:28 -07:00
env = UnityEnvironment ( file_name = " 3DBall " , seed = 1 , side_channels = [ ] )
2020-07-16 15:56:33 -07:00
# Start interacting with the environment.
2020-05-08 10:42:56 -07:00
env . reset ( )
2020-07-08 02:39:17 +09:00
behavior_names = env . behavior_specs . keys ( )
2020-05-08 10:42:56 -07:00
. . .
2018-09-04 21:16:20 -07:00
```
2020-05-08 10:42:56 -07:00
**NOTE: ** Please read [Interacting with a Unity Environment ](#interacting-with-a-unity-environment )
to read more about how you can interact with the Unity environment from Python.
2018-09-04 21:16:20 -07:00
- `file_name` is the name of the environment binary (located in the root
directory of the python project).
- `worker_id` indicates which port to use for communication with the
environment. For use in parallel training regimes such as A3C.
- `seed` indicates the seed to use when generating random numbers during the
2021-12-02 09:07:09 -08:00
training process. In environments which are stochastic, setting the seed
2020-04-30 09:58:28 -07:00
enables reproducible experimentation by ensuring that the environment and
trainers utilize the same random seed.
2019-12-09 15:58:26 -08:00
- `side_channels` provides a way to exchange data with the Unity simulation that
is not related to the reinforcement learning loop. For example: configurations
2021-09-16 14:35:01 -07:00
or properties. More on them in the [Side Channels ](Custom-SideChannels.md ) doc.
2018-09-04 21:16:20 -07:00
If you want to directly interact with the Editor, you need to use
2020-04-30 09:58:28 -07:00
`file_name=None` , then press the **Play ** button in the Editor when the message
_ "Start training by pressing the Play button in the Unity Editor" _ is displayed
on the screen
2018-09-04 21:16:20 -07:00
### Interacting with a Unity Environment
2019-12-09 15:58:26 -08:00
#### The BaseEnv interface
A `BaseEnv` has the following methods:
2020-04-30 09:58:28 -07:00
- **Reset : `env.reset()` ** Sends a signal to reset the environment. Returns
None.
- **Step : `env.step()` ** Sends a signal to step the environment. Returns None.
Note that a "step" for Python does not correspond to either Unity `Update` nor
`FixedUpdate` . When `step()` or `reset()` is called, the Unity simulation will
move forward until an Agent in the simulation needs a input from Python to
act.
- **Close : `env.close()` ** Sends a shutdown signal to the environment and
terminates the communication.
2020-05-12 15:24:40 -07:00
- **Behavior Specs : `env.behavior_specs` ** Returns a Mapping of
`BehaviorName` to `BehaviorSpec` objects (read only).
2020-12-09 17:12:55 -05:00
A `BehaviorSpec` contains the observation shapes and the
`ActionSpec` (which defines the action shape). Note that
2020-05-12 15:24:40 -07:00
the `BehaviorSpec` for a specific group is fixed throughout the simulation.
The number of entries in the Mapping can change over time in the simulation
if new Agent behaviors are created in the simulation.
2020-04-30 09:58:28 -07:00
- **Get Steps : `env.get_steps(behavior_name: str)` ** Returns a tuple
`DecisionSteps, TerminalSteps` corresponding to the behavior_name given as
input. The `DecisionSteps` contains information about the state of the agents
**that need an action this step ** and have the behavior behavior_name. The
`TerminalSteps` contains information about the state of the agents **whose
episode ended** and have the behavior behavior_name. Both `DecisionSteps` and
`TerminalSteps` contain information such as the observations, the rewards and
the agent identifiers. `DecisionSteps` also contains action masks for the next
action while `TerminalSteps` contains the reason for termination (did the
Agent reach its maximum step and was interrupted). The data is in `np.array`
of which the first dimension is always the number of agents note that the
number of agents is not guaranteed to remain constant during the simulation
and it is not unusual to have either `DecisionSteps` or `TerminalSteps`
contain no Agents at all.
2020-12-09 17:12:55 -05:00
- **Set Actions :`env.set_actions(behavior_name: str, action: ActionTuple)` ** Sets
the actions for a whole agent group. `action` is an `ActionTuple` , which
is made up of a 2D `np.array` of `dtype=np.int32` for discrete actions, and
`dtype=np.float32` for continuous actions. The first dimension of `np.array`
in the tuple is the number of agents that requested a decision since the
last call to `env.step()` . The second dimension is the number of discrete or
continuous actions for the corresponding array.
2020-04-30 09:58:28 -07:00
- **Set Action for Agent :
2020-12-09 17:12:55 -05:00
`env.set_action_for_agent(agent_group: str, agent_id: int, action: ActionTuple)` **
2020-04-30 09:58:28 -07:00
Sets the action for a specific Agent in an agent group. `agent_group` is the
name of the group the Agent belongs to and `agent_id` is the integer
2020-12-09 17:12:55 -05:00
identifier of the Agent. `action` is an `ActionTuple` as described above.
2020-04-30 09:58:28 -07:00
**Note: ** If no action is provided for an agent group between two calls to
2020-12-14 14:53:02 -08:00
`env.step()` then the default action will be all zeros.
2019-12-09 15:58:26 -08:00
2020-04-06 12:50:32 -07:00
#### DecisionSteps and DecisionStep
2020-04-30 09:58:28 -07:00
`DecisionSteps` (with `s` ) contains information about a whole batch of Agents
while `DecisionStep` (no `s` ) only contains information about a single Agent.
2020-04-06 12:50:32 -07:00
A `DecisionSteps` has the following fields :
2019-12-09 15:58:26 -08:00
2020-04-30 09:58:28 -07:00
- `obs` is a list of numpy arrays observations collected by the group of agent.
The first dimension of the array corresponds to the batch size of the group
(number of agents requesting a decision since the last call to `env.step()` ).
- `reward` is a float vector of length batch size. Corresponds to the rewards
collected by each agent since the last simulation step.
- `agent_id` is an int vector of length batch size containing unique identifier
for the corresponding Agent. This is used to track Agents across simulation
steps.
2020-12-09 17:12:55 -05:00
- `action_mask` is an optional list of two dimensional arrays of booleans which is only
available when using multi-discrete actions. Each array corresponds to an
2020-04-30 09:58:28 -07:00
action branch. The first dimension of each array is the batch size and the
second contains a mask for each action of the branch. If true, the action is
not available for the agent during this simulation step.
2019-12-09 15:58:26 -08:00
It also has the two following methods:
2020-04-30 09:58:28 -07:00
- `len(DecisionSteps)` Returns the number of agents requesting a decision since
the last call to `env.step()` .
- `DecisionSteps[agent_id]` Returns a `DecisionStep` for the Agent with the
`agent_id` unique identifier.
2019-12-09 15:58:26 -08:00
2020-04-06 12:50:32 -07:00
A `DecisionStep` has the following fields:
2019-12-09 15:58:26 -08:00
2020-04-30 09:58:28 -07:00
- `obs` is a list of numpy arrays observations collected by the agent. (Each
array has one less dimension than the arrays in `DecisionSteps` )
- `reward` is a float. Corresponds to the rewards collected by the agent since
the last simulation step.
- `agent_id` is an int and an unique identifier for the corresponding Agent.
2020-12-09 17:12:55 -05:00
- `action_mask` is an optional list of one dimensional arrays of booleans which is only
available when using multi-discrete actions. Each array corresponds to an
2020-04-30 09:58:28 -07:00
action branch. Each array contains a mask for each action of the branch. If
true, the action is not available for the agent during this simulation step.
2019-12-09 15:58:26 -08:00
2020-04-06 12:50:32 -07:00
#### TerminalSteps and TerminalStep
2020-04-30 09:58:28 -07:00
Similarly to `DecisionSteps` and `DecisionStep` , `TerminalSteps` (with `s` )
contains information about a whole batch of Agents while `TerminalStep` (no `s` )
only contains information about a single Agent.
2019-12-09 15:58:26 -08:00
2020-04-06 12:50:32 -07:00
A `TerminalSteps` has the following fields :
2020-04-30 09:58:28 -07:00
- `obs` is a list of numpy arrays observations collected by the group of agent.
The first dimension of the array corresponds to the batch size of the group
(number of agents requesting a decision since the last call to `env.step()` ).
- `reward` is a float vector of length batch size. Corresponds to the rewards
collected by each agent since the last simulation step.
- `agent_id` is an int vector of length batch size containing unique identifier
for the corresponding Agent. This is used to track Agents across simulation
steps.
2020-05-01 14:37:34 -07:00
- `interrupted` is an array of booleans of length batch size. Is true if the
associated Agent was interrupted since the last decision step. For example,
if the Agent reached the maximum number of steps for the episode.
2020-04-06 12:50:32 -07:00
It also has the two following methods:
2020-04-30 09:58:28 -07:00
- `len(TerminalSteps)` Returns the number of agents requesting a decision since
the last call to `env.step()` .
- `TerminalSteps[agent_id]` Returns a `TerminalStep` for the Agent with the
`agent_id` unique identifier.
2020-04-06 12:50:32 -07:00
A `TerminalStep` has the following fields:
2020-04-30 09:58:28 -07:00
- `obs` is a list of numpy arrays observations collected by the agent. (Each
array has one less dimension than the arrays in `TerminalSteps` )
- `reward` is a float. Corresponds to the rewards collected by the agent since
the last simulation step.
- `agent_id` is an int and an unique identifier for the corresponding Agent.
2020-05-01 14:37:34 -07:00
- `interrupted` is a bool. Is true if the Agent was interrupted since the last
decision step. For example, if the Agent reached the maximum number of steps for
the episode.
2020-04-06 12:50:32 -07:00
#### BehaviorSpec
A `BehaviorSpec` has the following fields :
2019-12-09 15:58:26 -08:00
2021-01-07 11:25:59 -08:00
- `observation_specs` is a List of `ObservationSpec` objects : Each `ObservationSpec`
2020-12-22 09:01:30 -08:00
corresponds to an observation's properties: `shape` is a tuple of ints that
corresponds to the shape of the observation (without the number of agents dimension).
`dimension_property` is a tuple of flags containing extra information about how the
2021-01-07 11:25:59 -08:00
data should be processed in the corresponding dimension. `observation_type` is an enum
corresponding to what type of observation is generating the data (i.e., default, goal,
etc). Note that the `ObservationSpec` have the same ordering as the ordering of observations
2021-01-05 14:33:04 -08:00
in the DecisionSteps, DecisionStep, TerminalSteps and TerminalStep.
2020-12-09 17:12:55 -05:00
- `action_spec` is an `ActionSpec` namedtuple that defines the number and types
of actions for the Agent.
An `ActionSpec` has the following fields and properties:
- `continuous_size` is the number of floats that constitute the continuous actions.
- `discrete_size` is the number of branches (the number of independent actions) that
constitute the multi-discrete actions.
- `discrete_branches` is a Tuple of ints. Each int corresponds to the number of
different options for each branch of the action. For example:
In a game direction input (no movement, left, right) and
2020-04-30 09:58:28 -07:00
jump input (no jump, jump) there will be two branches (direction and jump),
2020-12-09 17:12:55 -05:00
the first one with 3 options and the second with 2 options. (`discrete_size = 2`
2020-04-30 09:58:28 -07:00
and `discrete_action_branches = (3,2,)` )
2018-09-04 21:16:20 -07:00
2020-12-09 17:12:55 -05:00
2020-03-09 12:04:24 -07:00
### Communicating additional information with the Environment
2020-04-06 12:50:32 -07:00
2020-04-30 09:58:28 -07:00
In addition to the means of communicating between Unity and python described
above, we also provide methods for sharing agent-agnostic information. These
additional methods are referred to as side channels. ML-Agents includes two
ready-made side channels, described below. It is also possible to create custom
side channels to communicate any additional data between a Unity environment and
Python. Instructions for creating custom side channels can be found
[here ](Custom-SideChannels.md ).
2019-12-03 16:40:01 -08:00
2020-04-30 09:58:28 -07:00
Side channels exist as separate classes which are instantiated, and then passed
as list to the `side_channels` argument of the constructor of the
`UnityEnvironment` class.
2020-03-09 12:04:24 -07:00
``` python
channel = MyChannel ( )
env = UnityEnvironment ( side_channels = [ channel ] )
```
2020-04-30 09:58:28 -07:00
**Note ** : A side channel will only send/receive messages when `env.step` or
`env.reset()` is called.
2019-12-03 16:40:01 -08:00
#### EngineConfigurationChannel
2020-03-09 12:04:24 -07:00
2020-04-30 09:58:28 -07:00
The `EngineConfiguration` side channel allows you to modify the time-scale,
resolution, and graphics quality of the environment. This can be useful for
adjusting the environment to perform better during training, or be more
interpretable during inference.
2019-12-03 16:40:01 -08:00
2020-04-30 09:58:28 -07:00
`EngineConfigurationChannel` has two methods :
2019-12-03 16:40:01 -08:00
2020-04-30 09:58:28 -07:00
- `set_configuration_parameters` which takes the following arguments:
- `width` : Defines the width of the display. (Must be set alongside height)
- `height` : Defines the height of the display. (Must be set alongside width)
- `quality_level` : Defines the quality level of the simulation.
- `time_scale` : Defines the multiplier for the deltatime in the simulation. If
set to a higher value, time will pass faster in the simulation but the
physics may perform unpredictably.
- `target_frame_rate` : Instructs simulation to try to render at a specified
frame rate.
- `capture_frame_rate` Instructs the simulation to consider time between
updates to always be constant, regardless of the actual frame rate.
- `set_configuration` with argument config which is an `EngineConfig` NamedTuple
object.
For example, the following code would adjust the time-scale of the simulation to
be 2x realtime.
2020-03-09 12:04:24 -07:00
2019-12-03 16:40:01 -08:00
``` python
2019-12-14 14:56:46 -08:00
from mlagents_envs . environment import UnityEnvironment
from mlagents_envs . side_channel . engine_configuration_channel import EngineConfigurationChannel
2019-12-03 16:40:01 -08:00
channel = EngineConfigurationChannel ( )
2020-03-23 20:48:28 -07:00
env = UnityEnvironment ( side_channels = [ channel ] )
2019-12-03 16:40:01 -08:00
channel . set_configuration_parameters ( time_scale = 2.0 )
i = env . reset ( )
. . .
```
2020-04-23 16:46:55 -07:00
#### EnvironmentParameters
2020-04-30 09:58:28 -07:00
The `EnvironmentParameters` will allow you to get and set pre-defined numerical
values in the environment. This can be useful for adjusting environment-specific
settings, or for reading non-agent related information from the environment. You
can call `get_property` and `set_property` on the side channel to read and write
properties.
2020-03-09 12:04:24 -07:00
2020-04-23 16:46:55 -07:00
`EnvironmentParametersChannel` has one methods:
2019-12-03 16:40:01 -08:00
2020-04-30 09:58:28 -07:00
- `set_float_parameter` Sets a float parameter in the Unity Environment.
- key: The string identifier of the property.
- value: The float value of the property.
2020-02-10 16:45:28 -08:00
2019-12-03 16:40:01 -08:00
``` python
2019-12-14 14:56:46 -08:00
from mlagents_envs . environment import UnityEnvironment
2020-04-23 16:46:55 -07:00
from mlagents_envs . side_channel . environment_parameters_channel import EnvironmentParametersChannel
2019-12-03 16:40:01 -08:00
2020-04-23 16:46:55 -07:00
channel = EnvironmentParametersChannel ( )
2019-12-03 16:40:01 -08:00
2020-03-23 20:48:28 -07:00
env = UnityEnvironment ( side_channels = [ channel ] )
2019-12-03 16:40:01 -08:00
2020-04-23 16:46:55 -07:00
channel . set_float_parameter ( " parameter_1 " , 2.0 )
2019-12-03 16:40:01 -08:00
i = env . reset ( )
. . .
```
2020-04-30 09:58:28 -07:00
Once a property has been modified in Python, you can access it in C# after the
next call to `step` as follows:
2019-12-03 16:40:01 -08:00
``` csharp
2020-04-23 16:46:55 -07:00
var envParameters = Academy . Instance . EnvironmentParameters ;
float property1 = envParameters . GetWithDefault ( "parameter_1" , 0.0f ) ;
2019-12-03 16:40:01 -08:00
```
2020-02-10 16:45:28 -08:00
2020-03-09 12:04:24 -07:00
#### Custom side channels
2020-02-10 16:45:28 -08:00
2020-04-30 09:58:28 -07:00
For information on how to make custom side channels for sending additional data
types, see the documentation [here ](Custom-SideChannels.md ).
