2019-11-19 20:38:21 -05:00
|
|
|
|
---
|
|
|
|
|
|
id: python
|
|
|
|
|
|
title: Python User Guide
|
|
|
|
|
|
---
|
|
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
Perspective for Python uses the exact same C++ data engine used by the
|
2020-01-15 11:28:08 -05:00
|
|
|
|
[WebAssembly version](https://perspective.finos.org/docs/md/js.html). The
|
2021-09-12 14:08:32 -04:00
|
|
|
|
library consists of many of the same abstractions and API as in JavaScript, as
|
|
|
|
|
|
well as Python-specific data loading support for [NumPy](https://numpy.org/),
|
2019-11-22 09:15:36 -05:00
|
|
|
|
[Pandas](https://pandas.pydata.org/) (and
|
2021-01-03 16:53:18 -08:00
|
|
|
|
[Apache Arrow](https://arrow.apache.org/), as in JavaScript).
|
2019-11-22 09:15:36 -05:00
|
|
|
|
|
|
|
|
|
|
Additionally, `perspective-python` provides a session manager suitable for
|
2019-11-22 14:15:27 -05:00
|
|
|
|
integration into server systems such as
|
|
|
|
|
|
[Tornado websockets](https://www.tornadoweb.org/en/stable/websocket.html), which
|
|
|
|
|
|
allows fully _virtual_ Perspective tables to be interacted with by multiple
|
2021-01-03 16:53:18 -08:00
|
|
|
|
`<perspective-viewer>` in a web browser.
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
|
|
|
|
|
As `<perspective-viewer>` will only consume the data necessary to render the
|
2021-01-03 16:53:18 -08:00
|
|
|
|
current screen, this runtime mode allows _ludicrously-sized_ datasets with
|
2020-01-15 11:28:08 -05:00
|
|
|
|
instant-load after they've been manifest on the server (at the expense of
|
|
|
|
|
|
network latency on UI interaction).
|
|
|
|
|
|
|
|
|
|
|
|
The included `PerspectiveWidget` allows running such a viewer in
|
|
|
|
|
|
[JupyterLab](https://jupyterlab.readthedocs.io/en/stable/) in either server or
|
2019-11-22 09:15:36 -05:00
|
|
|
|
client (via WebAssembly) mode, and the included `PerspectiveTornadoHandler`
|
|
|
|
|
|
makes it simple to extend a Tornado server with virtual Perspective support.
|
|
|
|
|
|
|
|
|
|
|
|
The `perspective` module exports several tools:
|
|
|
|
|
|
|
|
|
|
|
|
- `Table`, the table constructor for Perspective, which implements the `table`
|
2021-01-03 16:53:18 -08:00
|
|
|
|
and `view` API in the same manner as the JavaScript library.
|
2019-11-22 09:15:36 -05:00
|
|
|
|
- `PerspectiveWidget` the JupyterLab widget for interactive visualization.
|
2021-09-12 14:08:32 -04:00
|
|
|
|
- `PerspectiveTornadoHandler`, an integration with
|
|
|
|
|
|
[Tornado](https://www.tornadoweb.org/) that interfaces seamlessly with
|
|
|
|
|
|
`<perspective-viewer>` in JavaScript.
|
2020-05-27 15:23:55 -04:00
|
|
|
|
- `PerspectiveManager` the session manager for a shared server deployment of
|
2021-09-12 14:08:32 -04:00
|
|
|
|
`perspective-python`.
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
This user's guide provides an overview of the most common ways to use
|
|
|
|
|
|
Perspective in Python: the `Table` API, the JupyterLab widget, and the Tornado
|
2020-01-15 11:28:08 -05:00
|
|
|
|
handler.
|
|
|
|
|
|
|
|
|
|
|
|
For an understanding of Perspective's core concepts, see the
|
2021-09-12 14:08:32 -04:00
|
|
|
|
[Table](/docs/md/table.html), [View](/docs/md/view.html), and
|
|
|
|
|
|
[Data Binding](/docs/md/server.html) documentation. For API documentation, see
|
|
|
|
|
|
the [Python API](/docs/obj/perspective-python.html).
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
[More Examples](https://github.com/finos/perspective/tree/master/examples) are
|
|
|
|
|
|
available on GitHub.
|
2020-10-11 06:23:44 -04:00
|
|
|
|
|
|
|
|
|
|
## Installation
|
|
|
|
|
|
|
|
|
|
|
|
`perspective-python` contains full bindings to the Perspective API, a JupyterLab
|
|
|
|
|
|
widget, and a [Tornado](http://www.tornadoweb.org/en/stable/) WebSocket handler
|
|
|
|
|
|
that allows you to host Perspective using server-side Python.
|
|
|
|
|
|
|
|
|
|
|
|
In addition to supporting row/columnar formats of data using `dict` and `list`,
|
|
|
|
|
|
`pandas.DataFrame`, dictionaries of NumPy arrays, NumPy structured arrays, and
|
|
|
|
|
|
NumPy record arrays are all supported in `perspective-python`.
|
|
|
|
|
|
|
|
|
|
|
|
`perspective-python` can be installed from `pip`:
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
|
pip install perspective-python
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### Jupyterlab
|
|
|
|
|
|
|
|
|
|
|
|
`PerspectiveWidget` is a JupyterLab widget that implements the same API as
|
|
|
|
|
|
`<perspective-viewer>`, allowing for fast, intuitive
|
|
|
|
|
|
transformations/visualizations of various data formats within JupyterLab.
|
2021-01-22 20:55:28 -05:00
|
|
|
|
|
|
|
|
|
|
`PerspectiveWidget` is compatible with Jupyterlab 3. To use it, make sure you
|
|
|
|
|
|
have installed `perspective-python` and then install the extension from the
|
|
|
|
|
|
Jupyter lab extension directory:
|
2020-10-11 06:23:44 -04:00
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
|
jupyter labextension install @finos/perspective-jupyterlab
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
If the widget does not display, you might be missing the
|
|
|
|
|
|
[ipywidgets extension](https://ipywidgets.readthedocs.io/en/latest/user_install.html#installing-the-jupyterlab-extension).
|
|
|
|
|
|
Install it from the extension directory:
|
2020-10-11 06:23:44 -04:00
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
|
jupyter labextension install @jupyter-widgets/jupyterlab-manager
|
|
|
|
|
|
```
|
2020-05-27 15:23:55 -04:00
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
## `Table`
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2019-11-22 14:15:27 -05:00
|
|
|
|
A `Table` can be created from a dataset or a schema, the specifics of which are
|
2021-01-03 16:53:18 -08:00
|
|
|
|
[discussed](#loading-data-with-table) in the JavaScript section of the user's
|
2019-11-22 14:15:27 -05:00
|
|
|
|
guide. In Python, however, Perspective supports additional data types that are
|
|
|
|
|
|
commonly used when processing data:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
- `pandas.DataFrame`
|
|
|
|
|
|
- `numpy.ndarray`
|
2019-11-22 09:15:36 -05:00
|
|
|
|
- `bytes` (encoding an Apache Arrow)
|
2020-03-14 16:07:39 -04:00
|
|
|
|
- `objects` (either extracting a repr or via reference)
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2021-01-03 16:53:18 -08:00
|
|
|
|
A `Table` is created in a similar fashion to its JavaScript equivalent:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
from datetime import date, datetime
|
|
|
|
|
|
import numpy as np
|
|
|
|
|
|
import pandas as pd
|
|
|
|
|
|
import perspective
|
|
|
|
|
|
|
|
|
|
|
|
data = pd.DataFrame({
|
|
|
|
|
|
"int": np.arange(100),
|
|
|
|
|
|
"float": [i * 1.5 for i in range(100)],
|
|
|
|
|
|
"bool": [True for i in range(100)],
|
|
|
|
|
|
"date": [date.today() for i in range(100)],
|
|
|
|
|
|
"datetime": [datetime.now() for i in range(100)],
|
|
|
|
|
|
"string": [str(i) for i in range(100)]
|
|
|
|
|
|
})
|
2020-10-11 06:23:44 -04:00
|
|
|
|
|
2019-11-19 20:38:21 -05:00
|
|
|
|
table = perspective.Table(data, index="float")
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
Likewise, a `View` can be created via the `view()` method:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
2022-01-31 00:52:38 -05:00
|
|
|
|
view = table.view(group_by=["float"], filter=[["bool", "==", True]])
|
2019-11-19 20:38:21 -05:00
|
|
|
|
column_data = view.to_dict()
|
|
|
|
|
|
row_data = view.to_records()
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
### Pandas & Numpy Support
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2021-01-03 16:53:18 -08:00
|
|
|
|
Perspective supports dictionaries of one-dimensional `numpy.ndarray`, as well as
|
2019-11-22 14:15:27 -05:00
|
|
|
|
structured arrays and record arrays. When passing in dictionaries of NumPy
|
2019-11-22 09:15:36 -05:00
|
|
|
|
arrays, make sure that your dataset contains only NumPy arrays, and not a
|
|
|
|
|
|
mixture of arrays and Python lists — this will raise an exception. Numpy
|
|
|
|
|
|
structured/record arrays are parsed according to their field name and dtype.
|
|
|
|
|
|
|
2020-05-26 03:47:55 +05:30
|
|
|
|
`Table` can also be constructed from `pandas.DataFrame` and `pandas.Series`
|
2019-11-22 09:15:36 -05:00
|
|
|
|
objects. Because Perspective is designed for applying its own transformations on
|
|
|
|
|
|
top of a flat dataset, dataframes that are passed in will be flattened and have
|
2019-11-22 14:15:27 -05:00
|
|
|
|
its `index` treated as another column (through the
|
|
|
|
|
|
[`reset_index()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reset_index.html)
|
2020-05-27 15:23:55 -04:00
|
|
|
|
method).
|
|
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
If the dataframe does not have an index set, an integer-typed column named
|
|
|
|
|
|
`"index"` is created. If you want to preserve the indexing behavior of the
|
2019-11-22 09:15:36 -05:00
|
|
|
|
dataframe passed into Perspective, simply create the `Table` with
|
|
|
|
|
|
`index="index"` as a keyword argument. This tells Perspective to once again
|
|
|
|
|
|
treat the index as a primary key:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
data.set_index("datetime")
|
|
|
|
|
|
table = perspective.Table(data, index="index")
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
### Schemas & Supported Data Types
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2021-01-03 16:53:18 -08:00
|
|
|
|
Unlike JavaScript, where schemas must be created using string representations of
|
2019-11-19 20:38:21 -05:00
|
|
|
|
their types, `perspective-python` leverages Python's type system for schema
|
2021-09-12 14:08:32 -04:00
|
|
|
|
creation. A schema can be created with the following types:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
- `int` (and `long` in Python 2)
|
|
|
|
|
|
- `float`
|
|
|
|
|
|
- `bool`
|
|
|
|
|
|
- `datetime.date`
|
|
|
|
|
|
- `datetime.datetime`
|
|
|
|
|
|
- `str` (and `unicode` in Python 2)
|
2020-03-14 16:07:39 -04:00
|
|
|
|
- `object`
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2020-05-27 15:23:55 -04:00
|
|
|
|
#### Loading Custom Objects
|
|
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
Custom objects can also be loaded into Perspective by using `object` in the
|
|
|
|
|
|
schema, or implementing `_psp_repr_` to return `object`. Perspective stores a
|
|
|
|
|
|
reference to your object as an unsigned 64-bit integer (e.g. a pointer), and
|
|
|
|
|
|
uses `__repr__` (or `_psp_repr` if implemented) to represent the object.
|
2020-03-14 16:07:39 -04:00
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
You can customize how Perspective extracts data from your objects by
|
|
|
|
|
|
implementing these two methods into your object:
|
2020-05-27 15:23:55 -04:00
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
- `_psp_repr_`: Since `__repr__` can only return strings, this lets you return
|
|
|
|
|
|
other values
|
|
|
|
|
|
- `_psp_dtype_`: Perpspective will look at this to determine how to cast your
|
|
|
|
|
|
objects' repr.
|
2020-03-14 16:07:39 -04:00
|
|
|
|
|
2020-01-15 11:28:08 -05:00
|
|
|
|
#### Time Zone Handling
|
|
|
|
|
|
|
2020-07-29 18:46:36 -04:00
|
|
|
|
- ["Naive"](https://docs.python.org/3/library/datetime.html#aware-and-naive-objects)
|
|
|
|
|
|
datetimes are assumed to be local time.
|
|
|
|
|
|
- ["Aware"](https://docs.python.org/3/library/datetime.html#aware-and-naive-objects)
|
2021-01-03 16:53:18 -08:00
|
|
|
|
datetimes use the time zone specified in the `tzinfo`.
|
2020-05-27 15:23:55 -04:00
|
|
|
|
|
2021-01-03 16:53:18 -08:00
|
|
|
|
All `datetime` columns (regardless of input time zone) are output to the user as
|
2020-07-29 18:46:36 -04:00
|
|
|
|
`datetime.datetime` objects in _local time_ according to the Python runtime.
|
2020-05-27 15:23:55 -04:00
|
|
|
|
|
2021-01-03 16:53:18 -08:00
|
|
|
|
This behavior is consistent with Perspective's behavior in JavaScript. For more
|
2021-09-12 14:08:32 -04:00
|
|
|
|
details, see this in-depth
|
|
|
|
|
|
[explanation](https://github.com/finos/perspective/pull/867) of
|
|
|
|
|
|
`perspective-python` semantics around time zone handling.
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-05-27 15:23:55 -04:00
|
|
|
|
##### Pandas Timestamps
|
|
|
|
|
|
|
2020-07-29 18:46:36 -04:00
|
|
|
|
- Naive `pandas.Timestamp` objects are _always_ treated as UTC times, and will
|
|
|
|
|
|
be converted to local time when output to the user.
|
2021-01-03 16:53:18 -08:00
|
|
|
|
- Aware `pandas.Timestamp` objects use the time zone specified in `tzinfo`. Use
|
2020-07-29 18:46:36 -04:00
|
|
|
|
`tz_localize` or `tz_convert` to provide the `Timestamp` with a time zone.
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2019-11-19 20:38:21 -05:00
|
|
|
|
### Callbacks and Events
|
|
|
|
|
|
|
|
|
|
|
|
`perspective.Table` allows for `on_update` and `on_delete` callbacks to be
|
|
|
|
|
|
set—simply call `on_update` or `on_delete` with a reference to a function or a
|
|
|
|
|
|
lambda without any parameters:
|
|
|
|
|
|
|
|
|
|
|
|
```python
|
2020-05-27 15:23:55 -04:00
|
|
|
|
def update_callback():
|
2019-11-19 20:38:21 -05:00
|
|
|
|
print("Updated!")
|
2020-05-27 15:23:55 -04:00
|
|
|
|
|
|
|
|
|
|
# set the update callback
|
|
|
|
|
|
view.on_update(update_callback)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def delete_callback():
|
|
|
|
|
|
print("Deleted!")
|
|
|
|
|
|
|
|
|
|
|
|
# set the delete callback
|
|
|
|
|
|
view.on_delete(delete_callback)
|
|
|
|
|
|
|
|
|
|
|
|
# set a lambda as a callback
|
|
|
|
|
|
view.on_delete(lambda: print("Deleted x2!"))
|
2019-11-19 20:38:21 -05:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
If the callback is a named reference to a function, it can be removed with
|
2020-05-27 15:23:55 -04:00
|
|
|
|
`remove_update` or `remove_delete`:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
2020-05-27 15:23:55 -04:00
|
|
|
|
view.remove_update(update_callback)
|
|
|
|
|
|
view.remove_delete(delete_callback)
|
2019-11-19 20:38:21 -05:00
|
|
|
|
```
|
|
|
|
|
|
|
2020-10-11 06:23:44 -04:00
|
|
|
|
Callbacks defined with a lambda function cannot be removed, as lambda functions
|
|
|
|
|
|
have no identifier.
|
2020-05-27 15:23:55 -04:00
|
|
|
|
|
2020-01-15 11:28:08 -05:00
|
|
|
|
## `PerspectiveManager`
|
|
|
|
|
|
|
|
|
|
|
|
`PerspectiveManager` offers an interface for hosting multiple
|
2021-09-12 14:08:32 -04:00
|
|
|
|
`perspective.Table` and `perspective.View` instances, extending their interfaces
|
|
|
|
|
|
to operate with the [JavaScript library](/docs/md/js.html) over a websocket
|
|
|
|
|
|
connection. `PerspectiveManager` is required to enable `perspective-python` to
|
2020-01-15 11:28:08 -05:00
|
|
|
|
[operate remotely](/docs/md/js.html#remote-perspective-via-perspective-python-and-tornado)
|
|
|
|
|
|
using a websocket API.
|
|
|
|
|
|
|
2020-09-18 02:17:53 -04:00
|
|
|
|
### Async Mode
|
|
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
By default, `perspective` will run with a synchronous interface. Using the
|
2020-09-18 02:17:53 -04:00
|
|
|
|
`PerspectiveManager.set_loop_callback()` method, `perspective` can be configured
|
|
|
|
|
|
to defer the application of side-effectful calls like `update()` to an event
|
2021-09-12 14:08:32 -04:00
|
|
|
|
loop, such as `tornado.ioloop.IOLoop`. When running in Async mode, Perspective
|
2021-04-05 19:05:15 -04:00
|
|
|
|
will release the GIL for some operations, enabling better parallelism and
|
2021-09-12 14:08:32 -04:00
|
|
|
|
overall better server performance. There are a few important differences when
|
2021-04-05 19:05:15 -04:00
|
|
|
|
running `PerspectiveManager` in this mode:
|
2020-09-18 02:17:53 -04:00
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
- Calls to methods like `update()` will return immediately, and the reciprocal
|
|
|
|
|
|
`on_update()` callbacks will be invoked on an event later scheduled. Calls to
|
|
|
|
|
|
other methods which require an up-to-date object, but will still synchronously
|
|
|
|
|
|
apply the pending update.
|
|
|
|
|
|
- Updates will be _conflated_ when multiple calls to `update()` occur before the
|
|
|
|
|
|
scheduled application. In such cases, you may receive a single `on_update()`
|
|
|
|
|
|
notification for multiple `update()` calls.
|
|
|
|
|
|
- Once `set_loop_callback()` has been called, you may not call Perspective
|
|
|
|
|
|
functions from any other thread.
|
2020-09-18 02:17:53 -04:00
|
|
|
|
|
|
|
|
|
|
For example, using Tornado `IOLoop` you can create a dedicated thread for a
|
|
|
|
|
|
`PerspectiveManager`:
|
|
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
manager = perspective.PerspectiveManager()
|
|
|
|
|
|
|
|
|
|
|
|
def perspective_thread():
|
|
|
|
|
|
loop = tornado.ioloop.IOLoop()
|
|
|
|
|
|
manager.set_loop_callback(loop.add_callback)
|
|
|
|
|
|
loop.start()
|
|
|
|
|
|
|
|
|
|
|
|
thread = threading.Thread(target=perspective_thread)
|
|
|
|
|
|
thread.daemon = True
|
|
|
|
|
|
thread.start()
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2020-01-15 11:28:08 -05:00
|
|
|
|
### Hosting `Table` and `View` instances
|
|
|
|
|
|
|
|
|
|
|
|
`PerspectiveManager` has the ability to "host" `perspective.Table` and
|
2021-09-12 14:08:32 -04:00
|
|
|
|
`perspective.View` instances. Hosted tables/views can have their methods called
|
|
|
|
|
|
from other sources than the Python server, i.e. by a `perspective-viewer`
|
|
|
|
|
|
running in a JavaScript client over the network, interfacing with
|
|
|
|
|
|
`perspective-python` through the websocket API.
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
The server has full control of all hosted `Table` and `View` instances, and can
|
|
|
|
|
|
call any public API method on hosted instances. This makes it extremely easy to
|
|
|
|
|
|
stream data to a hosted `Table` using `.update()`:
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
manager = PerspectiveManager()
|
|
|
|
|
|
table = Table(data)
|
|
|
|
|
|
manager.host_table("data_source", table)
|
|
|
|
|
|
|
|
|
|
|
|
for i in range(10):
|
|
|
|
|
|
# updates continue to propagate automatically
|
|
|
|
|
|
table.update(new_data)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
In situations where clients should only be able to view the table and not modify
|
|
|
|
|
|
it through `update`, `delete`, etc., initialize the `PerspectiveManager` with
|
|
|
|
|
|
`lock=True`, or call the `lock()` method on a manager instance:
|
2020-05-27 15:23:55 -04:00
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
# lock prevents clients from calling methods that may mutate the state
|
|
|
|
|
|
# of the table.
|
|
|
|
|
|
manager = PerspectiveManager(lock=True)
|
|
|
|
|
|
table = Table(data)
|
|
|
|
|
|
manager.host_table("data_source", table)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2020-01-15 11:28:08 -05:00
|
|
|
|
A `PerspectiveManager` instance can host as many `Table`s and `View`s as
|
|
|
|
|
|
necessary, but each `Table` should only be hosted by _one_ `PerspectiveManager`.
|
|
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
To host a `Table` or a `View`, call the corresponding method on an instance of
|
|
|
|
|
|
`PerspectiveManager` with a string name and the instance to be hosted:
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
manager = PerspectiveManager()
|
|
|
|
|
|
table = Table(data)
|
|
|
|
|
|
manager.host_table("data_source", table)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
The `name` provided is important, as it enables Perspective in JavaScript to
|
|
|
|
|
|
look up a `Table` and get a handle to it over the network. This enables several
|
|
|
|
|
|
powerful server/client implementations of Perspective, as explained in the next
|
|
|
|
|
|
section.
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-10-11 06:23:44 -04:00
|
|
|
|
### Client/Server Replicated Mode
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
Using Tornado and
|
|
|
|
|
|
[`PerspectiveTornadoHandler`](/docs/md/python.html#perspectivetornadohandler),
|
2021-01-03 16:53:18 -08:00
|
|
|
|
as well as `Perspective`'s JavaScript library, we can set up "distributed"
|
2021-09-12 14:08:32 -04:00
|
|
|
|
Perspective instances that allows multiple browser `perspective-viewer` clients
|
|
|
|
|
|
to read from a common `perspective-python` server, as in the
|
2020-10-11 06:23:44 -04:00
|
|
|
|
[Tornado Example Project](https://github.com/finos/perspective/tree/master/examples/tornado-python).
|
2020-10-26 21:40:38 -04:00
|
|
|
|
|
|
|
|
|
|
This architecture works by maintaining two `Tables`—one on the server, and one
|
|
|
|
|
|
on the client that mirrors the server's `Table` automatically using `on_update`.
|
|
|
|
|
|
All updates to the table on the server are automatically applied to each client,
|
|
|
|
|
|
which makes this architecture a natural fit for streaming dashboards and other
|
2021-09-12 14:08:32 -04:00
|
|
|
|
distributed use-cases. In conjunction with [Async Mode](#async-mode),
|
|
|
|
|
|
distributed Perspective offers consistently high performance over large numbers
|
|
|
|
|
|
of clients and large datasets.
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
|
|
|
|
|
_*server.py*_
|
|
|
|
|
|
|
|
|
|
|
|
```python
|
2020-03-20 16:26:51 +00:00
|
|
|
|
from perspective import Table, PerspectiveManager, PerspectiveTornadoHandler
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
|
|
|
|
|
# Create an instance of PerspectiveManager, and host a Table
|
|
|
|
|
|
MANAGER = PerspectiveManager()
|
2020-10-26 21:40:38 -04:00
|
|
|
|
TABLE = Table(data)
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
|
|
|
|
|
# The Table is exposed at `localhost:8888/websocket` with the name `data_source`
|
|
|
|
|
|
MANAGER.host_table("data_source", TABLE)
|
|
|
|
|
|
|
|
|
|
|
|
app = tornado.web.Application([
|
|
|
|
|
|
(r"/", MainHandler),
|
2021-01-03 16:53:18 -08:00
|
|
|
|
# create a websocket endpoint that the client JavaScript can access
|
2020-01-15 11:28:08 -05:00
|
|
|
|
(r"/websocket", PerspectiveTornadoHandler, {"manager": MANAGER, "check_origin": True})
|
|
|
|
|
|
])
|
|
|
|
|
|
|
|
|
|
|
|
# Start the Tornado server
|
|
|
|
|
|
app.listen(8888)
|
|
|
|
|
|
loop = tornado.ioloop.IOLoop.current()
|
|
|
|
|
|
loop.start()
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2020-10-26 21:40:38 -04:00
|
|
|
|
Instead of calling `load(server_table)`, create a `View` using `server_table`
|
|
|
|
|
|
and pass that into `viewer.load()`. This will automatically register an
|
|
|
|
|
|
`on_update` callback that synchronizes state between the server and the client.
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
|
|
|
|
|
_*index.html*_
|
|
|
|
|
|
|
|
|
|
|
|
```html
|
|
|
|
|
|
<perspective-viewer id="viewer" editable></perspective-viewer>
|
|
|
|
|
|
|
|
|
|
|
|
<script>
|
2021-04-20 04:28:58 -04:00
|
|
|
|
window.addEventListener("DOMContentLoaded", async function () {
|
2020-05-26 03:47:55 +05:30
|
|
|
|
// Create a client that expects a Perspective server
|
|
|
|
|
|
// to accept connections at the specified URL.
|
|
|
|
|
|
const websocket = perspective.websocket("ws://localhost:8888/websocket");
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-10-26 21:40:38 -04:00
|
|
|
|
// Get a handle to the Table on the server
|
|
|
|
|
|
const server_table = websocket.open_table("data_source_one");
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-10-26 21:40:38 -04:00
|
|
|
|
// Create a new view
|
Add async view() constructor
WIP: await table.view()
clean up server
Fix rest of viewer, refactor perspective/api/server
Fix examples, JS tests, jupyterlab to use async view constructor
Fix examples and JS core tests, TODO: fix websocket tests
Node server returns promise to view
Fix filter test for computed
Fix jupyterlab - use async view
Async table constructor
Fix examples and docs
Fix tests, jupyterlab for async table constructor
Fix engine tests
Fix jupyterlab for remote table
Fix jlab build - use fat build
WIP: fix workspace tests
wait_for_update must be true otherwise the screenshot happens before table is loaded async
refactor jupyter widget
rebase on master, fix docs, tests, benchmark
rebase on master, fix docs, tests, benchmark suite
fix jlab tests
update jlab, webpack plugin to 0.6.0, remove test.run
update jlab, webpack plugin to perspective 0.6.0
remove test.run
async table/view on tornado client
fix all tests
Errors thrown with verbose message from WASM, remove closure compiler
assert that we can load a promise
Skip flaky d3fc tests
rebase master
2020-12-14 18:43:24 -05:00
|
|
|
|
const server_view = await table.view();
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-10-26 21:40:38 -04:00
|
|
|
|
// Create a Table on the client using `perspective.worker()`
|
|
|
|
|
|
const worker = perspective.worker();
|
Add async view() constructor
WIP: await table.view()
clean up server
Fix rest of viewer, refactor perspective/api/server
Fix examples, JS tests, jupyterlab to use async view constructor
Fix examples and JS core tests, TODO: fix websocket tests
Node server returns promise to view
Fix filter test for computed
Fix jupyterlab - use async view
Async table constructor
Fix examples and docs
Fix tests, jupyterlab for async table constructor
Fix engine tests
Fix jupyterlab for remote table
Fix jlab build - use fat build
WIP: fix workspace tests
wait_for_update must be true otherwise the screenshot happens before table is loaded async
refactor jupyter widget
rebase on master, fix docs, tests, benchmark
rebase on master, fix docs, tests, benchmark suite
fix jlab tests
update jlab, webpack plugin to 0.6.0, remove test.run
update jlab, webpack plugin to perspective 0.6.0
remove test.run
async table/view on tornado client
fix all tests
Errors thrown with verbose message from WASM, remove closure compiler
assert that we can load a promise
Skip flaky d3fc tests
rebase master
2020-12-14 18:43:24 -05:00
|
|
|
|
const client_table = await worker.table(view);
|
2020-10-26 21:40:38 -04:00
|
|
|
|
|
|
|
|
|
|
// Load the client table in the `<perspective-viewer>`.
|
|
|
|
|
|
document.getElementById("viewer").load(client_table);
|
2020-05-26 03:47:55 +05:30
|
|
|
|
});
|
2020-01-15 11:28:08 -05:00
|
|
|
|
</script>
|
|
|
|
|
|
```
|
|
|
|
|
|
|
2020-10-26 21:40:38 -04:00
|
|
|
|
For a more complex example that offers distributed editing of the server
|
2021-09-12 14:08:32 -04:00
|
|
|
|
dataset, see
|
|
|
|
|
|
[client_server_editing.html](https://github.com/finos/perspective/blob/master/examples/tornado-python/client_server_editing.html).
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-10-11 06:23:44 -04:00
|
|
|
|
### Server-only Mode
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-10-26 21:40:38 -04:00
|
|
|
|
The server setup is identical to [Distributed Mode](#distributed-mode) above,
|
2021-09-12 14:08:32 -04:00
|
|
|
|
but instead of creating a view, the client calls `load(server_table)`: In
|
|
|
|
|
|
Python, use `PerspectiveManager` and `PerspectiveTornadoHandler` to create a
|
|
|
|
|
|
websocket server that exposes a `Table`. In this example, `table` is a proxy for
|
|
|
|
|
|
the `Table` we created on the server. All API methods are available on
|
|
|
|
|
|
_proxies_, the.g.us calling `view()`, `schema()`, `update()` on `table` will
|
2020-10-11 06:23:44 -04:00
|
|
|
|
pass those operations to the Python `Table`, execute the commands, and return
|
|
|
|
|
|
the result back to Javascript.
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-10-26 21:40:38 -04:00
|
|
|
|
```html
|
|
|
|
|
|
<perspective-viewer id="viewer" editable></perspective-viewer>
|
2020-10-11 06:23:44 -04:00
|
|
|
|
```
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2020-10-11 06:23:44 -04:00
|
|
|
|
```javascript
|
|
|
|
|
|
const websocket = perspective.websocket("ws://localhost:8888/websocket");
|
|
|
|
|
|
const table = websocket.open_table("data_source_one");
|
|
|
|
|
|
document.getElementById("viewer").load(table);
|
2020-10-26 21:40:38 -04:00
|
|
|
|
```
|
2020-01-15 11:28:08 -05:00
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
## `PerspectiveWidget`
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
Building on top of the API provided by `perspective.Table`, the
|
|
|
|
|
|
`PerspectiveWidget` is a JupyterLab plugin that offers the entire functionality
|
|
|
|
|
|
of Perspective within the Jupyter environment. It supports the same API
|
|
|
|
|
|
semantics of `<perspective-viewer>`, along with the additional data types
|
2021-09-12 14:08:32 -04:00
|
|
|
|
supported by `perspective.Table`. `PerspectiveWidget` takes keyword arguments
|
|
|
|
|
|
for the managed `View`; additioanl arguments `index` and `limit` will be passed
|
|
|
|
|
|
to the `Table`. For convenience are the
|
|
|
|
|
|
[`Aggregate`](https://github.com/finos/perspective/blob/master/python/perspective/perspective/core/aggregate.py),
|
2020-10-11 06:23:44 -04:00
|
|
|
|
[`Sort`](https://github.com/finos/perspective/blob/master/python/perspective/perspective/core/sort.py),
|
|
|
|
|
|
and
|
|
|
|
|
|
[`Plugin`](https://github.com/finos/perspective/blob/master/python/perspective/perspective/core/plugin.py)
|
|
|
|
|
|
enums, which can be used as replacements to string values in the API:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
from perspective import PerspectiveWidget, Aggregate, Sort, Plugin
|
|
|
|
|
|
w = perspective.PerspectiveWidget(
|
2020-10-11 06:23:44 -04:00
|
|
|
|
data,
|
2021-09-12 14:08:32 -04:00
|
|
|
|
plugin=Plugin.XBAR,
|
2020-10-11 06:23:44 -04:00
|
|
|
|
aggregates={"datetime": Aggregate.ANY},
|
|
|
|
|
|
sort=[["date", Sort.DESC]]
|
|
|
|
|
|
)
|
2019-11-19 20:38:21 -05:00
|
|
|
|
```
|
|
|
|
|
|
|
2020-01-13 17:27:48 -05:00
|
|
|
|
### Creating a widget
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
A widget is created through the `PerspectiveWidget` constructor, which takes as
|
|
|
|
|
|
its first, required parameter a `perspective.Table`, a dataset, a schema, or
|
2020-10-11 06:23:44 -04:00
|
|
|
|
`None`, which serves as a special value that tells the Widget to defer loading
|
|
|
|
|
|
any data until later. In maintaining consistency with the Javascript API,
|
|
|
|
|
|
Widgets cannot be created with empty dictionaries or lists—`None` should be used
|
2021-09-12 14:08:32 -04:00
|
|
|
|
if the intention is to await data for loading later on. A widget can be
|
2020-10-11 06:23:44 -04:00
|
|
|
|
constructed from a dataset:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
from perspective import PerspectiveWidget, Table
|
2022-01-31 00:52:38 -05:00
|
|
|
|
PerspectiveWidget(data, group_by=["date"])
|
2019-11-19 20:38:21 -05:00
|
|
|
|
```
|
|
|
|
|
|
|
2020-10-11 06:23:44 -04:00
|
|
|
|
.. or a schema:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
2020-10-11 06:23:44 -04:00
|
|
|
|
PerspectiveWidget({"a": int, "b": str})
|
2019-11-19 20:38:21 -05:00
|
|
|
|
```
|
|
|
|
|
|
|
2020-10-11 06:23:44 -04:00
|
|
|
|
.. or an instance of a `perspective.Table`:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
table = Table(data)
|
2020-10-11 06:23:44 -04:00
|
|
|
|
PerspectiveWidget(table)
|
2019-11-19 20:38:21 -05:00
|
|
|
|
```
|
|
|
|
|
|
|
2020-10-11 06:23:44 -04:00
|
|
|
|
.. or `None`:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```python
|
2020-10-11 06:23:44 -04:00
|
|
|
|
PerspectiveWidget(None)
|
2019-11-19 20:38:21 -05:00
|
|
|
|
```
|
|
|
|
|
|
|
2021-04-12 18:45:16 -04:00
|
|
|
|
## `PerspectiveRenderer`
|
|
|
|
|
|
|
2021-09-12 14:08:32 -04:00
|
|
|
|
Perspective also exposes a JS-only `mimerender-extension`. This lets you view
|
|
|
|
|
|
`csv`, `json`, and `arrow` files directly from the file browser. You can see
|
|
|
|
|
|
this by right clicking one of these files and `Open With->CSVPerspective` (or
|
|
|
|
|
|
`JSONPerspective` or `ArrowPerspective`). Perspective will also install itself
|
|
|
|
|
|
as the default handler for opening `.arrow` files.
|
2021-04-12 18:45:16 -04:00
|
|
|
|
|
2019-11-19 20:38:21 -05:00
|
|
|
|
## `PerspectiveTornadoHandler`
|
|
|
|
|
|
|
|
|
|
|
|
Perspective ships with a pre-built Tornado handler that makes integration with
|
|
|
|
|
|
`tornado.websockets` extremely easy. This allows you to run an instance of
|
|
|
|
|
|
`Perspective` on a server using Python, open a websocket to a `Table`, and
|
2021-09-12 14:08:32 -04:00
|
|
|
|
access the `Table` in JavaScript and through `<perspective-viewer>`. All
|
2020-10-11 06:23:44 -04:00
|
|
|
|
instructions sent to the `Table` are processed in Python, which executes the
|
|
|
|
|
|
commands, and returns its output through the websocket back to Javascript.
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
### Python setup
|
|
|
|
|
|
|
|
|
|
|
|
To use the handler, we need to first have an instance of a `Table` and a
|
2021-01-03 16:53:18 -08:00
|
|
|
|
`PerspectiveManager`. The manager acts as the interface between the JavaScript
|
|
|
|
|
|
and Python layers, implementing a JSON API that allows the two Perspective
|
2019-11-19 20:38:21 -05:00
|
|
|
|
runtimes to communicate.
|
|
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
MANAGER = PerspectiveManager()
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Once the manager has been created, create a `Table` instance and call
|
|
|
|
|
|
`host_table` on the manager with a name, passing through a reference to the
|
|
|
|
|
|
`Table` you just created. `host_table()` registers the Table with the manager
|
|
|
|
|
|
and allows the manager to send instructions to the Table.
|
|
|
|
|
|
|
|
|
|
|
|
The name that you host the table under is important—it acts as a unique accessor
|
2021-01-03 16:53:18 -08:00
|
|
|
|
on the JavaScript side, which will look for a Table hosted at the websocket with
|
2019-11-19 20:38:21 -05:00
|
|
|
|
the name you specify.
|
|
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
TABLE = Table(data)
|
|
|
|
|
|
MANAGER.host_table("data_source_one", TABLE)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
After the manager and table setup is complete, create a websocket endpoint and
|
2021-01-03 16:53:18 -08:00
|
|
|
|
provide it a reference to `PerspectiveTornadoHandler`. You must provide the
|
2019-11-19 20:38:21 -05:00
|
|
|
|
configuration object in the route tuple, and it must contain `manager`, which is
|
|
|
|
|
|
a reference to the `PerspectiveManager` you just created.
|
|
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
app = tornado.web.Application([
|
|
|
|
|
|
(r"/", MainHandler),
|
2021-01-03 16:53:18 -08:00
|
|
|
|
# create a websocket endpoint that the client JavaScript can access
|
2019-11-19 20:38:21 -05:00
|
|
|
|
(r"/websocket", PerspectiveTornadoHandler, {"manager": MANAGER, "check_origin": True})
|
|
|
|
|
|
])
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Optionally, the configuration object can also include `check_origin`, a boolean
|
|
|
|
|
|
that determines whether the websocket accepts requests from origins other than
|
|
|
|
|
|
where the server is hosted. See
|
|
|
|
|
|
[Tornado docs](https://www.tornadoweb.org/en/stable/websocket.html#tornado.websocket.WebSocketHandler.check_origin)
|
|
|
|
|
|
for more details.
|
|
|
|
|
|
|
2021-01-03 16:53:18 -08:00
|
|
|
|
### JavaScript setup
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
Once the server is up and running, you can access the Table you just hosted
|
2021-01-03 16:53:18 -08:00
|
|
|
|
using `perspective.websocket` and `open_table()`. First, create a client that
|
2019-11-22 09:15:36 -05:00
|
|
|
|
expects a Perspective server to accept connections at the specified URL:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
|
|
|
|
|
```javascript
|
|
|
|
|
|
const websocket = perspective.websocket("ws://localhost:8888/websocket");
|
2019-11-22 09:15:36 -05:00
|
|
|
|
```
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
Next open the `Table` we created on the server by name:
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
```javascript
|
2019-11-19 20:38:21 -05:00
|
|
|
|
const table = websocket.open_table("data_source_one");
|
2019-11-22 09:15:36 -05:00
|
|
|
|
```
|
2019-11-19 20:38:21 -05:00
|
|
|
|
|
2019-11-22 14:15:27 -05:00
|
|
|
|
`table` is a proxy for the `Table` we created on the server. All operations that
|
2021-01-03 16:53:18 -08:00
|
|
|
|
are possible through the JavaScript API are possible on the Python API as well,
|
|
|
|
|
|
thus calling `view()`, `schema()`, `update()` etc. on `const table` will pass
|
2019-11-22 14:15:27 -05:00
|
|
|
|
those operations to the Python `Table`, execute the commands, and return the
|
2021-01-03 16:53:18 -08:00
|
|
|
|
result back to JavaScript. Similarly, providing this `table` to a
|
2019-11-22 09:15:36 -05:00
|
|
|
|
`<perspective-viewer>` instance will allow virtual rendering:
|
|
|
|
|
|
|
|
|
|
|
|
```javascript
|
2019-11-19 20:38:21 -05:00
|
|
|
|
viewer.load(table);
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
`perspective.websocket` expects a Websocket URL where it will send instructions.
|
|
|
|
|
|
When `open_table` is called, the name to a hosted Table is passed through, and a
|
2021-09-12 14:08:32 -04:00
|
|
|
|
request is sent through the socket to fetch the Table. No actual `Table`
|
|
|
|
|
|
instance is passed inbetween the runtimes; all instructions are proxied through
|
2019-11-19 20:38:21 -05:00
|
|
|
|
websockets.
|
|
|
|
|
|
|
2019-11-22 09:15:36 -05:00
|
|
|
|
This provides for great flexibility — while `Perspective.js` is full of
|
|
|
|
|
|
features, browser WebAssembly runtimes currently have some performance
|
|
|
|
|
|
restrictions on memory and CPU feature utilization, and the architecture in
|
|
|
|
|
|
general suffers when the dataset itself is too large to download to the client
|
|
|
|
|
|
in full.
|
|
|
|
|
|
|
2022-01-05 01:01:33 -06:00
|
|
|
|
The Python runtime does not suffer from memory limitations, utilizes Apache
|
|
|
|
|
|
Arrow internal threadpools for threading and parallel processing, and generates
|
|
|
|
|
|
architecture optimized code, which currently makes it more suitable as a
|
|
|
|
|
|
server-side runtime than `node.js`.
|
