Morph MORPH
SIGN IN SIGN UP
ray-project / ray UNCLAIMED

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

0 0 104 Python
Normal View History Raw
data tests (#40658) - Add java build option (one of the test required java build) - Move starting mongodb to mongo_test setup Signed-off-by: can <can@anyscale.com>
2023-10-26 19:55:41 -07:00
# syntax=docker/dockerfile:1.3-labs
[ci] updating data ci tests to py310 (#58378) upgrading data ci tests to py3.10 postmerge build: https://buildkite.com/ray-project/postmerge/builds/14192 --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Lonnie Liu <95255098+aslonnie@users.noreply.github.com>
2025-11-03 20:12:38 -08:00
ARG DOCKER_IMAGE_BASE_BUILD=cr.ray.io/rayproject/oss-ci-base_ml-py3.10
[civ2] move data tests to CI v2 Move data tests to CI v2 Signed-off-by: can <can@anyscale.com>
2023-09-06 16:23:47 -07:00
FROM $DOCKER_IMAGE_BASE_BUILD
data tests (#40658) - Add java build option (one of the test required java build) - Move starting mongodb to mongo_test setup Signed-off-by: can <can@anyscale.com>
2023-10-26 19:55:41 -07:00
ARG RAY_CI_JAVA_BUILD=
[ci][deps] Generating dependency sets for Data CI images (#61532) Adding depsets for data ci tests: - Data base ci depset (to expand ray requirements with data & data test requirements) - Relaxed data ci depset (to remove pyarrow and other packages to be expanded for different versions of pyarrow) - Data pyarrow latest depset (test against pyarrow v23) - Data pyarrow v9 depset (test against pyarrow v9) - Data mongo depset (test against pyarrow mongo) - Data pyarrow nightly (test against pyarrow nightly releases) Data ci images: - installing depsets on image based on new Docker arg IMAGE_TYPE which maps to the depset lock file name Updating tests - adding `env = {"TF_USE_LEGACY_KERAS": "1"},` to tensorflow related tests. Tensorflow upgrade to 2.20.0 pulls in keras 3.0 (which ray is not compatible with currently). Adding TF_USE_LEGACY_KERAS=1 forces tensorflow to use keras 2.0 via tf-keras (ref [here](https://keras.io/getting_started/)) - Skip lance tests for older pylance versions (<=0.3.19) due to incompatible lance API changes with pyarrow v9.0.0 Postmerge build: https://buildkite.com/ray-project/postmerge/builds/16606 --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
2026-04-02 15:28:50 -07:00
ARG IMAGE_TYPE=base
ARG PYTHON=3.10
ARG PYTHON_DEPSET=python/deplocks/ci/data-$IMAGE_TYPE-ci_depset_py$PYTHON.lock
Fix data docker file (#39780) ARROW_VERSION is currently declared before the FROM keyword so it is not read in properly into the docker image. As a result, all data tests are currently running pyarrow=12 (which is part of requirement files). Fix by moving the arg to below the FROM keyword Signed-off-by: can <can@anyscale.com>
2023-09-21 04:02:46 -07:00
[ci][deps] Generating dependency sets for Data CI images (#61532) Adding depsets for data ci tests: - Data base ci depset (to expand ray requirements with data & data test requirements) - Relaxed data ci depset (to remove pyarrow and other packages to be expanded for different versions of pyarrow) - Data pyarrow latest depset (test against pyarrow v23) - Data pyarrow v9 depset (test against pyarrow v9) - Data mongo depset (test against pyarrow mongo) - Data pyarrow nightly (test against pyarrow nightly releases) Data ci images: - installing depsets on image based on new Docker arg IMAGE_TYPE which maps to the depset lock file name Updating tests - adding `env = {"TF_USE_LEGACY_KERAS": "1"},` to tensorflow related tests. Tensorflow upgrade to 2.20.0 pulls in keras 3.0 (which ray is not compatible with currently). Adding TF_USE_LEGACY_KERAS=1 forces tensorflow to use keras 2.0 via tf-keras (ref [here](https://keras.io/getting_started/)) - Skip lance tests for older pylance versions (<=0.3.19) due to incompatible lance API changes with pyarrow v9.0.0 Postmerge build: https://buildkite.com/ray-project/postmerge/builds/16606 --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
2026-04-02 15:28:50 -07:00
COPY $PYTHON_DEPSET /home/ray/python_depset.lock
[civ2] move data tests to CI v2 Move data tests to CI v2 Signed-off-by: can <can@anyscale.com>
2023-09-06 16:23:47 -07:00
[ci][deps] Generating dependency sets for Data CI images (#61532) Adding depsets for data ci tests: - Data base ci depset (to expand ray requirements with data & data test requirements) - Relaxed data ci depset (to remove pyarrow and other packages to be expanded for different versions of pyarrow) - Data pyarrow latest depset (test against pyarrow v23) - Data pyarrow v9 depset (test against pyarrow v9) - Data mongo depset (test against pyarrow mongo) - Data pyarrow nightly (test against pyarrow nightly releases) Data ci images: - installing depsets on image based on new Docker arg IMAGE_TYPE which maps to the depset lock file name Updating tests - adding `env = {"TF_USE_LEGACY_KERAS": "1"},` to tensorflow related tests. Tensorflow upgrade to 2.20.0 pulls in keras 3.0 (which ray is not compatible with currently). Adding TF_USE_LEGACY_KERAS=1 forces tensorflow to use keras 2.0 via tf-keras (ref [here](https://keras.io/getting_started/)) - Skip lance tests for older pylance versions (<=0.3.19) due to incompatible lance API changes with pyarrow v9.0.0 Postmerge build: https://buildkite.com/ray-project/postmerge/builds/16606 --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
2026-04-02 15:28:50 -07:00
SHELL ["/bin/bash", "-ice"]
[civ2] move data tests to CI v2 Move data tests to CI v2 Signed-off-by: can <can@anyscale.com>
2023-09-06 16:23:47 -07:00
data tests (#40658) - Add java build option (one of the test required java build) - Move starting mongodb to mongo_test setup Signed-off-by: can <can@anyscale.com>
2023-10-26 19:55:41 -07:00
RUN <<EOF
#!/bin/bash
[ci][7up/3] initial dependency upgrades (#42104) Upgrade ray dependencies to versions that are available on both python 3.9 and python 3.11. ray 3.8 is no longer build-able with these new versions. All tests passed except for two ml tests which the team has agreed to deprecate them w00t w00t. Signed-off-by: can <can@anyscale.com>
2024-03-14 13:37:42 -07:00
set -ex
[ci][deps] Generating dependency sets for Data CI images (#61532) Adding depsets for data ci tests: - Data base ci depset (to expand ray requirements with data & data test requirements) - Relaxed data ci depset (to remove pyarrow and other packages to be expanded for different versions of pyarrow) - Data pyarrow latest depset (test against pyarrow v23) - Data pyarrow v9 depset (test against pyarrow v9) - Data mongo depset (test against pyarrow mongo) - Data pyarrow nightly (test against pyarrow nightly releases) Data ci images: - installing depsets on image based on new Docker arg IMAGE_TYPE which maps to the depset lock file name Updating tests - adding `env = {"TF_USE_LEGACY_KERAS": "1"},` to tensorflow related tests. Tensorflow upgrade to 2.20.0 pulls in keras 3.0 (which ray is not compatible with currently). Adding TF_USE_LEGACY_KERAS=1 forces tensorflow to use keras 2.0 via tf-keras (ref [here](https://keras.io/getting_started/)) - Skip lance tests for older pylance versions (<=0.3.19) due to incompatible lance API changes with pyarrow v9.0.0 Postmerge build: https://buildkite.com/ray-project/postmerge/builds/16606 --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
2026-04-02 15:28:50 -07:00
uv pip install -r /home/ray/python_depset.lock --no-deps --system --index-strategy unsafe-best-match
data tests (#40658) - Add java build option (one of the test required java build) - Move starting mongodb to mongo_test setup Signed-off-by: can <can@anyscale.com>
2023-10-26 19:55:41 -07:00
[data] Install mongodb-org from official repo, start mongod directly in tests (#61657) Switches from the Ubuntu universe `mongodb` package (only available on focal and providing an old 2.x version) to mongodb-org 8.0 from the official MongoDB apt repo. The focal codename is hardcoded; the follow-on Ubuntu 22.04 upgrade PR updates it to jammy. Replaces the SysV `service mongodb start/stop` wrapper in test_mongo.py with a direct `mongod --dbpath` invocation, since mongodb-org 8.0 ships without a SysV init script. The fixture creates a temp dbpath, polls for readiness, and terminates the process on teardown. Signed-off-by: andrew andrew@anyscale.com Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 10:19:04 -07:00
curl -fsSL https://pgp.mongodb.com/server-8.0.asc | \
sudo gpg -o /usr/share/keyrings/mongodb-server-8.0.gpg --dearmor
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] \
[ci] upgrade CI test containers from Ubuntu 20.04 to 22.04 (#61533) Upgrades CI test container base images from Ubuntu 20.04 to 22.04 (jammy), including the mandatory clang-12 to clang-14 toolchain upgrade available in 22.04. - focal → jammy for all base images and wanda configs (base.test, base.gpu, base.cu128, forge wanda) - clang-12 → clang-14 in base.test, base.gpu, base.build Dockerfiles and install-llvm-binaries.sh - libncurses5 → libncurses6 - libgl1-mesa-glx → libgl1 (dropped in 22.04) - netcat → netcat-openbsd (22.04 no longer provides the bare netcat metapackage) - mongodb-org apt source: focal → jammy - Java 8 update-alternatives: 22.04 defaults to Java 11, pin to Java 8 where needed - test_dashboard_profiler.py: skip py-spy --native on Linux (known issue, see ray-project/ray#30566) - onnx 1.15.0 → 1.16.0 (compatibility with newer environment) - Simplified Podman doc (20.04 workaround removed) Signed-off-by: andrew <andrew@anyscale.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 17:42:25 -07:00
https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/8.0 multiverse" | \
[data] Install mongodb-org from official repo, start mongod directly in tests (#61657) Switches from the Ubuntu universe `mongodb` package (only available on focal and providing an old 2.x version) to mongodb-org 8.0 from the official MongoDB apt repo. The focal codename is hardcoded; the follow-on Ubuntu 22.04 upgrade PR updates it to jammy. Replaces the SysV `service mongodb start/stop` wrapper in test_mongo.py with a direct `mongod --dbpath` invocation, since mongodb-org 8.0 ships without a SysV init script. The fixture creates a temp dbpath, polls for readiness, and terminates the process on teardown. Signed-off-by: andrew andrew@anyscale.com Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 10:19:04 -07:00
sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
data tests (#40658) - Add java build option (one of the test required java build) - Move starting mongodb to mongo_test setup Signed-off-by: can <can@anyscale.com>
2023-10-26 19:55:41 -07:00
if [[ $RAY_CI_JAVA_BUILD == 1 ]]; then
Fix lint (#50061) Signed-off-by: dentiny <dentinyhao@gmail.com>
2025-01-24 13:43:38 -08:00
# These packages increase the image size quite a bit, so we only install them
data tests (#40658) - Add java build option (one of the test required java build) - Move starting mongodb to mongo_test setup Signed-off-by: can <can@anyscale.com>
2023-10-26 19:55:41 -07:00
# as needed.
sudo apt-get install -y -qq maven openjdk-8-jre openjdk-8-jdk
[ci] upgrade CI test containers from Ubuntu 20.04 to 22.04 (#61533) Upgrades CI test container base images from Ubuntu 20.04 to 22.04 (jammy), including the mandatory clang-12 to clang-14 toolchain upgrade available in 22.04. - focal → jammy for all base images and wanda configs (base.test, base.gpu, base.cu128, forge wanda) - clang-12 → clang-14 in base.test, base.gpu, base.build Dockerfiles and install-llvm-binaries.sh - libncurses5 → libncurses6 - libgl1-mesa-glx → libgl1 (dropped in 22.04) - netcat → netcat-openbsd (22.04 no longer provides the bare netcat metapackage) - mongodb-org apt source: focal → jammy - Java 8 update-alternatives: 22.04 defaults to Java 11, pin to Java 8 where needed - test_dashboard_profiler.py: skip py-spy --native on Linux (known issue, see ray-project/ray#30566) - onnx 1.15.0 → 1.16.0 (compatibility with newer environment) - Simplified Podman doc (20.04 workaround removed) Signed-off-by: andrew <andrew@anyscale.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 17:42:25 -07:00
# Ensure Java 8 is the default; Ubuntu 22.04 defaults to Java 11 which
# breaks Spark's reflective access to DirectByteBuffer.
if [[ "$(dpkg --print-architecture)" == "arm64" ]]; then
sudo update-alternatives --set java /usr/lib/jvm/java-8-openjdk-arm64/jre/bin/java
else
sudo update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
fi
data tests (#40658) - Add java build option (one of the test required java build) - Move starting mongodb to mongo_test setup Signed-off-by: can <can@anyscale.com>
2023-10-26 19:55:41 -07:00
fi
EOF
Reference in New Issue Copy Permalink