Morph MORPH
SIGN IN SIGN UP
apache / arrow UNCLAIMED

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

0 0 2 C++
Normal View History Raw
ARROW-988 [JS] Add entry to Travis CI matrix Author: Brian Hulette <brian.hulette@ccri.com> Closes #677 from TheNeuralBit/js_travis and squashes the following commits: 3d5b54b5 [Brian Hulette] Revert "Testing a failure" a84b3641 [Brian Hulette] Testing a failure 70310b68 [Brian Hulette] add flatbuffers download e0c1100d [Brian Hulette] Add travis config and scripts for JS
2017-05-13 15:53:31 -04:00
#!/usr/bin/env bash
ARROW-7101: [CI] Refactor docker-compose setup and use it with GitHub Actions ## Projecting ideas from ursabot ### Parametric docker images The images are better parameterized now, meaning that we can build more variant of the same service. Couple of examples: ```console UBUNTU=16.04 docker-compose build ubuntu-cpp ARCH=arm64v8 UBUNTU=18.04 docker-compose build ubuntu-cpp PYTHON=3.6 docker-compose build conda-python ARCH=arm32v7 PYTHON=3.6 PANDAS=0.25 docker-compose build conda-python-pandas ``` Each variant has it's own docker image following a string naming schema: `{org}/{arch}-{platform}-{platform-version}[[-{variant}-{variant-version}]..]:latest` ### Use *_build.sh and *_test.sh for each job The docker images provide the environment, and each language backend usually should implement two scripts, a `build.sh` and a `test.sh`. This way dependent build like the docker python, r or c glib are able to reuse the build script of the ancestor without running its tests. With small enough scripts, if the environment is properly set up even the non-docker builds should be reproducible locally. GitHub Actions support bash scripts across all three platforms, so we can reuse the same `*_build.sh` and `*_test.sh` scripts to execute the builds either in docker, on the CI or locally. ## Using GitHub Actions for running the builds Regardless of the CI we're going to choose, the isolation constraint of different platforms requires some sort of virtualisation. Currently linux (and windows, but I have not tried it yet) has lightweight containerisation, so we should keep the linux builds isolated in docker containers. The rest of the platforms (windows and macOS) should be executed on the CI system. GitHub Actions support all three major platforms, linux, windows and macOS. I've added cross platform builds for a couple of languages, like Rust, and Go, the rest are work in progress. ### Workflow A workflow should define all builds of a language, mostly because the path filters can be defined on workflow level. For example the python builds should be triggered if either a cpp/** or a python/** file changes which can be covered in the same workflow file. ## Feature parity with the current builds Reaching feature parity with all of the builds below is not a goal for this PR, the difficult ones should at least have a tracking JIRA ticket. ### Travis-CI - [x] **Lint, Release tests**: - `Lint / C++, Python, R, Rust, Docker, RAT` - `Dev / Source Release` - [x] **C++ unit tests w/ conda-forge toolchain, coverage**: without coverage - `C++ / AMD64 Conda C++` - [x] **Python 3.6 unit tests, conda-forge toolchain, coverage**: without coverage - `Python / AMD64 Conda Python 3.6` - [x] **[OS X] C++ w/ Xcode 9.3**: - `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3 - [x] **[OS X] Python w/ Xcode 9.3**: - `Python / AMD64 MacOS 10.14 Python 3`: with Xcode 10.3 - [x] **Java OpenJDK8 and OpenJDK11**: - `Java / AMD64 Debian Java JDK 8 Maven 3.5.2` - `Java / AMD64 Debian Java JDK 11 Maven 3.6.2` - [x] **Protocol / Flight Integration Tests**: - `Dev / Protocol Test` - [x] **NodeJS**: without running lint and coverage - `NodeJS / AMD64 Debian NodeJS 11` - [x] **C++ & GLib & Ruby w/ gcc 5.4**: - `C++ / AMD64 Debian 10 C++`: with GCC 8.3 - `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4 - `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4 - `C GLib / AMD64 Ubuntu 18.04 C GLib` - `Ruby / AMD64 Ubuntu 18.04 Ruby` - [x] **[OS X] C++ & GLib & Ruby w/ XCode 10.2 & Homebrew** - `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3 - `C GLib / AMD64 MacOS 10.14 C Glib`: with Xcode 10.3 - `Ruby / AMD64 MacOS 10.14 Ruby`: with Xcode 10.3 - [x] **Go**: without coverage - `Go / AMD64 Debian Go 1.12` - [x] **R (with and without libarrow)**: - `R / AMD64 Conda R 3.6`: with libarrow - `R / AMD64 Ubuntu 18.04 R 3.6` with libarrow ### Appveyor - ~JOB=Build, GENERATOR=Ninja, CONFIGURATION=Release, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017~ - ~JOB=Toolchain, GENERATOR=Ninja, CONFIGURATION=Release, ARROW_S3=ON, ARROW_BUILD_FLIGHT=ON, ARROW_BUILD_GANDIVA=ON~ - ~JOB=Build_Debug, GENERATOR=Ninja, CONFIGURATION=Debug~ - ~JOB=MinGW32, MINGW_ARCH=i686, MINGW_PACKAGE_PREFIX=mingw-w64-i686, MINGW_PREFIX=c:\msys64\mingw32, MSYSTEM=MINGW32, USE_CLCACHE=false~ - ~JOB=MinGW64, MINGW_ARCH=x86_64, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64, MINGW_PREFIX=c:\msys64\mingw64, MSYSTEM=MINGW64, USE_CLCACHE=false~ - [x] **JOB=Rust, TARGET=x86_64-pc-windows-msvc, USE_CLCACHE=false**: - `Rust / AMD64 Windows 2019 Rust nightly-2019-09-25` - [x] **JOB=C#, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017, USE_CLCACHE=false** - `C# / AMD64 Windows 2019 C# 2.2.103` - [x] **JOB=Go, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 ...**: - `Go / AMD64 Windows 2019 Go 1.12` - ~JOB=R with libarrow, USE_CLCACHE=false, TEST_R_WITH_ARROW=TRUE, RWINLIB_LOCAL=%APPVEYOR_BUILD_FOLDER%\libarrow.zip~ ### Github Actions - [x] **Windows MSVC C++ / Build (Visual Studio 16 2019)**: - `C++ / AMD64 Windows 2019 C++`: without tests - [x] **Windows MSVC C++ / Build (Visual Studio 15 2017)**: - `C++ / AMD64 Windows 2016 C++`: without tests - [x] **Linux docker-compose / Test (C++ w/ clang-7 & system packages)**: all have llvm for gandiva but the compiler is set to gcc - `C++ / AMD64 Debian 10 C++`: with GCC 8.3 - `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4 - `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4 - [x] **Linux docker-compose / Test (Rust)**: without rustfmt - `Rust / AMD64 Debian Rust nightly-2019-09-25` - [x] **Linux docker-compose / Test (Lint, Release tests)**: - `Lint / C++, Python, R, Rust, Docker, RAT` - `Dev / Source Release` ### Nightly Crossbow tests The packaging builds are out of the scope if this PR, but the nightly **dockerized test** task are in. Nightly tests: - [x] docker-r - [x] docker-r-conda - [x] docker-r-sanitizer - [x] docker-rust - [x] docker-cpp - [x] docker-cpp-cmake32 - [x] docker-cpp-release - [x] docker-cpp-static-only - [x] docker-c_glib - [x] docker-go - [x] docker-python-2.7 - [x] docker-python-3.6 - [x] docker-python-3.7 - [x] docker-python-2.7-nopandas - [x] docker-python-3.6-nopandas - [x] docker-java - [x] docker-js - [x] docker-docs - [x] docker-lint - [x] docker-iwyu: included in the lint - [x] docker-clang-format: included in the lint - [x] docker-pandas-master - [x] docker-dask-integration - [x] docker-hdfs-integration - [x] docker-spark-integration - [x] docker-turbodbc-integration # TODOs left: - [x] Fix the Apidoc generation for c_glib - [x] Fix the JNI test for Gandiva and ORC - [x] Test that crossbow tests are passing - ~Optionally restore the travis configuration to incrementally decommission old builds~ ## Follow-up JIRAs: - [Archery] Consider porting the docker tool of ursabot to archery - [Archery] Consider to use archery with or instead of the pre-commit hooks - [Archery] Create a wrapper script in archery for docker compose in order to run the containers with the host's user and group - [C++] GCC 5.4.0 has a compile errors, reproduce with UBUNTU=16.04 docker-compose run ubuntu-cpp - [C++][CI] Test the ported fuzzit integration image - [C++][CI] Turn off unnecessary features in the integration tests (spark/turbodbc/dask/hdfs) - [C++][CI] Revisit ASAN UBSAN settings in every C++ based image - [CI] Consider re-adding the removed debian testing image is removed - [Go][CI] Pre-install the go dependencies in the dockerfile using go get - [JS][CI] Pre-install the JS dependencies in the dockerfile - [Rust][CI] Pre-install the rust dependencies in the dockerfile - [Java][CI] Pre-install the java dependencies in the dockerfile - [Ruby][CI] Pre-install the ruby dependencies in the dockerfile and remove it from the test script - [C#][CI] Pre-install the C# dependencies in the dockerfile - [R][CI] Fix the r-sanitizer build https://issues.apache.org/jira/browse/ARROW-6957 - [GLIB][MacOS] Fail to execute lua examples (fails to load 'lgi.corelgilua51' despite that lgi is installed) - [C++][CMake] Automatically set ARROW_GANDIVA_PC_CXX_FLAGS for conda and OSX sdk (see cpp_build.sh) - [C++][CI] Hiveserver2 instegration test fails to connect to impala container - [CI][Spark] Support specific Spark version in the integration tet including latest - [JS][CI] Move nodejs linting from js_build.sh to archery - [Python][CI] create a docker image for python ASV benchmarks and fix the script - [CI] Find a short but related prefix for the env vars used for the docker-compose file to prevent collisions - [C#] the docker container fails to run because of the ubuntu host versions, see https://github.com/dotnet/core/issues/3509 - [C++][Windows] Enable more features on the windows GHA build - [Doc] document docker-compose usage in the developer sphinx guide - [CI][C++] Add .ccache to the docker-compose mounts - [Archery][CI] Refactor the ci/scripts to a sourceable bash functions or to archery directly - [C++][CI] Use scripts/util_coredump.sh to show automatic backtraces - [C++] Fix the hanging C++ tests in Windows 2019 - [CI] Ask INFRA to set up the DOCKERHUB_* secrets for GitHub actions - [C++][CI] Running Gandiva tests fails on Fedora: Reproduce with: `docker-compose run -e ARROW_GANDIVA=ON fedora-cpp` ``` Running gandiva-internals-test, redirecting output into /build/cpp/build/test-logs/gandiva-internals-test.txt (attempt 1/1) 1364 : CommandLine Error: Option 'x86-experimental-vector-widening-legalization' registered more than once! 1365 LLVM ERROR: inconsistency in registered CommandLine options 1366 /build/cpp/src/gandiva ``` - [JS][CI] NodeJS build fails on Github Actions Windows node ``` > NODE_NO_WARNINGS=1 gulp build # 'NODE_NO_WARNINGS' is not recognized as an internal or external command, # operable program or batch file. # npm ERR! code ELIFECYCLE # npm ERR! errno 1 # npm ERR! apache-arrow@1.0.0-SNAPSHOT build: `NODE_NO_WARNINGS=1 gulp build` # npm ERR! Exit status 1 # npm ERR! # npm ERR! Failed at the apache-arrow@1.0.0-SNAPSHOT build script. # npm ERR! This is probably not a problem with npm. There is likely additional logging output above. ``` Closes #5589 from kszucs/docker-refactor and squashes the following commits: 5105d12e6 <Krisztián Szűcs> Rename pull-request folder to dev_cron e9e9a7eec <Krisztián Szűcs> Use underscores for naming the workflow files a92c99d03 <Krisztián Szűcs> Disable hanging C++ tests on windows f158c89b5 <Krisztián Szűcs> Attempt to push from apache/arrow master; Don't push from crossbow tasks 0e1d470a1 <Krisztián Szűcs> Turn off ORC on macOS C++ test due to link error 258db5cff <Krisztián Szűcs> Only push docker images from apache/arrow repository acdfcf086 <Krisztián Szűcs> Remove ORC from the brewfile 5102b85b1 <Krisztián Szűcs> Fix nodeJS workflow 032d6a388 <Krisztián Szűcs> Turn off 2 python builds 7f15b97a8 <Krisztián Szűcs> Filter branches 48b8d128a <Krisztián Szűcs> Fix workflows 36ad9d297 <Krisztián Szűcs> Disable builds 0f603af0c <Krisztián Szűcs> master only and cron workflows 28cc2d78d <Krisztián Szűcs> Rename Java JNI workflow bcd8af7b7 <Krisztián Szűcs> Port the remaining travis utility scripts ed5688154 <Krisztián Szűcs> Usage comments; recommend installing pandas from the docs because of its removal from conda_env_python 3c8c023ce <Krisztián Szűcs> Use Arch in volumes; some comments; remove conda version 'latest' from the images 771b023a8 <Krisztián Szűcs> Cleanup files; separate JNI builds 97ff8a122 <Krisztián Szűcs> Push docker images only from master dc00b4297 <Krisztián Szűcs> Enable path filters e0e2e1f46 <Krisztián Szűcs> Fix pandas master build 3814e0828 <Krisztián Szűcs> Fix manylinux volumes c18edda70 <Krisztián Szűcs> Add CentOS version to the manylinux image names c8b9dd6b1 <Krisztián Szűcs> Missing --pyargs argument for the python test command 33e646981 <Krisztián Szűcs> Turn off gandiva and flight for the HDFS test b9c547889 <Krisztián Szűcs> Refactor docker-compose file and use it with github actions. Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2019-11-12 11:07:48 +01:00
#
ARROW-1288: Fix many license headers to use proper ASF one Author: Wes McKinney <wes.mckinney@twosigma.com> Closes #900 from wesm/fix-license-headers and squashes the following commits: 73941bf2 [Wes McKinney] Fix many license headers to use proper ASF one
2017-07-27 14:18:45 -04:00
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
ARROW-988 [JS] Add entry to Travis CI matrix Author: Brian Hulette <brian.hulette@ccri.com> Closes #677 from TheNeuralBit/js_travis and squashes the following commits: 3d5b54b5 [Brian Hulette] Revert "Testing a failure" a84b3641 [Brian Hulette] Testing a failure 70310b68 [Brian Hulette] add flatbuffers download e0c1100d [Brian Hulette] Add travis config and scripts for JS
2017-05-13 15:53:31 -04:00
#
# http://www.apache.org/licenses/LICENSE-2.0
#
ARROW-1288: Fix many license headers to use proper ASF one Author: Wes McKinney <wes.mckinney@twosigma.com> Closes #900 from wesm/fix-license-headers and squashes the following commits: 73941bf2 [Wes McKinney] Fix many license headers to use proper ASF one
2017-07-27 14:18:45 -04:00
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
ARROW-988 [JS] Add entry to Travis CI matrix Author: Brian Hulette <brian.hulette@ccri.com> Closes #677 from TheNeuralBit/js_travis and squashes the following commits: 3d5b54b5 [Brian Hulette] Revert "Testing a failure" a84b3641 [Brian Hulette] Testing a failure 70310b68 [Brian Hulette] add flatbuffers download e0c1100d [Brian Hulette] Add travis config and scripts for JS
2017-05-13 15:53:31 -04:00
set -ex
ARROW-7101: [CI] Refactor docker-compose setup and use it with GitHub Actions ## Projecting ideas from ursabot ### Parametric docker images The images are better parameterized now, meaning that we can build more variant of the same service. Couple of examples: ```console UBUNTU=16.04 docker-compose build ubuntu-cpp ARCH=arm64v8 UBUNTU=18.04 docker-compose build ubuntu-cpp PYTHON=3.6 docker-compose build conda-python ARCH=arm32v7 PYTHON=3.6 PANDAS=0.25 docker-compose build conda-python-pandas ``` Each variant has it's own docker image following a string naming schema: `{org}/{arch}-{platform}-{platform-version}[[-{variant}-{variant-version}]..]:latest` ### Use *_build.sh and *_test.sh for each job The docker images provide the environment, and each language backend usually should implement two scripts, a `build.sh` and a `test.sh`. This way dependent build like the docker python, r or c glib are able to reuse the build script of the ancestor without running its tests. With small enough scripts, if the environment is properly set up even the non-docker builds should be reproducible locally. GitHub Actions support bash scripts across all three platforms, so we can reuse the same `*_build.sh` and `*_test.sh` scripts to execute the builds either in docker, on the CI or locally. ## Using GitHub Actions for running the builds Regardless of the CI we're going to choose, the isolation constraint of different platforms requires some sort of virtualisation. Currently linux (and windows, but I have not tried it yet) has lightweight containerisation, so we should keep the linux builds isolated in docker containers. The rest of the platforms (windows and macOS) should be executed on the CI system. GitHub Actions support all three major platforms, linux, windows and macOS. I've added cross platform builds for a couple of languages, like Rust, and Go, the rest are work in progress. ### Workflow A workflow should define all builds of a language, mostly because the path filters can be defined on workflow level. For example the python builds should be triggered if either a cpp/** or a python/** file changes which can be covered in the same workflow file. ## Feature parity with the current builds Reaching feature parity with all of the builds below is not a goal for this PR, the difficult ones should at least have a tracking JIRA ticket. ### Travis-CI - [x] **Lint, Release tests**: - `Lint / C++, Python, R, Rust, Docker, RAT` - `Dev / Source Release` - [x] **C++ unit tests w/ conda-forge toolchain, coverage**: without coverage - `C++ / AMD64 Conda C++` - [x] **Python 3.6 unit tests, conda-forge toolchain, coverage**: without coverage - `Python / AMD64 Conda Python 3.6` - [x] **[OS X] C++ w/ Xcode 9.3**: - `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3 - [x] **[OS X] Python w/ Xcode 9.3**: - `Python / AMD64 MacOS 10.14 Python 3`: with Xcode 10.3 - [x] **Java OpenJDK8 and OpenJDK11**: - `Java / AMD64 Debian Java JDK 8 Maven 3.5.2` - `Java / AMD64 Debian Java JDK 11 Maven 3.6.2` - [x] **Protocol / Flight Integration Tests**: - `Dev / Protocol Test` - [x] **NodeJS**: without running lint and coverage - `NodeJS / AMD64 Debian NodeJS 11` - [x] **C++ & GLib & Ruby w/ gcc 5.4**: - `C++ / AMD64 Debian 10 C++`: with GCC 8.3 - `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4 - `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4 - `C GLib / AMD64 Ubuntu 18.04 C GLib` - `Ruby / AMD64 Ubuntu 18.04 Ruby` - [x] **[OS X] C++ & GLib & Ruby w/ XCode 10.2 & Homebrew** - `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3 - `C GLib / AMD64 MacOS 10.14 C Glib`: with Xcode 10.3 - `Ruby / AMD64 MacOS 10.14 Ruby`: with Xcode 10.3 - [x] **Go**: without coverage - `Go / AMD64 Debian Go 1.12` - [x] **R (with and without libarrow)**: - `R / AMD64 Conda R 3.6`: with libarrow - `R / AMD64 Ubuntu 18.04 R 3.6` with libarrow ### Appveyor - ~JOB=Build, GENERATOR=Ninja, CONFIGURATION=Release, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017~ - ~JOB=Toolchain, GENERATOR=Ninja, CONFIGURATION=Release, ARROW_S3=ON, ARROW_BUILD_FLIGHT=ON, ARROW_BUILD_GANDIVA=ON~ - ~JOB=Build_Debug, GENERATOR=Ninja, CONFIGURATION=Debug~ - ~JOB=MinGW32, MINGW_ARCH=i686, MINGW_PACKAGE_PREFIX=mingw-w64-i686, MINGW_PREFIX=c:\msys64\mingw32, MSYSTEM=MINGW32, USE_CLCACHE=false~ - ~JOB=MinGW64, MINGW_ARCH=x86_64, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64, MINGW_PREFIX=c:\msys64\mingw64, MSYSTEM=MINGW64, USE_CLCACHE=false~ - [x] **JOB=Rust, TARGET=x86_64-pc-windows-msvc, USE_CLCACHE=false**: - `Rust / AMD64 Windows 2019 Rust nightly-2019-09-25` - [x] **JOB=C#, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017, USE_CLCACHE=false** - `C# / AMD64 Windows 2019 C# 2.2.103` - [x] **JOB=Go, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 ...**: - `Go / AMD64 Windows 2019 Go 1.12` - ~JOB=R with libarrow, USE_CLCACHE=false, TEST_R_WITH_ARROW=TRUE, RWINLIB_LOCAL=%APPVEYOR_BUILD_FOLDER%\libarrow.zip~ ### Github Actions - [x] **Windows MSVC C++ / Build (Visual Studio 16 2019)**: - `C++ / AMD64 Windows 2019 C++`: without tests - [x] **Windows MSVC C++ / Build (Visual Studio 15 2017)**: - `C++ / AMD64 Windows 2016 C++`: without tests - [x] **Linux docker-compose / Test (C++ w/ clang-7 & system packages)**: all have llvm for gandiva but the compiler is set to gcc - `C++ / AMD64 Debian 10 C++`: with GCC 8.3 - `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4 - `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4 - [x] **Linux docker-compose / Test (Rust)**: without rustfmt - `Rust / AMD64 Debian Rust nightly-2019-09-25` - [x] **Linux docker-compose / Test (Lint, Release tests)**: - `Lint / C++, Python, R, Rust, Docker, RAT` - `Dev / Source Release` ### Nightly Crossbow tests The packaging builds are out of the scope if this PR, but the nightly **dockerized test** task are in. Nightly tests: - [x] docker-r - [x] docker-r-conda - [x] docker-r-sanitizer - [x] docker-rust - [x] docker-cpp - [x] docker-cpp-cmake32 - [x] docker-cpp-release - [x] docker-cpp-static-only - [x] docker-c_glib - [x] docker-go - [x] docker-python-2.7 - [x] docker-python-3.6 - [x] docker-python-3.7 - [x] docker-python-2.7-nopandas - [x] docker-python-3.6-nopandas - [x] docker-java - [x] docker-js - [x] docker-docs - [x] docker-lint - [x] docker-iwyu: included in the lint - [x] docker-clang-format: included in the lint - [x] docker-pandas-master - [x] docker-dask-integration - [x] docker-hdfs-integration - [x] docker-spark-integration - [x] docker-turbodbc-integration # TODOs left: - [x] Fix the Apidoc generation for c_glib - [x] Fix the JNI test for Gandiva and ORC - [x] Test that crossbow tests are passing - ~Optionally restore the travis configuration to incrementally decommission old builds~ ## Follow-up JIRAs: - [Archery] Consider porting the docker tool of ursabot to archery - [Archery] Consider to use archery with or instead of the pre-commit hooks - [Archery] Create a wrapper script in archery for docker compose in order to run the containers with the host's user and group - [C++] GCC 5.4.0 has a compile errors, reproduce with UBUNTU=16.04 docker-compose run ubuntu-cpp - [C++][CI] Test the ported fuzzit integration image - [C++][CI] Turn off unnecessary features in the integration tests (spark/turbodbc/dask/hdfs) - [C++][CI] Revisit ASAN UBSAN settings in every C++ based image - [CI] Consider re-adding the removed debian testing image is removed - [Go][CI] Pre-install the go dependencies in the dockerfile using go get - [JS][CI] Pre-install the JS dependencies in the dockerfile - [Rust][CI] Pre-install the rust dependencies in the dockerfile - [Java][CI] Pre-install the java dependencies in the dockerfile - [Ruby][CI] Pre-install the ruby dependencies in the dockerfile and remove it from the test script - [C#][CI] Pre-install the C# dependencies in the dockerfile - [R][CI] Fix the r-sanitizer build https://issues.apache.org/jira/browse/ARROW-6957 - [GLIB][MacOS] Fail to execute lua examples (fails to load 'lgi.corelgilua51' despite that lgi is installed) - [C++][CMake] Automatically set ARROW_GANDIVA_PC_CXX_FLAGS for conda and OSX sdk (see cpp_build.sh) - [C++][CI] Hiveserver2 instegration test fails to connect to impala container - [CI][Spark] Support specific Spark version in the integration tet including latest - [JS][CI] Move nodejs linting from js_build.sh to archery - [Python][CI] create a docker image for python ASV benchmarks and fix the script - [CI] Find a short but related prefix for the env vars used for the docker-compose file to prevent collisions - [C#] the docker container fails to run because of the ubuntu host versions, see https://github.com/dotnet/core/issues/3509 - [C++][Windows] Enable more features on the windows GHA build - [Doc] document docker-compose usage in the developer sphinx guide - [CI][C++] Add .ccache to the docker-compose mounts - [Archery][CI] Refactor the ci/scripts to a sourceable bash functions or to archery directly - [C++][CI] Use scripts/util_coredump.sh to show automatic backtraces - [C++] Fix the hanging C++ tests in Windows 2019 - [CI] Ask INFRA to set up the DOCKERHUB_* secrets for GitHub actions - [C++][CI] Running Gandiva tests fails on Fedora: Reproduce with: `docker-compose run -e ARROW_GANDIVA=ON fedora-cpp` ``` Running gandiva-internals-test, redirecting output into /build/cpp/build/test-logs/gandiva-internals-test.txt (attempt 1/1) 1364 : CommandLine Error: Option 'x86-experimental-vector-widening-legalization' registered more than once! 1365 LLVM ERROR: inconsistency in registered CommandLine options 1366 /build/cpp/src/gandiva ``` - [JS][CI] NodeJS build fails on Github Actions Windows node ``` > NODE_NO_WARNINGS=1 gulp build # 'NODE_NO_WARNINGS' is not recognized as an internal or external command, # operable program or batch file. # npm ERR! code ELIFECYCLE # npm ERR! errno 1 # npm ERR! apache-arrow@1.0.0-SNAPSHOT build: `NODE_NO_WARNINGS=1 gulp build` # npm ERR! Exit status 1 # npm ERR! # npm ERR! Failed at the apache-arrow@1.0.0-SNAPSHOT build script. # npm ERR! This is probably not a problem with npm. There is likely additional logging output above. ``` Closes #5589 from kszucs/docker-refactor and squashes the following commits: 5105d12e6 <Krisztián Szűcs> Rename pull-request folder to dev_cron e9e9a7eec <Krisztián Szűcs> Use underscores for naming the workflow files a92c99d03 <Krisztián Szűcs> Disable hanging C++ tests on windows f158c89b5 <Krisztián Szűcs> Attempt to push from apache/arrow master; Don't push from crossbow tasks 0e1d470a1 <Krisztián Szűcs> Turn off ORC on macOS C++ test due to link error 258db5cff <Krisztián Szűcs> Only push docker images from apache/arrow repository acdfcf086 <Krisztián Szűcs> Remove ORC from the brewfile 5102b85b1 <Krisztián Szűcs> Fix nodeJS workflow 032d6a388 <Krisztián Szűcs> Turn off 2 python builds 7f15b97a8 <Krisztián Szűcs> Filter branches 48b8d128a <Krisztián Szűcs> Fix workflows 36ad9d297 <Krisztián Szűcs> Disable builds 0f603af0c <Krisztián Szűcs> master only and cron workflows 28cc2d78d <Krisztián Szűcs> Rename Java JNI workflow bcd8af7b7 <Krisztián Szűcs> Port the remaining travis utility scripts ed5688154 <Krisztián Szűcs> Usage comments; recommend installing pandas from the docs because of its removal from conda_env_python 3c8c023ce <Krisztián Szűcs> Use Arch in volumes; some comments; remove conda version 'latest' from the images 771b023a8 <Krisztián Szűcs> Cleanup files; separate JNI builds 97ff8a122 <Krisztián Szűcs> Push docker images only from master dc00b4297 <Krisztián Szűcs> Enable path filters e0e2e1f46 <Krisztián Szűcs> Fix pandas master build 3814e0828 <Krisztián Szűcs> Fix manylinux volumes c18edda70 <Krisztián Szűcs> Add CentOS version to the manylinux image names c8b9dd6b1 <Krisztián Szűcs> Missing --pyargs argument for the python test command 33e646981 <Krisztián Szűcs> Turn off gandiva and flight for the HDFS test b9c547889 <Krisztián Szűcs> Refactor docker-compose file and use it with github actions. Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2019-11-12 11:07:48 +01:00
arrow_dir=${1}
ARROW-16340: [C++][Python] Move all Python related code into PyArrow (#13311) This PR moves `src/arrow/python` directory into `pyarrow` and arranges PyArrow to build it. The build on the Python side is made in two steps: 1. `_run_cmake_pyarrow_cpp()` where the C++ part of the pyarrow is build first (the part that was moved in the refactoring) 2. `_run_cmake()` where pyarrow is built as before No changes are needed in the build process from the user side to successfully build pyarrow after this refactoring. The test for PyArrow CPP will however be moved into Cython and can currently be run with: ```shell >>> pushd python/build/dist/temp >>> ctest ``` Lead-authored-by: Alenka Frim <frim.alenka@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2022-08-26 09:27:32 +02:00
test_dir=${1}/python/build/dist
ARROW-988 [JS] Add entry to Travis CI matrix Author: Brian Hulette <brian.hulette@ccri.com> Closes #677 from TheNeuralBit/js_travis and squashes the following commits: 3d5b54b5 [Brian Hulette] Revert "Testing a failure" a84b3641 [Brian Hulette] Testing a failure 70310b68 [Brian Hulette] add flatbuffers download e0c1100d [Brian Hulette] Add travis config and scripts for JS
2017-05-13 15:53:31 -04:00
GH-40454: [CI][Debian] Update Debian to 12 from 11 (#40455) ### Rationale for this change Debian 11 will reach EOL on 2024-07. ### What changes are included in this PR? * Use Debian 12. * Enable build options that can't be enabled with Debian 11. * `ARROW_AZURE` * `ARROW_FLIGHT_SQL` * `ARROW_WITH_OPENTELEMETRY` ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #40454 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
2024-03-13 19:19:38 +09:00
if [ -n "${ARROW_PYTHON_VENV:-}" ]; then
. "${ARROW_PYTHON_VENV}/bin/activate"
fi
ARROW-7930: [CI][Python] Test jpype integration Closes #6484 from pitrou/ARROW-7930-jpype-integration and squashes the following commits: 686ec6537 <Antoine Pitrou> ARROW-7930: Test jpype integration Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2020-02-25 09:56:04 +01:00
export ARROW_SOURCE_DIR=${arrow_dir}
ARROW-7101: [CI] Refactor docker-compose setup and use it with GitHub Actions ## Projecting ideas from ursabot ### Parametric docker images The images are better parameterized now, meaning that we can build more variant of the same service. Couple of examples: ```console UBUNTU=16.04 docker-compose build ubuntu-cpp ARCH=arm64v8 UBUNTU=18.04 docker-compose build ubuntu-cpp PYTHON=3.6 docker-compose build conda-python ARCH=arm32v7 PYTHON=3.6 PANDAS=0.25 docker-compose build conda-python-pandas ``` Each variant has it's own docker image following a string naming schema: `{org}/{arch}-{platform}-{platform-version}[[-{variant}-{variant-version}]..]:latest` ### Use *_build.sh and *_test.sh for each job The docker images provide the environment, and each language backend usually should implement two scripts, a `build.sh` and a `test.sh`. This way dependent build like the docker python, r or c glib are able to reuse the build script of the ancestor without running its tests. With small enough scripts, if the environment is properly set up even the non-docker builds should be reproducible locally. GitHub Actions support bash scripts across all three platforms, so we can reuse the same `*_build.sh` and `*_test.sh` scripts to execute the builds either in docker, on the CI or locally. ## Using GitHub Actions for running the builds Regardless of the CI we're going to choose, the isolation constraint of different platforms requires some sort of virtualisation. Currently linux (and windows, but I have not tried it yet) has lightweight containerisation, so we should keep the linux builds isolated in docker containers. The rest of the platforms (windows and macOS) should be executed on the CI system. GitHub Actions support all three major platforms, linux, windows and macOS. I've added cross platform builds for a couple of languages, like Rust, and Go, the rest are work in progress. ### Workflow A workflow should define all builds of a language, mostly because the path filters can be defined on workflow level. For example the python builds should be triggered if either a cpp/** or a python/** file changes which can be covered in the same workflow file. ## Feature parity with the current builds Reaching feature parity with all of the builds below is not a goal for this PR, the difficult ones should at least have a tracking JIRA ticket. ### Travis-CI - [x] **Lint, Release tests**: - `Lint / C++, Python, R, Rust, Docker, RAT` - `Dev / Source Release` - [x] **C++ unit tests w/ conda-forge toolchain, coverage**: without coverage - `C++ / AMD64 Conda C++` - [x] **Python 3.6 unit tests, conda-forge toolchain, coverage**: without coverage - `Python / AMD64 Conda Python 3.6` - [x] **[OS X] C++ w/ Xcode 9.3**: - `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3 - [x] **[OS X] Python w/ Xcode 9.3**: - `Python / AMD64 MacOS 10.14 Python 3`: with Xcode 10.3 - [x] **Java OpenJDK8 and OpenJDK11**: - `Java / AMD64 Debian Java JDK 8 Maven 3.5.2` - `Java / AMD64 Debian Java JDK 11 Maven 3.6.2` - [x] **Protocol / Flight Integration Tests**: - `Dev / Protocol Test` - [x] **NodeJS**: without running lint and coverage - `NodeJS / AMD64 Debian NodeJS 11` - [x] **C++ & GLib & Ruby w/ gcc 5.4**: - `C++ / AMD64 Debian 10 C++`: with GCC 8.3 - `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4 - `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4 - `C GLib / AMD64 Ubuntu 18.04 C GLib` - `Ruby / AMD64 Ubuntu 18.04 Ruby` - [x] **[OS X] C++ & GLib & Ruby w/ XCode 10.2 & Homebrew** - `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3 - `C GLib / AMD64 MacOS 10.14 C Glib`: with Xcode 10.3 - `Ruby / AMD64 MacOS 10.14 Ruby`: with Xcode 10.3 - [x] **Go**: without coverage - `Go / AMD64 Debian Go 1.12` - [x] **R (with and without libarrow)**: - `R / AMD64 Conda R 3.6`: with libarrow - `R / AMD64 Ubuntu 18.04 R 3.6` with libarrow ### Appveyor - ~JOB=Build, GENERATOR=Ninja, CONFIGURATION=Release, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017~ - ~JOB=Toolchain, GENERATOR=Ninja, CONFIGURATION=Release, ARROW_S3=ON, ARROW_BUILD_FLIGHT=ON, ARROW_BUILD_GANDIVA=ON~ - ~JOB=Build_Debug, GENERATOR=Ninja, CONFIGURATION=Debug~ - ~JOB=MinGW32, MINGW_ARCH=i686, MINGW_PACKAGE_PREFIX=mingw-w64-i686, MINGW_PREFIX=c:\msys64\mingw32, MSYSTEM=MINGW32, USE_CLCACHE=false~ - ~JOB=MinGW64, MINGW_ARCH=x86_64, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64, MINGW_PREFIX=c:\msys64\mingw64, MSYSTEM=MINGW64, USE_CLCACHE=false~ - [x] **JOB=Rust, TARGET=x86_64-pc-windows-msvc, USE_CLCACHE=false**: - `Rust / AMD64 Windows 2019 Rust nightly-2019-09-25` - [x] **JOB=C#, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017, USE_CLCACHE=false** - `C# / AMD64 Windows 2019 C# 2.2.103` - [x] **JOB=Go, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 ...**: - `Go / AMD64 Windows 2019 Go 1.12` - ~JOB=R with libarrow, USE_CLCACHE=false, TEST_R_WITH_ARROW=TRUE, RWINLIB_LOCAL=%APPVEYOR_BUILD_FOLDER%\libarrow.zip~ ### Github Actions - [x] **Windows MSVC C++ / Build (Visual Studio 16 2019)**: - `C++ / AMD64 Windows 2019 C++`: without tests - [x] **Windows MSVC C++ / Build (Visual Studio 15 2017)**: - `C++ / AMD64 Windows 2016 C++`: without tests - [x] **Linux docker-compose / Test (C++ w/ clang-7 & system packages)**: all have llvm for gandiva but the compiler is set to gcc - `C++ / AMD64 Debian 10 C++`: with GCC 8.3 - `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4 - `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4 - [x] **Linux docker-compose / Test (Rust)**: without rustfmt - `Rust / AMD64 Debian Rust nightly-2019-09-25` - [x] **Linux docker-compose / Test (Lint, Release tests)**: - `Lint / C++, Python, R, Rust, Docker, RAT` - `Dev / Source Release` ### Nightly Crossbow tests The packaging builds are out of the scope if this PR, but the nightly **dockerized test** task are in. Nightly tests: - [x] docker-r - [x] docker-r-conda - [x] docker-r-sanitizer - [x] docker-rust - [x] docker-cpp - [x] docker-cpp-cmake32 - [x] docker-cpp-release - [x] docker-cpp-static-only - [x] docker-c_glib - [x] docker-go - [x] docker-python-2.7 - [x] docker-python-3.6 - [x] docker-python-3.7 - [x] docker-python-2.7-nopandas - [x] docker-python-3.6-nopandas - [x] docker-java - [x] docker-js - [x] docker-docs - [x] docker-lint - [x] docker-iwyu: included in the lint - [x] docker-clang-format: included in the lint - [x] docker-pandas-master - [x] docker-dask-integration - [x] docker-hdfs-integration - [x] docker-spark-integration - [x] docker-turbodbc-integration # TODOs left: - [x] Fix the Apidoc generation for c_glib - [x] Fix the JNI test for Gandiva and ORC - [x] Test that crossbow tests are passing - ~Optionally restore the travis configuration to incrementally decommission old builds~ ## Follow-up JIRAs: - [Archery] Consider porting the docker tool of ursabot to archery - [Archery] Consider to use archery with or instead of the pre-commit hooks - [Archery] Create a wrapper script in archery for docker compose in order to run the containers with the host's user and group - [C++] GCC 5.4.0 has a compile errors, reproduce with UBUNTU=16.04 docker-compose run ubuntu-cpp - [C++][CI] Test the ported fuzzit integration image - [C++][CI] Turn off unnecessary features in the integration tests (spark/turbodbc/dask/hdfs) - [C++][CI] Revisit ASAN UBSAN settings in every C++ based image - [CI] Consider re-adding the removed debian testing image is removed - [Go][CI] Pre-install the go dependencies in the dockerfile using go get - [JS][CI] Pre-install the JS dependencies in the dockerfile - [Rust][CI] Pre-install the rust dependencies in the dockerfile - [Java][CI] Pre-install the java dependencies in the dockerfile - [Ruby][CI] Pre-install the ruby dependencies in the dockerfile and remove it from the test script - [C#][CI] Pre-install the C# dependencies in the dockerfile - [R][CI] Fix the r-sanitizer build https://issues.apache.org/jira/browse/ARROW-6957 - [GLIB][MacOS] Fail to execute lua examples (fails to load 'lgi.corelgilua51' despite that lgi is installed) - [C++][CMake] Automatically set ARROW_GANDIVA_PC_CXX_FLAGS for conda and OSX sdk (see cpp_build.sh) - [C++][CI] Hiveserver2 instegration test fails to connect to impala container - [CI][Spark] Support specific Spark version in the integration tet including latest - [JS][CI] Move nodejs linting from js_build.sh to archery - [Python][CI] create a docker image for python ASV benchmarks and fix the script - [CI] Find a short but related prefix for the env vars used for the docker-compose file to prevent collisions - [C#] the docker container fails to run because of the ubuntu host versions, see https://github.com/dotnet/core/issues/3509 - [C++][Windows] Enable more features on the windows GHA build - [Doc] document docker-compose usage in the developer sphinx guide - [CI][C++] Add .ccache to the docker-compose mounts - [Archery][CI] Refactor the ci/scripts to a sourceable bash functions or to archery directly - [C++][CI] Use scripts/util_coredump.sh to show automatic backtraces - [C++] Fix the hanging C++ tests in Windows 2019 - [CI] Ask INFRA to set up the DOCKERHUB_* secrets for GitHub actions - [C++][CI] Running Gandiva tests fails on Fedora: Reproduce with: `docker-compose run -e ARROW_GANDIVA=ON fedora-cpp` ``` Running gandiva-internals-test, redirecting output into /build/cpp/build/test-logs/gandiva-internals-test.txt (attempt 1/1) 1364 : CommandLine Error: Option 'x86-experimental-vector-widening-legalization' registered more than once! 1365 LLVM ERROR: inconsistency in registered CommandLine options 1366 /build/cpp/src/gandiva ``` - [JS][CI] NodeJS build fails on Github Actions Windows node ``` > NODE_NO_WARNINGS=1 gulp build # 'NODE_NO_WARNINGS' is not recognized as an internal or external command, # operable program or batch file. # npm ERR! code ELIFECYCLE # npm ERR! errno 1 # npm ERR! apache-arrow@1.0.0-SNAPSHOT build: `NODE_NO_WARNINGS=1 gulp build` # npm ERR! Exit status 1 # npm ERR! # npm ERR! Failed at the apache-arrow@1.0.0-SNAPSHOT build script. # npm ERR! This is probably not a problem with npm. There is likely additional logging output above. ``` Closes #5589 from kszucs/docker-refactor and squashes the following commits: 5105d12e6 <Krisztián Szűcs> Rename pull-request folder to dev_cron e9e9a7eec <Krisztián Szűcs> Use underscores for naming the workflow files a92c99d03 <Krisztián Szűcs> Disable hanging C++ tests on windows f158c89b5 <Krisztián Szűcs> Attempt to push from apache/arrow master; Don't push from crossbow tasks 0e1d470a1 <Krisztián Szűcs> Turn off ORC on macOS C++ test due to link error 258db5cff <Krisztián Szűcs> Only push docker images from apache/arrow repository acdfcf086 <Krisztián Szűcs> Remove ORC from the brewfile 5102b85b1 <Krisztián Szűcs> Fix nodeJS workflow 032d6a388 <Krisztián Szűcs> Turn off 2 python builds 7f15b97a8 <Krisztián Szűcs> Filter branches 48b8d128a <Krisztián Szűcs> Fix workflows 36ad9d297 <Krisztián Szűcs> Disable builds 0f603af0c <Krisztián Szűcs> master only and cron workflows 28cc2d78d <Krisztián Szűcs> Rename Java JNI workflow bcd8af7b7 <Krisztián Szűcs> Port the remaining travis utility scripts ed5688154 <Krisztián Szűcs> Usage comments; recommend installing pandas from the docs because of its removal from conda_env_python 3c8c023ce <Krisztián Szűcs> Use Arch in volumes; some comments; remove conda version 'latest' from the images 771b023a8 <Krisztián Szűcs> Cleanup files; separate JNI builds 97ff8a122 <Krisztián Szűcs> Push docker images only from master dc00b4297 <Krisztián Szűcs> Enable path filters e0e2e1f46 <Krisztián Szűcs> Fix pandas master build 3814e0828 <Krisztián Szűcs> Fix manylinux volumes c18edda70 <Krisztián Szűcs> Add CentOS version to the manylinux image names c8b9dd6b1 <Krisztián Szűcs> Missing --pyargs argument for the python test command 33e646981 <Krisztián Szűcs> Turn off gandiva and flight for the HDFS test b9c547889 <Krisztián Szűcs> Refactor docker-compose file and use it with github actions. Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2019-11-12 11:07:48 +01:00
export ARROW_TEST_DATA=${arrow_dir}/testing/data
export PARQUET_TEST_DATA=${arrow_dir}/cpp/submodules/parquet-testing/data
export LD_LIBRARY_PATH=${ARROW_HOME}/lib:${LD_LIBRARY_PATH}
ARROW-18339: [Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job (#14643) Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2022-12-14 21:58:45 +01:00
export DYLD_LIBRARY_PATH=${ARROW_HOME}/lib:${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
ARROW-12735: [C++] Write GDB plugin Add a GDB plugin (written in Python) that registers custom printers for common Arrow C++ types, and makes their values much more readable when printed on the GDB prompt. Currently covered types are: - `string_view`, `optional`, `Variant` - `Status`, `Result` - `Buffer` (and subclasses) - `BasicDecimal{128,256}` (and subclasses) - `DataType` (and subclasses), `Field`, `Schema`, `KeyValueMetadata` - `Scalar` (and subclasses) - `ArrayData`, `Array` (and subclasses) - `ChunkedArray`, `RecordBatch`, `Table`, `Datum` The plugin is tested using a test harness in PyArrow, for convenience. Here is for example `arrow::util::Variant<int, bool, std::string>`: * with the GDB plugin: ```gdb (gdb) p int_variant $1 = arrow::util::Variant of index 0 (actual type int), value 42 ``` * without the GDB plugin: ```gdb (gdb) p int_variant $1 = { <arrow::util::detail::VariantImpl<arrow::util::Variant<int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = { <arrow::util::detail::VariantImpl<arrow::util::Variant<int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = { <arrow::util::detail::VariantImpl<arrow::util::Variant<int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = { <arrow::util::detail::VariantImpl<arrow::util::Variant<int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = { <arrow::util::detail::VariantStorage<int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = { [ snip more lines of inscrutable implementation details ] ``` Closes #12092 from pitrou/ARROW-12735-gdb Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-01-20 18:49:16 +01:00
export ARROW_GDB_SCRIPT=${arrow_dir}/cpp/gdb_arrow.py
ARROW-988 [JS] Add entry to Travis CI matrix Author: Brian Hulette <brian.hulette@ccri.com> Closes #677 from TheNeuralBit/js_travis and squashes the following commits: 3d5b54b5 [Brian Hulette] Revert "Testing a failure" a84b3641 [Brian Hulette] Testing a failure 70310b68 [Brian Hulette] add flatbuffers download e0c1100d [Brian Hulette] Add travis config and scripts for JS
2017-05-13 15:53:31 -04:00
ARROW-7915: [CI][Python] Enable development mode in tests Python's "development mode" enables some runtime checks and warnings that help developers spot errors in their code (such as resource leaks). Closes #6473 from pitrou/ARROW-7915-py-dev-mode and squashes the following commits: 1c2b91dcd <Antoine Pitrou> Enable Python dev mode also on AppVeyor e4363961d <Antoine Pitrou> ARROW-7915: Enable development mode in tests Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2020-02-24 11:36:05 +01:00
# Enable some checks inside Python itself
export PYTHONDEVMODE=1
GH-39973: [C++][CI] Disable debug memory pool for ASAN and Valgrind (#39975) ### Rationale for this change Disable debug memory pool for ASAN and Valgrind so that they can detect more subtle memory issues regarding to buffer tail bytes. ### What changes are included in this PR? 1. Add a `none` option to debug memory pool env var to make other things slightly easier. 2. Change `*_test.sh` scripts to conditionally set debug memory pool env var. 3. Top-level docker compose change to pass none to debug memory pool env var for ASAN and Valgrind. ### Are these changes tested? The CI should cover it well. ### Are there any user-facing changes? No. * Closes: #39973 Authored-by: Ruoxi Sun <zanmato1984@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2024-02-09 01:02:04 +08:00
# Enable memory debug checks if the env is not set already
if [ -z "${ARROW_DEBUG_MEMORY_POOL}" ]; then
export ARROW_DEBUG_MEMORY_POOL=trap
fi
ARROW-15550: [C++] Add optional debug memory checks Debug memory checks can be enabled by setting the environment variable ARROW_DEBUG_MEMORY_POOL to one of the recognized values "abort", "trap" or "warn" (regardless of whether Arrow was compiled in debug or release mode). The only implemented check adds a suffix past the allocation and checks that it isn't clobbered on deallocation. Adding a prefix would be more expensive because of alignment constraints. Also add a fix by Will Jones for ARROW-14047 (Parquet Arrow reader sets null values in buffer overflow), since otherwise CI would fail. Closes #12330 from pitrou/ARROW-15550-debug-memory-checks Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-02-09 09:44:12 +01:00
ARROW-15504: [Python][CI] Ensure that optional components are tested Tests for optional components could be silently skipped if by accident the component would fail importing despite being built. Closes #12301 from pitrou/ARROW-15504-py-test-optional-components Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-02-01 14:13:09 +01:00
# By default, force-test all optional components
GH-15280: [C++][Python][GLib] add libarrow_acero containing everything previously in compute/exec (#34711) ### Rationale for this change See the linked issue ### What changes are included in this PR? C++: * remove all compute/exec/* from libarrow * rename compute/exec -> acero and make libarrow_acero * add new ARROW_ACERO option, required if ARROW_DATASET is on * libarrow_dataset now depends on libarrow_acero c_glib: add the new libarrow_acero dependency - we disallow building glib without it python: added PYARROW_BUILD_ACERO, set to on if DATASETS are built ### Are these changes tested? All the standard tests do work properly. I manually compiled C++ with: * no ARROW_ACERO * ARROW_ACERO and no ARROW_DATASET * ARROW_ACERO and ARROW_DATASET and no ARROW_SUBSTRAIT I manually compiled python without ACERO & DATASET and with ACERO and without DATASET ### Are there any user-facing changes? If users include compute/exec files directly then they'll have to update their code. * Closes: #15280 Lead-authored-by: Davide Pasetto <dpasetto69@gmail.com> Co-authored-by: Li Jin <ice.xelloss@gmail.com> Co-authored-by: Weston Pace <weston.pace@gmail.com> Signed-off-by: Li Jin <ice.xelloss@gmail.com>
2023-04-01 06:47:16 -04:00
: ${PYARROW_TEST_ACERO:=${ARROW_ACERO:-ON}}
GH-39352: [FS][Azure] Enable azure in builds (#39971) ### Rationale for this change ### What changes are included in this PR? Enable Azure in linux and mac os wheel builds. Tried to copy GCS Don't enable Azure for windows builds because windows builds where all failing. Failures were a combination of cmake version or `Could not find a package configuration file provided by "wil"`. I think it makes sense to come back to windows builds in another PR. ### Are these changes tested? There is no new functionality to test. ### Are there any user-facing changes? No * Closes: #39352 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2024-02-09 01:41:36 +00:00
: ${PYARROW_TEST_AZURE:=${ARROW_AZURE:-ON}}
ARROW-15504: [Python][CI] Ensure that optional components are tested Tests for optional components could be silently skipped if by accident the component would fail importing despite being built. Closes #12301 from pitrou/ARROW-15504-py-test-optional-components Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-02-01 14:13:09 +01:00
: ${PYARROW_TEST_CUDA:=${ARROW_CUDA:-ON}}
: ${PYARROW_TEST_DATASET:=${ARROW_DATASET:-ON}}
: ${PYARROW_TEST_FLIGHT:=${ARROW_FLIGHT:-ON}}
: ${PYARROW_TEST_GANDIVA:=${ARROW_GANDIVA:-ON}}
ARROW-14892: [Python][C++] GCS Bindings (#12763) Incorporate GCS file system into python and other bug fixes. Bugs/Other changes: - Add GCS bindings mostly based on AWS bindings in Python and associated unit tests - Tell was incorrect, it double counted when the stream was constructed with an offset. - Missed setting the define in config.cmake which means `FileSystemFromUri was never tested and didn't compile this is now fixed` - Refine logic for GetFileInfo with a single path to recognize prefixes followed by a slash as a directory. This allows datasets to work as expected with a toy dataset generated on local-filesystem and copied to the cloud (I believe this is typical of how other systems write to GCS as well. - Switch convention for creating directories to always end in "/" and make use of this as another indicator. From testing with a sample iceberg table it appears this is the convention used for hive-partitioning, so I assume this is common practice for other Hive related writers (i.e. what we want to support). - Fix bug introduced in https://github.com/apache/arrow/commit/a5e45cecb24229433b825dac64e0ffd10d400e8c which caused failures when a deletion occurred on a bucket (not an object in the bucket). - Ensure output streams are closed on destruction (this is consistent with S3) Lead-authored-by: Micah Kornfield <micahk@google.com> Co-authored-by: emkornfield <emkornfield@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-06-12 02:50:28 -07:00
: ${PYARROW_TEST_GCS:=${ARROW_GCS:-ON}}
ARROW-15504: [Python][CI] Ensure that optional components are tested Tests for optional components could be silently skipped if by accident the component would fail importing despite being built. Closes #12301 from pitrou/ARROW-15504-py-test-optional-components Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-02-01 14:13:09 +01:00
: ${PYARROW_TEST_HDFS:=${ARROW_HDFS:-ON}}
: ${PYARROW_TEST_ORC:=${ARROW_ORC:-ON}}
: ${PYARROW_TEST_PARQUET:=${ARROW_PARQUET:-ON}}
GH-39006: [Python] Extract libparquet requirements out of libarrow_python.so to new libarrow_python_parquet_encryption.so (#39316) ### Rationale for this change If I build pyarrow with everything and then I remove some of the Arrow CPP .so in order to have a minimal build I can't import pyarrow because it requires libarrow and libparquet. This is relevant in order to have a minimal build for Conda. Please see the related issue for more information. ### What changes are included in this PR? Move libarrow parquet encryption for pyarrow to its own shared object. ### Are these changes tested? I will run extensive CI with extra python archery tests. ### Are there any user-facing changes? No, and yes :) There will be a new .so on pyarrow but shouldn't be relevant in my opinion. * Closes: #39006 Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2023-12-22 02:06:50 +01:00
: ${PYARROW_TEST_PARQUET_ENCRYPTION:=${PARQUET_REQUIRE_ENCRYPTION:-ON}}
ARROW-15504: [Python][CI] Ensure that optional components are tested Tests for optional components could be silently skipped if by accident the component would fail importing despite being built. Closes #12301 from pitrou/ARROW-15504-py-test-optional-components Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-02-01 14:13:09 +01:00
: ${PYARROW_TEST_S3:=${ARROW_S3:-ON}}
GH-15280: [C++][Python][GLib] add libarrow_acero containing everything previously in compute/exec (#34711) ### Rationale for this change See the linked issue ### What changes are included in this PR? C++: * remove all compute/exec/* from libarrow * rename compute/exec -> acero and make libarrow_acero * add new ARROW_ACERO option, required if ARROW_DATASET is on * libarrow_dataset now depends on libarrow_acero c_glib: add the new libarrow_acero dependency - we disallow building glib without it python: added PYARROW_BUILD_ACERO, set to on if DATASETS are built ### Are these changes tested? All the standard tests do work properly. I manually compiled C++ with: * no ARROW_ACERO * ARROW_ACERO and no ARROW_DATASET * ARROW_ACERO and ARROW_DATASET and no ARROW_SUBSTRAIT I manually compiled python without ACERO & DATASET and with ACERO and without DATASET ### Are there any user-facing changes? If users include compute/exec files directly then they'll have to update their code. * Closes: #15280 Lead-authored-by: Davide Pasetto <dpasetto69@gmail.com> Co-authored-by: Li Jin <ice.xelloss@gmail.com> Co-authored-by: Weston Pace <weston.pace@gmail.com> Signed-off-by: Li Jin <ice.xelloss@gmail.com>
2023-04-01 06:47:16 -04:00
export PYARROW_TEST_ACERO
GH-40079: [CI][Packaging] Enable Azure in more tests and builds (#40080) ### Rationale for this change We want python side tests of `AzureFileSystem` to run in CI. ### What changes are included in this PR? - Add missing `export` to enable Azure pyarrow tests - Enable azure in sdist tests. - Enable Azure on macos python builds - Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite) - Enable retries on C++ tests to mitigate https://github.com/apache/arrow/issues/40121 Probably all of this should have been included in https://github.com/apache/arrow/pull/39971 ### Are these changes tested? There is no new functionality to test ### Are there any user-facing changes? No * Closes: #40079 * GitHub Issue: #40079 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2024-02-22 22:16:31 +00:00
export PYARROW_TEST_AZURE
ARROW-15504: [Python][CI] Ensure that optional components are tested Tests for optional components could be silently skipped if by accident the component would fail importing despite being built. Closes #12301 from pitrou/ARROW-15504-py-test-optional-components Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-02-01 14:13:09 +01:00
export PYARROW_TEST_CUDA
export PYARROW_TEST_DATASET
export PYARROW_TEST_FLIGHT
export PYARROW_TEST_GANDIVA
ARROW-14892: [Python][C++] GCS Bindings (#12763) Incorporate GCS file system into python and other bug fixes. Bugs/Other changes: - Add GCS bindings mostly based on AWS bindings in Python and associated unit tests - Tell was incorrect, it double counted when the stream was constructed with an offset. - Missed setting the define in config.cmake which means `FileSystemFromUri was never tested and didn't compile this is now fixed` - Refine logic for GetFileInfo with a single path to recognize prefixes followed by a slash as a directory. This allows datasets to work as expected with a toy dataset generated on local-filesystem and copied to the cloud (I believe this is typical of how other systems write to GCS as well. - Switch convention for creating directories to always end in "/" and make use of this as another indicator. From testing with a sample iceberg table it appears this is the convention used for hive-partitioning, so I assume this is common practice for other Hive related writers (i.e. what we want to support). - Fix bug introduced in https://github.com/apache/arrow/commit/a5e45cecb24229433b825dac64e0ffd10d400e8c which caused failures when a deletion occurred on a bucket (not an object in the bucket). - Ensure output streams are closed on destruction (this is consistent with S3) Lead-authored-by: Micah Kornfield <micahk@google.com> Co-authored-by: emkornfield <emkornfield@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-06-12 02:50:28 -07:00
export PYARROW_TEST_GCS
ARROW-15504: [Python][CI] Ensure that optional components are tested Tests for optional components could be silently skipped if by accident the component would fail importing despite being built. Closes #12301 from pitrou/ARROW-15504-py-test-optional-components Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-02-01 14:13:09 +01:00
export PYARROW_TEST_HDFS
export PYARROW_TEST_ORC
export PYARROW_TEST_PARQUET
GH-39006: [Python] Extract libparquet requirements out of libarrow_python.so to new libarrow_python_parquet_encryption.so (#39316) ### Rationale for this change If I build pyarrow with everything and then I remove some of the Arrow CPP .so in order to have a minimal build I can't import pyarrow because it requires libarrow and libparquet. This is relevant in order to have a minimal build for Conda. Please see the related issue for more information. ### What changes are included in this PR? Move libarrow parquet encryption for pyarrow to its own shared object. ### Are these changes tested? I will run extensive CI with extra python archery tests. ### Are there any user-facing changes? No, and yes :) There will be a new .so on pyarrow but shouldn't be relevant in my opinion. * Closes: #39006 Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2023-12-22 02:06:50 +01:00
export PYARROW_TEST_PARQUET_ENCRYPTION
ARROW-15504: [Python][CI] Ensure that optional components are tested Tests for optional components could be silently skipped if by accident the component would fail importing despite being built. Closes #12301 from pitrou/ARROW-15504-py-test-optional-components Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-02-01 14:13:09 +01:00
export PYARROW_TEST_S3
ARROW-16340: [C++][Python] Move all Python related code into PyArrow (#13311) This PR moves `src/arrow/python` directory into `pyarrow` and arranges PyArrow to build it. The build on the Python side is made in two steps: 1. `_run_cmake_pyarrow_cpp()` where the C++ part of the pyarrow is build first (the part that was moved in the refactoring) 2. `_run_cmake()` where pyarrow is built as before No changes are needed in the build process from the user side to successfully build pyarrow after this refactoring. The test for PyArrow CPP will however be moved into Cython and can currently be run with: ```shell >>> pushd python/build/dist/temp >>> ctest ``` Lead-authored-by: Alenka Frim <frim.alenka@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
2022-08-26 09:27:32 +02:00
# Testing PyArrow
ARROW-17435: [CI][Python][CUDA] Install Numba for CUDA interop tests (#13899) Checked locally on a Ubuntu 20.04 host with: ``` archery docker run ubuntu-cuda-python ``` Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
2022-08-17 15:37:16 +02:00
pytest -r s ${PYTEST_ARGS} --pyargs pyarrow
GH-28859: [Doc][Python] Use only code-block directive and set up doctest for the python user guide (#48619) ### Rationale for this change In many places in the Python User Guide the code exampels are written with IPython directive (elsewhere code-block is used). IPython directives are converted to IPython format (`In` and `Out` during the doc build). This can lead to slower builds. ### What changes are included in this PR? IPython directives are converted to runnable code-block (with `>>>` and `...`) and pytest doctest support for `.rst` files is added to the `conda-python-docs` CI job. This means the code in the Python User Guide is tested separately to the building of the documentation. ### Are these changes tested? Yes, with the CI. ### Are there any user-facing changes? Changes to the Python User Guide examples will have to be tested with `pytest --doctest-glob='*.rst' docs/source/python/file.rst` * GitHub Issue: #28859 Lead-authored-by: AlenkaF <frim.alenka@gmail.com> Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com> Co-authored-by: tadeja <tadeja@users.noreply.github.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com>
2026-01-30 09:09:33 +01:00
# Testing RST documentation examples (if PYTEST_RST_ARGS is set)
if [ -n "${PYTEST_RST_ARGS}" ]; then
pytest ${PYTEST_RST_ARGS} ${arrow_dir}/docs/source/python
fi
Reference in New Issue Copy Permalink