Morph MORPH
SIGN IN SIGN UP
apache / arrow UNCLAIMED

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

0 0 16 C++
Normal View History Raw
ARROW-1424: [Python] Add CUDA support to pyarrow This PR implements CUDA support to pyarrow. In order to use it, the Arrow C++ library must be built with the following cmake options: ``` -DARROW_PYTHON=on -DARROW_GPU=ON -DARROW_IPC=ON ``` To enable CUDA support in pyarrow, it must be built with `--with-cuda` option, e.g. ``` python setup.py build_ext --with-cuda <other options> ``` or the environment must define ``` PYARROW_WITH_CUDA=1 ``` This CUDA support implementation is rather complete: all Arrow C++ GPU related functions and classes are exposed to Python, the new methods and functions are documented, the test coverage is close to 100%. However, there are some issues that need to be tackled and questions to be answered (hence the WIP attribute): 1. Is the naming convention of the new methods appropriate? Are there any changes needed in the new API? 2. The IPC test (see `test_IPC` in `python/pyarrow/tests/test_gpu.py`) fails when calling `open_ipc_buffer`: `cuIpcOpenMemHandle fails with code 201` . Currently, I don't know why. Any hint or help on this is much appreciated. [FIXED: using `multiprocessing.Process` in `spawn` mode] 3. Anything else? Author: Pearu Peterson <pearu.peterson@gmail.com> Closes #2536 from pearu/pearu-cuda-pyarrow and squashes the following commits: 7a018465a <Pearu Peterson> Minor clean up 2. c86e1651a <Pearu Peterson> Minor clean up. 5e171e8c2 <Pearu Peterson> Revised buffer_from_data. 7555c657b <Pearu Peterson> Raise BufferError in CudaBuffer.__getbuffer__. Fix pytest.raises usages. e1bcb0886 <Pearu Peterson> Apply feedback from PR. 2. 38ddfffba <Pearu Peterson> Apply feedback from PR. WIP. 4ece8092a <Pearu Peterson> Remove redudant -DPYARROW_BUILD_CUDA=on 66d704ea5 <Pearu Peterson> Silence flake8 on Arrow CUDA api module. 88961fa35 <Pearu Peterson> cmake: moved Arrow CUDA detection to FindArrowCuda. 2126eba85 <Pearu Peterson> Fixes for flake8 ff7faa2d1 <Pearu Peterson> Removed DeviceManager, use 'ctx=Context(<device_number>)'. Introduced Context.get_num_devices and Context.device_number. f3c41dbae <Pearu Peterson> Added cdefs to CMemoryPool varibales 106b07197 <Pearu Peterson> Removed cuda prefix everywhere except in CudaBuffer. dff5bd495 <Pearu Peterson> More space removal. Fix CudaBufferWriter and cuda_read_record_batch docs. ce1d3bb50 <Pearu Peterson> Remove _freed attribute from CudaHostBuffer 1defcb4c8 <Pearu Peterson> Remove spaces around * and & 94989a7cd <Pearu Peterson> Rename lib_gpu to _cuda, ARROW_GPU to CUDA, --with-arrow-gpu to --with-cuda. Remove --without-arrow-gpu option. To test availability of cuda support, try import pyarrow.cuda. cb89ee302 <Pearu Peterson> Remove usage of FreeHost to avoid double-freeing issue. 61659c4fe <Pearu Peterson> Introduce pyarrow.cuda module as the CUDA UI. 8faf1ee51 <Pearu Peterson> Add missing import of pandas_compat. a5447874d <Pearu Peterson> Fix formatting for flake8. 7053df860 <Pearu Peterson> Improve detecting availability of gpu support (2nd try). 3913e2904 <Pearu Peterson> Improve detecting availability of gpu support. 9e365bd15 <Pearu Peterson> Merge branch 'pearu-cuda-pyarrow' of github.com:Quansight/arrow into pearu-cuda-pyarrow d237b3493 <Pearu Peterson> Implement GPU support in pyarrow 584d94ba0 <Pearu Peterson> Flake it. c2799e645 <Pearu Peterson> Impl IPC tests. 7dc6d8833 <Pearu Peterson> Impl CudaBufferWriter.writeat, CudaBufferReader.read_buffer. Complete unittests for CudaBufferWriter/Reader e2b14df8b <Pearu Peterson> Impl CudaBuffer device_buffer and slice methods. Impl tests for CudaBuffer and CudaHostBuffer. 36396c689 <Pearu Peterson> Test memory management of CudaBuffer and CudaHostBuffer 5e2c1ba1e <Pearu Peterson> Unittest for allocate/free host buffer. 102f277e0 <Pearu Peterson> Document all methods and functions. 9b8cb1b9f <Pearu Peterson> Complete copy_from/to_host implementations and tests. 9eebf19aa <Pearu Peterson> Document copy_to_host and copy_from_host methods. Complete unittests for copy_to_host. 11cba54d2 <Pearu Peterson> Add copy_to_host and copy_from_host methods to CudaBuffer ae3cd3fda <Pearu Peterson> Exposed all Arrow GPU C++ classes and functions to pyarrow. WIP cf42941e9 <Pearu Peterson> Expose Message for lib_gpu. 49b0190ba <Pearu Peterson> Expose CudaDeviceManager, CudaContext, CudaIpcMemHandle to pyarrow. a2d755791 <Pearu Peterson> Minimal GPU support for pyarrow. WIP.
2018-09-13 19:07:46 -04:00
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
ARROW-3557: [Python] Set Cython language level This avoids a warning and makes us ready for the default setting of using Python 3 semantics. Author: Antoine Pitrou <antoine@python.org> Closes #2814 from pitrou/ARROW-3557-cython-language-level and squashes the following commits: 079835694 <Antoine Pitrou> Fix Python 2 04ecbaa52 <Antoine Pitrou> ARROW-3557: Set Cython language level
2018-10-23 06:27:38 -04:00
# cython: language_level = 3
ARROW-1424: [Python] Add CUDA support to pyarrow This PR implements CUDA support to pyarrow. In order to use it, the Arrow C++ library must be built with the following cmake options: ``` -DARROW_PYTHON=on -DARROW_GPU=ON -DARROW_IPC=ON ``` To enable CUDA support in pyarrow, it must be built with `--with-cuda` option, e.g. ``` python setup.py build_ext --with-cuda <other options> ``` or the environment must define ``` PYARROW_WITH_CUDA=1 ``` This CUDA support implementation is rather complete: all Arrow C++ GPU related functions and classes are exposed to Python, the new methods and functions are documented, the test coverage is close to 100%. However, there are some issues that need to be tackled and questions to be answered (hence the WIP attribute): 1. Is the naming convention of the new methods appropriate? Are there any changes needed in the new API? 2. The IPC test (see `test_IPC` in `python/pyarrow/tests/test_gpu.py`) fails when calling `open_ipc_buffer`: `cuIpcOpenMemHandle fails with code 201` . Currently, I don't know why. Any hint or help on this is much appreciated. [FIXED: using `multiprocessing.Process` in `spawn` mode] 3. Anything else? Author: Pearu Peterson <pearu.peterson@gmail.com> Closes #2536 from pearu/pearu-cuda-pyarrow and squashes the following commits: 7a018465a <Pearu Peterson> Minor clean up 2. c86e1651a <Pearu Peterson> Minor clean up. 5e171e8c2 <Pearu Peterson> Revised buffer_from_data. 7555c657b <Pearu Peterson> Raise BufferError in CudaBuffer.__getbuffer__. Fix pytest.raises usages. e1bcb0886 <Pearu Peterson> Apply feedback from PR. 2. 38ddfffba <Pearu Peterson> Apply feedback from PR. WIP. 4ece8092a <Pearu Peterson> Remove redudant -DPYARROW_BUILD_CUDA=on 66d704ea5 <Pearu Peterson> Silence flake8 on Arrow CUDA api module. 88961fa35 <Pearu Peterson> cmake: moved Arrow CUDA detection to FindArrowCuda. 2126eba85 <Pearu Peterson> Fixes for flake8 ff7faa2d1 <Pearu Peterson> Removed DeviceManager, use 'ctx=Context(<device_number>)'. Introduced Context.get_num_devices and Context.device_number. f3c41dbae <Pearu Peterson> Added cdefs to CMemoryPool varibales 106b07197 <Pearu Peterson> Removed cuda prefix everywhere except in CudaBuffer. dff5bd495 <Pearu Peterson> More space removal. Fix CudaBufferWriter and cuda_read_record_batch docs. ce1d3bb50 <Pearu Peterson> Remove _freed attribute from CudaHostBuffer 1defcb4c8 <Pearu Peterson> Remove spaces around * and & 94989a7cd <Pearu Peterson> Rename lib_gpu to _cuda, ARROW_GPU to CUDA, --with-arrow-gpu to --with-cuda. Remove --without-arrow-gpu option. To test availability of cuda support, try import pyarrow.cuda. cb89ee302 <Pearu Peterson> Remove usage of FreeHost to avoid double-freeing issue. 61659c4fe <Pearu Peterson> Introduce pyarrow.cuda module as the CUDA UI. 8faf1ee51 <Pearu Peterson> Add missing import of pandas_compat. a5447874d <Pearu Peterson> Fix formatting for flake8. 7053df860 <Pearu Peterson> Improve detecting availability of gpu support (2nd try). 3913e2904 <Pearu Peterson> Improve detecting availability of gpu support. 9e365bd15 <Pearu Peterson> Merge branch 'pearu-cuda-pyarrow' of github.com:Quansight/arrow into pearu-cuda-pyarrow d237b3493 <Pearu Peterson> Implement GPU support in pyarrow 584d94ba0 <Pearu Peterson> Flake it. c2799e645 <Pearu Peterson> Impl IPC tests. 7dc6d8833 <Pearu Peterson> Impl CudaBufferWriter.writeat, CudaBufferReader.read_buffer. Complete unittests for CudaBufferWriter/Reader e2b14df8b <Pearu Peterson> Impl CudaBuffer device_buffer and slice methods. Impl tests for CudaBuffer and CudaHostBuffer. 36396c689 <Pearu Peterson> Test memory management of CudaBuffer and CudaHostBuffer 5e2c1ba1e <Pearu Peterson> Unittest for allocate/free host buffer. 102f277e0 <Pearu Peterson> Document all methods and functions. 9b8cb1b9f <Pearu Peterson> Complete copy_from/to_host implementations and tests. 9eebf19aa <Pearu Peterson> Document copy_to_host and copy_from_host methods. Complete unittests for copy_to_host. 11cba54d2 <Pearu Peterson> Add copy_to_host and copy_from_host methods to CudaBuffer ae3cd3fda <Pearu Peterson> Exposed all Arrow GPU C++ classes and functions to pyarrow. WIP cf42941e9 <Pearu Peterson> Expose Message for lib_gpu. 49b0190ba <Pearu Peterson> Expose CudaDeviceManager, CudaContext, CudaIpcMemHandle to pyarrow. a2d755791 <Pearu Peterson> Minimal GPU support for pyarrow. WIP.
2018-09-13 19:07:46 -04:00
from pyarrow.lib cimport *
from pyarrow.includes.common cimport *
from pyarrow.includes.libarrow cimport *
from pyarrow.includes.libarrow_cuda cimport *
ARROW-9469: [Python] Make more objects weakrefable By default, Cython extension classes (defined with "cdef class") don't have a weakref slot, so add one to all of them. This adds just one memory word to each object, which IMHO is acceptable. Closes #7758 from pitrou/ARROW-9469-py-weakrefable-objects Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2020-07-29 12:24:36 +02:00
cdef class Context(_Weakrefable):
ARROW-1424: [Python] Add CUDA support to pyarrow This PR implements CUDA support to pyarrow. In order to use it, the Arrow C++ library must be built with the following cmake options: ``` -DARROW_PYTHON=on -DARROW_GPU=ON -DARROW_IPC=ON ``` To enable CUDA support in pyarrow, it must be built with `--with-cuda` option, e.g. ``` python setup.py build_ext --with-cuda <other options> ``` or the environment must define ``` PYARROW_WITH_CUDA=1 ``` This CUDA support implementation is rather complete: all Arrow C++ GPU related functions and classes are exposed to Python, the new methods and functions are documented, the test coverage is close to 100%. However, there are some issues that need to be tackled and questions to be answered (hence the WIP attribute): 1. Is the naming convention of the new methods appropriate? Are there any changes needed in the new API? 2. The IPC test (see `test_IPC` in `python/pyarrow/tests/test_gpu.py`) fails when calling `open_ipc_buffer`: `cuIpcOpenMemHandle fails with code 201` . Currently, I don't know why. Any hint or help on this is much appreciated. [FIXED: using `multiprocessing.Process` in `spawn` mode] 3. Anything else? Author: Pearu Peterson <pearu.peterson@gmail.com> Closes #2536 from pearu/pearu-cuda-pyarrow and squashes the following commits: 7a018465a <Pearu Peterson> Minor clean up 2. c86e1651a <Pearu Peterson> Minor clean up. 5e171e8c2 <Pearu Peterson> Revised buffer_from_data. 7555c657b <Pearu Peterson> Raise BufferError in CudaBuffer.__getbuffer__. Fix pytest.raises usages. e1bcb0886 <Pearu Peterson> Apply feedback from PR. 2. 38ddfffba <Pearu Peterson> Apply feedback from PR. WIP. 4ece8092a <Pearu Peterson> Remove redudant -DPYARROW_BUILD_CUDA=on 66d704ea5 <Pearu Peterson> Silence flake8 on Arrow CUDA api module. 88961fa35 <Pearu Peterson> cmake: moved Arrow CUDA detection to FindArrowCuda. 2126eba85 <Pearu Peterson> Fixes for flake8 ff7faa2d1 <Pearu Peterson> Removed DeviceManager, use 'ctx=Context(<device_number>)'. Introduced Context.get_num_devices and Context.device_number. f3c41dbae <Pearu Peterson> Added cdefs to CMemoryPool varibales 106b07197 <Pearu Peterson> Removed cuda prefix everywhere except in CudaBuffer. dff5bd495 <Pearu Peterson> More space removal. Fix CudaBufferWriter and cuda_read_record_batch docs. ce1d3bb50 <Pearu Peterson> Remove _freed attribute from CudaHostBuffer 1defcb4c8 <Pearu Peterson> Remove spaces around * and & 94989a7cd <Pearu Peterson> Rename lib_gpu to _cuda, ARROW_GPU to CUDA, --with-arrow-gpu to --with-cuda. Remove --without-arrow-gpu option. To test availability of cuda support, try import pyarrow.cuda. cb89ee302 <Pearu Peterson> Remove usage of FreeHost to avoid double-freeing issue. 61659c4fe <Pearu Peterson> Introduce pyarrow.cuda module as the CUDA UI. 8faf1ee51 <Pearu Peterson> Add missing import of pandas_compat. a5447874d <Pearu Peterson> Fix formatting for flake8. 7053df860 <Pearu Peterson> Improve detecting availability of gpu support (2nd try). 3913e2904 <Pearu Peterson> Improve detecting availability of gpu support. 9e365bd15 <Pearu Peterson> Merge branch 'pearu-cuda-pyarrow' of github.com:Quansight/arrow into pearu-cuda-pyarrow d237b3493 <Pearu Peterson> Implement GPU support in pyarrow 584d94ba0 <Pearu Peterson> Flake it. c2799e645 <Pearu Peterson> Impl IPC tests. 7dc6d8833 <Pearu Peterson> Impl CudaBufferWriter.writeat, CudaBufferReader.read_buffer. Complete unittests for CudaBufferWriter/Reader e2b14df8b <Pearu Peterson> Impl CudaBuffer device_buffer and slice methods. Impl tests for CudaBuffer and CudaHostBuffer. 36396c689 <Pearu Peterson> Test memory management of CudaBuffer and CudaHostBuffer 5e2c1ba1e <Pearu Peterson> Unittest for allocate/free host buffer. 102f277e0 <Pearu Peterson> Document all methods and functions. 9b8cb1b9f <Pearu Peterson> Complete copy_from/to_host implementations and tests. 9eebf19aa <Pearu Peterson> Document copy_to_host and copy_from_host methods. Complete unittests for copy_to_host. 11cba54d2 <Pearu Peterson> Add copy_to_host and copy_from_host methods to CudaBuffer ae3cd3fda <Pearu Peterson> Exposed all Arrow GPU C++ classes and functions to pyarrow. WIP cf42941e9 <Pearu Peterson> Expose Message for lib_gpu. 49b0190ba <Pearu Peterson> Expose CudaDeviceManager, CudaContext, CudaIpcMemHandle to pyarrow. a2d755791 <Pearu Peterson> Minimal GPU support for pyarrow. WIP.
2018-09-13 19:07:46 -04:00
cdef:
shared_ptr[CCudaContext] context
int device_number
cdef void init(self, const shared_ptr[CCudaContext]& ctx)
ARROW-9469: [Python] Make more objects weakrefable By default, Cython extension classes (defined with "cdef class") don't have a weakref slot, so add one to all of them. This adds just one memory word to each object, which IMHO is acceptable. Closes #7758 from pitrou/ARROW-9469-py-weakrefable-objects Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
2020-07-29 12:24:36 +02:00
cdef class IpcMemHandle(_Weakrefable):
ARROW-1424: [Python] Add CUDA support to pyarrow This PR implements CUDA support to pyarrow. In order to use it, the Arrow C++ library must be built with the following cmake options: ``` -DARROW_PYTHON=on -DARROW_GPU=ON -DARROW_IPC=ON ``` To enable CUDA support in pyarrow, it must be built with `--with-cuda` option, e.g. ``` python setup.py build_ext --with-cuda <other options> ``` or the environment must define ``` PYARROW_WITH_CUDA=1 ``` This CUDA support implementation is rather complete: all Arrow C++ GPU related functions and classes are exposed to Python, the new methods and functions are documented, the test coverage is close to 100%. However, there are some issues that need to be tackled and questions to be answered (hence the WIP attribute): 1. Is the naming convention of the new methods appropriate? Are there any changes needed in the new API? 2. The IPC test (see `test_IPC` in `python/pyarrow/tests/test_gpu.py`) fails when calling `open_ipc_buffer`: `cuIpcOpenMemHandle fails with code 201` . Currently, I don't know why. Any hint or help on this is much appreciated. [FIXED: using `multiprocessing.Process` in `spawn` mode] 3. Anything else? Author: Pearu Peterson <pearu.peterson@gmail.com> Closes #2536 from pearu/pearu-cuda-pyarrow and squashes the following commits: 7a018465a <Pearu Peterson> Minor clean up 2. c86e1651a <Pearu Peterson> Minor clean up. 5e171e8c2 <Pearu Peterson> Revised buffer_from_data. 7555c657b <Pearu Peterson> Raise BufferError in CudaBuffer.__getbuffer__. Fix pytest.raises usages. e1bcb0886 <Pearu Peterson> Apply feedback from PR. 2. 38ddfffba <Pearu Peterson> Apply feedback from PR. WIP. 4ece8092a <Pearu Peterson> Remove redudant -DPYARROW_BUILD_CUDA=on 66d704ea5 <Pearu Peterson> Silence flake8 on Arrow CUDA api module. 88961fa35 <Pearu Peterson> cmake: moved Arrow CUDA detection to FindArrowCuda. 2126eba85 <Pearu Peterson> Fixes for flake8 ff7faa2d1 <Pearu Peterson> Removed DeviceManager, use 'ctx=Context(<device_number>)'. Introduced Context.get_num_devices and Context.device_number. f3c41dbae <Pearu Peterson> Added cdefs to CMemoryPool varibales 106b07197 <Pearu Peterson> Removed cuda prefix everywhere except in CudaBuffer. dff5bd495 <Pearu Peterson> More space removal. Fix CudaBufferWriter and cuda_read_record_batch docs. ce1d3bb50 <Pearu Peterson> Remove _freed attribute from CudaHostBuffer 1defcb4c8 <Pearu Peterson> Remove spaces around * and & 94989a7cd <Pearu Peterson> Rename lib_gpu to _cuda, ARROW_GPU to CUDA, --with-arrow-gpu to --with-cuda. Remove --without-arrow-gpu option. To test availability of cuda support, try import pyarrow.cuda. cb89ee302 <Pearu Peterson> Remove usage of FreeHost to avoid double-freeing issue. 61659c4fe <Pearu Peterson> Introduce pyarrow.cuda module as the CUDA UI. 8faf1ee51 <Pearu Peterson> Add missing import of pandas_compat. a5447874d <Pearu Peterson> Fix formatting for flake8. 7053df860 <Pearu Peterson> Improve detecting availability of gpu support (2nd try). 3913e2904 <Pearu Peterson> Improve detecting availability of gpu support. 9e365bd15 <Pearu Peterson> Merge branch 'pearu-cuda-pyarrow' of github.com:Quansight/arrow into pearu-cuda-pyarrow d237b3493 <Pearu Peterson> Implement GPU support in pyarrow 584d94ba0 <Pearu Peterson> Flake it. c2799e645 <Pearu Peterson> Impl IPC tests. 7dc6d8833 <Pearu Peterson> Impl CudaBufferWriter.writeat, CudaBufferReader.read_buffer. Complete unittests for CudaBufferWriter/Reader e2b14df8b <Pearu Peterson> Impl CudaBuffer device_buffer and slice methods. Impl tests for CudaBuffer and CudaHostBuffer. 36396c689 <Pearu Peterson> Test memory management of CudaBuffer and CudaHostBuffer 5e2c1ba1e <Pearu Peterson> Unittest for allocate/free host buffer. 102f277e0 <Pearu Peterson> Document all methods and functions. 9b8cb1b9f <Pearu Peterson> Complete copy_from/to_host implementations and tests. 9eebf19aa <Pearu Peterson> Document copy_to_host and copy_from_host methods. Complete unittests for copy_to_host. 11cba54d2 <Pearu Peterson> Add copy_to_host and copy_from_host methods to CudaBuffer ae3cd3fda <Pearu Peterson> Exposed all Arrow GPU C++ classes and functions to pyarrow. WIP cf42941e9 <Pearu Peterson> Expose Message for lib_gpu. 49b0190ba <Pearu Peterson> Expose CudaDeviceManager, CudaContext, CudaIpcMemHandle to pyarrow. a2d755791 <Pearu Peterson> Minimal GPU support for pyarrow. WIP.
2018-09-13 19:07:46 -04:00
cdef:
shared_ptr[CCudaIpcMemHandle] handle
cdef void init(self, shared_ptr[CCudaIpcMemHandle]& h)
cdef class CudaBuffer(Buffer):
cdef:
shared_ptr[CCudaBuffer] cuda_buffer
ARROW-4486: [Python][CUDA] Add base argument to foreign_buffer Resolves [ARROW-4486](https://issues.apache.org/jira/browse/ARROW-4486) Author: Pearu Peterson <pearu.peterson@gmail.com> Closes #3850 from pearu/arrow-4486 and squashes the following commits: e96265df0 <Pearu Peterson> Add base argument to foreign_buffer, resolves ARROW-4486
2019-03-14 09:47:20 -05:00
object base
ARROW-1424: [Python] Add CUDA support to pyarrow This PR implements CUDA support to pyarrow. In order to use it, the Arrow C++ library must be built with the following cmake options: ``` -DARROW_PYTHON=on -DARROW_GPU=ON -DARROW_IPC=ON ``` To enable CUDA support in pyarrow, it must be built with `--with-cuda` option, e.g. ``` python setup.py build_ext --with-cuda <other options> ``` or the environment must define ``` PYARROW_WITH_CUDA=1 ``` This CUDA support implementation is rather complete: all Arrow C++ GPU related functions and classes are exposed to Python, the new methods and functions are documented, the test coverage is close to 100%. However, there are some issues that need to be tackled and questions to be answered (hence the WIP attribute): 1. Is the naming convention of the new methods appropriate? Are there any changes needed in the new API? 2. The IPC test (see `test_IPC` in `python/pyarrow/tests/test_gpu.py`) fails when calling `open_ipc_buffer`: `cuIpcOpenMemHandle fails with code 201` . Currently, I don't know why. Any hint or help on this is much appreciated. [FIXED: using `multiprocessing.Process` in `spawn` mode] 3. Anything else? Author: Pearu Peterson <pearu.peterson@gmail.com> Closes #2536 from pearu/pearu-cuda-pyarrow and squashes the following commits: 7a018465a <Pearu Peterson> Minor clean up 2. c86e1651a <Pearu Peterson> Minor clean up. 5e171e8c2 <Pearu Peterson> Revised buffer_from_data. 7555c657b <Pearu Peterson> Raise BufferError in CudaBuffer.__getbuffer__. Fix pytest.raises usages. e1bcb0886 <Pearu Peterson> Apply feedback from PR. 2. 38ddfffba <Pearu Peterson> Apply feedback from PR. WIP. 4ece8092a <Pearu Peterson> Remove redudant -DPYARROW_BUILD_CUDA=on 66d704ea5 <Pearu Peterson> Silence flake8 on Arrow CUDA api module. 88961fa35 <Pearu Peterson> cmake: moved Arrow CUDA detection to FindArrowCuda. 2126eba85 <Pearu Peterson> Fixes for flake8 ff7faa2d1 <Pearu Peterson> Removed DeviceManager, use 'ctx=Context(<device_number>)'. Introduced Context.get_num_devices and Context.device_number. f3c41dbae <Pearu Peterson> Added cdefs to CMemoryPool varibales 106b07197 <Pearu Peterson> Removed cuda prefix everywhere except in CudaBuffer. dff5bd495 <Pearu Peterson> More space removal. Fix CudaBufferWriter and cuda_read_record_batch docs. ce1d3bb50 <Pearu Peterson> Remove _freed attribute from CudaHostBuffer 1defcb4c8 <Pearu Peterson> Remove spaces around * and & 94989a7cd <Pearu Peterson> Rename lib_gpu to _cuda, ARROW_GPU to CUDA, --with-arrow-gpu to --with-cuda. Remove --without-arrow-gpu option. To test availability of cuda support, try import pyarrow.cuda. cb89ee302 <Pearu Peterson> Remove usage of FreeHost to avoid double-freeing issue. 61659c4fe <Pearu Peterson> Introduce pyarrow.cuda module as the CUDA UI. 8faf1ee51 <Pearu Peterson> Add missing import of pandas_compat. a5447874d <Pearu Peterson> Fix formatting for flake8. 7053df860 <Pearu Peterson> Improve detecting availability of gpu support (2nd try). 3913e2904 <Pearu Peterson> Improve detecting availability of gpu support. 9e365bd15 <Pearu Peterson> Merge branch 'pearu-cuda-pyarrow' of github.com:Quansight/arrow into pearu-cuda-pyarrow d237b3493 <Pearu Peterson> Implement GPU support in pyarrow 584d94ba0 <Pearu Peterson> Flake it. c2799e645 <Pearu Peterson> Impl IPC tests. 7dc6d8833 <Pearu Peterson> Impl CudaBufferWriter.writeat, CudaBufferReader.read_buffer. Complete unittests for CudaBufferWriter/Reader e2b14df8b <Pearu Peterson> Impl CudaBuffer device_buffer and slice methods. Impl tests for CudaBuffer and CudaHostBuffer. 36396c689 <Pearu Peterson> Test memory management of CudaBuffer and CudaHostBuffer 5e2c1ba1e <Pearu Peterson> Unittest for allocate/free host buffer. 102f277e0 <Pearu Peterson> Document all methods and functions. 9b8cb1b9f <Pearu Peterson> Complete copy_from/to_host implementations and tests. 9eebf19aa <Pearu Peterson> Document copy_to_host and copy_from_host methods. Complete unittests for copy_to_host. 11cba54d2 <Pearu Peterson> Add copy_to_host and copy_from_host methods to CudaBuffer ae3cd3fda <Pearu Peterson> Exposed all Arrow GPU C++ classes and functions to pyarrow. WIP cf42941e9 <Pearu Peterson> Expose Message for lib_gpu. 49b0190ba <Pearu Peterson> Expose CudaDeviceManager, CudaContext, CudaIpcMemHandle to pyarrow. a2d755791 <Pearu Peterson> Minimal GPU support for pyarrow. WIP.
2018-09-13 19:07:46 -04:00
ARROW-4486: [Python][CUDA] Add base argument to foreign_buffer Resolves [ARROW-4486](https://issues.apache.org/jira/browse/ARROW-4486) Author: Pearu Peterson <pearu.peterson@gmail.com> Closes #3850 from pearu/arrow-4486 and squashes the following commits: e96265df0 <Pearu Peterson> Add base argument to foreign_buffer, resolves ARROW-4486
2019-03-14 09:47:20 -05:00
cdef void init_cuda(self,
const shared_ptr[CCudaBuffer]& buffer,
object base)
ARROW-1424: [Python] Add CUDA support to pyarrow This PR implements CUDA support to pyarrow. In order to use it, the Arrow C++ library must be built with the following cmake options: ``` -DARROW_PYTHON=on -DARROW_GPU=ON -DARROW_IPC=ON ``` To enable CUDA support in pyarrow, it must be built with `--with-cuda` option, e.g. ``` python setup.py build_ext --with-cuda <other options> ``` or the environment must define ``` PYARROW_WITH_CUDA=1 ``` This CUDA support implementation is rather complete: all Arrow C++ GPU related functions and classes are exposed to Python, the new methods and functions are documented, the test coverage is close to 100%. However, there are some issues that need to be tackled and questions to be answered (hence the WIP attribute): 1. Is the naming convention of the new methods appropriate? Are there any changes needed in the new API? 2. The IPC test (see `test_IPC` in `python/pyarrow/tests/test_gpu.py`) fails when calling `open_ipc_buffer`: `cuIpcOpenMemHandle fails with code 201` . Currently, I don't know why. Any hint or help on this is much appreciated. [FIXED: using `multiprocessing.Process` in `spawn` mode] 3. Anything else? Author: Pearu Peterson <pearu.peterson@gmail.com> Closes #2536 from pearu/pearu-cuda-pyarrow and squashes the following commits: 7a018465a <Pearu Peterson> Minor clean up 2. c86e1651a <Pearu Peterson> Minor clean up. 5e171e8c2 <Pearu Peterson> Revised buffer_from_data. 7555c657b <Pearu Peterson> Raise BufferError in CudaBuffer.__getbuffer__. Fix pytest.raises usages. e1bcb0886 <Pearu Peterson> Apply feedback from PR. 2. 38ddfffba <Pearu Peterson> Apply feedback from PR. WIP. 4ece8092a <Pearu Peterson> Remove redudant -DPYARROW_BUILD_CUDA=on 66d704ea5 <Pearu Peterson> Silence flake8 on Arrow CUDA api module. 88961fa35 <Pearu Peterson> cmake: moved Arrow CUDA detection to FindArrowCuda. 2126eba85 <Pearu Peterson> Fixes for flake8 ff7faa2d1 <Pearu Peterson> Removed DeviceManager, use 'ctx=Context(<device_number>)'. Introduced Context.get_num_devices and Context.device_number. f3c41dbae <Pearu Peterson> Added cdefs to CMemoryPool varibales 106b07197 <Pearu Peterson> Removed cuda prefix everywhere except in CudaBuffer. dff5bd495 <Pearu Peterson> More space removal. Fix CudaBufferWriter and cuda_read_record_batch docs. ce1d3bb50 <Pearu Peterson> Remove _freed attribute from CudaHostBuffer 1defcb4c8 <Pearu Peterson> Remove spaces around * and & 94989a7cd <Pearu Peterson> Rename lib_gpu to _cuda, ARROW_GPU to CUDA, --with-arrow-gpu to --with-cuda. Remove --without-arrow-gpu option. To test availability of cuda support, try import pyarrow.cuda. cb89ee302 <Pearu Peterson> Remove usage of FreeHost to avoid double-freeing issue. 61659c4fe <Pearu Peterson> Introduce pyarrow.cuda module as the CUDA UI. 8faf1ee51 <Pearu Peterson> Add missing import of pandas_compat. a5447874d <Pearu Peterson> Fix formatting for flake8. 7053df860 <Pearu Peterson> Improve detecting availability of gpu support (2nd try). 3913e2904 <Pearu Peterson> Improve detecting availability of gpu support. 9e365bd15 <Pearu Peterson> Merge branch 'pearu-cuda-pyarrow' of github.com:Quansight/arrow into pearu-cuda-pyarrow d237b3493 <Pearu Peterson> Implement GPU support in pyarrow 584d94ba0 <Pearu Peterson> Flake it. c2799e645 <Pearu Peterson> Impl IPC tests. 7dc6d8833 <Pearu Peterson> Impl CudaBufferWriter.writeat, CudaBufferReader.read_buffer. Complete unittests for CudaBufferWriter/Reader e2b14df8b <Pearu Peterson> Impl CudaBuffer device_buffer and slice methods. Impl tests for CudaBuffer and CudaHostBuffer. 36396c689 <Pearu Peterson> Test memory management of CudaBuffer and CudaHostBuffer 5e2c1ba1e <Pearu Peterson> Unittest for allocate/free host buffer. 102f277e0 <Pearu Peterson> Document all methods and functions. 9b8cb1b9f <Pearu Peterson> Complete copy_from/to_host implementations and tests. 9eebf19aa <Pearu Peterson> Document copy_to_host and copy_from_host methods. Complete unittests for copy_to_host. 11cba54d2 <Pearu Peterson> Add copy_to_host and copy_from_host methods to CudaBuffer ae3cd3fda <Pearu Peterson> Exposed all Arrow GPU C++ classes and functions to pyarrow. WIP cf42941e9 <Pearu Peterson> Expose Message for lib_gpu. 49b0190ba <Pearu Peterson> Expose CudaDeviceManager, CudaContext, CudaIpcMemHandle to pyarrow. a2d755791 <Pearu Peterson> Minimal GPU support for pyarrow. WIP.
2018-09-13 19:07:46 -04:00
cdef class HostBuffer(Buffer):
cdef:
shared_ptr[CCudaHostBuffer] host_buffer
cdef void init_host(self, const shared_ptr[CCudaHostBuffer]& buffer)
cdef class BufferReader(NativeFile):
cdef:
CCudaBufferReader* reader
CudaBuffer buffer
cdef class BufferWriter(NativeFile):
cdef:
CCudaBufferWriter* writer
CudaBuffer buffer
Reference in New Issue Copy Permalink