SIGN IN SIGN UP
apache / arrow UNCLAIMED

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

0 0 16 C++
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# cython: language_level = 3
from pyarrow.includes.common cimport *
from pyarrow.includes.libarrow_fs cimport *
from pyarrow.lib import _detect_compression, frombytes, tobytes
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
from pyarrow.lib cimport *
cpdef enum FileType:
NotFound = <int8_t> CFileType_NotFound
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
Unknown = <int8_t> CFileType_Unknown
File = <int8_t> CFileType_File
Directory = <int8_t> CFileType_Directory
cdef class FileInfo(_Weakrefable):
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
cdef:
CFileInfo info
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
@staticmethod
cdef wrap(CFileInfo info)
ARROW-6341: [Python] Implement low-level bindings for Dataset Closes #5237 from kszucs/ARROW-6341 and squashes the following commits: 45121f77a <Krisztián Szűcs> Fix tests for MockFs 069d8f55e <Krisztián Szűcs> Don't expose SimpleDataSource 48b5556ef <Krisztián Szűcs> Test projected partitions 27dbe56ab <Krisztián Szűcs> Don't deprecate RandomFileAccess just remove in favor of CRandomFileAccess 64ca71245 <Krisztián Szűcs> Execute instead of scan bd0d1d2e7 <Krisztián Szűcs> more type_name f2ab5ebfe <Krisztián Szűcs> type_name 3988865ca <Krisztián Szűcs> Rebase again b8949a3ce <Krisztián Szűcs> Result iterator api 7553caa96 <Krisztián Szűcs> Clang format a1d82546f <Krisztián Szűcs> Remove ScanContext 6260734a3 <Krisztián Szűcs> Fix api changes e6a562356 <Krisztián Szűcs> Expose root_partition setter; resolve a couple of review issues c9ba0fb93 <Krisztián Szűcs> Fix review comments f589ecb23 <Krisztián Szűcs> Removed todo notes 9b38a40cf <Krisztián Szűcs> Docstring additions 3210e9160 <Krisztián Szűcs> Fixing review issues 4384b74cf <Krisztián Szűcs> Enable PYARROW_BUILD_DATASET f52e735a0 <Krisztián Szűcs> Remove DataFragment and ScanOptions 13eaf46a0 <Krisztián Szűcs> Remove move workaround 620ba6ffa <Krisztián Szűcs> schema as property e9f77bd6b <Krisztián Szűcs> Expose root_partition 01510bcf8 <Krisztián Szűcs> Some docstrings 5beb0d26c <Krisztián Szűcs> Pxd definition fixes f89dc4913 <Krisztián Szűcs> Data fragments c9881c858 <Krisztián Szűcs> Downcast data fragments 032a4358c <Krisztián Szűcs> More expressions and testing 1a4a8544a <Krisztián Szűcs> Fix import errors if dataset is not enabled 2da1b5c76 <Krisztián Szűcs> Please the linters d1bc74efe <Krisztián Szűcs> HivePartitionScheme bf5dd17f4 <Krisztián Szűcs> Release the gil for std::move a76dc6c3c <Krisztián Szűcs> Remove the move headers from flight 8cdfe1054 <Krisztián Szűcs> Expose more methods 53b64910e <Krisztián Szűcs> Expose Scalar/Comparison/Boolean expressions 444ae58a0 <Krisztián Szűcs> Expressions and scalar wrapping 01029a666 <Krisztián Szűcs> Test parquet data discovery 2e416ea8f <Krisztián Szűcs> Expressions d14cf502b <Krisztián Szűcs> PartitionScheme bd6e1d656 <Krisztián Szűcs> FileSystemDataSourceDiscovery 0c0e3752f <Krisztián Szűcs> Working scanner 18cfd949b <Krisztián Szűcs> Resolve issues with handling iterator results 43c3d2bb2 <Krisztián Szűcs> MockFileSystem and use Results 3bbc51bd0 <Krisztián Szűcs> MockFileSystem 5e279ead2 <Krisztián Szűcs> Basic scanning c0d6715aa <Krisztián Szűcs> Rebase datasets 132e2f553 <Krisztián Szűcs> license d51b560d7 <Krisztián Szűcs> data fragments and data sources a0b35c7a0 <Krisztián Szűcs> skeleton and headers Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
2019-12-13 12:26:07 -05:00
cdef inline CFileInfo unwrap(self) nogil
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
@staticmethod
cdef CFileInfo unwrap_safe(obj)
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
cdef class FileSelector(_Weakrefable):
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
cdef:
CFileSelector selector
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
@staticmethod
cdef FileSelector wrap(CFileSelector selector)
cdef inline CFileSelector unwrap(self) nogil
ARROW-6341: [Python] Implement low-level bindings for Dataset Closes #5237 from kszucs/ARROW-6341 and squashes the following commits: 45121f77a <Krisztián Szűcs> Fix tests for MockFs 069d8f55e <Krisztián Szűcs> Don't expose SimpleDataSource 48b5556ef <Krisztián Szűcs> Test projected partitions 27dbe56ab <Krisztián Szűcs> Don't deprecate RandomFileAccess just remove in favor of CRandomFileAccess 64ca71245 <Krisztián Szűcs> Execute instead of scan bd0d1d2e7 <Krisztián Szűcs> more type_name f2ab5ebfe <Krisztián Szűcs> type_name 3988865ca <Krisztián Szűcs> Rebase again b8949a3ce <Krisztián Szűcs> Result iterator api 7553caa96 <Krisztián Szűcs> Clang format a1d82546f <Krisztián Szűcs> Remove ScanContext 6260734a3 <Krisztián Szűcs> Fix api changes e6a562356 <Krisztián Szűcs> Expose root_partition setter; resolve a couple of review issues c9ba0fb93 <Krisztián Szűcs> Fix review comments f589ecb23 <Krisztián Szűcs> Removed todo notes 9b38a40cf <Krisztián Szűcs> Docstring additions 3210e9160 <Krisztián Szűcs> Fixing review issues 4384b74cf <Krisztián Szűcs> Enable PYARROW_BUILD_DATASET f52e735a0 <Krisztián Szűcs> Remove DataFragment and ScanOptions 13eaf46a0 <Krisztián Szűcs> Remove move workaround 620ba6ffa <Krisztián Szűcs> schema as property e9f77bd6b <Krisztián Szűcs> Expose root_partition 01510bcf8 <Krisztián Szűcs> Some docstrings 5beb0d26c <Krisztián Szűcs> Pxd definition fixes f89dc4913 <Krisztián Szűcs> Data fragments c9881c858 <Krisztián Szűcs> Downcast data fragments 032a4358c <Krisztián Szűcs> More expressions and testing 1a4a8544a <Krisztián Szűcs> Fix import errors if dataset is not enabled 2da1b5c76 <Krisztián Szűcs> Please the linters d1bc74efe <Krisztián Szűcs> HivePartitionScheme bf5dd17f4 <Krisztián Szűcs> Release the gil for std::move a76dc6c3c <Krisztián Szűcs> Remove the move headers from flight 8cdfe1054 <Krisztián Szűcs> Expose more methods 53b64910e <Krisztián Szűcs> Expose Scalar/Comparison/Boolean expressions 444ae58a0 <Krisztián Szűcs> Expressions and scalar wrapping 01029a666 <Krisztián Szűcs> Test parquet data discovery 2e416ea8f <Krisztián Szűcs> Expressions d14cf502b <Krisztián Szűcs> PartitionScheme bd6e1d656 <Krisztián Szűcs> FileSystemDataSourceDiscovery 0c0e3752f <Krisztián Szűcs> Working scanner 18cfd949b <Krisztián Szűcs> Resolve issues with handling iterator results 43c3d2bb2 <Krisztián Szűcs> MockFileSystem and use Results 3bbc51bd0 <Krisztián Szűcs> MockFileSystem 5e279ead2 <Krisztián Szűcs> Basic scanning c0d6715aa <Krisztián Szűcs> Rebase datasets 132e2f553 <Krisztián Szűcs> license d51b560d7 <Krisztián Szűcs> data fragments and data sources a0b35c7a0 <Krisztián Szűcs> skeleton and headers Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
2019-12-13 12:26:07 -05:00
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
cdef class FileSystem(_Weakrefable):
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
cdef:
shared_ptr[CFileSystem] wrapped
CFileSystem* fs
cdef init(self, const shared_ptr[CFileSystem]& wrapped)
ARROW-6341: [Python] Implement low-level bindings for Dataset Closes #5237 from kszucs/ARROW-6341 and squashes the following commits: 45121f77a <Krisztián Szűcs> Fix tests for MockFs 069d8f55e <Krisztián Szűcs> Don't expose SimpleDataSource 48b5556ef <Krisztián Szűcs> Test projected partitions 27dbe56ab <Krisztián Szűcs> Don't deprecate RandomFileAccess just remove in favor of CRandomFileAccess 64ca71245 <Krisztián Szűcs> Execute instead of scan bd0d1d2e7 <Krisztián Szűcs> more type_name f2ab5ebfe <Krisztián Szűcs> type_name 3988865ca <Krisztián Szűcs> Rebase again b8949a3ce <Krisztián Szűcs> Result iterator api 7553caa96 <Krisztián Szűcs> Clang format a1d82546f <Krisztián Szűcs> Remove ScanContext 6260734a3 <Krisztián Szűcs> Fix api changes e6a562356 <Krisztián Szűcs> Expose root_partition setter; resolve a couple of review issues c9ba0fb93 <Krisztián Szűcs> Fix review comments f589ecb23 <Krisztián Szűcs> Removed todo notes 9b38a40cf <Krisztián Szűcs> Docstring additions 3210e9160 <Krisztián Szűcs> Fixing review issues 4384b74cf <Krisztián Szűcs> Enable PYARROW_BUILD_DATASET f52e735a0 <Krisztián Szűcs> Remove DataFragment and ScanOptions 13eaf46a0 <Krisztián Szűcs> Remove move workaround 620ba6ffa <Krisztián Szűcs> schema as property e9f77bd6b <Krisztián Szűcs> Expose root_partition 01510bcf8 <Krisztián Szűcs> Some docstrings 5beb0d26c <Krisztián Szűcs> Pxd definition fixes f89dc4913 <Krisztián Szűcs> Data fragments c9881c858 <Krisztián Szűcs> Downcast data fragments 032a4358c <Krisztián Szűcs> More expressions and testing 1a4a8544a <Krisztián Szűcs> Fix import errors if dataset is not enabled 2da1b5c76 <Krisztián Szűcs> Please the linters d1bc74efe <Krisztián Szűcs> HivePartitionScheme bf5dd17f4 <Krisztián Szűcs> Release the gil for std::move a76dc6c3c <Krisztián Szűcs> Remove the move headers from flight 8cdfe1054 <Krisztián Szűcs> Expose more methods 53b64910e <Krisztián Szűcs> Expose Scalar/Comparison/Boolean expressions 444ae58a0 <Krisztián Szűcs> Expressions and scalar wrapping 01029a666 <Krisztián Szűcs> Test parquet data discovery 2e416ea8f <Krisztián Szűcs> Expressions d14cf502b <Krisztián Szűcs> PartitionScheme bd6e1d656 <Krisztián Szűcs> FileSystemDataSourceDiscovery 0c0e3752f <Krisztián Szűcs> Working scanner 18cfd949b <Krisztián Szűcs> Resolve issues with handling iterator results 43c3d2bb2 <Krisztián Szűcs> MockFileSystem and use Results 3bbc51bd0 <Krisztián Szűcs> MockFileSystem 5e279ead2 <Krisztián Szűcs> Basic scanning c0d6715aa <Krisztián Szűcs> Rebase datasets 132e2f553 <Krisztián Szűcs> license d51b560d7 <Krisztián Szűcs> data fragments and data sources a0b35c7a0 <Krisztián Szűcs> skeleton and headers Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
2019-12-13 12:26:07 -05:00
@staticmethod
cdef wrap(const shared_ptr[CFileSystem]& sp)
ARROW-6341: [Python] Implement low-level bindings for Dataset Closes #5237 from kszucs/ARROW-6341 and squashes the following commits: 45121f77a <Krisztián Szűcs> Fix tests for MockFs 069d8f55e <Krisztián Szűcs> Don't expose SimpleDataSource 48b5556ef <Krisztián Szűcs> Test projected partitions 27dbe56ab <Krisztián Szűcs> Don't deprecate RandomFileAccess just remove in favor of CRandomFileAccess 64ca71245 <Krisztián Szűcs> Execute instead of scan bd0d1d2e7 <Krisztián Szűcs> more type_name f2ab5ebfe <Krisztián Szűcs> type_name 3988865ca <Krisztián Szűcs> Rebase again b8949a3ce <Krisztián Szűcs> Result iterator api 7553caa96 <Krisztián Szűcs> Clang format a1d82546f <Krisztián Szűcs> Remove ScanContext 6260734a3 <Krisztián Szűcs> Fix api changes e6a562356 <Krisztián Szűcs> Expose root_partition setter; resolve a couple of review issues c9ba0fb93 <Krisztián Szűcs> Fix review comments f589ecb23 <Krisztián Szűcs> Removed todo notes 9b38a40cf <Krisztián Szűcs> Docstring additions 3210e9160 <Krisztián Szűcs> Fixing review issues 4384b74cf <Krisztián Szűcs> Enable PYARROW_BUILD_DATASET f52e735a0 <Krisztián Szűcs> Remove DataFragment and ScanOptions 13eaf46a0 <Krisztián Szűcs> Remove move workaround 620ba6ffa <Krisztián Szűcs> schema as property e9f77bd6b <Krisztián Szűcs> Expose root_partition 01510bcf8 <Krisztián Szűcs> Some docstrings 5beb0d26c <Krisztián Szűcs> Pxd definition fixes f89dc4913 <Krisztián Szűcs> Data fragments c9881c858 <Krisztián Szűcs> Downcast data fragments 032a4358c <Krisztián Szűcs> More expressions and testing 1a4a8544a <Krisztián Szűcs> Fix import errors if dataset is not enabled 2da1b5c76 <Krisztián Szűcs> Please the linters d1bc74efe <Krisztián Szűcs> HivePartitionScheme bf5dd17f4 <Krisztián Szűcs> Release the gil for std::move a76dc6c3c <Krisztián Szűcs> Remove the move headers from flight 8cdfe1054 <Krisztián Szűcs> Expose more methods 53b64910e <Krisztián Szűcs> Expose Scalar/Comparison/Boolean expressions 444ae58a0 <Krisztián Szűcs> Expressions and scalar wrapping 01029a666 <Krisztián Szűcs> Test parquet data discovery 2e416ea8f <Krisztián Szűcs> Expressions d14cf502b <Krisztián Szűcs> PartitionScheme bd6e1d656 <Krisztián Szűcs> FileSystemDataSourceDiscovery 0c0e3752f <Krisztián Szűcs> Working scanner 18cfd949b <Krisztián Szűcs> Resolve issues with handling iterator results 43c3d2bb2 <Krisztián Szűcs> MockFileSystem and use Results 3bbc51bd0 <Krisztián Szűcs> MockFileSystem 5e279ead2 <Krisztián Szűcs> Basic scanning c0d6715aa <Krisztián Szűcs> Rebase datasets 132e2f553 <Krisztián Szűcs> license d51b560d7 <Krisztián Szűcs> data fragments and data sources a0b35c7a0 <Krisztián Szűcs> skeleton and headers Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
2019-12-13 12:26:07 -05:00
cdef inline shared_ptr[CFileSystem] unwrap(self) nogil
ARROW-6655: [Python] Filesystem bindings for S3 - Add support for S3FileSystem in the python bindings. - Fixed issue with reading all the content of an S3 object - Introduce `minio_server` fixture for parametrized testing of all filesystem implementations - Fixed s3fs parquet test and updated it to use minio_server fixture Closes #5423 from kszucs/s3 and squashes the following commits: 384c96052 <Krisztián Szűcs> Resolve review comments 73e6625f9 <Krisztián Szűcs> S3Options 98bd91ad1 <Krisztián Szűcs> remove commented tests db89859ca <Krisztián Szűcs> rename to s3fs 44784582a <Krisztián Szűcs> fix read() issue c1df10b92 <Krisztián Szűcs> initialization in first use 192ab6547 <Krisztián Szűcs> flake8 f70f9fbd8 <Krisztián Szűcs> remove minio-client dependency d399643dc <Krisztián Szűcs> simplify test suite fee57a9a4 <Krisztián Szűcs> remove accidentally committed files 751cfd429 <Krisztián Szűcs> resolve a couple of review comments; enum workaround 45436f7b3 <Krisztián Szűcs> cython flake8 38dcb88d0 <Krisztián Szűcs> rat c541b3e15 <Krisztián Szűcs> comment left 098048a8a <Krisztián Szűcs> more compat 00340ed4c <Krisztián Szűcs> fixture error handling 2be25ce29 <Krisztián Szűcs> auto initialize s3 on import 88e0c9f79 <Krisztián Szűcs> py2 compat 8585a6085 <Krisztián Szűcs> py2 compat d37228711 <Krisztián Szűcs> install minio in the conda-toolchain build 041cad42a <Krisztián Szűcs> executable flag fb0f2813a <Krisztián Szűcs> travis minio install script 8cbe0eeef <Krisztián Szűcs> travis osx 72e56a68f <Krisztián Szűcs> enable S3 in travis python builds 7800c75d8 <Krisztián Szűcs> appveyor flag 7daf5668d <Krisztián Szűcs> fix syntax error in travis script 68eb59161 <Krisztián Szűcs> enable PYARROW_WITH_S3 on appveyor 2cb19d1ff <Krisztián Szűcs> conditional import of test dependencies efa05d28e <Krisztián Szűcs> use minio for dask.s3fs test too f25ae5aed <Krisztián Szűcs> travis 9042c7e4c <Krisztián Szűcs> use S3FS_DIR 9ce7180d1 <Krisztián Szűcs> cmake format; fix orc cimport 45a2a17ba <Krisztián Szűcs> docstrings c0b91621e <Krisztián Szűcs> test requirements; flake8 44aedfd10 <Krisztián Szűcs> stat test a343950e2 <Krisztián Szűcs> testing suite 1551b525c <Krisztián Szűcs> wip dd41d21c2 <Krisztián Szűcs> imports 5200af16f <Krisztián Szűcs> s3 filesystem bindings Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
2019-10-01 15:22:19 +02:00
cdef class LocalFileSystem(FileSystem):
cdef init(self, const shared_ptr[CFileSystem]& wrapped)
cdef class SubTreeFileSystem(FileSystem):
cdef:
CSubTreeFileSystem* subtreefs
cdef init(self, const shared_ptr[CFileSystem]& wrapped)
ARROW-6341: [Python] Implement low-level bindings for Dataset Closes #5237 from kszucs/ARROW-6341 and squashes the following commits: 45121f77a <Krisztián Szűcs> Fix tests for MockFs 069d8f55e <Krisztián Szűcs> Don't expose SimpleDataSource 48b5556ef <Krisztián Szűcs> Test projected partitions 27dbe56ab <Krisztián Szűcs> Don't deprecate RandomFileAccess just remove in favor of CRandomFileAccess 64ca71245 <Krisztián Szűcs> Execute instead of scan bd0d1d2e7 <Krisztián Szűcs> more type_name f2ab5ebfe <Krisztián Szűcs> type_name 3988865ca <Krisztián Szűcs> Rebase again b8949a3ce <Krisztián Szűcs> Result iterator api 7553caa96 <Krisztián Szűcs> Clang format a1d82546f <Krisztián Szűcs> Remove ScanContext 6260734a3 <Krisztián Szűcs> Fix api changes e6a562356 <Krisztián Szűcs> Expose root_partition setter; resolve a couple of review issues c9ba0fb93 <Krisztián Szűcs> Fix review comments f589ecb23 <Krisztián Szűcs> Removed todo notes 9b38a40cf <Krisztián Szűcs> Docstring additions 3210e9160 <Krisztián Szűcs> Fixing review issues 4384b74cf <Krisztián Szűcs> Enable PYARROW_BUILD_DATASET f52e735a0 <Krisztián Szűcs> Remove DataFragment and ScanOptions 13eaf46a0 <Krisztián Szűcs> Remove move workaround 620ba6ffa <Krisztián Szűcs> schema as property e9f77bd6b <Krisztián Szűcs> Expose root_partition 01510bcf8 <Krisztián Szűcs> Some docstrings 5beb0d26c <Krisztián Szűcs> Pxd definition fixes f89dc4913 <Krisztián Szűcs> Data fragments c9881c858 <Krisztián Szűcs> Downcast data fragments 032a4358c <Krisztián Szűcs> More expressions and testing 1a4a8544a <Krisztián Szűcs> Fix import errors if dataset is not enabled 2da1b5c76 <Krisztián Szűcs> Please the linters d1bc74efe <Krisztián Szűcs> HivePartitionScheme bf5dd17f4 <Krisztián Szűcs> Release the gil for std::move a76dc6c3c <Krisztián Szűcs> Remove the move headers from flight 8cdfe1054 <Krisztián Szűcs> Expose more methods 53b64910e <Krisztián Szűcs> Expose Scalar/Comparison/Boolean expressions 444ae58a0 <Krisztián Szűcs> Expressions and scalar wrapping 01029a666 <Krisztián Szűcs> Test parquet data discovery 2e416ea8f <Krisztián Szűcs> Expressions d14cf502b <Krisztián Szűcs> PartitionScheme bd6e1d656 <Krisztián Szűcs> FileSystemDataSourceDiscovery 0c0e3752f <Krisztián Szűcs> Working scanner 18cfd949b <Krisztián Szűcs> Resolve issues with handling iterator results 43c3d2bb2 <Krisztián Szűcs> MockFileSystem and use Results 3bbc51bd0 <Krisztián Szűcs> MockFileSystem 5e279ead2 <Krisztián Szűcs> Basic scanning c0d6715aa <Krisztián Szűcs> Rebase datasets 132e2f553 <Krisztián Szűcs> license d51b560d7 <Krisztián Szűcs> data fragments and data sources a0b35c7a0 <Krisztián Szűcs> skeleton and headers Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
2019-12-13 12:26:07 -05:00
cdef class _MockFileSystem(FileSystem):
cdef:
CMockFileSystem* mockfs
cdef init(self, const shared_ptr[CFileSystem]& wrapped)
cdef class PyFileSystem(FileSystem):
cdef:
CPyFileSystem* pyfs
cdef init(self, const shared_ptr[CFileSystem]& wrapped)