Skip to content

Commit d6d425d

Browse files
authored
Declare pyarrow for Parquet output of SqlToS3Operator (#74691)
Writing Parquet files and reading with the pyarrow dtype backend both need pyarrow, but the provider never declared it. It only worked when another installed package happened to bring pyarrow in, and otherwise failed at runtime with pandas' "Unable to find a usable engine" error. A separate extra keeps the roughly 40 MB dependency out of installs that only write CSV or JSON.
1 parent a23aa81 commit d6d425d

4 files changed

Lines changed: 23 additions & 5 deletions

File tree

‎providers/amazon/docs/index.rst‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -174,9 +174,9 @@ Install them when installing from PyPI. For example:
174174
pip install apache-airflow-providers-amazon[aiobotocore]
175175
176176
177-
==================== ============================================================================================================================================================
177+
==================== ======================================================================================================================================================================
178178
Extra Dependencies
179-
==================== ============================================================================================================================================================
179+
==================== ======================================================================================================================================================================
180180
``aiobotocore`` ``aiobotocore>=3.0.0``
181181
``cncf.kubernetes`` ``apache-airflow-providers-cncf-kubernetes>=7.2.0``
182182
``s3fs`` ``s3fs>=2023.10.0``
@@ -191,6 +191,7 @@ Extra Dependencies
191191
``microsoft.azure`` ``apache-airflow-providers-microsoft-azure``
192192
``mongo`` ``apache-airflow-providers-mongo``
193193
``pandas`` ``pandas>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"``
194+
``parquet`` ``pyarrow>=16.1.0; python_version < '3.13'``, ``pyarrow>=18.0.0; python_version >= '3.13' and python_version < '3.14'``, ``pyarrow>=22.0.0; python_version >= '3.14'``
194195
``openlineage`` ``apache-airflow-providers-openlineage>=2.3.0``
195196
``salesforce`` ``apache-airflow-providers-salesforce``
196197
``ssh`` ``apache-airflow-providers-ssh``
@@ -199,7 +200,7 @@ Extra Dependencies
199200
``common.messaging`` ``apache-airflow-providers-common-messaging>=2.0.0``
200201
``duckdb`` ``apache-airflow-providers-duckdb>=0.1.0``
201202
``sqlalchemy`` ``sqlalchemy>=1.4.54,!=2.1.0``
202-
==================== ============================================================================================================================================================
203+
==================== ======================================================================================================================================================================
203204

204205
Downloading official packages
205206
-----------------------------

‎providers/amazon/docs/transfer/sql_to_s3.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ Use ``SqlToS3Operator`` to copy data from a SQL server to an Amazon Simple Stora
2424
converts the SQL result to `pandas dataframe <https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html>`__
2525
(e.g. MySQL, Hive, ...).
2626

27+
With the default ``df_type="pandas"`` the operator needs the ``pandas`` extra. Writing Parquet files
28+
(``file_format="parquet"``) or reading with ``read_kwargs={"dtype_backend": "pyarrow"}`` also needs
29+
pyarrow, which the ``parquet`` extra installs: ``apache-airflow-providers-amazon[pandas,parquet]``.
30+
2731
Prerequisite Tasks
2832
------------------
2933

‎providers/amazon/pyproject.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,11 @@ dependencies = [
136136
'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
137137
'pandas>=2.3.3; python_version >="3.14"',
138138
]
139+
"parquet" = [
140+
"pyarrow>=16.1.0; python_version < '3.13'",
141+
"pyarrow>=18.0.0; python_version >= '3.13' and python_version < '3.14'",
142+
"pyarrow>=22.0.0; python_version >= '3.14'",
143+
]
139144
"openlineage" = [
140145
"apache-airflow-providers-openlineage>=2.3.0"
141146
]
@@ -201,6 +206,7 @@ dev = [
201206
"responses>=0.25.0",
202207
"apache-airflow-providers-common-sql[pandas,polars]",
203208
"apache-airflow-providers-amazon[sqlalchemy]",
209+
"apache-airflow-providers-amazon[parquet]",
204210
"apache-airflow-providers-duckdb"
205211
]
206212

‎uv.lock‎

Lines changed: 9 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)