Repository navigation
Add .sql template extension rendering for BigQueryInsertJobOperator #22859
Description
Description
Add .sql as value to template_ext in airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator
Use case/motivation
It would be helpful if Jinja would also read .sql templates in airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator, especially for its configuration.query.query parameter. It will render that parameter as sql correctly, but only the outer configuration body -- which is based on Bigquery/rest/v2/Job#jobconfiguration -- is considered for extension-based rendering and only json is considered.
I think there is a clear use-case to support templating SQL-only files. I know that BigQueryExecuteQueryOperator has this but it's deprecated in favour of BigQueryInsertJobOperator.
I currently work around this using a really basic override:
class BigQueryTemplateFileOperator(BigQueryInsertJobOperator):
template_ext = (".json", ".sql")
However, I'd like to avoid having to replicate this operator everywhere in our codebase if this might actually already be implemented soon.
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Apr 8, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
josh-fell commentedon Apr 9, 2022
@LennyKLB Assigned to you!
implements apache#22859 - Add .sql as templatable extension
LennyKLB commentedon Apr 11, 2022
The change itself is so small that I refrained from adding/editing tests. I found that the BigQuery-related tests didn't feature any file-based templating.
If this is nonetheless still necessary, could someone point me in the right direction in terms of inspiration?
implements #22859 - Add .sql as templatable extension (#22920)
josh-fell commentedon Apr 11, 2022
Closing via #22920.