Repository navigation
Use Hash / SHA to compare Serialized DAGs #10116
Description
Description
Currently, we compare entire Serialized blobs to determine if the DAG changed or not before we write the Serialized DAGs to the Database which can be expensive.
Instead, we will store the HASH / SHA of the Serialized JSON and store that in a separate column in the serialized_dag table.
And before writing the Serialized Blob, we will compare the SHA to decide if DAG changed.
Activity
[-]Use SHA to compare Serialized DAGs[/-][+]Use Hash / SHA to compare Serialized DAGs[/+]potiuk commentedon Aug 2, 2020
Oh yeah!
kaxil commentedon Aug 2, 2020
Planning to get to this by tomorrow so we can release it with Airflow 1.10.12
potiuk commentedon Aug 2, 2020
What are the plans for 1.10.12?
kaxil commentedon Aug 2, 2020
Mostly to fix all the Kubernetes related issues that came of the Kubernetes Refactors in Airflow 1.10.11 -- https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Ak8s+milestone%3A%22Airflow+1.10.12%22
We can hopefully get those fixed (PRs are directly to v1-10-* branches) so hopefully, we can get RCs at the end of next week
Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
Use Hash of Serialized DAG to determine DAG is changed or not (apache…