Migrate Alibaba example DAGs to new design #22437

Activity

1. potiuk

   added

   AIP-47AIP-47 New Design of System TestsAIP-47 New Design of System Tests

   kind:metaHigh-level information important to the communityHigh-level information important to the community

   on Mar 22, 2022
2. EricGao888 commented on Mar 24, 2022

   

   EricGao888

   on Mar 24, 2022

   Member

   More actions

   I took a quick at AIP-47. May I ask is there any example pr for provider example DAGs migration?
3. potiuk commented on Mar 25, 2022

   

   potiuk

   on Mar 25, 2022

   MemberAuthor

   More actions

   We are just about to merge #22311

   👍React with 👍1Reacted by Eric Gao
4. bhirsz commented on Mar 25, 2022

   

   bhirsz

   on Mar 25, 2022

   Last edited by bhirsz

   Contributor

   More actions

   I took a quick look into your examples - there are short so they could also make good example (since our PR is bloated with all changes required). You don't have any system tests - which in old design were usually pytest classes that were invoking example dags. In a new design it's somehow similar (in a sense that we have a file, that works both as a system test and example).
   From my POV action points include:

   1. Moving example dags under tests/system/providers/alibaba
   2. Create tests/system/providers/README.md explaining how to run your examples (if there is need for preconfigured environment, created account etc).
   3. If your DAG creates any object, that shall be deleted - add trigger rule ALL_DONE to operators doing the removal (that's to ensure it actually happens).
   4. Append watcher at the end of DAG:

   from tests.system.utils.watcher import watcher
   
   # This test needs watcher in order to properly mark success/failure
   # when "tearDown" task with trigger rule is part of the DAG
   list(dag.tasks) >> watcher()
   

   6. Append test method at the end of file:

   from tests.system.utils import get_test_run  # noqa: E402
   
   # Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
   test_run = get_test_run(dag)
   
   

   7. Update documentation pointing to your examples. Whenever you used tags like "# [START howto_operator_oss_bucket]" you need update path in rst includes to new one (under system/tests). For example here https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers-alibaba/operators/oss.rst .

   System tests should be runnable using following command (use Breeze to ensure you have all dependencies):

   pytest --system alibaba tests/system
   

   I don't know your package and what's it's doing so I don't know if there are extra steps required for preparing environment. It would be best to describe in a way that would allow other users easily run it and of course enable community to include your system tests in CI checks.

   ❤️React with ❤️1Reacted by Eric Gao
5. EricGao888 commented on Mar 25, 2022

   

   EricGao888

   on Mar 25, 2022

   Member

   More actions

   I took a quick look into your examples - there are short so they could also make good example (since our PR is bloated with all changes required). You don't have any system tests - which in old design were usually pytest classes that were invoking example dags. In a new design it's somehow similar (in a sense that we have a file, that works both as a system test and example). From my POV action points include:

   1. Moving example dags under tests/system/providers/alibaba
   2. Create tests/system/providers/README.md explaining how to run your examples (if there is need for preconfigured environment, created account etc).
   3. If your DAG creates any object, that shall be deleted - add trigger rule ALL_DONE to operators doing the removal (that's to ensure it actually happens).
   4. Append watcher at the end of DAG:

   from tests.system.utils import get_test_run  # noqa: E402
   
   # Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
   test_run = get_test_run(dag)
   

   6. Append test method at the end of file:

   from tests.system.utils import get_test_run  # noqa: E402
   
   # Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
   test_run = get_test_run(dag)
   

   7. Update documentation pointing to your examples. Whenever you used tags like "# [START howto_operator_oss_bucket]" you need update path in rst includes to new one (under system/tests). For example here https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers-alibaba/operators/oss.rst .

   System tests should be runnable using following command (use Breeze to ensure you have all dependencies):

   pytest --system alibaba tests/system
   

   I don't know your package and what's it's doing so I don't know if there are extra steps required for preparing environment. It would be best to describe in a way that would allow other users easily run it and of course enable community to include your system tests in CI checks.

   Thanks for the detailed explanation! It's very coooool!

   👍React with 👍1Reacted by Jarek Potiuk
6. chethanuk

   added a commit that references this issue on Jun 2, 2022

   Migrate Alibaba example DAGs to new design apache#22437

   43d0414
7. potiuk

   added a commit that references this issue on Jun 3, 2022

   Migrate Alibaba example DAGs to new design #22437 (#24130)

   ...

   c887275
8. potiuk

   closed this as completedon Jun 3, 2022