add try catch around provenance tracking #159266
Conversation
Summary: Add try-except around provenance tracking logic to make it more safe Test Plan: CI Rollback Plan: Differential Revision: D79008234
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/159266
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 11da12f with merge base 14d67ee ( UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
This pull request was exported from Phabricator. Differential Revision: D79008234 |
|
I'm kind of down on this sort of thing. While I do understand that it's annoying to have a run fail because of some debugging code, the alternative is that debugging information is just missing and no one knows why (because this sort of thing is a lot harder to fix.) |
@ezyang We're trying to enable this provenance tracking debugging information by default for all production models, and the enablement team's people really want these try-except to make sure no production model is impacted by turning this on by default. Since we have the errors logged when we have any exception, we should be able to know why the debugging information is missing by looking at the logs. |
|
I wouldn't rely on logview to find these problems, in that case. Why not write it to our Scuba table using signpost or something similar? |
Summary: Add try-except around provenance tracking logic to make it more safe
Test Plan:
CI
Rollback Plan:
Differential Revision: D79008234
cc @ezyang @SherlockNoMad @EikanWang @jgong5 @wenzhe-nrv @voznesenskym @penguinwu @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben