Data cleaning region tags #4785
Conversation
|
fixing lint now ugh 🙈 |
| temp_df = spark.read.format("bigquery").option("table", TABLE).load() | ||
| df = temp_df.drop("gender") |
There was a problem hiding this comment.
Should this be more explicitly explained in the code why we are removing gender? Either way, I would move it outside of this try-block. It's also ok to do df = df.<operation> and not use the temp_df var name as this ends up happening in multiple other places in the script.
There was a problem hiding this comment.
Np for moving it outside of try block and making that df = df.operation change. Reasoning for removal of the column is in the Putting it together section of the doc I sent you - do you think it also needs to be mentioned here?
There was a problem hiding this comment.
I think that's fair, maybe it should have its own region tag then?
There was a problem hiding this comment.
Added region tag, but have to wait a few min before re-staging
| @@ -109,7 +106,7 @@ def setup_and_teardown_cluster(): | |||
| project_id=PROJECT_ID, | |||
| region=CLUSTER_REGION, | |||
| cluster_name=DATAPROC_CLUSTER, | |||
| timeout=300 | |||
| timeout=300, | |||
There was a problem hiding this comment.
This doesn't hurt but you shouldn't need to do this. Were there timeout errors deleting the cluster?
There was a problem hiding this comment.
leftover from Tushar - can remove if need be
There was a problem hiding this comment.
Removing is fine unless it somehow breaks
There was a problem hiding this comment.
Will remove next, just testing locally now
There was a problem hiding this comment.
local tests passed, just pushed removal
|
ugh, ofc these were passing locally. I'll figure out why they don't pass here! |
Description
Adds region tags for the data cleaning section, removes the gender column from the dataframe. Link to doc in staging available if you ping me :)
Note: It's a good idea to open an issue first for discussion.
Checklist
nox -s py-3.6(see Test Environment Setup)nox -s lint(see Test Environment Setup)