Skip to content

Commit 53ad6ca

Browse files
leahecoleleahecole
authored
Merge branch 'master' into fix_monitoring_tests
2 parents b3404fd + fe705fc commit 53ad6ca

File tree

63 files changed

+962
-417
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

63 files changed

+962
-417
lines changed

‎.kokoro/tests/run_tests.sh

Lines changed: 2 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -103,26 +103,8 @@ for file in **/requirements.txt; do
103103
# If this is a continuous build, send the test log to the Build Cop Bot.
104104
# See https://github.com/googleapis/repo-automation-bots/tree/master/packages/buildcop.
105105
if [[ $KOKORO_BUILD_ARTIFACTS_SUBDIR = *"continuous"* ]]; then
106-
XML=$(base64 -w 0 sponge_log.xml)
107-
108-
# See https://github.com/apps/build-cop-bot/installations/5943459.
109-
MESSAGE=$(cat <<EOF
110-
{
111-
"Name": "buildcop",
112-
"Type" : "function",
113-
"Location": "us-central1",
114-
"installation": {"id": "5943459"},
115-
"repo": "GoogleCloudPlatform/python-docs-samples",
116-
"buildID": "$KOKORO_GIT_COMMIT",
117-
"buildURL": "https://source.cloud.google.com/results/invocations/$KOKORO_BUILD_ID",
118-
"xunitXML": "$XML"
119-
}
120-
EOF
121-
)
122-
123-
# Use a service account with access to the repo-automation-bots project.
124-
gcloud auth activate-service-account --key-file $GOOGLE_APPLICATION_CREDENTIALS
125-
gcloud pubsub topics publish passthrough --project=repo-automation-bots --message="$MESSAGE"
106+
chmod +x $KOKORO_GFILE_DIR/linux_amd64/buildcop
107+
$KOKORO_GFILE_DIR/linux_amd64/buildcop
126108
fi
127109

128110
if [[ $EXIT -ne 0 ]]; then

‎AUTHORING_GUIDE.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,8 @@ style expected of samples that live in
77

88
## Canonical sample
99

10-
The [Cloud Storage Sample](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/storage/cloud-client/snippets.py)
11-
is a great example of what we expect from samples. It's a great sample to copy
12-
and start from.
10+
The [Cloud Storage Samples](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/storage/cloud-client)
11+
are great examples of what we expect from samples. Take a look at some of the files within to gain better understanding of the patterns we use.
1312

1413
## The basics
1514

‎MAC_SETUP.md

Lines changed: 70 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,10 @@ test their code.
1212

1313
1. Install [homebrew](https://brew.sh/) if you do not already have it.
1414

15-
**Note:** If you are running Catalina (MacOS 10.15.x), ensure that you have a
16-
compatible version of Homebrew (2.1.13 or later). Running `brew update` on
15+
**Note:** If you are running Catalina (MacOS 10.15.x), ensure that you have
16+
a compatible version of Homebrew (2.1.13 or later). Running `brew update` on
1717
Catalina does not always result in a compatible version, so uninstall and
18-
reinstall homebrew, if necessary.
18+
reinstall homebrew, if necessary
1919

2020
## Installing pyenv and pyenv-virtualenv
2121

@@ -51,7 +51,7 @@ test their code.
5151
## Installing multiple Python versions
5252

5353

54-
1. See the available Python versions with
54+
1. See the available Python versions with [pyenv](https://github.com/pyenv/pyenv).
5555

5656
```console
5757
pyenv install --list
@@ -88,6 +88,10 @@ test their code.
8888
```console
8989
$ pyenv install 3.7.6
9090
```
91+
* 3.8.1 (latest 3.8.x release)
92+
```console
93+
$ pyenv install 3.8.1
94+
```
9195

9296
1. After you have installed a python version through pyenv,
9397
verify that you are now using the pyenv Python shim.
@@ -97,35 +101,76 @@ test their code.
97101
~/.pyenv/shims/python
98102
```
99103

104+
## Managing python versions using Pyenv global
105+
Pyenv allows you to configure the priority order for your python installs.
100106

101-
## Using pyenv and pyenv-virtualenv to manage your Python versions
107+
```
108+
pyenv global 3.8.1 3.7.6 3.6.10 3.5.9 2.7.17
109+
```
102110
103-
1. Change to the desired source directory.
111+
This will make python and python3 point to Python 3.8.1. python2 will use
112+
2.7.17. You can also further specify versions, such as python3.6 to use that
113+
version.
104114
105-
```console
106-
cd ~/src/python-docs-samples
107-
```
115+
## Python virtual environments
116+
Using [Virtual Environments](https://docs.python.org/3/library/venv.html)
117+
prevents inadvertent modifications to your global python install. Once
118+
created and sourced, calls to `python` will use this virtual environment, not
119+
a global python install. Each virtual environment can have its own set of
120+
packages that can be different from others.
108121
109-
1. Create a virtualenv using `pyenv virtualenv`.
110122
111-
```console
112-
pyenv virtualenv 3.7.6 python-docs-samples
113-
```
123+
### Using Python 3+ venv
124+
Python has builtin support for creating virtual environments, accessible by
125+
running the `venv` module.
114126
115-
This creates a virtualenv folder within `~/.pyenv/versions/`.
127+
```
128+
cd python-docs-samples
129+
python -m venv [venv-name]
130+
source [venv-name]/bin/activate
131+
```
116132
117-
1. Set the local Python version(s) with `pyenv local`
133+
Typically you will name the venv `venv`, or `venv38` for a python 3.8 venv.
118134
119-
```console
120-
# pyenv local [name of virtualenv] [list of python versions to use]
121-
pyenv local python-docs-samples 3.6.10 3.7.6 3.5.9 2.7.17
122-
```
123135
124-
1. Now, when you `cd` into the source directory or a subdirectory within it,
125-
pyenv will make your virtualenv the default Python. Since you specified
126-
more than one version, it will also add binaries like `python36` and
127-
`python27` to your PATH, which nox uses when picking Python interpreters.
136+
### Using pyenv-virtualenv
137+
You can also use an extension for pyenv that will assist in managing virtual
138+
environments. This allows you to use `pyenv local` to automatically use the
139+
created virtual environment. You can install this by running
140+
`$ brew install pyenv-virtualenv`
141+
142+
1. Change to the desired source directory.
128143
129-
1. Add `.python-version` to your [global gitignore
130-
file](https://help.github.com/articles/ignoring-files/#create-a-global-gitignore),
144+
```console
145+
cd ~/src/python-docs-samples
146+
```
147+
148+
1. Create a virtualenv for python 3.8.1 using `pyenv virtualenv`.
149+
150+
```console
151+
pyenv virtualenv 3.8.1 python-docs-samples
152+
```
153+
154+
This creates a virtualenv folder within `~/.pyenv/versions/`.
155+
156+
1. Set the local Python version(s) with `pyenv local`
157+
158+
```console
159+
# pyenv local [name of virtualenv] [list of python versions to use]
160+
pyenv local python-docs-samples 3.8.1 3.7.6 3.6.10 3.5.9 2.7.17
161+
```
162+
163+
1. Now, when you `cd` into the source directory or a subdirectory within it,
164+
pyenv will make your virtualenv the default Python. Since you specified
165+
more than one version, it will also add binaries like `python36` and
166+
`python27` to your PATH, which
167+
[nox](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#using-nox)
168+
uses when picking Python interpreters.
169+
170+
1. Add `.python-version` to your
171+
[global gitignore file](https://help.github.com/articles/ignoring-files/#create-a-global-gitignore),
131172
so it won't be committed into the repository.
173+
174+
## More on authoring samples
175+
If you are looking for more information on how to author samples, please view
176+
the [Authoring Guide](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md)

‎bigquery/bqml/data_scientist_tutorial_test.py

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -16,29 +16,34 @@
1616
from google.cloud import bigquery
1717
# [END bqml_data_scientist_tutorial_import_and_client]
1818
import pytest
19+
import uuid
1920

2021
# [START bqml_data_scientist_tutorial_import_and_client]
2122
client = bigquery.Client()
23+
# We use a unique dataset ID for this example to avoid collisions with
24+
# other invocations of this tutorial. In practice, you could leverage
25+
# a persistent dataset and not create/destroy it with each invocation.
26+
dataset_id = "bqml_tutorial_{}".format(str(uuid.uuid4().hex))
2227
# [END bqml_data_scientist_tutorial_import_and_client]
2328

2429

2530
@pytest.fixture
2631
def delete_dataset():
2732
yield
2833
client.delete_dataset(
29-
client.dataset('bqml_tutorial'), delete_contents=True)
34+
client.dataset(dataset_id), delete_contents=True)
3035

3136

3237
def test_data_scientist_tutorial(delete_dataset):
3338
# [START bqml_data_scientist_tutorial_create_dataset]
34-
dataset = bigquery.Dataset(client.dataset('bqml_tutorial'))
39+
dataset = bigquery.Dataset(client.dataset(dataset_id))
3540
dataset.location = 'US'
3641
client.create_dataset(dataset)
3742
# [END bqml_data_scientist_tutorial_create_dataset]
3843

3944
# [START bqml_data_scientist_tutorial_create_model]
4045
sql = """
41-
CREATE OR REPLACE MODEL `bqml_tutorial.sample_model`
46+
CREATE OR REPLACE MODEL `{}.sample_model`
4247
OPTIONS(model_type='logistic_reg') AS
4348
SELECT
4449
IF(totals.transactions IS NULL, 0, 1) AS label,
@@ -50,7 +55,7 @@ def test_data_scientist_tutorial(delete_dataset):
5055
`bigquery-public-data.google_analytics_sample.ga_sessions_*`
5156
WHERE
5257
_TABLE_SUFFIX BETWEEN '20160801' AND '20170630'
53-
"""
58+
""".format(dataset_id)
5459
df = client.query(sql).to_dataframe()
5560
print(df)
5661
# [END bqml_data_scientist_tutorial_create_model]
@@ -60,8 +65,8 @@ def test_data_scientist_tutorial(delete_dataset):
6065
SELECT
6166
*
6267
FROM
63-
ML.TRAINING_INFO(MODEL `bqml_tutorial.sample_model`)
64-
"""
68+
ML.TRAINING_INFO(MODEL `{}.sample_model`)
69+
""".format(dataset_id)
6570
df = client.query(sql).to_dataframe()
6671
print(df)
6772
# [END bqml_data_scientist_tutorial_get_training_statistics]
@@ -70,7 +75,7 @@ def test_data_scientist_tutorial(delete_dataset):
7075
sql = """
7176
SELECT
7277
*
73-
FROM ML.EVALUATE(MODEL `bqml_tutorial.sample_model`, (
78+
FROM ML.EVALUATE(MODEL `{}.sample_model`, (
7479
SELECT
7580
IF(totals.transactions IS NULL, 0, 1) AS label,
7681
IFNULL(device.operatingSystem, "") AS os,
@@ -81,7 +86,7 @@ def test_data_scientist_tutorial(delete_dataset):
8186
`bigquery-public-data.google_analytics_sample.ga_sessions_*`
8287
WHERE
8388
_TABLE_SUFFIX BETWEEN '20170701' AND '20170801'))
84-
"""
89+
""".format(dataset_id)
8590
df = client.query(sql).to_dataframe()
8691
print(df)
8792
# [END bqml_data_scientist_tutorial_evaluate_model]
@@ -91,7 +96,7 @@ def test_data_scientist_tutorial(delete_dataset):
9196
SELECT
9297
country,
9398
SUM(predicted_label) as total_predicted_purchases
94-
FROM ML.PREDICT(MODEL `bqml_tutorial.sample_model`, (
99+
FROM ML.PREDICT(MODEL `{}.sample_model`, (
95100
SELECT
96101
IFNULL(device.operatingSystem, "") AS os,
97102
device.isMobile AS is_mobile,
@@ -104,7 +109,7 @@ def test_data_scientist_tutorial(delete_dataset):
104109
GROUP BY country
105110
ORDER BY total_predicted_purchases DESC
106111
LIMIT 10
107-
"""
112+
""".format(dataset_id)
108113
df = client.query(sql).to_dataframe()
109114
print(df)
110115
# [END bqml_data_scientist_tutorial_predict_transactions]
@@ -114,7 +119,7 @@ def test_data_scientist_tutorial(delete_dataset):
114119
SELECT
115120
fullVisitorId,
116121
SUM(predicted_label) as total_predicted_purchases
117-
FROM ML.PREDICT(MODEL `bqml_tutorial.sample_model`, (
122+
FROM ML.PREDICT(MODEL `{}.sample_model`, (
118123
SELECT
119124
IFNULL(device.operatingSystem, "") AS os,
120125
device.isMobile AS is_mobile,
@@ -128,7 +133,7 @@ def test_data_scientist_tutorial(delete_dataset):
128133
GROUP BY fullVisitorId
129134
ORDER BY total_predicted_purchases DESC
130135
LIMIT 10
131-
"""
136+
""".format(dataset_id)
132137
df = client.query(sql).to_dataframe()
133138
print(df)
134139
# [END bqml_data_scientist_tutorial_predict_purchases]

0 commit comments

Comments
 (0)