Skip to content

Commit e60ddbb

Browse files
hfwangshubha-rajankurtisvgleahecole
authored
feat: add Cloud SQL samples for Client-side encryption (GoogleCloudPlatform#5510)
* Copy existing sqlalchemy example as base for tink encryption example * Create a CLI app demonstrating reading/writing encrypted fields via Tink * Finish refactoring init_tink->init_tink_eaead * Respond to some review comments * reorganize files and add tests * fix header check * linting * update README and region tags * linting * update environment variable names * add python 3.6 and 3.9 to excluded versions * fix typo in test * use capsys in tests * fix linting * remove dockerignore * Update noxfile.py * Update cloud-sql/mysql/client-side-encryption/README.md Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> * deleted gitignore * update comments * add tests and address review comments * Update cloud-sql/mysql/client-side-encryption/requirements-test.txt Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * update sqlalchemy version * update environment variable names * fix typos * fix failing tests * linting fixes * set enforce_type_hints to False * add typing annotations * linting * linting fixes * Update README.md Co-authored-by: Shubha Rajan <shubhadayini@google.com> Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
1 parent 81a588f commit e60ddbb

13 files changed

+876
-0
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Encrypting fields in Cloud SQL - MySQL with Tink
2+
3+
## Before you begin
4+
5+
1. If you haven't already, set up a Python Development Environment by following the [python setup guide](https://cloud.google.com/python/setup) and
6+
[create a project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project).
7+
8+
1. Create a 2nd Gen Cloud SQL Instance by following these
9+
[instructions](https://cloud.google.com/sql/docs/mysql/create-instance). Note the connection string,
10+
database user, and database password that you create.
11+
12+
1. Create a database for your application by following these
13+
[instructions](https://cloud.google.com/sql/docs/mysql/create-manage-databases). Note the database
14+
name.
15+
16+
1. Create a KMS key for your application by following these
17+
[instructions](https://cloud.google.com/kms/docs/creating-keys). Copy the resource name of your
18+
created key.
19+
20+
1. Create a service account with the 'Cloud SQL Client' permissions by following these
21+
[instructions](https://cloud.google.com/sql/docs/mysql/connect-external-app#4_if_required_by_your_authentication_method_create_a_service_account).
22+
Download a JSON key to use to authenticate your connection.
23+
24+
1. **macOS / Windows only**: Configure gRPC Root Certificates: On some platforms you may need to
25+
accept the Google server certificates, see instructions for setting up
26+
[root certs](https://github.com/googleapis/google-cloud-cpp/blob/master/google/cloud/bigtable/examples/README.md#configure-grpc-root-certificates).
27+
28+
## Running locally
29+
30+
To run this application locally, download and install the `cloud_sql_proxy` by
31+
following the instructions
32+
[here](https://cloud.google.com/sql/docs/mysql/sql-proxy#install).
33+
34+
Instructions are provided below for using the proxy with a TCP connection or a Unix Domain Socket.
35+
On Linux or Mac OS you can use either option, but on Windows the proxy currently requires a TCP
36+
connection.
37+
38+
### Launch proxy with TCP
39+
40+
To run the sample locally with a TCP connection, set environment variables and launch the proxy as
41+
shown below.
42+
43+
#### Linux / Mac OS
44+
Use these terminal commands to initialize environment variables:
45+
```bash
46+
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service/account/key.json
47+
export DB_HOST='127.0.0.1:3306'
48+
export DB_USER='<DB_USER_NAME>'
49+
export DB_PASS='<DB_PASSWORD>'
50+
export DB_NAME='<DB_NAME>'
51+
export GCP_KMS_URI='<GCP_KMS_URI>'
52+
```
53+
Note: Saving credentials in environment variables is convenient, but not secure - consider a more
54+
secure solution such as [Secret Manager](https://cloud.google.com/secret-manager/docs/overview) to
55+
help keep secrets safe.
56+
57+
Then use this command to launch the proxy in the background:
58+
```bash
59+
./cloud_sql_proxy -instances=<project-id>:<region>:<instance-name>=tcp:3306 -credential_file=$GOOGLE_APPLICATION_CREDENTIALS &
60+
```
61+
62+
#### Windows/PowerShell
63+
Use these PowerShell commands to initialize environment variables:
64+
```powershell
65+
$env:GOOGLE_APPLICATION_CREDENTIALS="<CREDENTIALS_JSON_FILE>"
66+
$env:DB_HOST="127.0.0.1:3306"
67+
$env:DB_USER="<DB_USER_NAME>"
68+
$env:DB_PASS="<DB_PASSWORD>"
69+
$env:DB_NAME="<DB_NAME>"
70+
$env:GCP_KMS_URI='<GCP_KMS_URI>'
71+
```
72+
Note: Saving credentials in environment variables is convenient, but not secure - consider a more
73+
secure solution such as [Secret Manager](https://cloud.google.com/secret-manager/docs/overview) to
74+
help keep secrets safe.
75+
76+
Then use this command to launch the proxy in a separate PowerShell session:
77+
```powershell
78+
Start-Process -filepath "C:\<path to proxy exe>" -ArgumentList "-instances=<project-id>:<region>:<instance-name>=tcp:3306 -credential_file=<CREDENTIALS_JSON_FILE>"
79+
```
80+
81+
### Launch proxy with Unix Domain Socket
82+
NOTE: this option is currently only supported on Linux and Mac OS. Windows users should use the
83+
[Launch proxy with TCP](#launch-proxy-with-tcp) option.
84+
85+
To use a Unix socket, you'll need to create a directory and give write access to the user running
86+
the proxy. For example:
87+
88+
```bash
89+
sudo mkdir /cloudsql
90+
sudo chown -R $USER /cloudsql
91+
```
92+
93+
You'll also need to initialize an environment variable containing the directory you just created:
94+
```bash
95+
export DB_SOCKET_DIR=/path/to/the/new/directory
96+
```
97+
98+
Use these terminal commands to initialize other environment variables as well:
99+
```bash
100+
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service/account/key.json
101+
export INSTANCE_CONNECTION_NAME='<MY-PROJECT>:<INSTANCE-REGION>:<INSTANCE-NAME>'
102+
export DB_USER='<DB_USER_NAME>'
103+
export DB_PASS='<DB_PASSWORD>'
104+
export DB_NAME='<DB_NAME>'
105+
export GCP_KMS_URI='<GCP_KMS_URI>'
106+
```
107+
Note: Saving credentials in environment variables is convenient, but not secure - consider a more
108+
secure solution such as [Secret Manager](https://cloud.google.com/secret-manager/docs/overview) to
109+
help keep secrets safe.
110+
111+
Then use this command to launch the proxy in the background:
112+
```bash
113+
./cloud_sql_proxy -dir=$DB_SOCKET_DIR --instances=$INSTANCE_CONNECTION_NAME --credential_file=$GOOGLE_APPLICATION_CREDENTIALS &
114+
```
115+
116+
### Install requirements
117+
118+
Next, setup install the requirements into a virtual enviroment:
119+
```bash
120+
virtualenv --python python3 env
121+
source env/bin/activate
122+
pip install -r requirements.txt
123+
```
124+
125+
### Run the demo
126+
127+
Add new votes and the collected votes:
128+
```bash
129+
python snippets/query_and_decrypt_data.py
130+
```
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Default TEST_CONFIG_OVERRIDE for python repos.
16+
17+
# You can copy this file into your directory, then it will be inported from
18+
# the noxfile.py.
19+
20+
# The source of truth:
21+
# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/noxfile_config.py
22+
23+
TEST_CONFIG_OVERRIDE = {
24+
# You can opt out from the test for specific Python versions.
25+
"ignored_versions": ["2.7", "3.6"],
26+
# Old samples are opted out of enforcing Python type hints
27+
# All new samples should feature them
28+
"enforce_type_hints": True,
29+
# An envvar key for determining the project id to use. Change it
30+
# to 'BUILD_SPECIFIC_GCLOUD_PROJECT' if you want to opt in using a
31+
# build specific Cloud project. You can also use your own string
32+
# to use your own Cloud project.
33+
"gcloud_project_env": "GOOGLE_CLOUD_PROJECT",
34+
# 'gcloud_project_env': 'BUILD_SPECIFIC_GCLOUD_PROJECT',
35+
# A dictionary you want to inject into your test. Don't put any
36+
# secrets here. These values will override predefined values.
37+
"envs": {},
38+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
pytest==6.2.2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
SQLAlchemy==1.4.4
2+
PyMySQL==1.0.2
3+
tink==1.5.0

cloud-sql/mysql/client-side-encryption/snippets/__init__.py

Whitespace-only changes.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# [START cloud_sql_mysql_cse_key]
16+
import logging
17+
18+
import tink
19+
from tink import aead
20+
from tink.integration import gcpkms
21+
22+
logger = logging.getLogger(__name__)
23+
24+
25+
def init_tink_env_aead(
26+
key_uri: str,
27+
credentials: str) -> tink.aead.KmsEnvelopeAead:
28+
aead.register()
29+
30+
try:
31+
gcp_client = gcpkms.GcpKmsClient(key_uri, credentials)
32+
gcp_aead = gcp_client.get_aead(key_uri)
33+
except tink.TinkError as e:
34+
logger.error("Error initializing GCP client: %s", e)
35+
raise e
36+
37+
# Create envelope AEAD primitive using AES256 GCM for encrypting the data
38+
# This key should only be used for client-side encryption to ensure authenticity and integrity
39+
# of data.
40+
key_template = aead.aead_key_templates.AES256_GCM
41+
env_aead = aead.KmsEnvelopeAead(key_template, gcp_aead)
42+
43+
print(f"Created envelope AEAD Primitive using KMS URI: {key_uri}")
44+
45+
return env_aead
46+
47+
48+
# [END cloud_sql_mysql_cse_key]
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Copyright 2021 Google LLC
16+
#
17+
# Licensed under the Apache License, Version 2.0 (the "License");
18+
# you may not use this file except in compliance with the License.
19+
# You may obtain a copy of the License at
20+
#
21+
# http://www.apache.org/licenses/LICENSE-2.0
22+
#
23+
# Unless required by applicable law or agreed to in writing, software
24+
# distributed under the License is distributed on an "AS IS" BASIS,
25+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
26+
# See the License for the specific language governing permissions and
27+
# limitations under the License.
28+
29+
import os
30+
31+
import pytest
32+
33+
from snippets.cloud_kms_env_aead import init_tink_env_aead
34+
35+
36+
@pytest.fixture(name="kms_uri")
37+
def setup() -> str:
38+
kms_uri = "gcp-kms://" + os.environ["CLOUD_KMS_KEY"]
39+
40+
yield kms_uri
41+
42+
43+
def test_cloud_kms_env_aead(
44+
capsys: pytest.CaptureFixture, kms_uri: str) -> None:
45+
credentials = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", "")
46+
47+
# Create env_aead primitive
48+
init_tink_env_aead(kms_uri, credentials)
49+
50+
captured = capsys.readouterr().out
51+
assert f"Created envelope AEAD Primitive using KMS URI: {kms_uri}" in captured
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# [START cloud_sql_mysql_cse_db]
16+
import sqlalchemy
17+
18+
19+
def init_tcp_connection_engine(
20+
db_user: str, db_pass: str, db_name: str, db_host: str
21+
) -> sqlalchemy.engine.base.Engine:
22+
# Remember - storing secrets in plaintext is potentially unsafe. Consider using
23+
# something like https://cloud.google.com/secret-manager/docs/overview to help keep
24+
# secrets secret.
25+
26+
# Extract host and port from db_host
27+
host_args = db_host.split(":")
28+
db_hostname, db_port = host_args[0], int(host_args[1])
29+
30+
pool = sqlalchemy.create_engine(
31+
# Equivalent URL:
32+
# mysql+pymysql://<db_user>:<db_pass>@<db_host>:<db_port>/<db_name>
33+
sqlalchemy.engine.url.URL(
34+
drivername="mysql+pymysql",
35+
username=db_user, # e.g. "my-database-user"
36+
password=db_pass, # e.g. "my-database-password"
37+
host=db_hostname, # e.g. "127.0.0.1"
38+
port=db_port, # e.g. 3306
39+
database=db_name, # e.g. "my-database-name"
40+
),
41+
)
42+
print("Created TCP connection pool")
43+
return pool
44+
45+
46+
def init_unix_connection_engine(
47+
db_user: str,
48+
db_pass: str,
49+
db_name: str,
50+
cloud_sql_connection_name: str,
51+
db_socket_dir: str,
52+
) -> sqlalchemy.engine.base.Engine:
53+
# Remember - storing secrets in plaintext is potentially unsafe. Consider using
54+
# something like https://cloud.google.com/secret-manager/docs/overview to help keep
55+
# secrets secret.
56+
57+
pool = sqlalchemy.create_engine(
58+
# Equivalent URL:
59+
# mysql+pymysql://<db_user>:<db_pass>@/<db_name>?unix_socket=<socket_path>/<cloud_sql_instance_name>
60+
sqlalchemy.engine.url.URL(
61+
drivername="mysql+pymysql",
62+
username=db_user, # e.g. "my-database-user"
63+
password=db_pass, # e.g. "my-database-password"
64+
database=db_name, # e.g. "my-database-name"
65+
query={"unix_socket": f"{db_socket_dir}/{cloud_sql_connection_name}"},
66+
),
67+
)
68+
print("Created Unix socket connection pool")
69+
return pool
70+
71+
72+
def init_db(
73+
db_user: str,
74+
db_pass: str,
75+
db_name: str,
76+
table_name: str,
77+
cloud_sql_connection_name: str = None,
78+
db_socket_dir: str = None,
79+
db_host: str = None,
80+
) -> sqlalchemy.engine.base.Engine:
81+
82+
if db_host:
83+
db = init_tcp_connection_engine(db_user, db_pass, db_name, db_host)
84+
else:
85+
db = init_unix_connection_engine(
86+
db_user, db_pass, db_name, cloud_sql_connection_name, db_socket_dir
87+
)
88+
89+
# Create tables (if they don't already exist)
90+
with db.connect() as conn:
91+
conn.execute(
92+
f"CREATE TABLE IF NOT EXISTS {table_name} "
93+
"( vote_id SERIAL NOT NULL, time_cast timestamp NOT NULL, "
94+
"team CHAR(6) NOT NULL, voter_email VARBINARY(255), "
95+
"PRIMARY KEY (vote_id) );"
96+
)
97+
98+
print(f"Created table {table_name} in db {db_name}")
99+
return db
100+
101+
102+
# [END cloud_sql_mysql_cse_db]

0 commit comments

Comments
 (0)