Skip to content

Commit 4980e85

Browse files
vglafirovbwplotka
vglafirov
authored and
bwplotka
committed
Azure Storage support (thanos-io#502)
* add Azure storage * Container create * Added container creation logic * added couple of functions * Upload function * Added k8s manifests * rebased to master * acceptance tests * all storage tests passed * fix cmd params * added metrics and logs * Added docs * Replaced imports * Rollback deployment changes * Rolled back deployment changes * Using the same container name across all components * adding debug messages * Ensure dependencies * Use make deps * trick or treat * adde azure config to all commands * removed close method * fixed Close() method * added errcheck * Rebuilt docs * Skip unconfigured Azure test * fixes after the review * Add unified flags for bucket configuration Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Use the unified bucket config for components Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Use double quotes instead of single quotes Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Fixed missing flags in man page Signed-off-by: jojohappy <sarahdj0917@gmail.com> * To use value of bucket config instead of pointer to get value from command-line arguments Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Remove useless code Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update documents Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update - Rename provider to objstore for flags - To use objProvider instead of string for Provider Type - Get rid of bucket configuration errors Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Change errors package Signed-off-by: jojohappy <sarahdj0917@gmail.com> * To validate the configuration in each provider client Signed-off-by: jojohappy <sarahdj0917@gmail.com> * To support to make bucket configuration flag using suffix Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update documents Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Refactor: - Remove all flags of objstore - Add flag as objstore.config to pass the configuration for bucket with yaml - To define the configuration for each provider - Add new method to get the name of bucket Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update documents * Fixed missing method for inmem bucket * Update the describe for objstore.config * To setup bucket flags required for component bucket and downsample * Rename Content to Config for Bucket.Config Signed-off-by: jojohappy <sarahdj0917@gmail.com> * To change error handler idiom Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update describe for component store Signed-off-by: jojohappy <sarahdj0917@gmail.com> * To setup secret-key just use envvar Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update the placeholder of flags and documents Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update example for bucket Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update documents Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update CHANGELOG Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Fixed something nits * To distinguish no bucket is configured or not supported Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update CHANGELOG Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Remove unrequired unit test Signed-off-by: jojohappy <sarahdj0917@gmail.com> * To set bucket flag required for component store and compact Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Rename GetBucket to Name & Set context as first argument Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Wrap error to give more information Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Rename field bucket to name for the struct of Bucket Signed-off-by: jojohappy <sarahdj0917@gmail.com> * Update documents Signed-off-by: jojohappy <sarahdj0917@gmail.com> * implement objstore.config * rebuild docs * added Azure provider to factory * update test documentation * Modified Azure documentation * regenerate docs * Changes after the review * Wrapped errors * wrap error * removed switch * check Exists error * Fixes after review * resolved comments * removed blobReaderCloser * fixed markdown lint * Resolved comments * type assert and syntax * replaced with regexp * use snake_case for parameters * another portion of fixes * fix broken build
1 parent 8f247d6 commit 4980e85

File tree

13 files changed

+420
-22
lines changed

13 files changed

+420
-22
lines changed

‎.circleci/config.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,9 @@ jobs:
3434
export THANOS_SKIP_S3_AWS_TESTS="true"
3535
echo "Skipping AWS tests."
3636
37+
export THANOS_SKIP_AZURE_TESTS="true"
38+
echo "Skipping Azure tests."
39+
3740
make test
3841
3942
# Cross build is needed for publish_release but needs to be done outside of docker.

‎CONTRIBUTING.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,10 @@ $ git push origin <your_branch_for_new_pr>
3535
5. If you don't have a live object store ready add these envvars to skip tests for these:
3636
- THANOS_SKIP_GCS_TESTS to skip GCS tests.
3737
- THANOS_SKIP_S3_AWS_TESTS to skip AWS tests.
38+
- THANOS_SKIP_AZURE_TESTS to skip Azure tests.
3839

39-
If you skip both of these, the store specific tests will be run against memory object storage only.
40-
CI runs GCS and inmem tests only for now. Not having these variables will produce auth errors against GCS or AWS tests.
40+
If you skip all of these, the store specific tests will be run against memory object storage only.
41+
CI runs GCS and inmem tests only for now. Not having these variables will produce auth errors against GCS, AWS or Azure tests.
4142

4243
6. If your change affects users (adds or removes feature) consider adding the item to [CHANGELOG](CHANGELOG.md)
4344
7. You may merge the Pull Request in once you have the sign-off of at least one developers with write access, or if you

‎Gopkg.lock

Lines changed: 24 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Gopkg.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,3 +76,6 @@ ignored = ["github.com/improbable-eng/thanos/benchmark/*"]
7676
name = "github.com/minio/minio-go"
7777
version = "6.0.1"
7878

79+
[[constraint]]
80+
name = "github.com/Azure/azure-storage-blob-go"
81+
revision = "197d1c0aea1b9eedbbaee0a1a32bf81e879bde80"

‎Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -140,7 +140,7 @@ tarballs-release: $(PROMU)
140140
# test runs all Thanos golang tests against each supported version of Prometheus.
141141
.PHONY: test
142142
test: test-deps
143-
@echo ">> running all tests. Do export THANOS_SKIP_GCS_TESTS='true' or/and export THANOS_SKIP_S3_AWS_TESTS='true' if you want to skip e2e tests against real store buckets"
143+
@echo ">> running all tests. Do export THANOS_SKIP_GCS_TESTS='true' or/and export THANOS_SKIP_S3_AWS_TESTS='true' or/and THANOS_SKIP_AZURE_TESTS='true' if you want to skip e2e tests against real store buckets"
144144
@for ver in $(SUPPORTED_PROM_VERSIONS); do \
145145
THANOS_TEST_PROMETHEUS_PATH="prometheus-$$ver" THANOS_TEST_ALERTMANAGER_PATH="alertmanager-$(ALERTMANAGER_VERSION)" go test $(shell go list ./... | grep -v /vendor/ | grep -v /benchmark/); \
146146
done

‎cmd/thanos/store.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ import (
2323

2424
// registerStore registers a store command.
2525
func registerStore(m map[string]setupFunc, app *kingpin.Application, name string) {
26-
cmd := app.Command(name, "store node giving access to blocks in a bucket provider. Now supported GCS / S3.")
26+
cmd := app.Command(name, "store node giving access to blocks in a bucket provider. Now supported GCS, S3 and Azure.")
2727

2828
grpcBindAddr, httpBindAddr, cert, key, clientCA, newPeerFn := regCommonServerFlags(cmd)
2929

‎docs/components/bucket.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Bucket
22

33
The bucket component of Thanos is a set of commands to inspect data in object storage buckets.
4-
It is normally run as a stand alone command to aid with troubleshooting.
4+
It is normally run as a stand alone command to aid with troubleshooting.
55

66
Example:
77

@@ -73,7 +73,7 @@ Example:
7373

7474
```
7575
$ thanos bucket verify --gcs.bucket example-bucket
76-
```
76+
```
7777

7878
[embedmd]:# (flags/bucket_verify.txt)
7979
```txt
@@ -125,7 +125,7 @@ Example:
125125

126126
```
127127
$ thanos bucket ls -o json --gcs.bucket example-bucket
128-
```
128+
```
129129

130130
[embedmd]:# (flags/bucket_ls.txt)
131131
```txt

‎docs/components/store.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,8 @@ In general about 1MB of local disk space is required per TSDB block stored in th
2727
```$
2828
usage: thanos store [<flags>]
2929

30-
store node giving access to blocks in a bucket provider. Now supported GCS / S3.
30+
store node giving access to blocks in a bucket provider. Now supported GCS, S3
31+
and Azure.
3132

3233
Flags:
3334
-h, --help Show context-sensitive help (also try

‎docs/storage.md

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Current object storage client implementations:
88
|----------------------|-------------------|-----------|---------------|
99
| Google Cloud Storage | Stable (production usage) | yes | @bplotka |
1010
| AWS S3 | Beta (working PoCs, testing usage) | no | ? |
11+
| Azure Storage Account | Alpha | yes | @vglafirov |
1112

1213
NOTE: Currently Thanos requires strong consistency (write-read) for object store implementation.
1314

@@ -27,7 +28,8 @@ At that point, anyone can use your provider!
2728
Thanos uses minio client to upload Prometheus data into AWS S3.
2829

2930
To configure S3 bucket as an object store you need to set these mandatory S3 variables in yaml format stored in a file:
30-
```
31+
32+
```yaml
3133
type: S3
3234
config:
3335
bucket: <bucket>
@@ -83,6 +85,7 @@ Example working AWS IAM policy for user:
8385
]
8486
}
8587
```
88+
8689
(No bucket policy)
8790

8891
To test the policy, set env vars for S3 access for *empty, not used* bucket as well as:
@@ -123,12 +126,13 @@ With this policy you should be able to run set `THANOS_SKIP_GCS_TESTS=true` and
123126

124127
Details about AWS policies: https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
125128

126-
## GCP Configuration
129+
## GCP Configuration
127130

128131
To configure Google Cloud Storage bucket as an object store you need to set `bucket` with GCS bucket name and configure Google Application credentials.
129132

130133
For example:
131-
```
134+
135+
```yaml
132136
type: GCS
133137
config:
134138
bucket: <bucket>
@@ -160,9 +164,24 @@ For testing:
160164

161165
`Storage Object Admin` for ability to create and delete temporary buckets.
162166

163-
164167
## Other minio supported S3 object storages
165168

166169
Minio client used for AWS S3 can be potentially configured against other S3-compatible object storages.
167170

168171
<TBD>
172+
173+
## Azure Configuration
174+
175+
To use Azure Storage as Thanos object store, you need to precreate storage account from Azure portal or using Azure CLI. Follow the instructions from Azure Storage Documentation: [https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=portal)
176+
177+
To configure Azure Storage account as an object store you need to provide a path to Azure storage config file in flag `--objstore.config-file`.
178+
179+
Config file format is the following:
180+
181+
```yaml
182+
type: AZURE
183+
config:
184+
storage_account: <Name of Azure Storage Account>
185+
storage_account_key: <Storage Account key>
186+
container: <Blob container>
187+
```

0 commit comments

Comments
 (0)