Skip to content

Commit ba60e15

Browse files
jonathanbeberjonathanbeber
authored
Add ServiceAnnotations cluster config (zalando#803)
The [operator parameters][1] already support the `custom_service_annotations` config.With this parameter is possible to define custom annotations that will be used on the services created by the operator. The `custom_service_annotations` as all the other [operator parameters][1] are defined on the operator level and do not allow customization on the cluster level. A cluster may require different service annotations, as for example, set up different cloud load balancers timeouts, different ingress annotations, and/or enable more customizable environments. This commit introduces a new parameter on the cluster level, called `serviceAnnotations`, responsible for defining custom annotations just for the services created by the operator to the specifically defined cluster. It allows a mix of configuration between `custom_service_annotations` and `serviceAnnotations` where the latest one will have priority. In order to allow custom service annotations to be used on services without LoadBalancers (as for example, service mesh services annotations) both `custom_service_annotations` and `serviceAnnotations` are applied independently of load-balancing configuration. For retro-compatibility purposes, `custom_service_annotations` is still under [Load balancer related options][2]. The two default annotations when using LoadBalancer services, `external-dns.alpha.kubernetes.io/hostname` and `service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` are still defined by the operator. `service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` can be overridden by `custom_service_annotations` or `serviceAnnotations`, allowing a more customizable environment. `external-dns.alpha.kubernetes.io/hostname` can not be overridden once there is no differentiation between custom service annotations for replicas and masters. It updates the documentation and creates the necessary unit and e2e tests to the above-described feature too. [1]: https://github.com/zalando/postgres-operator/blob/master/docs/reference/operator_parameters.md [2]: https://github.com/zalando/postgres-operator/blob/master/docs/reference/operator_parameters.md#load-balancer-related-options
1 parent a660d75 commit ba60e15

15 files changed

+565
-37
lines changed

‎charts/postgres-operator/crds/postgresqls.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,6 +266,10 @@ spec:
266266
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
267267
# Note: the value specified here must not be zero or be higher
268268
# than the corresponding limit.
269+
serviceAnnotations:
270+
type: object
271+
additionalProperties:
272+
type: string
269273
sidecars:
270274
type: array
271275
nullable: true

‎docs/administrator.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,17 @@ cluster manifest. In the case any of these variables are omitted from the
376376
manifest, the operator configuration settings `enable_master_load_balancer` and
377377
`enable_replica_load_balancer` apply. Note that the operator settings affect
378378
all Postgresql services running in all namespaces watched by the operator.
379+
If load balancing is enabled two default annotations will be applied to its
380+
services:
381+
382+
- `external-dns.alpha.kubernetes.io/hostname` with the value defined by the
383+
operator configs `master_dns_name_format` and `replica_dns_name_format`.
384+
This value can't be overwritten. If any changing in its value is needed, it
385+
MUST be done changing the DNS format operator config parameters; and
386+
- `service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` with
387+
a default value of "3600". This value can be overwritten with the operator
388+
config parameter `custom_service_annotations` or the cluster parameter
389+
`serviceAnnotations`.
379390

380391
To limit the range of IP addresses that can reach a load balancer, specify the
381392
desired ranges in the `allowedSourceRanges` field (applies to both master and

‎docs/reference/cluster_manifest.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,11 @@ These parameters are grouped directly under the `spec` key in the manifest.
122122
A map of key value pairs that gets attached as [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
123123
to each pod created for the database.
124124

125+
* **serviceAnnotations**
126+
A map of key value pairs that gets attached as [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
127+
to the services created for the database cluster. Check the
128+
[administrator docs](https://github.com/zalando/postgres-operator/blob/master/docs/administrator.md#load-balancers-and-allowed-ip-ranges)
129+
for more information regarding default values and overwrite rules.
125130

126131
* **enableShmVolume**
127132
Start a database pod without limitations on shm memory. By default Docker

‎docs/reference/operator_parameters.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -388,8 +388,9 @@ In the CRD-based configuration they are grouped under the `load_balancer` key.
388388
`false`.
389389

390390
* **custom_service_annotations**
391-
when load balancing is enabled, LoadBalancer service is created and
392-
this parameter takes service annotations that are applied to service.
391+
This key/value map provides a list of annotations that get attached to each
392+
service of a cluster created by the operator. If the annotation key is also
393+
provided by the cluster definition, the manifest value is used.
393394
Optional.
394395

395396
* **master_dns_name_format** defines the DNS name string template for the

‎e2e/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,3 +44,4 @@ The current tests are all bundled in [`test_e2e.py`](tests/test_e2e.py):
4444
* taint-based eviction of Postgres pods
4545
* invoking logical backup cron job
4646
* uniqueness of master pod
47+
* custom service annotations

‎e2e/tests/test_e2e.py

Lines changed: 46 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -211,8 +211,8 @@ def test_logical_backup_cron_job(self):
211211
schedule = "7 7 7 7 *"
212212
pg_patch_enable_backup = {
213213
"spec": {
214-
"enableLogicalBackup": True,
215-
"logicalBackupSchedule": schedule
214+
"enableLogicalBackup": True,
215+
"logicalBackupSchedule": schedule
216216
}
217217
}
218218
k8s.api.custom_objects_api.patch_namespaced_custom_object(
@@ -234,7 +234,7 @@ def test_logical_backup_cron_job(self):
234234
image = "test-image-name"
235235
patch_logical_backup_image = {
236236
"data": {
237-
"logical_backup_docker_image": image,
237+
"logical_backup_docker_image": image,
238238
}
239239
}
240240
k8s.update_config(patch_logical_backup_image)
@@ -247,7 +247,7 @@ def test_logical_backup_cron_job(self):
247247
# delete the logical backup cron job
248248
pg_patch_disable_backup = {
249249
"spec": {
250-
"enableLogicalBackup": False,
250+
"enableLogicalBackup": False,
251251
}
252252
}
253253
k8s.api.custom_objects_api.patch_namespaced_custom_object(
@@ -257,6 +257,37 @@ def test_logical_backup_cron_job(self):
257257
self.assertEqual(0, len(jobs),
258258
"Expected 0 logical backup jobs, found {}".format(len(jobs)))
259259

260+
@timeout_decorator.timeout(TEST_TIMEOUT_SEC)
261+
def test_service_annotations(self):
262+
'''
263+
Create a Postgres cluster with service annotations and check them.
264+
'''
265+
k8s = self.k8s
266+
patch_custom_service_annotations = {
267+
"data": {
268+
"custom_service_annotations": "foo:bar",
269+
}
270+
}
271+
k8s.update_config(patch_custom_service_annotations)
272+
273+
k8s.create_with_kubectl("manifests/postgres-manifest-with-service-annotations.yaml")
274+
annotations = {
275+
"annotation.key": "value",
276+
"foo": "bar",
277+
}
278+
self.assertTrue(k8s.check_service_annotations(
279+
"version=acid-service-annotations,spilo-role=master", annotations))
280+
self.assertTrue(k8s.check_service_annotations(
281+
"version=acid-service-annotations,spilo-role=replica", annotations))
282+
283+
# clean up
284+
unpatch_custom_service_annotations = {
285+
"data": {
286+
"custom_service_annotations": "",
287+
}
288+
}
289+
k8s.update_config(unpatch_custom_service_annotations)
290+
260291
def assert_master_is_unique(self, namespace='default', version="acid-minimal-cluster"):
261292
'''
262293
Check that there is a single pod in the k8s cluster with the label "spilo-role=master"
@@ -322,6 +353,16 @@ def wait_for_pod_start(self, pod_labels, namespace='default'):
322353
pod_phase = pods[0].status.phase
323354
time.sleep(self.RETRY_TIMEOUT_SEC)
324355

356+
def check_service_annotations(self, svc_labels, annotations, namespace='default'):
357+
svcs = self.api.core_v1.list_namespaced_service(namespace, label_selector=svc_labels, limit=1).items
358+
for svc in svcs:
359+
if len(svc.metadata.annotations) != len(annotations):
360+
return False
361+
for key in svc.metadata.annotations:
362+
if svc.metadata.annotations[key] != annotations[key]:
363+
return False
364+
return True
365+
325366
def wait_for_pg_to_scale(self, number_of_instances, namespace='default'):
326367

327368
body = {
@@ -330,7 +371,7 @@ def wait_for_pg_to_scale(self, number_of_instances, namespace='default'):
330371
}
331372
}
332373
_ = self.api.custom_objects_api.patch_namespaced_custom_object(
333-
"acid.zalan.do", "v1", namespace, "postgresqls", "acid-minimal-cluster", body)
374+
"acid.zalan.do", "v1", namespace, "postgresqls", "acid-minimal-cluster", body)
334375

335376
labels = 'version=acid-minimal-cluster'
336377
while self.count_pods_with_label(labels) != number_of_instances:

‎manifests/complete-postgres-manifest.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,8 @@ spec:
3232
# spiloFSGroup: 103
3333
# podAnnotations:
3434
# annotation.key: value
35+
# serviceAnnotations:
36+
# annotation.key: value
3537
# podPriorityClassName: "spilo-pod-priority"
3638
# tolerations:
3739
# - key: postgres

‎manifests/postgres-manifest-with-service-annotations.yaml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: "acid.zalan.do/v1"
2+
kind: postgresql
3+
metadata:
4+
name: acid-service-annotations
5+
spec:
6+
teamId: "acid"
7+
volume:
8+
size: 1Gi
9+
numberOfInstances: 2
10+
users:
11+
zalando: # database owner
12+
- superuser
13+
- createdb
14+
foo_user: [] # role for application foo
15+
databases:
16+
foo: zalando # dbname: owner
17+
postgresql:
18+
version: "11"
19+
serviceAnnotations:
20+
annotation.key: value

‎manifests/postgresql.crd.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,10 @@ spec:
230230
pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$'
231231
# Note: the value specified here must not be zero or be higher
232232
# than the corresponding limit.
233+
serviceAnnotations:
234+
type: object
235+
additionalProperties:
236+
type: string
233237
sidecars:
234238
type: array
235239
nullable: true

‎pkg/apis/acid.zalan.do/v1/crds.go

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,14 @@ var PostgresCRDResourceValidation = apiextv1beta1.CustomResourceValidation{
383383
},
384384
},
385385
},
386+
"serviceAnnotations": {
387+
Type: "object",
388+
AdditionalProperties: &apiextv1beta1.JSONSchemaPropsOrBool{
389+
Schema: &apiextv1beta1.JSONSchemaProps{
390+
Type: "string",
391+
},
392+
},
393+
},
386394
"sidecars": {
387395
Type: "array",
388396
Items: &apiextv1beta1.JSONSchemaPropsOrArray{

‎pkg/apis/acid.zalan.do/v1/postgresql_type.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ type PostgresSpec struct {
6060
LogicalBackupSchedule string `json:"logicalBackupSchedule,omitempty"`
6161
StandbyCluster *StandbyDescription `json:"standby"`
6262
PodAnnotations map[string]string `json:"podAnnotations"`
63+
ServiceAnnotations map[string]string `json:"serviceAnnotations"`
6364

6465
// deprecated json tags
6566
InitContainersOld []v1.Container `json:"init_containers,omitempty"`

‎pkg/apis/acid.zalan.do/v1/util_test.go

Lines changed: 94 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -456,18 +456,84 @@ var postgresqlList = []struct {
456456
PostgresqlList{},
457457
errors.New("unexpected end of JSON input")}}
458458

459-
var annotations = []struct {
459+
var podAnnotations = []struct {
460460
about string
461461
in []byte
462462
annotations map[string]string
463463
err error
464464
}{{
465-
about: "common annotations",
466-
in: []byte(`{"kind": "Postgresql","apiVersion": "acid.zalan.do/v1","metadata": {"name": "acid-testcluster1"}, "spec": {"podAnnotations": {"foo": "bar"},"teamId": "acid", "clone": {"cluster": "team-batman"}}}`),
465+
about: "common annotations",
466+
in: []byte(`{
467+
"kind": "Postgresql",
468+
"apiVersion": "acid.zalan.do/v1",
469+
"metadata": {
470+
"name": "acid-testcluster1"
471+
},
472+
"spec": {
473+
"podAnnotations": {
474+
"foo": "bar"
475+
},
476+
"teamId": "acid",
477+
"clone": {
478+
"cluster": "team-batman"
479+
}
480+
}
481+
}`),
467482
annotations: map[string]string{"foo": "bar"},
468483
err: nil},
469484
}
470485

486+
var serviceAnnotations = []struct {
487+
about string
488+
in []byte
489+
annotations map[string]string
490+
err error
491+
}{
492+
{
493+
about: "common single annotation",
494+
in: []byte(`{
495+
"kind": "Postgresql",
496+
"apiVersion": "acid.zalan.do/v1",
497+
"metadata": {
498+
"name": "acid-testcluster1"
499+
},
500+
"spec": {
501+
"serviceAnnotations": {
502+
"foo": "bar"
503+
},
504+
"teamId": "acid",
505+
"clone": {
506+
"cluster": "team-batman"
507+
}
508+
}
509+
}`),
510+
annotations: map[string]string{"foo": "bar"},
511+
err: nil,
512+
},
513+
{
514+
about: "common two annotations",
515+
in: []byte(`{
516+
"kind": "Postgresql",
517+
"apiVersion": "acid.zalan.do/v1",
518+
"metadata": {
519+
"name": "acid-testcluster1"
520+
},
521+
"spec": {
522+
"serviceAnnotations": {
523+
"foo": "bar",
524+
"post": "gres"
525+
},
526+
"teamId": "acid",
527+
"clone": {
528+
"cluster": "team-batman"
529+
}
530+
}
531+
}`),
532+
annotations: map[string]string{"foo": "bar", "post": "gres"},
533+
err: nil,
534+
},
535+
}
536+
471537
func mustParseTime(s string) metav1.Time {
472538
v, err := time.Parse("15:04", s)
473539
if err != nil {
@@ -517,21 +583,42 @@ func TestWeekdayTime(t *testing.T) {
517583
}
518584
}
519585

520-
func TestClusterAnnotations(t *testing.T) {
521-
for _, tt := range annotations {
586+
func TestPodAnnotations(t *testing.T) {
587+
for _, tt := range podAnnotations {
522588
t.Run(tt.about, func(t *testing.T) {
523589
var cluster Postgresql
524590
err := cluster.UnmarshalJSON(tt.in)
525591
if err != nil {
526592
if tt.err == nil || err.Error() != tt.err.Error() {
527-
t.Errorf("Unable to marshal cluster with annotations: expected %v got %v", tt.err, err)
593+
t.Errorf("Unable to marshal cluster with podAnnotations: expected %v got %v", tt.err, err)
528594
}
529595
return
530596
}
531597
for k, v := range cluster.Spec.PodAnnotations {
532598
found, expected := v, tt.annotations[k]
533599
if found != expected {
534-
t.Errorf("Didn't find correct value for key %v in for podAnnotations: Expected %v found %v", k, expected, found)
600+
t.Errorf("Didn't find correct value for key %v in for podAnnotations: Expected %v found %v", k, expected, found)
601+
}
602+
}
603+
})
604+
}
605+
}
606+
607+
func TestServiceAnnotations(t *testing.T) {
608+
for _, tt := range serviceAnnotations {
609+
t.Run(tt.about, func(t *testing.T) {
610+
var cluster Postgresql
611+
err := cluster.UnmarshalJSON(tt.in)
612+
if err != nil {
613+
if tt.err == nil || err.Error() != tt.err.Error() {
614+
t.Errorf("Unable to marshal cluster with serviceAnnotations: expected %v got %v", tt.err, err)
615+
}
616+
return
617+
}
618+
for k, v := range cluster.Spec.ServiceAnnotations {
619+
found, expected := v, tt.annotations[k]
620+
if found != expected {
621+
t.Errorf("Didn't find correct value for key %v in for serviceAnnotations: Expected %v found %v", k, expected, found)
535622
}
536623
}
537624
})

‎pkg/apis/acid.zalan.do/v1/zz_generated.deepcopy.go

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)