Skip to content

Commit 9b21bda

Browse files
mikhail-klimkomikhail-klimko
committed
updated volume selectors and runtime cleaners
1 parent 7f057d6 commit 9b21bda

File tree

1 file changed

+124
-0
lines changed

1 file changed

+124
-0
lines changed

‎_docs/administration/codefresh-runner.md

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -413,6 +413,41 @@ codefresh attach runtime --agent-name $AGENT_NAME --agent-kube-namespace codefre
413413
414414
You can fine tune the installation of the runner to better match your environment and cloud provider.
415415
416+
### Volume reusage policy
417+
418+
The behavior of how the volumes are reused depends on volume selector configuration.
419+
`reuseVolumeSelector` option is configurable in runtime environment spec.
420+
421+
The following options are available:
422+
423+
- `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName'` - determined PV can be used by **ANY** pipeline of your account (it's a **default** volume selector).
424+
425+
- `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id'` - determined PV can be used only by a **single pipeline**.
426+
427+
- `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id,codefresh-app,io.codefresh.branch_name'` - determined PV can be used only by **single pipeline AND single branch**.
428+
429+
- `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id,trigger'` - determined PV can be used only by **single pipeline AND single trigger**.
430+
431+
To change volume selector follow this procedure:
432+
433+
```shell
434+
#get runtime environmet spec yaml
435+
codefresh get re $RUNTIME_NAME -o yaml > runtime.yaml
436+
```
437+
Under `dockerDaemonScheduler.pvcs.dind` block specify `reuseVolumeSelector`:
438+
{% highlight yaml %}
439+
{% raw %}
440+
pvcs:
441+
dind:
442+
volumeSize: 30Gi
443+
reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id'
444+
{% endraw %}
445+
{% endhighlight %}
446+
```shell
447+
#apply changes to runtime environment
448+
codefresh patch re -f runtime.yaml
449+
```
450+
416451
### Custom global environment variables
417452

418453
You can add your own environment variables in the runtime environment, so that all pipeline steps have access to the same set of external files. A typical
@@ -1474,8 +1509,97 @@ For example, let's say Venona-zoneA is the default RE, then, that means that for
14741509

14751510
Regarding [Regional Persistent Disks](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/regional-pd), their support is not currently implemented in the Codefresh runner.
14761511

1512+
## Runtime Cleaners
1513+
1514+
##### Key points:
1515+
- Codefresh pipeline require disk space for:
1516+
- [Pipeline Shared Volume](https://codefresh.io/docs/docs/yaml-examples/examples/shared-volumes-between-builds/) (`/codefresh/volume`, implemented as [docker volume](https://docs.docker.com/storage/volumes/))
1517+
- Docker containers - running and stopped
1518+
- Docker images and cached layers
1519+
- To improve performance, `volume-provisioner` is able to provision previously used disk with docker images and pipeline volume from previously running builds. It improves performance by using docker cache and decreasing I/O rate.
1520+
- Least recently docker images and volumes should be cleaned to avoid out-of-space errors.
1521+
- There are several places where pipeline volume cleanup is required, so there are several kinds of cleaner.
1522+
1523+
##### Cleaners:
1524+
- [IN-DIND cleaner](https://github.com/codefresh-io/dind/tree/master/cleaner) - deletes extra docker containers, volumes, images on dind pod
1525+
- [External volumes cleaner](https://github.com/codefresh-io/runtime-cluster-monitor/blob/master/chart/templates/dind-volume-cleanup.yaml) - deletes unused **external** PVs (EBS, GCE/Azure disks)
1526+
- [Local volumes cleaner](https://github.com/codefresh-io/dind-volume-utils/blob/master/local-volumes/lv-cleaner.sh) - deletes **local** volumes in case node disk space is close to the threshold
1527+
1528+
***
1529+
1530+
##### IN-DIND cleaner
1531+
**Purpose:** removes unneeded *docker containers, images, volumes* inside kubernetes volume mounted to the dind pod
1532+
1533+
**Where it runs:** Running inside each dind pod as script
1534+
1535+
**Triggered by:** SIGTERM and also during the run when disk usage (cleaner-agent ) > 90% (configurable)
1536+
1537+
**Configured by:** Environment Variables which can be set in Runtime Environment configuration
1538+
1539+
**Configuration/Logic:** [README.md](https://github.com/codefresh-io/dind/tree/master/cleaner#readme)
1540+
1541+
Override `dockerDaemonScheduler.envVars` on Runtime Environment if necessary (the following are **defaults**):
1542+
{% highlight yaml %}
1543+
{% raw %}
1544+
dockerDaemonScheduler:
1545+
envVars:
1546+
CLEAN_DOCKER: 'true'
1547+
CLEAN_PERIOD_BUILDS: '5'
1548+
IMAGE_RETAIN_PERIOD: '14400'
1549+
VOLUMES_RETAIN_PERIOD: '14400'
1550+
{% endraw %}
1551+
{% endhighlight %}
1552+
1553+
***
1554+
1555+
##### External volumes cleaner
1556+
**Purpose:** removes unused *kubernetes volumes and related backend volumes*
1557+
1558+
**Where it runs:** on Runtime Cluster as cron job
1559+
(`kubectl get cronjobs -n codefresh -l app=dind-volume-cleanup`). Installed in case the Runner uses non-local volumes (`Storage.Backend != local`)
1560+
1561+
**Triggered by:** CronJob every 10min (configurable), part of [runtime-cluster-monitor](https://github.com/codefresh-io/runtime-cluster-monitor/blob/master/chart/templates/dind-volume-cleanup.yaml) and runner deployment
1562+
1563+
**Configuration:**
1564+
1565+
Set `codefresh.io/volume-retention` annotation on Runtime Environment:
1566+
{% highlight yaml %}
1567+
{% raw %}
1568+
dockerDaemonScheduler:
1569+
pvcs:
1570+
dind:
1571+
storageClassName: dind-ebs-volumes-runner-codefresh
1572+
reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id'
1573+
volumeSize: 32Gi
1574+
annotations:
1575+
codefresh.io/volume-retention: 7d
1576+
{% endraw %}
1577+
{% endhighlight %}
1578+
1579+
Override environment variables for `dind-volume-cleanup` cronjob if necessary:
1580+
- `RETENTION_DAYS` (defaults to 4)
1581+
- `MOUNT_MIN` (defaults to 3)
1582+
- `PROVISIONED_BY` (defaults to `codefresh.io/dind-volume-provisioner`)
1583+
1584+
About *optional* `-m` argument:
1585+
- `dind-volume-cleanup` to clean volumes that were last used more than `RETENTION_DAYS` ago
1586+
- `dind-volume-cleanup-m` to clean volumes that were used more than a day ago, but mounted less than `MOUNT_MIN` times
1587+
1588+
***
1589+
1590+
##### Local volumes cleaner
1591+
**Purpose:** deletes local volumes in case node disk space is close to the threshold
1592+
1593+
**Where it runs:** on each node on runtime cluster as DaemonSet `dind-lv-monitor`. Installed in case the Runner use local volumes (`Storage.Backend == local`)
1594+
1595+
**Triggered by:** starts clean if disk space usage or inodes usage is more than thresholds (configurable)
14771596

1597+
**Configuration:**
14781598

1599+
Override environment variables for `dind-lv-monitor` daemonset if necessary:
1600+
- `VOLUME_PARENT_DIR` - default `/var/lib/codefresh`
1601+
- `KB_USAGE_THRESHOLD` - default 80 (percentage)
1602+
- `INODE_USAGE_THRESHOLD` - default 80
14791603

14801604
## Troubleshooting
14811605

0 commit comments

Comments
 (0)