Self-hosted agents allow you to run env0 deployment workloads on your own Kubernetes cluster.
- Execution is contained on your own servers/infrastructure.
- The agent requires an internet connection but no inbound network access.
- Secrets can be stored on your own infrastructure.
Self-hosted agents are only available to Business and Enterprise level customers. Click here for more details
- Kubernetes cluster at version >= 1.18
- The agent will be installed using a Helm chart.
Use our repo k8s-modules which contains Terraform code for an easier cluster installation. You can use the main provider folder for a "full-blown" installation or a specific module to complete the requirements.
- The env0 agent will scale pods up and down according to deployment usage. Please notice, the ability to scale cluster nodes up and down must be provided.
- A pod running a single deployment requires
memory: 1500Mi, so the cluster nodes must be able to provide this resource request. Limits can be adjusted by providing custom configuration during chart installation.
- Minimum node requirements: an instance with at least 2 CPU and 8GiB memory
For the EKS cluster, you can use this TF example
- env0 will store the deployment state and working directory, on a persistent volume in the cluster.
- Must support Dynamic Provisioning and ReadWriteMany access mode.
- The requested storage space is
- The cluster must include a
- The StorageClass should be set up with
reclaimPolicy: Retain, so in case the agent needs to be replaced or uninstalled, data won't be lost.
We recommend the current implementations for the major cloud providers:
By default, the deployment state and working directory is stored in a PV (Persistent Volume) which is configured on your Kubernetes cluster. In the scenario where PV creation or management is difficult, or not required, you can use env0-Hosted Encrypted State with
- As Self Hosted agents allow you to store secrets on your own infrastructure, using secrets stored in the env0 platform is not allowed for self-hosted agents.
- If you are migrating from the SaaS to a self-hosted agent, deployments attempting to use these secrets will fail.
- This includes sensitive configuration variables, SSH keys, and Cloud Deployment credentials. The values for these secrets should be replaced with references to your secret store, as detailed in the table below.
- In order to use an external secret store, authentication to the secret store must be configured using a custom Helm values file. The required parameters are detailed below.
- Storing secrets is supported using these secret stores:
|Secret reference format
|AWS Secrets Manager (us-east-1)
|Set by the
awsSecretsRegion helm value. Defaults to
|GCP Secrets Manager
|Your GCP project's default region
|Access to the secret must be possible using the
customerGoogleCredentials configuration or using GKE workload identity. The
customerGoogleProject configuration must be supplied and will be used to access secrets in that project only. The permission 'secrets.versions.access' is required.
|Azure Key Vault
|Your Azure subscription's default region
@<namespace> is optional
values.yml will be provided by env0 with the configuration env0 provides.
The customer will need to provide a
values.customer.yml with the following values (optional), to enable specific features :
|Required for feature
|Custom Docker image URI and Base64 encoded
|Custom Docker image. See Using a custom image in an agent
|Base64 encoded Infracost API key
|Base64 encoded AWS Access Key ID & Secret
|AWS Assume role for deploy credentials. Also, see Authenticating the agent on AWS EKS
|Container resource limits
Read more about resource allocation
|Custom deployment pod size
|Container resource requests
|Custom deployment container resources
|An array of
toleration objects to apply to the deployment containers.
|Allows you to constrain which nodes env0 pods are eligible to be scheduled on (see https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)
|Custom node affinity
|Affinity for deployment pods. This will override the default affinity for deployment pods.
|Custom node affinity
|Base64 encoded AWS Access Key ID & Secret. Requires the
|Using AWS Secrets Manager to store secrets for the agent
|Base64 encoded GCP project name and JSON service-key contents. Requires the
Secret Manager Secret Access role.
|Using GCP Secret Manager to store secrets for the agent. These credentials are not used for the deployment itself. If
deploymentJobServiceAccountName is set - Workload identity will override any supplied credentials.
|Base64 encoded Azure Credentials.
|Using Azure Key Vault Secrets to store secrets for the agent
|Base64 encoded HCP Vault token, and the cluster's URL (also base64 encoded)
|Using HCP Vault to store secrets for the agent
|Base64 Bitbucket server credentials in the format
username:token (using a Personal Access token).
|On-premise Bitbucket Server installation.
|Base64 Gitlab Enterprise credentials in the form of a Personal Access token.
|On-premise Gitlab Enterprise installation
|In cases where your GitLab instance base url is not at the root of the url, and in a separate path, e.g
https://gitlab.acme.com/prod you should define that added suffix to this value
|On-premise Gitlab Enterprise installation
|Github Enterprise Integration (see step 3)
|On-premise GitHub Enterprise installation
|When set, cloning a git repository will only be permitted if the git url matches the regular expression set.
|VCS URL Whitelisting
|An array of strings. Each represents a name of Kubernetes secret that contains custom CA certificates. Those certificates will be available during deployments.
|Custom CA Certificates. More details here.
|When set to
true, cloning a git repo will not verify SSL/TLS certs
|Ignoring SSL/TLS certs for on-premise git servers.
|Ability to change the default PVC storage class name for env0 self-hosted agent
|the default is
Please pay attention, when you change this - you should also change your storage class name to match this configuration
|Customize the Kubernetes service account used by the deployment pod. Primarily for pod-level IAM permissions.
|the default is
|How many successful and failed deployment jobs should be kept in the Kubernetes cluster history.
|The default is 10 for each one.
|When set to
true, the pod operates under
node user instead of
|Increased agent pod security
|A base64 string. When set, deployment state and working directory will be encrypted and persisted on Env0's end.
|Env0-Hosted Encrypted State
level - debug/info/warn/error
format - json/cli
imagePullPolicy attribute - Always/Never/IfNotPresent
|Agent's Proxy pod config:
install - true/false
limits - k8s (cpu & memory) limits - default is 250m and 500Mi
|A number of deployment pods that should be left "warm" (running & idle) and ready for new deployments.
|Additional Environment variables to be passed to the agent pods, which will also be passed to the deployment process. Those are set plain yaml object i.e:
|Additional labels to be set on deployment pods. Those are set plain yaml object i.e:
|Additional annotations to be set on deployment pods. Those are set plain yaml object i.e:
|Additional labels to be set on agent (trigger/proxy) pods. Those are set plain yaml object i.e:
|Additional annotations to be set on agent (trigger/proxy) pods. Those are set plain yaml object i.e:
Base64 Encoding Values
To ensure no additional new line characters are being encoded, please use the following command in your terminal:
echo -n $VALUE | base64
We do our best to support all common configuration case scenarios, but sometimes a more exotic or pre-released configuration is required.
You may desire to add a limit on the number of concurrent runs. To do so add a "Resource Quota" for the agent namespace with a parameter on
count/jobs.batch See: https://kubernetes.io/docs/concepts/policy/resource-quotas/ for more details.
Add our Helm Repo
helm repo add env0 https://env0.github.io/self-hosted
Update Helm Repo
helm repo update
Download the configuration file:
<your_agent_key>_values.yamlfrom Organization Settings -> Agents tab
- Install the Helm Charts
helm install --create-namespace env0-agent env0/env0-agent --namespace env0-agent -f <your_agent_key>_values.yaml -f values.customer.yaml # values.customer.yaml should contain any optional configuration options as detailed above
Example for helm install
helm upgrade env0-agent env0/env0-agent --namespace env0-agent
You were previously requested to download the values.yaml file, which is not required anymore for an upgrade.
Custom Agent Docker Image
If you extended the docker image on the agent, you should update the agent version in your custom image as well.
After installing a new version of the env0 agent helm chart is is highly recommended to verify the installation by running:
helm test env0-agent --namespace env0-agent --logs --timeout 1m
The agent needs the following outbound domains access:
|env0 SaaS platform, the agent needs to communicate with the SaaS platform.
|GitHub Docker registry which holds the Docker container of the agent.
|Downloading Terraform binaries
|Downloading public modules from the Terraform Registry
|github.com, gitlab.com, bitbucket.org
|Git VCS providers ( ports 22, 9418, 80, 443 )
|Cost estimation by Infracost
- Make sure to allow access to your cloud providers, VCS domains, and any other tool that creates an outbound request.
Note that if you have the cluster behind a managed firewall, you might need to whitelist the Cluster's API server's FQDN and its corresponding Public IP.
Updated about 1 month ago