Installing Yatai#

Welcome to Yatai! You will learn the system requirements, software dependencies, instructions for installation. See Yatai architecture for a detailed introduction of the yatai component.

Prerequisites#

Kubernetes

Kubernetes cluster with version 1.20 or newer (<=1.25 if you need yatai-deployment)

Note

If you do not have a production Kubernetes cluster and want to install yatai for development and testing purposes. You can use minikube to set up a local Kubernetes cluster for testing. If you are using macOS, you should use hyperkit driver to prevent the macOS docker desktop network limitation
Dynamic Volume Provisioning

If you use MinIO as the object storage, you need to enable dynamic volume provisioning in your Kubernetes cluster. See Dynamic Volume Provisioning for more details.
Helm

Yatai uses Helm to install yatai.

Quick Installation#

Note

This quick installation script should only be used for development and testing purposes.

This script will automatically install the following dependencies inside the yatai-system namespace of the Kubernetes cluster:

PostgreSQL
MinIO

bash <(curl -s "https://raw.githubusercontent.com/bentoml/yatai/main/scripts/quick-install-yatai.sh")

Installation Steps#

Note

If you don’t have kubectl installed and you are using minikube, you can use minikube kubectl -- instead of kubectl, for more details on using it, please check: minikube kubectl

1. Create Namespace#

kubectl create namespace yatai-system

2. Prepare PostgreSQL#

Use Existing PostgreSQL

Prepare PostgreSQL connection params

export PG_PASSWORD=xxx
export PG_HOST=1.1.1.1
export PG_PORT=5432
export PG_DATABASE=yatai
export PG_USER=postgres
export PG_SSLMODE=disable

Create the PostgreSQL database $PG_DATABASE

PGPASSWORD=$PG_PASSWORD psql \
    -h $PG_HOST \
    -p $PG_PORT \
    -U $PG_USER \
    -d postgres \
    -c "create database $PG_DATABASE"

Create AWS RDS

Prerequisites:

jq command line tool. Follow the official installation guide to install jq.
AWS CLI with RDS permission. Follow the official installation guide to install AWS CLI.

Prepare params

export PG_PASSWORD=$(LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 20)
export PG_USER=yatai
export PG_DATABASE=yatai
export PG_SSLMODE=disable
export RDS_INSTANCE_IDENTIFIER=yatai-postgresql

aws rds create-db-instance \
    --db-name $PG_DATABASE \
    --db-instance-identifier $RDS_INSTANCE_IDENTIFIER \
    --db-instance-class db.t3.micro \
    --engine postgres \
    --master-username $PG_USER \
    --master-user-password $PG_PASSWORD \
    --allocated-storage 20

Get the RDS instance host and port

read PG_HOST PG_PORT < <(echo $(aws rds describe-db-instances --db-instance-identifier $RDS_INSTANCE_IDENTIFIER | jq '.DBInstances[0].Endpoint.Address, .DBInstances[0].Endpoint.Port'))
PG_HOST=$(sh -c "echo $PG_HOST")

Test the connection

kubectl -n yatai-system delete pod postgresql-ha-client 2> /dev/null || true; \
kubectl run postgresql-ha-client --rm --tty -i --restart='Never' \
    --namespace yatai-system \
    --image docker.io/bitnami/postgresql-repmgr:14.4.0-debian-11-r13 \
    --env="PGPASSWORD=$PG_PASSWORD" \
    --command -- psql -h $PG_HOST -p $PG_PORT -U $PG_USER -d $PG_DATABASE -c "select 1"

Expected output:

?column?
----------
        1
(1 row)

pod "postgresql-ha-client" deleted

Note

If there is no response for a long time, you can check if the VPC security group of the AWS RDS instance has opened port 5432 for public access

Install New PostgreSQL

Note

Do not recommend for production because this installation of PostgreSQL does not provide high availability and data replication.

Install the postgresql-ha helm chart:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update bitnami
helm upgrade --install postgresql-ha bitnami/postgresql-ha -n yatai-system --version 10.0.6

Verify the postgresql-ha installation:

Monitor the postgresql-ha components until all of the components show a STATUS of Running or Completed. You can do this by running the following command and inspecting the output:

kubectl -n yatai-system get pod -l app.kubernetes.io/name=postgresql-ha

Example output:

Note

You need to be patient for a while until the status of all pods becomes Running, the number of pods depends on how many nodes you have

NAME                                    READY   STATUS    RESTARTS   AGE
postgresql-ha-postgresql-0              1/1     Running   0          3m42s
postgresql-ha-pgpool-56cf7b6b98-fs7g4   1/1     Running   0          3m42s
postgresql-ha-postgresql-1              1/1     Running   0          3m41s
postgresql-ha-postgresql-2              1/1     Running   0          3m41s

Get the PostgreSQL connection params

export PG_PASSWORD=$(kubectl get secret --namespace yatai-system postgresql-ha-postgresql -o jsonpath="{.data.password}" | base64 -d)
export PG_HOST=postgresql-ha-pgpool.yatai-system.svc.cluster.local
export PG_PORT=5432
export PG_DATABASE=yatai
export PG_USER=postgres
export PG_SSLMODE=disable

Test PostgreSQL connection

You can create a connection test by running the following command and inspecting the output:

kubectl -n yatai-system delete pod postgresql-ha-client 2> /dev/null || true; \
kubectl run postgresql-ha-client --rm --tty -i --restart='Never' \
    --namespace yatai-system \
    --image docker.io/bitnami/postgresql-repmgr:14.4.0-debian-11-r13 \
    --env="PGPASSWORD=$PG_PASSWORD" \
    --command -- psql -h postgresql-ha-pgpool -p 5432 -U postgres -d postgres -c "select 1"

Expected output:

?column?
----------
        1
(1 row)

pod "postgresql-ha-client" deleted

Create the PostgreSQL database $PG_DATABASE

You can create the database $PG_DATABASE by running the following command and inspecting the output:

kubectl -n yatai-system delete pod postgresql-ha-client 2> /dev/null || true; \
kubectl run postgresql-ha-client --rm --tty -i --restart='Never' \
    --namespace yatai-system \
    --image docker.io/bitnami/postgresql-repmgr:14.4.0-debian-11-r13 \
    --env="PGPASSWORD=$PG_PASSWORD" \
    --command -- psql -h postgresql-ha-pgpool -p 5432 -U postgres -d postgres -c "create database $PG_DATABASE"

Expected output:

If you don't see a command prompt, try pressing enter.
CREATE DATABASE
pod "postgresql-ha-client" deleted

Test PostgreSQL environment variables#

You can create a connection test by running the following command and inspecting the output:

kubectl -n yatai-system delete pod postgresql-ha-client 2> /dev/null || true; \
kubectl run postgresql-ha-client --rm --tty -i --restart='Never' \
    --namespace yatai-system \
    --image docker.io/bitnami/postgresql-repmgr:14.4.0-debian-11-r13 \
    --env="PGPASSWORD=$PG_PASSWORD" \
    --command -- psql -h $PG_HOST -p $PG_PORT -U $PG_USER -d $PG_DATABASE -c "select 1"

Expected output:

?column?
----------
        1
(1 row)

pod "postgresql-ha-client" deleted

Note

If the above command does not respond for a long time and you are using AWS RDS, you can check if the VPC security group of the AWS RDS instance has port 5432 open for public access

3. Prepare Object Storage#

Note

Now Yatai only support S3 protocol

Use Existing AWS S3

Prepare S3 connection params

export S3_REGION=YOUR-S3-REGION
export S3_ENDPOINT="s3.${S3_REGION}.amazonaws.com"
export S3_BUCKET_NAME=YOUR-BUCKET-NAME
export S3_ACCESS_KEY=$(aws configure get default.aws_access_key_id)
export S3_SECRET_KEY=$(aws configure get default.aws_secret_access_key)
export S3_SECURE=true

Note

Remember to replace YOUR-S3-REGION with your S3 region, replace YOUR-BUCKET-NAME with your S3 bucket name

Use Existing AWS S3 with IAM

Prepare S3 connection params

export S3_REGION=YOUR-S3-REGION
export S3_ENDPOINT="s3.amazonaws.com"
export S3_BUCKET_NAME=YOUR-BUCKET-NAME
export S3_ACCESS_KEY=""
export S3_SECRET_KEY=""
export S3_SECURE=true

Note

Remember to replace YOUR-S3-REGION with your S3 region, replace YOUR-BUCKET-NAME with your S3 bucket name

Create IAM policy for S3 bucket access

Create a file named s3-iam-policy.json with the following content:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action": "s3:ListAllMyBuckets",
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":["s3:ListBucket","s3:GetBucketLocation"],
         "Resource":"arn:aws:s3:::${S3_BUCKET_NAME}"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::${S3_BUCKET_NAME}/*"
      }
   ]
}

Replace ${S3_BUCKET_NAME} with your S3 bucket name:

envsubst < s3-iam-policy.json > s3-iam-policy.json

Note

If you don’t have envsubst installed and you are using macOS, you can install it by running brew install gettext && brew link --force gettext

Create the IAM policy:

aws iam create-policy \
    --policy-name yatai-s3-access \
    --policy-document file://s3-iam-policy.json

Note

Please store the Arn of the created policy, you will need it in the next step. The Arn format is like this: arn:aws:iam::ACCOUNT_ID:policy/yatai-s3-access

Create IAM ServiceAccount for S3 access

Create yatai-system namespace:

kubectl create namespace yatai-system

Create IAM ServiceAccount:

eksctl create iamserviceaccount \
    --name yatai \
    --namespace yatai-system \
    --cluster YOUR-CLUSTER \
    --region YOUR-REGION \
    --attach-policy-arn YOUR-IAM-POLICY-ARN \
    --approve

Note

Remember to replace YOUR-CLUSTER with your EKS cluster name, replace YOUR-REGION with your EKS cluster region, replace YOUR-IAM-POLICY-ARN with the Arn of the created IAM policy

Create New AWS S3

Prerequisites:

AWS CLI with AWS S3 permission. Follow the official installation guide to install AWS CLI

Prepare params

export S3_BUCKET_NAME=yatai-registry
export S3_REGION=ap-northeast-3
export S3_ENDPOINT="s3.${S3_REGION}.amazonaws.com"
export S3_SECURE=true

Create AWS S3 bucket

aws s3api create-bucket \
    --bucket $S3_BUCKET_NAME \
    --region $S3_REGION \
    --create-bucket-configuration LocationConstraint=$S3_REGION

Get ACCESS_KEY and SECRET_KEY

export S3_ACCESS_KEY=$(aws configure get default.aws_access_key_id)
export S3_SECRET_KEY=$(aws configure get default.aws_secret_access_key)

Verify S3 connection

kubectl -n yatai-system delete pod s3-client 2> /dev/null || true; \
kubectl run s3-client --rm --tty -i --restart='Never' \
    --namespace yatai-system \
    --env "AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY" \
    --env "AWS_SECRET_ACCESS_KEY=$S3_SECRET_KEY" \
    --image quay.io/bentoml/s3-client:0.0.1 \
    --command -- sh -c "s3-client -e https://$S3_ENDPOINT listobj $S3_BUCKET_NAME && echo successfully"

The output should be:

successfully
pod "s3-client" deleted

Install MinIO

Note

Do not recommend for production. Because you need to maintain the stability and data security of this important blob storage cluster yourself, it is recommended to use the blob storage provided by the public cloud vendor since many public cloud vendors (e.g. AWS) already have very mature blob storage.

Install the minio-operator helm chart

helm repo add minio https://operator.min.io/
helm repo update minio

export S3_ACCESS_KEY=$(LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 20)
export S3_SECRET_KEY=$(LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 20)

cat <<EOF | helm upgrade --install minio-operator minio/minio-operator -n yatai-system -f -
tenants:
- image:
    pullPolicy: IfNotPresent
    repository: quay.io/bentoml/minio-minio
    tag: RELEASE.2021-10-06T23-36-31Z
  metrics:
    enabled: false
    port: 9000
  mountPath: /export
  name: yatai-minio
  namespace: yatai-system
  pools:
  - servers: 4
    size: 20Gi
    volumesPerServer: 4
  secrets:
    accessKey: $S3_ACCESS_KEY
    enabled: true
    name: yatai-minio
    secretKey: $S3_SECRET_KEY
  subPath: /data
EOF

Verify the minio-operator installation

Monitor the minio-operator components until all of the components show a STATUS of Running or Completed. You can do this by running the following command and inspecting the output:

kubectl -n yatai-system get pod -l app.kubernetes.io/name=minio-operator

Expected output:

Note

Wait until the status of all pods becomes Running before proceeding

NAME                                     READY   STATUS    RESTARTS   AGE
minio-operator-console-9d9cbbcc8-flzrw   1/1     Running   0          2m39s
minio-operator-6c984995c9-l8j2j          1/1     Running   0          2m39s

Verify the MinIO tenant installation

Monitor the MinIO tenant components until all of the components show a STATUS of Running or Completed. You can do this by running the following command and inspecting the output:

kubectl -n yatai-system get pod -l app=minio

Expected output:

Note

Since the pods are created by the minio-operator, it may take a minute for these pods to be created. Wait until the status of all pods becomes Running before proceeding.

NAME                 READY   STATUS    RESTARTS   AGE
yatai-minio-ss-0-0   1/1     Running   0          143m
yatai-minio-ss-0-1   1/1     Running   0          143m
yatai-minio-ss-0-2   1/1     Running   0          143m
yatai-minio-ss-0-3   1/1     Running   0          143m

Prepare S3 connection params

export S3_ENDPOINT=minio.yatai-system.svc.cluster.local
export S3_REGION=foo
export S3_BUCKET_NAME=yatai
export S3_SECURE=false
export S3_ACCESS_KEY=$(kubectl -n yatai-system get secret yatai-minio -o jsonpath='{.data.accesskey}' | base64 -d)
export S3_SECRET_KEY=$(kubectl -n yatai-system get secret yatai-minio -o jsonpath='{.data.secretkey}' | base64 -d)

Test S3 connection

kubectl -n yatai-system delete pod s3-client 2> /dev/null || true; \
kubectl run s3-client --rm --tty -i --restart='Never' \
    --namespace yatai-system \
    --env "AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY" \
    --env "AWS_SECRET_ACCESS_KEY=$S3_SECRET_KEY" \
    --image quay.io/bentoml/s3-client:0.0.1 \
    --command -- sh -c "s3-client -e http://$S3_ENDPOINT listbuckets && echo successfully"

The output should be:

Note

If the previous command reports an error that the service has not been initialized, please retry several times

successfully
pod "s3-client" deleted

4. Install Yatai#

1. Install the Yatai Helm chart#

helm upgrade --install yatai yatai \
    --repo https://bentoml.github.io/helm-charts \
    -n yatai-system \
    --set postgresql.host=$PG_HOST \
    --set postgresql.port=$PG_PORT \
    --set postgresql.user=$PG_USER \
    --set postgresql.database=$PG_DATABASE \
    --set postgresql.password=$PG_PASSWORD \
    --set postgresql.sslmode=$PG_SSLMODE \
    --set s3.endpoint=$S3_ENDPOINT \
    --set s3.region=$S3_REGION \
    --set s3.bucketName=$S3_BUCKET_NAME \
    --set s3.secure=$S3_SECURE \
    --set s3.accessKey=$S3_ACCESS_KEY \
    --set s3.secretKey=$S3_SECRET_KEY

Note

If you are using AWS S3 With IAM Role, you should add the following flags to the helm command:

--set serviceAccount.create=false \
--set serviceAccount.name=yatai

2. Verify the Yatai Installation#

kubectl -n yatai-system get pod -l app.kubernetes.io/name=yatai

The output should look like the following:

Note

Wait until the status of all pods becomes Running.

NAME                    READY   STATUS    RESTARTS   AGE
yatai-dbfbbb66f-67cq4   1/1     Running   0          45m