Join our No Code Enterprise RAG webinar

Reserve Your Spot

Tutorial

Firstly if you've followed the enterprise setup into Kubernetes you can install Airbyte using helm.

helm repo add airbyte https://airbytehq.github.io/helm-charts

And install into a namespace called airbyte.

helm install airbyte --create-namespace --namespace airbyte airbyte/airbyte

Open a port in K9s

In K9s move to the namespace airbyte to select a namespace type :ns.

You can now connect to Airbyte using from localhost http://localhost:8000

Install RabbitMQ

We use RabbitMQ as a way to communicate with Airbyte

Run the following...

echo "
apiVersion: v1
kind: Namespace
metadata:
  name: rabbitmq

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rabbitmq-deployment
  namespace: rabbitmq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      containers:
      - name: rabbitmq
        image: \"rabbitmq:3-management\"
        ports:
        - containerPort: 5672
        - containerPort: 15672
        env:
        - name: RABBITMQ_DEFAULT_USER
          value: \"admin\"
        - name: RABBITMQ_DEFAULT_PASS
          value: \"admin\"

---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-service
  namespace: rabbitmq
spec:
  selector:
    app: rabbitmq
  ports:
    - name: rabbitmq-port
      protocol: TCP
      port: 5672
      targetPort: 5672
    - name: rabbitmq-management-port
      protocol: TCP
      port: 15672
      targetPort: 15672
  type: ClusterIP
" | kubectl apply -f -

Create a rabbit MQ queue.

Go to the rabbitmq namespace in K9s and hit shift+f and open a port into the RabbitMQ admin interface which runs on port 15672.

You can then access the admin interface via http://localhost:15672.

  1. Create a queue called bionic-pipeline
  2. Create a binding to the queue from amq.topic to *.bionic-pipeline

Testing the Queue (Optional)

curl -i -u admin:admin -H "Content-Type: application/json" -X POST -d '{"properties":{},"routing_key":"123456.bionic-pipeline","payload":"Your_Message_Content","payload_encoding":"string"}' http://localhost:15672/api/exchanges/%2F/amq.topic/publish

If the message was successfully routed you should see

{"routed":true}

Create an Airbyte -> RabbitMQ destination

  1. From the Airbyte UI create a destination where the routing key is API_KEY.bionic-pipeline where API_KEY is the key we created in the bionic user interface for a document pipeline.
  2. The destination should be set to rabbitmq-service.rabbitmq.svc.cluster.local.
  3. Switch off SSL
  4. The username is admin
  5. The password is admin
  6. Exchange is amq.topic

Github connection

  1. Get an access token from Github
  2. Create a Github source and add the access token.

Install Bionic RabbitMQ Listener

echo '
apiVersion: batch/v1
kind: CronJob
metadata:
  name: rabbitmq-cronjob
  namespace: bionic-gpt
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: rabbitmq-container
            image: ghcr.io/bionic-gpt/bionicgpt-rabbitmq:latest
            env:
              - name: RABBITMQ_URL
                value: "http://rabbitmq-service.rabbitmq.svc.cluster.local:15672/api/queues/%2f/bionic-pipeline/get"
              - name: USERNAME
                value: "admin"
              - name: PASSWORD
                value: "admin"
              - name: UPLOAD_URL
                value: "http://bionic-gpt.bionic-gpt.svc.cluster.local:7903/v1/document_upload"
          restartPolicy: Never
' | kubectl apply -f -