30 Nov 2018

Conversational UI with Oracle Digital Assistant and Fn Project

Here and there we see numerous predictions that pretty soon chatbots will play a key role in the communication between the users and their systems. I don't have a crystal ball and I don't want to wait for this "pretty soon", so I decided to make these prophecies come true now and see what it looks like.

A flagman product of the company I am working for is FlexDeploy which is a fully automated DevOps solutions. One of the most popular activities in FlexDeploy is creating a release snapshot that actually builds all deployable artifacts and deploys them across environments with a pipeline.
So, I decided to have some fun over the weekend and implemented a conversational UI for this operation where I am able to talk to FlexDeploy. Literally. At the end of my work my family saw me talking to my laptop and they could hear something like that:

  "Calypso!" I said.
  "Hi, how can I help you?" was the answer.
  "Not sure" I tested her.
  "You gotta be kidding me!" she got it.
  "Can you build a snapshot?" I asked.
  "Sure, what release are you thinking of?"
  "Created a snapshot for release 1001" she reported.
  "Thank you" 
  "Have a nice day" she said with relief.

So,  basically, I was going to implement the following diagram:

As a core component of my UI I used a brand new Oracle product Oracle Digital Assistant. I built a new skill capable of basic chatting and implemented a new custom component so my bot was able to invoke an http request to have the backend system create a snapshot.  The export of the skill FlexDeployBot along with Node.js source code of the custom component custombotcomponent is available on GitHub repo for this post.
I used my MacBook as a communication device capable of listening and speaking and I defined a webhook  channel for my bot so I can send messages to it and get callbacks with responses.

It looks simple and nice on the diagram above. The only thing is that I wanted to decouple the brain, my chatbot, from the details of the communication device and from the details of the installation/version of my back-end system FlexDeploy. I needed an intermediate API layer, a buffer, something to put between ODA and the outer world. It looks like Serverless Functions is a perfect fit for this job.

As a serverless platform I used Fn Project. The beauty of it is that it's a container-native serverless platform, totally based on Docker containers and it can be easily run locally on my laptop (what I did for this post) or somewhere in the cloud, let's say on Oracle Kubernetes Engine.

Ok, let's get into the implementation details from left to right of the diagram.

So, the listener component, the ears, the one which recognizes my speech and converts it into text is implemented with Python:

The key code snippet of the component look like this (the full source code is available on GitHub):
r = sr.Recognizer()
mic = sr.Microphone()

with mic as source:
    r.energy_threshold = 2000

while True:  
        with mic as source: 
            audio = r.listen(source, phrase_time_limit=5)           
            transcript = r.recognize_google(audio)
            if active:
                requests.post(url = URL, data = transcript)
    except sr.UnknownValueError:
        print("Sorry, I don't understand you")

Why Python? There are plenty of available speech recognition libraries for Python, so you can play with them and choose the one which understands your accent better. I like Python.
So, once the listener recognizes my speech it invokes an Fn function passing the phrase as a request body.
The function sendToBotFn is implemented with Node.js:
function buildSignatureHeader(buf, channelSecretKey) {
    return 'sha256=' + buildSignature(buf, channelSecretKey);

function buildSignature(buf, channelSecretKey) {
   const hmac = crypto.createHmac('sha256', Buffer.from(channelSecretKey, 'utf8'));
   return hmac.digest('hex');

function performRequest(headers, data) {
  var dataString = JSON.stringify(data);
  var options = {
   body: dataString,   
   headers: headers
  request('POST', host+endpoint, options);             

function sendMessage(message) {
  let messagePayload = {
   type: 'text',
   text: message

  let messageToBot = {
    userId: userId,
    messagePayload: messagePayload

  let body = Buffer.from(JSON.stringify(messageToBot), 'utf8');
  let headers = {};
  headers['Content-Type'] = 'application/json; charset=utf-8';
  headers['X-Hub-Signature'] = buildSignatureHeader(body, channelKey);

  performRequest(headers, messageToBot);  

  return input; 

Why Node.js? It's not because I like it. No. It's because Oracle documentation on implementing a custom web hook channel is referring to Node.js. They like it.

When the chatbot is responding it is invoking a webhook referring to an Fn function receiveFromBotFn running on my laptop.  I use ngrok tunnel to expose my Fn application listening to localhost:8080 to the Internet. The receiveFromBotFn function is also implemented with Node.js:
const fdk=require('@fnproject/fdk');
const request = require('sync-request');
const url = 'http://localhost:4390';
    var sayItCall = request('POST', url,{
     body: input.messagePayload.text,
  return input;
The function sends an http request to a simple web server running locally and listening to 4390 port.
I have to admit that it's really easy to implement stuff like that with Node.js. The web server uses Mac OS X native utility say to pronounce whatever comes in the request body:
var http = require('http');
const exec = require("child_process").exec
const request = require('sync-request');

http.createServer(function (req, res) {
      let body = '';
      req.on('data', chunk => {
          body += chunk.toString();

      req.on('end', () => {       
          exec('say '+body, (error, stdout, stderr) => {


In order to actually invoke the back-end to create a snapshot with FlexDeploy the chatbot invokes with the custombotcomponent an Fn function createSnapshotFn:
var res=request('POST',fd_url+'/flexdeploy/rest/v1/releases/'+input+'/snapshot',  {
      json: {action : 'createSnapshot'},

  return JSON.parse(res.getBody('utf8'));

The function is simple, it just invokes FlexDeploy REST API to start building a snapshot for the given release. It is also implemented with Node.js, however I am going to rewrite it with Java. I love Java. Furthermore, instead of a simple function I am going to implement an Fn Flow that first checks if the given release exists and if it is valid and only after that it invokes the createSnapshotFn function for that release. In the next post.

That's it!

24 Nov 2018

Persistent Volumes for Database Containers running on a K8s cluster in the Cloud

In one of my previous posts I showed how we can run Oracle XE database on a K8s cluster. That approach works fine for the use-cases when we don't care about the data and we are fine with loosing it when the container is redeployed and the pod is restarted. But if we want to keep the data, if we want it to survive all rescheduling we'll want to reconsider K8s resources used to run the DB container on the cluster. That said, the yaml file defining the resources looks like this one:

apiVersion: apps/v1beta2
kind: StatefulSet
  name: oraclexe
    run: oraclexe
        run: oraclexe
  serviceName: "oraclexe-svc"
  replicas: 1
        run: oraclexe
       - name: dshm
           medium: Memory  
      - image: eugeneflexagon/database:
           - mountPath: /dev/shm
             name: dshm
           - mountPath: /u01/app/oracle/oradata
             name: db
        imagePullPolicy: Always
        name: oraclexe
        - containerPort: 1521
          protocol: TCP
   - metadata:
       name: db
       accessModes: [ "ReadWriteOnce" ]
           storage: 100M                   
apiVersion: v1
kind: Service
  name: oraclexe-svc
    run: oraclexe   
    run: oraclexe
    - port: 1521
      targetPort: 1521
  type: LoadBalancer

There are some interesting things here. First of all this is not a deployment. We are defining here another K8s resource which is called Stateful Set. Unlike a Deployment, a Stateful Set maintains a sticky identity for each of their Pods. These pods are created from the same specification, but they are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

This guy has been specially designed for stateful applications like database that save their data to a persistent storage. In order to define a persistent storage for our database we use a special K8s resource Persistent Volume and here in the yaml file we are defining a claim to create a 100mb Persistent Volume with name db. This volume provides read/write access mode for one assigned pod. The volume is called persistent because its lifespan is not maintained by a container and not even by a pod, it’s maintained by a K8s cluster. So it can outlive any containers and pods and save the data. We are referring to this persistence volume in the container definition mounting a volume on path /u01/app/oracle/oradata. This is where Oracle DB XE container stores its data.

That's it!