In one of my previous posts I described how to create a Docker container serving as a builder machine for ADF applications. Here I am going to show how to use this container as a function on Fn platform.
First of all let's update the container so that it meets requirements of a function, meaning that it can be invoked as a runnable binary accepting some arguments. In an empty folder I have created a Dockerfile (just a simple text file with this name) with the following content:
This file contains instructions for Docker on how to create a new Docker image out of existing one (efedorenko/adfbuilder from the previous post) and specifies an entry point, so that a container knows what to do once it has been initiated by the Docker run command. In this case whenever we run a container it executes Maven package goal for the pom file with the name fetched from stdin. This is important as Fn platform uses stdin/stdout for functions input/output as a standard approach.
In the same folder let's execute a command to build a new Docker image (fn_adfbuilder) out of our Docker file:
Now, if we run the container passing pom file name through stdin like this:
The container will execute inside itself what we actually need:
mvn package -DoracleHome=/opt/Oracle_Home -f /opt/MySampleApp/pom.xml
Basically, having done that, we got a container acting as a function. It builds an application for the given pom file.
Let's use this function in Fn platform. The installation of Fn on your local machine is as easy as invoking a single command and described on GitHub Fn project page. Once Fn is installed we can specify Docker registry where we store images of our functions-containers and start Fn server:
export FN_REGISTRY=efedorenko
fn start
The next step is to create an Fn application which is going to use our awesome function:
fn apps create adfbuilderapp
For this newly created app we have to specify a route to our function-confiner, so that the application knows when and how to invoke it:
We have created a route saying that whenever /build resource is requested for adfbuilderapp, Fn platform should create a new Docker container basing on the latest version of fn_adfbuilder image from efedorenko repository and run it granting with 1GB of memory and passing arguments to stdin (the default mode). Furthermore, since the building is a time/resource consuming job, we're going to invoke the function in async mode with an hour timeout. Having the route created we are able to invoke the function with Fn Cli:
In both cases the platform will put the call in a queue (since it is async) and return the call id:
The function is working now and we can check how it is going in a number of different ways. Since function invocation is just creating and running a Docker container, we can see it by getting a list of all running containers:
68940f3f0136 fnproject/fnserver 27 hours ago Up 27 hours fnserver
Fn has created a new container and used function call id as its name. We can attach our stdin/stdout to the container and see what is happening inside:
docker attach 01C5EJSJC847WK400000000000
Once the function has executed we can use Fn Rest API (or Fn Cli) to request information about the call:
http://localhost:8080/v1/apps/adfbuilderapp/calls/01C5EJSJC847WK400000000000
{"message":"Successfully loaded call","call":{"id":"01C5EJSJC847WK400000000000","status":"success","app_name":"adfbuilderapp","path":"/build","completed_at":"2018-02-03T19:52:33.204Z","created_at":"2018-02-03T19:46:56.071Z","started_at":"2018-02-03T19:46:57.050Z","stats":[{"timestamp":"2018-02-03T19:46:58.189Z","metrics":
We can also monitor function calls in a fancy way by using Fn UI dashboard:
The result of our work is a function that builds ADF applications. The beauty of it is that the consumer of the function, the caller, just uses Rest API over http to get the application built and the caller does not care how and where this job will be done. But the caller knows for sure that computing resources will be utilized no longer than it is needed to get the job done.
Next time we'll try to orchestrate the function in Fn Flow.
That's it!
First of all let's update the container so that it meets requirements of a function, meaning that it can be invoked as a runnable binary accepting some arguments. In an empty folder I have created a Dockerfile (just a simple text file with this name) with the following content:
FROM efedorenko/adfbuilder
ENTRYPOINT ["xargs","mvn","package","-DoracleHome=/opt/Oracle_Home","-f"]
In the same folder let's execute a command to build a new Docker image (fn_adfbuilder) out of our Docker file:
docker build -t efedorenko/fn_adfbuilder .
Now, if we run the container passing pom file name through stdin like this:
echo -n "/opt/MySampleApp/pom.xml" | docker run -i --rm efedorenko/fn_adfbuilder
The container will execute inside itself what we actually need:
mvn package -DoracleHome=/opt/Oracle_Home -f /opt/MySampleApp/pom.xml
Basically, having done that, we got a container acting as a function. It builds an application for the given pom file.
Let's use this function in Fn platform. The installation of Fn on your local machine is as easy as invoking a single command and described on GitHub Fn project page. Once Fn is installed we can specify Docker registry where we store images of our functions-containers and start Fn server:
export FN_REGISTRY=efedorenko
fn start
The next step is to create an Fn application which is going to use our awesome function:
fn apps create adfbuilderapp
For this newly created app we have to specify a route to our function-confiner, so that the application knows when and how to invoke it:
fn routes create --memory 1024 --timeout 3600 --type async adfbuilderapp /build efedorenko/fn_adfbuilder:latest
We have created a route saying that whenever /build resource is requested for adfbuilderapp, Fn platform should create a new Docker container basing on the latest version of fn_adfbuilder image from efedorenko repository and run it granting with 1GB of memory and passing arguments to stdin (the default mode). Furthermore, since the building is a time/resource consuming job, we're going to invoke the function in async mode with an hour timeout. Having the route created we are able to invoke the function with Fn Cli:
echo -n "/opt/MySampleApp/pom.xml" | fn call adfbuilderapp /build
or over http:
curl -d "/opt/MySampleApp/pom.xml" http://localhost:8080/r/adfbuilderapp/build
{"call_id":"01C5EJSJC847WK400000000000"}
docker ps
CONTAINER ID IMAGE CREATED STATUS NAMES
6e69a067b714 efedorenko/fn_adfbuilder:latest 3 seconds ago Up 2 seconds 01C5EJSJC847WK400000000000
e957cc54b638 fnproject/ui 21 hours ago Up 21 hours clever_turing68940f3f0136 fnproject/fnserver 27 hours ago Up 27 hours fnserver
Fn has created a new container and used function call id as its name. We can attach our stdin/stdout to the container and see what is happening inside:
docker attach 01C5EJSJC847WK400000000000
Once the function has executed we can use Fn Rest API (or Fn Cli) to request information about the call:
http://localhost:8080/v1/apps/adfbuilderapp/calls/01C5EJSJC847WK400000000000
{"message":"Successfully loaded call","call":{"id":"01C5EJSJC847WK400000000000","status":"success","app_name":"adfbuilderapp","path":"/build","completed_at":"2018-02-03T19:52:33.204Z","created_at":"2018-02-03T19:46:56.071Z","started_at":"2018-02-03T19:46:57.050Z","stats":[{"timestamp":"2018-02-03T19:46:58.189Z","metrics":
....
http://localhost:8080/v1/apps/adfbuilderapp/calls/01C5EJSJC847WK400000000000/log
{"message":"Successfully loaded log","log":{"call_id":"01C5EKA5Y747WK600000000000","log":"[INFO] Scanning for projects...\n[INFO] ------------------------------------------------------------------------\n[INFO] Reactor Build Order:\n[INFO] \n[INFO] Model\n[INFO] ViewController\n[INFO]
....
We can also monitor function calls in a fancy way by using Fn UI dashboard:
The result of our work is a function that builds ADF applications. The beauty of it is that the consumer of the function, the caller, just uses Rest API over http to get the application built and the caller does not care how and where this job will be done. But the caller knows for sure that computing resources will be utilized no longer than it is needed to get the job done.
Next time we'll try to orchestrate the function in Fn Flow.
That's it!