Page tree
Skip to end of metadata
Go to start of metadata


This post explains about how to build different BigData packages like Flink, Memcached, Kafka, Oozie, Kafka Streams, and Cassandra.  You need to install Maven v3.3.9 and Open JDK8u

About BigData components

There are many BigData components individually

Build and Test Instructions

Install OpenJDK 8


$ wget http://openjdk.linaro.org/releases/jdk8u-server-release-1708.tar.xz 
$ tar xvf jdk8u-server-release-1708.tar.xz 
$ cd jdk8u-server-release-1708 
$ export JAVA_HOME=$PWD 
$ cd jre/lib/security/ 
$ rm cacerts 

(for CentOS 7) $ ln --symbolic /etc/pki/java/cacerts . 
(for Debian Jessie) $ ln --symbolic /etc/ssl/certs/java/cacerts .  


$ cd jdk8u-server-release-1708/bin 
$ export PATH=$PWD:$PATH  

Install Maven v3.3.9

Download and export the Maven v3.3.9 


$ wget http://www.us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
$ tar -xvf apache-maven-3.3.9-bin.tar.gz
$ cd apache-maven-3.3.9/bin 
$ export PATH=$PWD:$PATH  

The flink package build instructions are LEG-250

$ git clone https://github.com/apache/flink.git
$ cd flink
$ mvn clean package -DskipTests  

Kafka

The kafka package build instructions are LEG-252 You need to install "gradle" before executing any other instructions.  The gradle install instructions are purely based on your distribution.

$ git clone https://github.com/apache/kafka.git
$ cd kafka
$ ./gradlew jar 

Oozie

The oozie package build instructions are LEG-276

$ git clone https://github.com/apache/oozie.git
$ cd oozie
$ mvn clean package -DskipTests

$ cd minitest
$ mvn clean test 

Nifi

The nifi package build instructions are LEG-677

$ git clone https://github.com/apache/nifi.git
$ cd nifi
$ mvn clean package -DskipTests  

Testing:
-------

nareshbhat@debian:~/nifi$ cd nifi-assembly/

nareshbhat@debian:~/nifi/nifi-assembly$ ls -lhd target/
drwxr-xr-x 5 nareshbhat nareshbhat 4.0K Feb 16 15:37 target/

nareshbhat@debian:~/nifi/nifi-assembly$ ls -lhd target/nifi*
drwxr-xr-x 3 nareshbhat nareshbhat 4.0K Feb 16 15:36 target/nifi-1.6.0-SNAPSHOT-bin
rw-rr- 1 nareshbhat nareshbhat 1.2G Feb 16 15:38 target/nifi-1.6.0-SNAPSHOT-bin.tar.gz
rw-rr- 1 nareshbhat nareshbhat 1.2G Feb 16 15:37 target/nifi-1.6.0-SNAPSHOT-bin.zip

nareshbhat@debian:~/nifi/nifi-assembly$ mkdir ~/example-nifi-deploy

nareshbhat@debian:~/nifi/nifi-assembly$ tar xzf target/nifi-*-bin.tar.gz -C ~/example-nifi-deploy

nareshbhat@debian:~/nifi/nifi-assembly$ ls -lh ~/example-nifi-deploy/
total 4.0K 
drwxr-xr-x 6 nareshbhat nareshbhat 4.0K Feb 16 15:49 nifi-1.6.0-SNAPSHOT

nareshbhat@debian:~/nifi/nifi-assembly$ cd ~/example-nifi-deploy/nifi-*

nareshbhat@debian:~/example-nifi-deploy/nifi-1.6.0-SNAPSHOT$ ./bin/nifi.sh start
Java home: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708
NiFi home: /home/nareshbhat/example-nifi-deploy/nifi-1.6.0-SNAPSHOT
Bootstrap Config File: /home/nareshbhat/example-nifi-deploy/nifi-1.6.0-SNAPSHOT/conf/bootstrap.conf
nareshbhat@debian:~/example-nifi-deploy/nifi-1.6.0-SNAPSHOT$

I could able to see the NiFi screen shot using web browser (Attached).  I did't use the https://nifi.apache.org/docs/nifi-docs/html/getting-started.html guide to very extended level.  But I just make sure that the basic things works fine.

nareshbhat@debian:~/example-nifi-deploy/nifi-1.6.0-SNAPSHOT$ ./bin/nifi.sh stop
Java home: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708
NiFi home: /home/nareshbhat/example-nifi-deploy/nifi-1.6.0-SNAPSHOT
Bootstrap Config File: /home/nareshbhat/example-nifi-deploy/nifi-1.6.0-SNAPSHOT/conf/bootstrap.conf
2018-02-16 16:02:54,540 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi has accepted the Shutdown Command and is shutting down now
2018-02-16 16:02:54,575 INFO [main] org.apache.nifi.bootstrap.Command Waiting for Apache NiFi to finish shutting down...
2018-02-16 16:02:56,602 INFO [main] org.apache.nifi.bootstrap.Command NiFi has finished shutting down.
nareshbhat@debian:~/example-nifi-deploy/nifi-1.6.0-SNAPSHOT$

Mini NiFi

The mini-nifi package build instructions are in Jira issue

$ git clone https://github.com/apache/nifi-minifi.git
$ cd nifi-minifi/
$ mvn clean package -DskipTests
nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi$ cd minifi-assembly
nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ ls -lhd target/minifi*
drwxr-xr-x 3 nareshbhat nareshbhat 4.0K Feb 27 11:27 target/minifi-0.5.0-SNAPSHOT-bin
-rw-r--r-- 1 nareshbhat nareshbhat  65M Feb 27 11:27 target/minifi-0.5.0-SNAPSHOT-bin.tar.gz
-rw-r--r-- 1 nareshbhat nareshbhat  65M Feb 27 11:27 target/minifi-0.5.0-SNAPSHOT-bin.zip
-rw-r--r-- 1 nareshbhat nareshbhat 112K Feb 27 11:26 target/minifi.exe
-rw-r--r-- 1 nareshbhat nareshbhat 116K Feb 27 11:26 target/minifiw.exe
nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ mkdir ~/example-minifi-deploy
nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ tar xzf target/minifi-*-bin.tar.gz -C ~/example-minifi-deploy/
nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ ls -lh ~/example-minifi-deploy/
total 4.0K
drwxr-xr-x 6 nareshbhat nareshbhat 4.0K Feb 27 11:29 minifi-0.5.0-SNAPSHOT
nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ cd ~/example-minifi-deploy/minifi-*
nareshbhat@debian:~/example-minifi-deploy/minifi-0.5.0-SNAPSHOT$ ./bin/minifi.sh start

Bootstrap Classpath: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/conf:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/lib/bootstrap/*:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/lib/*
Java home: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708
MiNiFi home: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT

Bootstrap Config File: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/conf/bootstrap.conf

2018-02-27 11:29:49,626 INFO [main] org.apache.nifi.minifi.bootstrap.Command Starting Apache MiNiFi...
2018-02-27 11:29:49,629 INFO [main] org.apache.nifi.minifi.bootstrap.Command Working Directory: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT
2018-02-27 11:29:49,631 INFO [main] org.apache.nifi.minifi.bootstrap.Command Command: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708/bin/java -classpath /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./conf:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-servlets-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/zookeeper-3.4.6.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-schema-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/osgi-resource-locator-1.0.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/quartz-2.2.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/httpcore-nio-4.4.5.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-client-dto-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-webapp-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/hk2-locator-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-server-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/httpclient-4.5.3.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.servlet.jsp-api-2.3.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-authorizer-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-api-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/minifi-nar-utils-0.5.0-SNAPSHOT.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/ecj-4.4.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-data-provenance-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-nar-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/minifi-framework-core-0.5.0-SNAPSHOT.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-annotations-2.6.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-runtime-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-schemas-3.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/guava-18.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-write-ahead-log-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-xml-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.el-3.0.1-b09.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-codec-1.10.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-collections4-4.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-mapper-asl-1.9.13.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-entity-filtering-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/httpasyncclient-4.1.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-hk2-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-repository-models-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.servlet.jsp.jstl-api-1.2.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-site-to-site-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/curator-client-2.11.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-api-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-core-2.6.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-cluster-protocol-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-registry-security-utils-0.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jul-to-slf4j-1.7.25.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-registry-data-model-0.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/curator-recipes-2.11.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/lucene-analyzers-common-4.10.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-socket-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/httpcore-4.4.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-common-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-registry-client-0.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/logback-core-1.2.3.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-registry-flow-diff-0.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/bcprov-jdk15on-1.55.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-user-actions-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/lucene-core-4.10.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-administration-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.ws.rs-api-2.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jline-0.9.94.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-databind-2.6.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/validation-api-2.0.0.Final.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-security-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/asm-1.0.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.inject-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-util-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-security-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-authorization-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-core-api-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/apache-el-8.0.33.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-properties-loader-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-continuation-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/minifi-utils-0.5.0-SNAPSHOT.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-io-2.5.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-io-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-logging-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-core-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/antlr-runtime-3.5.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-media-json-jackson-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/hk2-api-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-client-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/bcpkix-jdk15on-1.55.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jcl-over-slf4j-1.7.25.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-site-to-site-client-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-properties-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.inject-1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/logback-classic-1.2.3.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-web-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.servlet.jsp-2.3.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-jsp-jdt-2.3.3.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-servlet-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.servlet-api-3.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-persistent-provenance-repository-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-module-jaxb-annotations-2.8.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-flowfile-repo-serialization-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.annotation-api-1.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/spring-core-4.2.4.RELEASE.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/aopalliance-repackaged-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/asm-3.3.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/curator-framework-2.11.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/apache-jsp-9.3.9.v20160517.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javassist-3.22.0-CR2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/slf4j-api-1.7.25.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-lang3-3.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/json-smart-2.1.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/json-path-2.0.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/minifi-runtime-0.5.0-SNAPSHOT.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/apache-jsp-8.0.33.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.el-api-3.0.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-expression-language-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-logging-1.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-security-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/hk2-utils-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/netty-3.7.0.Final.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-http-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-core-asl-1.9.13.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx256m -Xms256m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.security.egd=file:/dev/urandom -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./conf/nifi.properties -Dnifi.bootstrap.listen.port=35271 -Dapp=MiNiFi -Dorg.apache.nifi.minifi.bootstrap.config.log.dir=/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/logs org.apache.nifi.minifi.MiNiFi 


nareshbhat@debian:~/example-minifi-deploy/minifi-0.5.0-SNAPSHOT$ cd ~/example-minifi-deploy/minifi-*
nareshbhat@debian:~/example-minifi-deploy/minifi-0.5.0-SNAPSHOT$ ./bin/minifi.sh stop

Bootstrap Classpath: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/conf:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/lib/bootstrap/*:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/lib/*
Java home: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708
MiNiFi home: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT

Bootstrap Config File: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/conf/bootstrap.conf

2018-02-27 11:30:57,260 INFO [main] org.apache.nifi.minifi.bootstrap.Command Apache MiNiFi has accepted the Shutdown Command and is shutting down now
2018-02-27 11:30:57,368 INFO [main] org.apache.nifi.minifi.bootstrap.Command MiNiFi has finished shutting down.

nareshbhat@debian:~/example-minifi-deploy/minifi-0.5.0-SNAPSHOT$

Memcached

The memcached package build instructions are LEG-686

Install dependency package

$ sudo apt-get install libevent-dev
$ sudo apt-get update
$ sudo apt-get install mysql-server php5-mysql php5 php5-memcached

Clone and build memcached

$ git clone https://github.com/memcached/memcached.git
$ cd memcached
$ ./autogen.sh
$ ./configure
$ make all

You can run the testapp

$ ./testapp

You can check memcached version and run 

$ ./memcached -V
memcached 1.5.4

I have started the memcached service on on console with below command

nareshbhat@debian:~/memcached$ memcached start

and test it on another console

nareshbhat@debian:~$ sudo netstat -tupan | grep 11211
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 20423/memcached 
tcp6 0 0 :::11211 :::* LISTEN 20423/memcached 
udp 0 0 0.0.0.0:11211 0.0.0.0:* 20423/memcached 
udp6 0 0 :::11211 :::* 20423/memcached
nareshbhat@debian:~$

To verify memcached is recognized by the web server:

1. Create a phpinfo.php file in the web server’s docroot:

$ sudo nano /var/www/html/phpinfo.php

=========phpinfo.php start============
<?php
// Show all information, defaults to INFO_ALL
phpinfo();
=========phpinfo.php end============

2. Go to that page in your web browser.
For example, http://<IP-address>/phpinfo.php

3. Make sure memcached displays all the information on web browser

I have also executed database_test.php, cache_test.php

Cassandra

The cassandra package build instructions are LEG-695 You have to install ant tool to build cassandra.

$ git clone https://github.com/apache/cassandra.git
$ cd cassandra
$ git tag
$ git checkout cassandra-3.11.1
$ ant clean
$ ant

After successful build you can run the cassandra using command

$ ./bin/cassandra -f

On another terminal I run the below example

$ bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.1-SNAPSHOT | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> CREATE SCHEMA schema1
   ... WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
cqlsh> USE schema1;
cqlsh:schema1> CREATE TABLE users (
           ...                  user_id varchar PRIMARY KEY,
           ...                  first varchar,
           ...                  last varchar,
           ...                  age int
           ...                );
cqlsh:schema1> INSERT INTO users (user_id, first, last, age)
           ...                VALUES ('jsmith', 'John', 'Smith', 42);
cqlsh:schema1> SELECT * FROM users;

 user_id | age | first | last
---------+-----+-------+-------
  jsmith |  42 |  John | Smith

(1 rows)
cqlsh:schema1> 

Difference between NiFi and Mini-NiFi

Apache NiFi is a robust and secure framework for routing, transforming, and delivering data across a multitude of systems. NiFi can run in parallel with other applications, but it performs best when the entire system (or multiple systems in a cluster) are dedicated to it. It often uses SAN or RAID storage at the TB level for the massive amounts of content it ingests and the provenance it generates. The UI allows multiple users to quickly modify flows simultaneously on the same machine or across a cluster. The latest release candidate of NiFi (1.1.0 RC1) includes over 170 processors for custom integration with various systems and operations, and is 762 MB when compressed for download. In other words, NiFi is a server-class application.

Apache MiNiFi was developed out of a recognized need to bring the capabilities of NiFi to the "edge" as "agents" -- accessing data from IoT and desktop-level devices, and applying primary features of NiFi at the earliest possible stage. Now data can be collected from a variety of protocols, have data provenance generated immediately for more holistic governance and transparency, have light transformations applied at source, be encrypted, be prioritized, and be redundantly routed back to the more powerful transformations done in the cloud or data center.

Now, all of these behaviors can be performed with custom scripts, but then the problem of command & control (C2) is encountered. With hundreds, thousands, or even millions of these devices existing, how can each be monitored and exfilled, and what happens when the flow needs to change? It could be to report back to a new endpoint, to update the frequency at which it is collected or transmitted, or to handle new metrics or metadata from the device. This manual process does not scale. With MiNiFi's integration with NiFi, a flow can be developed using the UI in NiFi and transparently translated to a MiNiFi flow and pushed out to classes of agents across the world.

With manual modification to remove unnecessary processors and features, NiFi can be trimmed to fit on a Raspberry Pi. But it still requires the JVM, and there are plenty of devices that won't support it. MiNiFi is offered in Java and C++, and the footprint is on a completely different scale -- 39 MBfor the Java agent (tar) and 310K for the C++ agent (tar).

A great example of the power and usefulness of MiNiFi is a recent demo at the TU-Automotive Detroit exhibition, where MiNiFi was loaded onto a custom Qualcomm modem located in a "connected car". As the car drives, massive amounts of data are generated by components throughout the car and routed via the CANBUS to be processed. Some data is important to stream back to a remote processing center in real-time -- this data is transmitted via an LTE connection. LTE is widely available, but bandwidth is expensive. Meanwhile, data that was much larger but less time-relevant (system diagnostics, etc.) could be batched and compressed, and then sent in bursts over WiFi when the car was in range of a known hotspot. MiNiFi coordinated all of the flow decisions and routing via geo-enrichment and control plane feedback. Here is a short video of Joe Niemiec explaining the process and showing the flow.

You can extrapolate that demo to many other use cases. It is helpful to think of MiNiFi as a "good guest" -- a lightweight agent that runs on hardware that is probably dedicated to a different primary purpose. Whether this is IoT, a cash register/point of sale system, a car modem, physical sensors, etc., is irrelevant to MiNiFi -- its job is to process and exfil this data while not taking unnecessary resources from the primary function. Contrast this with NiFi, which again, can run simultaneously with other applications, but ideally it has dedicated resources which it can maximize for its own performance.

Reference :

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-memcache-on-ubuntu-14-04

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-memcache-on-ubuntu-14-04
http://devdocs.magento.com/guides/v2.0/config-guide/memcache/memcache_ubuntu.html
https://github.com/linux-on-ibm-z/docs/wiki/Building-Apache-Cassandra-3.11
https://github.com/apache/cassandra

Page viewed times
  • No labels