Installing Apache Nifi Step by Step in Windows

Recently have been working with Apache Nifi for one my IoT Project. I would like to write a post on how to Install Apache Nifi in Windows 7.

Pre-requisites:

  1. Java Installation
  2. Environment variable JAVA_HOME

Step 1: Download Apache Nifi 1.0 from here.

Step 2: Download the zip file the mirror site as recommended. It would be around 740 MB patiently download it.

Step 3: Extract the folder and I moved nifi-1.0.0 to a root drive for each of access through windows command console.

Step 4: Now go to the command prompt and navigate to the nifi-1.0.0 directory

Step 5: Navigate to the bin directory

Step 6: You will find run run-nifi.bat, execute run-nifi

Step 7: You will see something like the below:

Step 8: Now open your favourite browser and type localhost:9090, I have changed the port from 8080 to 9090 in nifi.properties file. If you want, you can change that as well.

Step 9: Now you can access to the menu to get to know the version of you are running on

Step 10: Happy data flows with Apache Nifi!!!

Getting started with Polymer

In my interest to understand the material design concepts I was trying to put my hands on to it and working with it. Here are my initial challenges in doing so also this post would be useful for beginners and Web designers to understand the skills required.

Best way to get started is to have to go through the youtube videos available on search on the youtube for Google I/O 2014 material design or Google Design channel.  You will be able to get very intriguing insights on why we should adopt such material design concepts.

When I try to work on it I want to hit the ground very fast I was following Polymer Getting started guide . You need to heavily depend upon two things for get things going one is nodejs and another is git.

It uses packages like Bower, Gulp, etc., for you to use the Polymer start kit which will help you start with an App with ease. Make sure you know how to setup environment variables and stuff like that. If you have sublime editor to edit the files its easy you can get the app going within an hour without much hassles.

My quick result on localhost with gulp serve:

Material Design Approach

Give a try and give your feedback. 🙂

CWRAF – Common Weakness Risk Assessment Framework

500px-Risk_Management_Framework.svg
Source: Wikipedia

Anything which needs to be managed before an event occurs termed to be Risk. When it comes to building or using software in a business scenario needs lot of assessment of risk. I came across CWRAF from a not-for-profit organization MITRE which provides a comprehensive framework to address security risk for application in the business context using Vignettes.Typical risk management framework from wikipedia is given in wikipedia.

There is also an excellent paper which refer information security in analogy with “Clean Water Act” and how CWRAF can help in the process. Its worth reading. Interestingly CWRAF framework starts from weakness where any other risk framework starts from the Threat. Its always weakness becomes a Threat so CWRAF addresses the risk from that perspective.

Also its very important to understand that there could be generic Security guidelines such as OWASP Top 10 which are followed while developing or implementing a product, but there are very high chances that those security guidelines are CWRAF Framework Stepseither not sufficient or not needed. This is to be determined based completely on the business need. This is were CWRAF using the CWSS (Common Weakness Scoring System) comes to the rescue.

The steps outlined in the below given picture will hep you to understand the steps involved in establishing risk assessment for a business need. Please note that already vignettes are available form the CWRAF website for E-Commerce, Banking, SCADA based applications which can be refined or reused.

Need for Bridge in IoT world

Goshree_bridge_kochiBridges has a got significance in our lives:

  • It connect two cities or locations
  • It handles multiple transport (Two Wheeler, Four Wheeler, Pedestrians)
  • It helps crosses hurdles such as water, traffic, sea, hills, etc.,

Who would not have crossed bridges in their lifetime. Bridges in fact saves lot of time when you you build, consume it right!!

In the future there could be various devices need to talk to each other in different protocols using approaches like OPC, STOMP,, MQTT, etc., At that juncture bridging will be very much essential to succeed in IOT Projects

What is Bridging with respect to IoT?

My own definition of Bridging means “connecting interfaces of two brokers to communicate with each other seamlessly”.

When bridging is  required?

One broker has special capabilities than the other. Broker A might be very good at handling Text based protocols and Broker B may be very good with Pub/Sub mechanism and very light weight. These brokers A and B might be required to give a optimal solution. These kind of scenarios demand Bridging.

Real-time use cases could be:

  • Devices may be using some data formats and protocols due to connectivity constraints such as latency. We can use bridge to use the transfer data to anther broker which can handle or convert to another data format.
  • To aggregate data to one central location
  • To create restrain for the client directly not connect to the Broker for security reasons.

Installing and using MQTT Lens with Mosquitto

In this post I will try to outline how to use Mqtt Lens an useful chrome extension developed by Sanndro which can help you test the installation done as per my previous post Mosquitto broker installation. MqttLens Chrome Extension

MQTT Lens is a Chrome Extension available to use along with Chrome Browser for the purpose of connecting to a MQTT Broker and test with publish/subscribe scenarios of MQTT Messages. This is a very useful tool to check the connectivity to the MQTT broker and to check various scenarios of publishing and subscribing messages. I have tested this tool myself with Mosquitto and RabbitMQ+Mqtt Plug in scenarios.

In this post we will go through step by step on how to install and configure Mqtt Lens and use it 10 simple steps.

Step 1: Make sure you have chrome browser installed and have internet connectivity.

Step 2: You can click the URL here to go to the extension or chrome app instantly alternatively you can approach the method outline in the the github page of sandro-k (This post will give notes based on the URL provided for chrome extension installation)

Step 3: Once you have the extension installed by clicking the Install or Launch App as given in this blog post

Step 4: Once you have installed it navigate to Chrome App Launcher which would probably in the Windows Task bar as shown in the picture on to your right.Chrome App Launcher

Step 5: Once you click the Mqtt Lens Icon from the chrome app launcher you will see the following screen.

MqttLens Startup screen

Step 6: Before you add the connection ensure the broker is running. For this you might need the information about the broker and make sure the broker service is already running in the Windows with port 1883. You can check this using netstat -an.

Step 7: Now you need to add the Connection. For this you might need the IP address or Hostname, in this example I have provided the IP Address which will be most important parameter and click create connection.

MqttLens add new connection

Step 8: Now once you click create connection you will be taken to the screen where in you will need to provide the publish and subscribe information to test the broker or to work with the broker.

MqttLens Publish Subscribe Example Sensor

Step 9: In the above example you could realize that the # symbol in the Subscribe input box helps you to subscribe to all the content available in the queue to the available. You can look at the data in both Json format or plain text message

Step 10: Information on the message can be obtained by clicking the information button on the right hand side of the message. Screenshot given below:

MqttLens Message Details

You can add more connections and play with Mqtt Messages. Thanks. Please share your view and comments on this post.

Task Parallel Library – Internet of Things

We want to implement a energy monitoring system with the help of C# library which has interface with an Embedded System and relevant controllers attached to it. Its essential that we communicate with these devices and understand the data trends they provide. This would help to build the intelligence through some machine learning algorithms and there by improving the intelligence of these devices.Smart Home

The challenge in doing this is to have a master controller or a service which can talk to these devices are different intervals sequentially or parallel so that it can be more realistic and efficient. While I was looking at developing this kind of app I was looking at c# to implement a multitasking to communicate with these devices at different intervals in parallel. Thanks to TPL the Task Parallel Library of .NET Framework. Before I started writing these post I read about the TPL on  Introduction to TPL , Parallel Tasks in .NET 4.0 – Dotnetcurry and Understanding Task Parallelism Framework (TPL) using C#. These are really good articles to hit the ground pretty fast.

The rationale behind looking at this option is to evaluate how we can make best use of concurrency to collect data from devices and pass it on to the Message queues like Mqtt/RabbitMQ. This scenario would be very much helpful when you have the following scenarios with respect to M2M or IoT:

  1. Devices cannot be reprogrammed with the client to develop and deploy client app to the device.
  2. You have the exposure of data through connectivity means
  3. You can poll the devices for specific parameters through Usermanual
  4. You have connected your smart devices through LAN via Wi-Fi or Ethernet
  5. Communication can be done to the devices via standard protocols

Why TPL?

  • Need to interact with the devices concurrently
  • The topics needs to be monitored and can invoke parallel tasks based on the nature of message
  • The parallel tasks could be like sending Alerts, Notifications, Writing logs, etc.,
  • Tasks should all be done in an async manner

In the another post, I will try and see how we can accomplish this with the help of TPL.

Steps in reading a modbus register

Modbus.org has most of the information with respect to modbus. Modbus protocol has master/slave communication model. Master sends the queries to the slave to get a response. Masters have the capability to talk to individual slave or it can broadcast messages to all the slaves. Slave could be a Meter, Temperature reading device, etc.,  Modbus

It supports two modes of communication one is ASCII and another is RTU mode. Modbus ASCII vs Modbus RTU to know the differences between ASCII and RTU Mode. Any communication to modbus slave for reading the registers should have the following components with it.

  1. Slave Id
  2. Function Code
  3. Data address of the first register to read
  4. Total number of register to be read or length to be read
  5. CRC check

Typically holding registers will hold the data which will be available in the register value between 40001-49999.

Request:

Lets take an example of reading a value from the register 110. Slave Id could be 1. The function code would be 3 for reading the holding register. So the offset value added would be with 40001+110 which would be 40111.   Assuming we might need to  2 registers as the length. Maximum slave Id could be 247.

Important thing to understand while custom programming modbus devices is to ensure that modbus maps from the vendors are thoroughly read so that we can avoid overlaps.