Installing Apache Nifi Step by Step in Windows

Recently have been working with Apache Nifi for one my IoT Project. I would like to write a post on how to Install Apache Nifi in Windows 7.

Pre-requisites:

  1. Java Installation
  2. Environment variable JAVA_HOME

Step 1: Download Apache Nifi 1.0 from here.

Step 2: Download the zip file the mirror site as recommended. It would be around 740 MB patiently download it.

Step 3: Extract the folder and I moved nifi-1.0.0 to a root drive for each of access through windows command console.

Step 4: Now go to the command prompt and navigate to the nifi-1.0.0 directory

Step 5: Navigate to the bin directory

Step 6: You will find run run-nifi.bat, execute run-nifi

Step 7: You will see something like the below:

Step 8: Now open your favourite browser and type localhost:9090, I have changed the port from 8080 to 9090 in nifi.properties file. If you want, you can change that as well.

Step 9: Now you can access to the menu to get to know the version of you are running on

Step 10: Happy data flows with Apache Nifi!!!

Need for Bridge in IoT world

Goshree_bridge_kochiBridges has a got significance in our lives:

  • It connect two cities or locations
  • It handles multiple transport (Two Wheeler, Four Wheeler, Pedestrians)
  • It helps crosses hurdles such as water, traffic, sea, hills, etc.,

Who would not have crossed bridges in their lifetime. Bridges in fact saves lot of time when you you build, consume it right!!

In the future there could be various devices need to talk to each other in different protocols using approaches like OPC, STOMP,, MQTT, etc., At that juncture bridging will be very much essential to succeed in IOT Projects

What is Bridging with respect to IoT?

My own definition of Bridging means “connecting interfaces of two brokers to communicate with each other seamlessly”.

When bridging is  required?

One broker has special capabilities than the other. Broker A might be very good at handling Text based protocols and Broker B may be very good with Pub/Sub mechanism and very light weight. These brokers A and B might be required to give a optimal solution. These kind of scenarios demand Bridging.

Real-time use cases could be:

  • Devices may be using some data formats and protocols due to connectivity constraints such as latency. We can use bridge to use the transfer data to anther broker which can handle or convert to another data format.
  • To aggregate data to one central location
  • To create restrain for the client directly not connect to the Broker for security reasons.

Installing and using MQTT Lens with Mosquitto

In this post I will try to outline how to use Mqtt Lens an useful chrome extension developed by Sanndro which can help you test the installation done as per my previous post Mosquitto broker installation. MqttLens Chrome Extension

MQTT Lens is a Chrome Extension available to use along with Chrome Browser for the purpose of connecting to a MQTT Broker and test with publish/subscribe scenarios of MQTT Messages. This is a very useful tool to check the connectivity to the MQTT broker and to check various scenarios of publishing and subscribing messages. I have tested this tool myself with Mosquitto and RabbitMQ+Mqtt Plug in scenarios.

In this post we will go through step by step on how to install and configure Mqtt Lens and use it 10 simple steps.

Step 1: Make sure you have chrome browser installed and have internet connectivity.

Step 2: You can click the URL here to go to the extension or chrome app instantly alternatively you can approach the method outline in the the github page of sandro-k (This post will give notes based on the URL provided for chrome extension installation)

Step 3: Once you have the extension installed by clicking the Install or Launch App as given in this blog post

Step 4: Once you have installed it navigate to Chrome App Launcher which would probably in the Windows Task bar as shown in the picture on to your right.Chrome App Launcher

Step 5: Once you click the Mqtt Lens Icon from the chrome app launcher you will see the following screen.

MqttLens Startup screen

Step 6: Before you add the connection ensure the broker is running. For this you might need the information about the broker and make sure the broker service is already running in the Windows with port 1883. You can check this using netstat -an.

Step 7: Now you need to add the Connection. For this you might need the IP address or Hostname, in this example I have provided the IP Address which will be most important parameter and click create connection.

MqttLens add new connection

Step 8: Now once you click create connection you will be taken to the screen where in you will need to provide the publish and subscribe information to test the broker or to work with the broker.

MqttLens Publish Subscribe Example Sensor

Step 9: In the above example you could realize that the # symbol in the Subscribe input box helps you to subscribe to all the content available in the queue to the available. You can look at the data in both Json format or plain text message

Step 10: Information on the message can be obtained by clicking the information button on the right hand side of the message. Screenshot given below:

MqttLens Message Details

You can add more connections and play with Mqtt Messages. Thanks. Please share your view and comments on this post.

Task Parallel Library – Internet of Things

We want to implement a energy monitoring system with the help of C# library which has interface with an Embedded System and relevant controllers attached to it. Its essential that we communicate with these devices and understand the data trends they provide. This would help to build the intelligence through some machine learning algorithms and there by improving the intelligence of these devices.Smart Home

The challenge in doing this is to have a master controller or a service which can talk to these devices are different intervals sequentially or parallel so that it can be more realistic and efficient. While I was looking at developing this kind of app I was looking at c# to implement a multitasking to communicate with these devices at different intervals in parallel. Thanks to TPL the Task Parallel Library of .NET Framework. Before I started writing these post I read about the TPL on  Introduction to TPL , Parallel Tasks in .NET 4.0 – Dotnetcurry and Understanding Task Parallelism Framework (TPL) using C#. These are really good articles to hit the ground pretty fast.

The rationale behind looking at this option is to evaluate how we can make best use of concurrency to collect data from devices and pass it on to the Message queues like Mqtt/RabbitMQ. This scenario would be very much helpful when you have the following scenarios with respect to M2M or IoT:

  1. Devices cannot be reprogrammed with the client to develop and deploy client app to the device.
  2. You have the exposure of data through connectivity means
  3. You can poll the devices for specific parameters through Usermanual
  4. You have connected your smart devices through LAN via Wi-Fi or Ethernet
  5. Communication can be done to the devices via standard protocols

Why TPL?

  • Need to interact with the devices concurrently
  • The topics needs to be monitored and can invoke parallel tasks based on the nature of message
  • The parallel tasks could be like sending Alerts, Notifications, Writing logs, etc.,
  • Tasks should all be done in an async manner

In the another post, I will try and see how we can accomplish this with the help of TPL.

Steps in reading a modbus register

Modbus.org has most of the information with respect to modbus. Modbus protocol has master/slave communication model. Master sends the queries to the slave to get a response. Masters have the capability to talk to individual slave or it can broadcast messages to all the slaves. Slave could be a Meter, Temperature reading device, etc.,  Modbus

It supports two modes of communication one is ASCII and another is RTU mode. Modbus ASCII vs Modbus RTU to know the differences between ASCII and RTU Mode. Any communication to modbus slave for reading the registers should have the following components with it.

  1. Slave Id
  2. Function Code
  3. Data address of the first register to read
  4. Total number of register to be read or length to be read
  5. CRC check

Typically holding registers will hold the data which will be available in the register value between 40001-49999.

Request:

Lets take an example of reading a value from the register 110. Slave Id could be 1. The function code would be 3 for reading the holding register. So the offset value added would be with 40001+110 which would be 40111.   Assuming we might need to  2 registers as the length. Maximum slave Id could be 247.

Important thing to understand while custom programming modbus devices is to ensure that modbus maps from the vendors are thoroughly read so that we can avoid overlaps.

Step by step installing and configuring Mosquitto with Windows 7

Mosquitto is a MQTT (MQ Telemetry Transport) broker used for handling lightweight data transfer protocol. This broker is based on open source which implements MQTT v 3.1 and v3.1.1 as I write this post. This Post would be useful for someone who would like to configure Mosquitto in Windows 7 and test it.

Step 1: Download mosquito from the Download section of Mosquitto.org. In this example we will go through the use of the first link which would be native build applicable for Windows Vista and above.

Step 2: It will take you to the Eclipse repository with mirror options through which you can download the exe file.


Step 3: Double clicking the downloaded exe will throw you with the following screen through you might need to download the OpenSSL and pThreads.

Quick Links:

pThreads: ftp://sources.redhat.com/pub/pthreads-win32/dll-latest/dll/x86/

OpenSSL: http://slproweb.com/products/Win32OpenSSL.html

Step 4: Download the Win32 OpenSSL v1.0.2c Light setup file and install it. We need to get couple of DLL files from this installation which we will see it in the subsequent steps.

Pthreadvc2.dll will be found as given below which can right click and save it to the folder you need.

Step 5: The next step would be asking if you want to install the service too, select it and click next

Step 6: Select install location as given below

Step 7: Installation will get progressed once you press Install as given below

Step 8: Now the installation is done, we need to make sure the following files are copied to the folder where Mosquitto is installed.

Dependencies – win32

Required DLLs: libeay32.dll ssleay32.dll (Look for these files in the OpenSSL-Win32 or OpenSSL-Win32\Bin folder)

Required DLLs: pthreadVC2.dll

Important Note: Please ensure that the required DLLs are on the system path, or are in the same directory as the mosquitto executable.

Step 9: You need to reinstall Mosquitto so that it will also configure the windows Services as shown below as it has the necessary pre-requisites now in the folder C:\Program Files (x86)\mosquitto.

Step 10: How to check if Mosquitto broker has started and running? use the command netstat -an from the command prompt

In another post we will see how we can test the Mosquitto on MQTTLens using Chrome Extensions, mosquitto_pub and mosquitto_sub.

Connectivity challanges with respect to IoT Devices

With the emergence of Internet of things there are various challenges exists with respect to aggregating data to a centralized data repository because of connectivity issues. This post envisages to outline the challenges in detail.IoT Unique Characteristics

1. Unable to connect to the internet because of limited connectivity or range in remote locations where manufacturing facility exists

2. Bandwidth available with 2G options such as GPRS is very limited for data transfer

3. Intermittent loss of connectivity would hamper the data communication process from the IoT devices to the Private or Public Cloud

4. It must be secure enough as it might pose a great danger if its not well controlled

5. Must have the capability to Poll or Push data