Developing Apps with HEM¶
This section deals with developing and deploying apps locally on HEM.
Before you start¶
Make sure you are authorized to access the HEM boards¶
If you’re working with HEM systems connected to live microgrid, make sure you are authorized to access it. HEMs control critical infrastructure and any unverified access and modifications might cause power system failure. You should get necessary permissions and training before you proceed.
Accessing HEM boards¶
HEM boards can be accessed through SSH (wired or wireless). The boards are password protected. Talk to the microgrid admin/manufacturer for these details.
Making sure HEMApp is running¶
HEMApp is the core software on HEM which ensures these boards are up and running. Check if this is running. The easiest way is to check the touch-display and see if it displays the default screen as shown below.
Connect to SEUP Network¶
The SEUP network is a local network of all HEMs in the microgrid (wired or wireless). Once you connect your computer to the SEUP network, you can access information from all HEMs. If you’re using a wired connection, please plug in your computer to the router or switch. If you’re using a wireless connection, please connect your computer to the router. THe login details can be obtained from the microgrid admin.
Login¶
Each HEM board will have a static ip. This list is available from the microgrid admin.
To login
ssh hem@192.168.1.28
What is an app?¶
When people hear the word ‘app’, most think of a sophisticated iOS/Android application for phones. We are here to change that mindset. For us, an app is a piece of software than can perform a task(s). It can be something as simple as a Python script with few lines of code which sends you an alert when your energy consumption crosses a threshold. While this might seem simple (and it is), its a valuable feature to have when you’re battery is running low or your solar panel is not producing enough energy.
Where should you run your app?¶
- You can deploy your app on a computer connected to the SEUP network
- You can deploy your app on one of the HEMs in the SEUP network
When should you deploy your app on the computer?
- Your app does offline analysis
- It is computationally intensive
- Your app is UI (user interface) based and needs a large monitor, keyboard etc.
- Note: You can also run online apps if you have a dedicated computer on the SEUP network
When should you run your app on HEM?
- App has to be running continuously (24*7)
- It it computationally less intensive
- App is latency sensitive
Downloading the HEM Energy App Development Tool Kit¶
Create a new GitLab account¶
Upload your keys to GitLab¶
Here is the guide from GitLab
SEUP HEM Energy App Dev. Repository¶
Fork the repository¶
Here is the guide from GitLab - https://docs.gitlab.com/ce/gitlab-basics/fork-project.html
Clone the fork you created¶
git clone PASTE_YOUR_SSH_OR_HTTPS_PROJECT_NAME_HERE
Understanding development tool kit directory structure¶
In SeupAppKit
mainHem_
- any file starting with this name is a sample app which is used to demo a particular featuremodule
- directory which contains all the custom hem modules necessary to create an appdata
- this contains sample data from HEM for testing your apps3rdPartyApps
- this contains apps developed for HEM deployment
Creating your first app¶
To create your first app, copy the folder module
and the file mainHem_bare.py
to the desired location. Both of these entities should be in the same directory.
mainHem_bare.py
is a skeleton app. You can add your logic to this .
Sample apps included¶
There are a number of sample apps include to help you speed up your development. They are in microgridbootcamp/hem/code
Here is a list
mainHem_pullData.py
- Pull node(all) data from HEMApp Server through APImainHem_pullData_single.py
- Pull single node data from HEMApp Server through APImainHem_actuate.py
- Turn on/off nodes through APImainHem_demandManage.py
- Perform load management when power consumption crosses thresholdmainHem_email.py
- Send an email from your appmainHem_demandManage_email.py
- Send an email from your app when you do load managementmainHem_file.py
- Extract necessary information from data dumpmainHem_parsing.py
- Parsing data from data dumpmainHem_powerAggregate.py
- aggregate power data based on load/sources/charger for plotting and analyticsmainHem_report.py
- create a pdf report
Running a sample app¶
Let’s run an app which turns on/off a particular node on HEM
Copy mainHem_actuate.py
and module
.
Check server address
If you’re running this app on HEM, you can keep the SERVER_NAME
in mainHem_actuate.py
as localhost
.
If you’re running this app on your computer on a different HEM (which is different from where you’re pulling the data), change the SERVER_NAME
to the local ip address.
If your app has to run in the background, use nohup
nohup python mainHem_actuate.py &
Stopping your app¶
Get the process id (pid) of your app process
ps aux | grep mainHem_actuate
The output of this looks something like this
user 70413 0.0 0.1 110404 9804 pts/8 Sl 13:33 0:00 python mainHem_actuate.py
The pid in this case is 70413
pid is the number listed in the PID column for your app
Now kill that process
pkill -9 <pid>
In this case, it would be something like
pkill -9 70413
After you’re done developing and testing your app¶
After you’ve completed developing and testing your app you can add that as a folder to SeupAppKit/3rdPartyApps
directory in your local repository
File/directory structure for your app¶
Documentation for your app¶
Add a README to your app
What should your README contain? (adapted from https://guides.github.com/features/wikis/)
- Project Name - Your project’s name is the first thing people will see upon scrolling down to your README, and is included upon creation of your README file
- Description - A description of your project follows. A good description is clear, short, and to the point. Describe the importance of your project, and what it does.
- Installation - Installation is the next section in an effective README. Tell other users how to install your project locally. Optionally, include a gif to make the process even more clear for other people.
- Usage - The next section is usage, in which you instruct other people on how to use your project after they’ve installed it. This would also be a good place to include screenshots of your project in action.
- Bugs – Any bugs or quirks that exists in your app that developer or user should know
Make sure you comment your code so that it can be maintained and further developed
Here’s a simple guide to commenting - https://www.cs.utah.edu/~germain/PPS/Topics/commenting.html
Creating a pull request¶
Now you want to push back your app to the main microgrid repository. You can do this through a pull request.
Here are two tutorials
- https://docs.gitlab.com/ee/gitlab-basics/add-merge-request.html
- https://www.digitalocean.com/community/tutorials/how-to-create-a-pull-request-on-github - this is for GitHub but can be applied to GitLab
Skeleton app¶
Let’s look at a skeleton app that can be used as a starting point for app development
Open mainHem_bare.py
in microgridbootcamp/hem/code
to follow along with the rest of the section
#
# mainHem_bare.py
# Skeleton app that can be used for app deployments
# Author: Ashray Manur
import datetime
import threading
import time
import module.hemParsingData
from module.hemSuperClient import HemSuperClient
#Create a new HEM Client. This is used to send and receive data from your HEM
# It takes two arguments - IP/hostname and port
# If you're deploying your app on the same HEM you're getting data from use localhost.Otherwise give IP address
#Port is usually 9931
hemSuperClient = HemSuperClient("localhost", 9931)
#This is function which gets triggered whenever you get data from HEM
#You can add more logic here for post processing
def update(message, address):
print 'Rec:', message, address
#{'NODE': 'ALL', 'TYPE': 'DCPOWER', 'VALUE': [0.185, 5.9, 85.6, 10.4, 0, 0, 0, 12.5]} ('192.168.1.236', 9931)
#message is a list which gives you the type of response and the corresponding nodes
#address is a tuple giving you the server address and the port
#Subscribe to data from HEMs
# The argument to this is the name of the function you want triggered when you get data
hemSuperClient.subscribe(update)
def main():
while(1):
#This sends a request to HEM every 5 seconds
#Argument is the APIs
hemSuperClient.sendRequest("api/getdcpower/all")
time.sleep(5)
if __name__ == "__main__":
main()
The sections below are explaining the various components of the app framework using mainHem_bare.py
as an example
Importing Modules¶
The code sections below should be there in all your apps
Import your standard python modules
import datetime
import threading
import time
Import hem modules
from module.hemSuperClient import HemSuperClient
import module.hemParsingData
import module.hemEmail
Note: there might be other standard or custom modules that you might need to import based on what you’re doing
Defining a trigger function¶
The code sections below should be there in all your apps
def update(message, address):
print 'Rec:', message, address
This functions gets triggered every time your app gets a message from the HEM module.
It takes two arguments message
and address
. When it is triggered, the actual data gets passed to message
and the address of the server responding to your request gets passed to response
Creating a new HEM Client¶
This client is used to communicate with the HEM module. The HEM module runs a server to which you can send requests and receive responses This client abstracts the communication to make data communication between your app and the server easier.
The code sections below should be there in all your apps
#Creates a new HEM client to talk to the server
hemSuperClient = HemSuperClient("localhost", 9931)
If you’re pulling data from the HEM where your app is deployed, keep it as localhost
If you’re pulling data from other HEMs or your computer, change localhost
to server ip like 192.168.1.28
The port is usually 9931
Defining multiple HEM Clients¶
Not included in mainHem_bare.py
In case your app is talking to many HEMs
hemSuperClient1 = HemSuperClient('localhost', 9931)
hemSuperClient2 = HemSuperClient('192.168.1.28', 9931)
hemSuperClient3 = HemSuperClient('192.168.1.29, 9931)
You can request data from all these servers as along as they are in the same network
Subscribing to updates¶
After you define your trigger function and the HEM client, you need to subscribe to updates. This ensures that whenever you get a response from HEM. This can be done by
The code sections below should be there in all your apps
hemSuperClient.subscribe(update)
The argument to this function is the name of the trigger function.
Understanding response data from server¶
Whenever data is sent from the server, the trigger function gets called and it gives you two values
message
and address
.
message
contains the response to the request you made
address
contains the details of the server making that response
message
is a dictionary (key:value) and address
is a tuple
print 'Rec:', message, address
A sample message
is as follows. This response if for a request of power data for all nodes in HEM
{'NODE': 'ALL', 'TYPE': 'DCPOWER', 'VALUE': [0.185, 5.9, 85.6, 10.4, 0, 0, 0, 12.5]}
To extract the value:
message['VALUE']
The output would be
[0.185, 5.9, 85.6, 10.4, 0, 0, 0, 12.5]
To extract the node corresponding to this value
message['NODE']
The output would be
ALL
There is usually one key:value per request
API¶
API general format¶
/api/<action>/<node-number>
Actions can be
- Turn on/off
- Get electrical data
- Send messages
Node numbers usually go from 0-7
How to use the API¶
hemSuperClient.sendRequest(apiRequest)
Sample request¶
hemSuperClient.sendRequest("api/getdcpower/0")
Sample response¶
#message
{'NODE': '1', 'TYPE': 'DCPOWER', 'VALUE': [45.7]}
SEUP DC Systems¶
This section is for SEUP DC.
Turn off nodes¶
Format
/api/turnoff/<node-number>
Sample
/api/turnoff/3
Sample response
#message
{'NODE': '3', 'TYPE': 'TURNON', 'VALUE': [0]}
For ‘VALUE’ 1 means ON and 0 means OFF
Turn on nodes¶
Format
/api/turnon/<node-number>
Sample
/api/turnon/1
Sample response
#message
{'NODE': '1', 'TYPE': 'TURNON', 'VALUE': [1]}
For ‘VALUE’ 1 means ON and 0 means OFF
Status of single node¶
Format
/api/getnodestatus/<node-number>
Sample
/api/getnodestatus/1
Sample response
#message
{'NODE': '1', 'TYPE': 'STATUS', 'VALUE': [1]}
For ‘VALUE’ 1 means ON and 0 means OFF
Status for all nodes¶
Format
/api/getnodestatus/all
Sample response
#responseData
{'NODE': 'ALL', 'TYPE': 'STATUS', 'VALUE': [0,0,0,1,1,1,0,1]}
0 is off and 1 is on
DC voltage for single node¶
Format
/api/getdcvoltage/<node-number>
Sample
/api/getdcvoltage/3
Sample response
#message
{'NODE': '3', 'TYPE': 'DCVOLT', 'VALUE': [14.2]}
The value is in Volts
DC voltage values for all nodes¶
Format
/api/getdcvoltage/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'DCVOLT', 'VALUE': [13.625, 13.6, 13.625, 13.4, 13.7, 13.8, 13.6, 13.6]}
The value is in Volts
DC current for single node¶
Format
/api/getdccurrent/<node-number>
Sample
/api/getdccurrent/4
Sample response
#message
{'NODE': '4', 'TYPE': 'DCCURRENT', 'VALUE': [1.88375]}
The value is in Amps
DC current values for all nodes¶
Format
/api/getdccurrent/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'DCCURRENT', 'VALUE': [1.6, 1.2, 0, 0.4, 3.7, 1.8, 1.3, 3.6]}
The value is in Amps
DC power for single node¶
Format
/api/getdcpower/<node-number>
Sample
/api/getdccpower/5
Sample response
#message
{'NODE': '5', 'TYPE': 'DCPOWER', 'VALUE': [26.74925]}
The value is in Watts
DC power values for all nodes¶
Format
/api/getdcpower/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'DCPOWER', 'VALUE': [0.185, 0.185, 5.3,0,0,0,0,0]}
The value is in Watts
DC energy for single node¶
Format
/api/getdcenergy/<node-number>
Sample
/api/getdcenergy/1
Sample response
#message
{'NODE': '1', 'TYPE': 'DCENERGY', 'VALUE': [2252428.391018]}
The value is in Joules. You can convert it to Wh
DC energy values for all nodes¶
Format
/api/getdcenergy/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'DCENERGY', 'VALUE': [29.21, 16.688, 13.5,3.52,0,0,0.3358,0]}
The value is in Joules. You can convert that to Wh
DC charge for single node¶
Format
/api/getdccharge/<node-number>
Sample
/api/getdccharge/0
Sample response
#message
{'NODE': '0', 'TYPE': 'DCCHARGE', 'VALUE': [170782.33881]}
The value is in Coulombs
DC charge values for all nodes¶
Format
/api/getdccharge/all
Sample response
#responseData
{'NODE': 'ALL', 'TYPE': 'DCCHARGE', 'VALUE': [8779.87, 5014.19, 4066.334, 1040.689,0.000328,0.000328,15.080517,0]}
The value is in Coulombs
SEUP AC Systems¶
This section is for SEUP AC.
Turn off nodes¶
Format
/api/turnoff/<node-number>
Sample
/api/turnoff/3
Sample response
#message
{'NODE': '3', 'TYPE': 'TURNON', 'VALUE': [0]}
For ‘VALUE’ 1 means ON and 0 means OFF
Turn on nodes¶
Format
/api/turnon/<node-number>
Sample
/api/turnon/1
Sample response
#message
{'NODE': '1', 'TYPE': 'TURNON', 'VALUE': [1]}
For ‘VALUE’ 1 means ON and 0 means OFF
Status of single node¶
Format
/api/getnodestatus/<node-number>
Sample
/api/getnodestatus/1
Sample response
#message
{'NODE': '1', 'TYPE': 'STATUS', 'VALUE': [1]}
For ‘VALUE’ 1 means ON and 0 means OFF
Status for all nodes¶
Format
/api/getnodestatus/all
Sample response
#responseData
{'NODE': 'ALL', 'TYPE': 'STATUS', 'VALUE': [0,0,0,1,1,1,0,1]}
For ‘VALUE’ 1 means ON and 0 means OFF
AC voltage for single node¶
Format
api/getacvoltage/<node-number>
Sample
/api/getacvoltage/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACVOLT', 'VALUE': [115.717569]}
This is the RMS voltage and is in Volts
AC voltage for all nodes¶
Format
/api/getacvoltage/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'ACVOLT', 'VALUE': [115.82487, 115.82487, 115.82487, 115.82487, 115.82487, 115.82487]}
This is the RMS voltage and is in Volts
AC current for single node¶
Format
/api/getdcenergy/<node-number>
Sample
/api/getaccurrent/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACCURRENT', 'VALUE': [0.339167]}
This is the RMS current and is in Amps
AC current for all nodes¶
Format
/api/getaccurrent/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'ACCURRENT', 'VALUE': [0.339141, 0.304396, 0.009357, 0.005452, 0.003518, 0.003526]}
This is the RMS current and is in Amps
AC active power for single node¶
Format
api/getacpoweractive/<node-number>
Sample
api/getacpoweractive/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACPOWERACTIVE', 'VALUE': [39.164083]}
AC active power for all nodes¶
Format
/api/getacpoweractive/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'ACPOWERACTIVE', 'VALUE': [39.127099, 34.572928, 0, 0, 0, 0]}
AC reactive power for single node¶
Format
/api/getacpowerreactive/<node-number>
Sample
api/getacpowerreactive/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACPOWERREACTIVE', 'VALUE': [-0.74302]}
AC reactive power for all nodes¶
Format
/api/getacpowerreactive/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'ACPOWERREACTIVE', 'VALUE': [-0.744076, -5.766048, 0, 0, 0, 0]}}
AC active energy for single node¶
Format
/api/getacenergyactive/all
Sample
api/getacenergyactive/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACENERGYACTIVE', 'VALUE': [0.034566]}
AC active energy for all nodes¶
Format
/api/getdcenergy/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'ACENERGYACTIVE', 'VALUE': [0.034648, 0.030616, 0, 0, 0, 0]}
AC reactive energy for single node¶
Format
api/getacenergyreactive/<node-number>
Sample
api/getacenergyreactive/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACENERGYREACTIVE', 'VALUE': [-0.000741]}
AC reactive energy for all nodes¶
Format
api/getacenergyreactive/all
Sample response
#message
{u'NODE': u'ALL', u'TYPE': u'ACENERGYREACTIVE', u'VALUE': [-0.000658, -0.005185, 0, 0, 0, 0]}
AC frequency for single node¶
Format
api/getacfrequency/<node-number>
Sample
api/getacfrequency/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACFREQUENCY', 'VALUE': [59.967205]}
AC frequency for all nodes¶
Format
api/getacfrequency/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'ACFREQUENCY', 'VALUE': [59.967205, 59.967205, 59.967205, 59.967205, 59.967205, 59.967205, 0, 0]}
AC angle for single node¶
Format
"api/getacangle/<node-number>"
Sample
api/getacangle/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACANGLE', 'VALUE': [-0.815625]}
AC angle for all nodes¶
Format
api/getacangle/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'ACANGLE', 'VALUE': [-0.73125, -9.50625, -21.65625, -27.5625, 68.428125, -11.025]}
AC power factor for single node¶
Format
/api/getacangle/<node-number>
Sample
api/getacangle/0
Sample response
#message
{'NODE': '0', 'TYPE': 'ACPF', 'VALUE': [0.999797]}
AC power factor for all nodes¶
Format
api/getacangle/all
Sample response
#message
{'NODE': 'ALL', 'TYPE': 'ACPF', 'VALUE': [0.999797, 0.985777, 0.924442, 0.27144, 0.364928, 0.012272]}
HEM Modules¶
This section deals with the important modules that you can use to build your app.
These can be found in SeupAppKit/module
hemSuperClient.py¶
This module is used to communicate with the server
You can import it with
from module.hemSuperClient import HemSuperClient
Check out mainHem_pullData.py
to learn how to use it.
hemEmail.py¶
This module is used to send email with the server
You can import it with
import module.hemEmail
Check out mainHem_email.py
to learn how to use it
hemParsingData.py¶
This module parses data from HEM data dump to extract electrical and node data
You can import with
import module.hemParsingData
Check out mainHem_parsing.py
to learn how to use it
hemPowerAggregator.py¶
This module parses data and aggregates power data based on node types (sources, loads, charger)
You can import with
from module.hemPowerAggregator import HemPowerAggregator
Check out mainHem_powerAggregate.py
to learn how to use it
Tutorials¶
Understanding the HEM data dump¶
HEMApp (the core software which runs HEM) dumps all the electrical data (voltage, current, power etc. ) on the microSD card on the processor. The file is data.csv
and is located at /media/card
Here are few lines from data.csv
VOLT:13.65,NODE:0,DATE:12/8/17-2:28:40
VOLT:13.62,NODE:1,DATE:12/8/17-2:28:40
VOLT:13.65,NODE:2,DATE:12/8/17-2:28:40
VOLT:13.62,NODE:3,DATE:12/8/17-2:28:40
VOLT:13.65,NODE:4,DATE:12/8/17-2:28:40
VOLT:13.65,NODE:5,DATE:12/8/17-2:28:40
VOLT:13.62,NODE:6,DATE:12/8/17-2:28:40
VOLT:13.62,NODE:7,DATE:12/8/17-2:28:40
AMP:0.00,NODE:0,DATE:12/8/17-2:28:40
AMP:0.00,NODE:1,DATE:12/8/17-2:28:40
AMP:0.00,NODE:2,DATE:12/8/17-2:28:40
AMP:0.00,NODE:3,DATE:12/8/17-2:28:40
AMP:0.00,NODE:4,DATE:12/8/17-2:28:40
AMP:0.00,NODE:5,DATE:12/8/17-2:28:40
AMP:0.00,NODE:6,DATE:12/8/17-2:28:40
AMP:0.00,NODE:7,DATE:12/8/17-2:28:40
POW:1.00,NODE:0,DATE:12/8/17-2:28:40
POW:2.00,NODE:1,DATE:12/8/17-2:28:40
POW:3.00,NODE:2,DATE:12/8/17-2:28:40
POW:4.00,NODE:3,DATE:12/8/17-2:28:40
POW:5.00,NODE:4,DATE:12/8/17-2:28:40
POW:6.00,NODE:5,DATE:12/8/17-2:28:40
POW:7.00,NODE:6,DATE:12/8/17-2:28:40
POW:8.00,NODE:7,DATE:12/8/17-2:28:40
Here the labels mean the following
VOLT:13.65
-VOLT
is dc voltage and the value followed by:
is the voltage in volts. Similarly,AMP
,POW
,ENERGY
,CHARGE
are for dc current, power, energy and charge respectively.NODE:0
- indicates the node number. This can go from0-7
for dc systems and0-5
for ac systems.DATE:12/8/17-2:28:40
- indicates the date and time.
Extracting/copying data from HEM data dump¶
HEM Data Dump - HEM stores all the electrical data in a file data.csv
in /media/card
. This file is usually hundreds of megabytes. Copying the entire file for a small amount of specific data is unreasonable.
This tutorial shows how you can copy specific information from the data dump for your app. Three examples will be demonstrated.
- Only voltage data of all nodes for 2 days
- All electrical data of all nodes for a week
- Only power data of all nodes for 1 day
Open mainHem_file.py
to follow along
To copy one day’s worth of data from sample.csv
(you can replace this with /media/card/data.csv
) into a new file
oneDay.csv
:
cmd = "grep -e '10/1/17' ../data/sample.csv > ../data/oneDay.csv"
'10/1/17'
- the date of interest/data/sample.csv
- this is the source file (for this demo). If you want the actual data dump, replace this with/media/catd/data.csv
/data/oneDay.csv
- this is the destination file
Execute the command
result = subprocess.check_output(cmd, shell=True)
Check if your new file was created
fileExists = Path('../oneDay.csv')
if(fileExists.is_file()):
print 'file exists'
Copy one day’s worth of voltage data
The command
cmd = "grep -e 'VOLT.*10/1/17' ../data/sample.csv > ../data/oneDayVolt.csv"
The search should satisfy both date and data type. Here the electrical data type is VOLT
Execute the command
result = subprocess.check_output(cmd, shell=True)
Check if the file exists
fileExists = Path('../data/oneDayVolt.csv')
if(fileExists.is_file()):
print 'file exists'
Copying one week worth of all data
The command
cmd = "grep -e '10/[1-8]/17' ../data/sample.csv > ../data/oneWeek.csv"
Execute the command
result = subprocess.check_output(cmd, shell=True)
Check if the file exists
#Check if the file exists. This is to make sure the file was created
fileExists = Path('../data/oneWeek.csv')
if(fileExists.is_file()):
print 'file exists'
Parsing data from HEM data dump¶
Now that you’ve understood the format of the HEM data dump and know how to extract specific info let’s see how we can get the electrical values from it
Open mainHem_parsing.py
to follow along
Make sure you import the module
import module.hemParsingData
Open the data file and put it into read mode. In this example it is called data.csv
fh = open('../data/data.csv', 'r')
Iterate through each line of the file
for i, line in enumerate(fh):
Now pass this line into the module
parsedObj = module.hemParsingData.lineParser(line)
If you want to checkout parsedObj
print parsedObj['type'] #VOLT
print parsedObj['value'] #13.65
print parsedObj['node'] #0
print parsedObj['date'] # 2017-12-08 02:28:40
You can add your logic to use this data.
Developing a SCADA app for Monitoring and Control of HEMs on a local network¶
This tutorial deals with building a browser-based app for control and monitoring of all HEMs in a local network (or VPN).The HEM/computer running this SCADA app should also be connected to the same SEUP network either locally or through VPN.
A skeleton app already exists and you can use this to add more features such as live graphing, pricing, revenue management, remote on/off etc.
All the files necessary for this tutorial are available in microgridbootcamp/hemDashboard
Prerequisites
You should have experience and basic understanding of some web technologies and frameworks such as HTML, CSS, JavaScript, Bootstrap and jQuery.
Understanding the directory structure
module
- this contains the custom HEM modulestemplates
- this contains the HTML filesstatic
- contains all the design files and headers - CSS/JS/Bootstrap, images, videos etc.dash.py
- is the skeleton app. You can use this as a starting pointdash.html
- is the skeleton html file. This is what is rendered by the browser. You can edit this file to change the way your app is rendered on the browser.
Where can I run this app
- You can run this app on Linux-based OS/MacOS/Windows computer as long as you have/can install Python.
- You could also run this on HEM boards connected to a monitor via HDMI. However, this would require a special version of HEMApp. The first option is recommended.
Understanding the basic architecture
The app dash.py
runs on a computer which is on the same network (local or through VPN) as HEM. It pulls data from all these HEMs and also has the capability to send actuation signals to all the HEMs. Usually, the HEMs are all assigned static IPs and these are pre-defined and known to all applications.
The web app can be accessed on the same computer as the app or on a different computer. The web app can be accessed on the browser through an URL. The developer can modify the web dashboard and the Python app to suit the needs of the application.
How does it work
- The app
dash.py
requests data from all the HEMs and stores it locally. Accessing HEM data from Python app can be done through standard APIs defined earlier. See API - The web app
dash.html
and associated web files then request data fromdash.py
to render it on the browser. The browser app also requests data fromdash.py
through a standard WEB API defined later.
Starting the dashboard app
For testing you can run
python dash.py
For running it continuously in the background
nohup python dash.py &
As soon as you start running the app, you should see this output
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 318-503-355
From this we know the URL of the web app is http://127.0.0.1:5000/
Open a browser and type in this URL. You should see something like below
Features of the skeleton app
The skeleton already has a couple of basic features
Pulls data from all HEMs
It automatically pulls power
, voltage
, current
, energy
and charge
data from all HEMs you define. You can see hemList = ['192.168.1.28']
lists only one HEM. You can add more HEMs to the list. For example if there was another HEM with the ip address 192.168.1.29``your new list looks like this - ``hemList = ['192.168.1.28', '192.168.1.29']
.
How to pull additional data from HEM
If you want more data from your HEM apart from the standard electrical data, you can pull data from HEM through the standard APIs defined. See API.
This section in dash.py
is where the data is being requested from HEMs. You add your request to this section.
hemClient.sendRequest('/api/getdcpower/all', server_address)
time.sleep(0.1)
hemClient.sendRequest('/api/getdcvoltage/all', server_address)
time.sleep(0.1)
hemClient.sendRequest('/api/getdccurrent/all', server_address)
time.sleep(0.1)
hemClient.sendRequest('/api/getdcenergy/all', server_address)
time.sleep(0.1)
hemClient.sendRequest('/api/getdccharge/all', server_address)
time.sleep(0.1)
Maintains a global object with information of all HEMs
The app maintains hemData
which is a dictionary of all HEMs and their data.
Whenever you request data from HEMs, it automatically sorts it and adds it to the HEM
Understanding the global object
hemData
is a nested dictionary. The key is the ip address of the HEM. In this case it is 192.168.1.28. The value associated with is another dictionary. This nested dictionary has keys DCCURRENT
, DCVOLT
, DCENERGY
, DCCHARGE
etc.
For example,
For example for a two HEM network (192.168.1.28 and 192.168.1.29), hemData
looks like this
{'192.168.1.28': {u'DCVOLT': [13.625, 13.6, 13.625, 13.625, 13.625, 13.625, 13.6, 13.6], u'DCENERGY': [54.698562, 42.207546, 55.773058, 3.760736, 0, 0, 0.033578, 0], u'DCPOWER': [0.204375, 0.204, 0.204375, 0, 0, 0, 0, 0], u'DCCHARGE': [16438.2083, 12712.912803, 16767.096363, 1147.163972, 0.000328, 0.000328, 15.080517, 0], u'DCCURRENT': [0.015, 0.015, 0.015, 0, 0, 0, 0, 0]},
'192.168.1.29': {u'DCVOLT': [13.625, 13.6, 13.625, 13.625, 13.625, 13.625, 13.6, 13.6], u'DCENERGY': [54.698562, 42.207546, 55.773058, 3.760736, 0, 0, 0.033578, 0], u'DCPOWER': [0.204375, 0.204, 0.204375, 0, 0, 0, 0, 0], u'DCCHARGE': [16438.2083, 12712.912803, 16767.096363, 1147.163972, 0.000328, 0.000328, 15.080517, 0], u'DCCURRENT': [0.015, 0.015, 0.015, 0, 0, 0, 0, 0]}}
Now if we want to extract info for ip 192.168.1.28
, we can do hemSubData = hemData['192.168.1.28']
print hemSubData
would look something like this
{u'DCVOLT': [13.625, 13.6, 13.625, 13.625, 13.625, 13.625, 13.6, 13.6], u'DCENERGY': [54.698562, 42.207546, 55.773058, 3.760736, 0, 0, 0.033578, 0], u'DCPOWER': [0.204375, 0.204, 0.204375, 0, 0, 0, 0, 0], u'DCCHARGE': [16438.2083, 12712.912803, 16767.096363, 1147.163972, 0.000328, 0.000328, 15.080517, 0], u'DCCURRENT': [0.015, 0.015, 0.015, 0, 0, 0, 0, 0]}
Now if we want to extract info of voltage, power etc. you can access them through individual keys
print hemSubData['DCVOLT']
would give you
[13.625, 13.6, 13.625, 13.625, 13.625, 13.625, 13.6, 13.6]
print hemSubData['DCCURRENT']
would give you
[0.015, 0.015, 0.015, 0, 0, 0, 0, 0]
print hemSubData['DCCHARGE']
would give you
[16438.2083, 12712.912803, 16767.096363, 1147.163972, 0.000328, 0.000328, 15.080517, 0]
They each return a list of values for all nodes. The order is node 0 to 7. This is consistent across the code base.
Adding logic/intelligence
Let’s say you want to do something more than just displaying data - add logic/intelligence.
You can do this in two ways
- Add a function in
dash.py
- this works best when you want your logic/intelligence to run at certain instants in time (or called from other functions/threads) - Add a thread in
dash.py
- this works best when you want your logic to run continuously/periodically.
How to add a thread? (assuming you know how to add/define functions)
app.py
has two default threads
thread.start_new_thread(sendToServer, ())
thread.start_new_thread(receiveFromServer, ())
You can add your thread in this section of the code
thread.start_new_thread(sendToServer, ())
thread.start_new_thread(receiveFromServer, ())
thread.start_new_thread(myManagementAlgorithm, ())
Then define your myManagementAlgorithm
thread
def myManagementAlgorithm():
# run this continuously
while True:
#add your logic here
#loop (or execute your logic) every 2 seconds
time.sleep(2)
Adding a route
Adding routes means defining a new URL so that your web application can access certain data using this URL. URl which stands for uniform resource locator is a way to access resources from your web application.
dash.py
already defines many routes. Here’s a list of the default routes
/
. Whenever you type inhttp://localhost:5000
this route gets called./getdcpower/<hemip>
- this is used to get the dc power values for the hem with a specific ip./getdcvoltage/<hemip>
- this is used to get the dc voltage values for the hem with a specific ip./getdccurrent/<hemip>
- this is used to get the dc current values for the hem with a specific ip./getdcenergy/<hemip>
- this is used to get the dc energy values for the hem with a specific ip./getdccharge/<hemip>
- this is used to get the dc charge values for the hem with a specific ip.
How do you add a new route?
Add your code just below the default routes
Start with this line
@app.route("/getsomething")
When the browser accesses this route http://localhost:5000/getsomething
, it is handled by the above line of code.
Now you want to respond to the request at this route. So you define a function to respond
@app.route("/getsomething")
def getsomething():
return 'Hello, World'
Note: your function name should match the route. You can notice that both the route and function have a common name - getsomething
return 'Hello, World'
sends a response to the request. You can substitute this with any response object.
For more on routing, see this
Sample Apps for Deployment on HEM¶
Sample apps can be found in
microgridbootcamp/hem/code
You can copy code snippets from these apps into your app
HEM Data Acquisition App¶
mainHem_pullData_single.py
This app demos how to request data from server on a per node basis
HEM Data Acquisition App 2¶
mainHem_pullData.py
This app demos how to request data from server for all nodes
HEM Online Demand Management App¶
mainHem_demandManage.py
This app checks cumulative power consumption of all loads and shuts off the non-critical loads if the threshold is crossed
HEM Demand Management + Email Reporting App¶
mainHem_demandManage_email.py
This app shows you how to do demand management and send an email notification/report
HEM Data Extraction App¶
mainHem_file.py
Shows you how to copy the necessary information from the HEM data dump