Intro
With the amount of continuously generated data on the rise, the cost to upload and store that data in the cloud is increasing. Data is being gathered faster than it is stored and immediate action is often required. Sending all the data to the cloud can result in latency and presents risks when internet connectivity is intermittent. Fog computing, also known as edge computing, involves processing data at the edge for immediate analysis and decisions. Hence it can help reduce network load by sending only critical, processed data to the cloud. This proof of concept (POC) will explore fog computing in a digital signage scenario where the display is recording the number of people viewing the display throughout the day. OpenCV will be used to search the camera feed for faces and Dash* by Plotly* to graph out the results in real time without the cloud.
Figure 1: Up Squared* Board
Set-up
Hardware
The POC uses the Up Squared* board, a high performance maker board with an Apollo Lake processor, with Ubuntu* 16.04 LTS as the OS. For more information on the Up Squared board, please visit: http://www.up-board.org/upsquared/.
The camera used is the Logitech* C270 Webcam with HD 720P which connects to the board by USB.
Software
OpenCV needs to be installed on the Up Squared board to process the images. Reference here for instructions on how to install it on Ubuntu 16.04.
The processed data will be graphed with Dash by Plotly to create a web application in Python*. For more information on Dash: https://plot.ly/dash/
To install Plotly, Dash, and the needed dependencies:
pip install dash==0.19.0 # The core dash backend pip install dash-renderer==0.11.1 # The dash front-end pip install dash-html-components==0.8.0 # HTML components pip install dash-core-components==0.14.0 # Supercharged components pip install plotly --upgrade # Plotly graphing library used in examples #dependencies pip install pandas pip install flask apt get install squlite3 libsqlite3-dev
Face Detection
Face detection is done in Python with OpenCV using the Haar Cascades front face alt algorithm. Each found face is individually tracked in the frame so that each viewer can be counted. This data is recorded to a sqlite database with their face ID and timestamps. Possible expansions to this would be doing facial recognition or using other algorithms to detect demographics of viewers.
The code, seen below, will create and connect to sqlite3 to initialize the database faces.db. The tables inside the database can only be created the first time the code is run, hence it is inside a try clause. OpenCV will then connect to the camera and loop over the feed looking for faces. When it finds a face or faces, it will write send the array of face rectangles to the face_tracking method which will which give each face an ID and then track it based on position. Each face class object stores its current x and y location as well as the time it was last seen and its face ID. If the face disappears for more than 2 seconds, it is aged out of the array and assumed the person moved on or looked away for too long; this will also help with any in-between frames where OpenCV might fail to detect the face where there really is one. This data is then written to the database as the face ID and the timestamp it was seen and to another table is inputted the total number of viewer faces at that timestamp.
Figure 2: Data in the visitorFace table with time stamps and ID
#!/usr/bin/env python
from __future__ import print_function
# Import Python modules
import numpy as np
import cv2
import Face
import sys
import os
import pandas as pd
import sqlite3
from datetime import datetime
import math
#array to store visitor faces in
facesArray = []
#connect to sqlite database
conn = sqlite3.connect('/home/upsquare/Desktop/dash-wind-streaming-master/Data/faces.db')
c = conn.cursor()
# Create tables
try:
c.execute('''CREATE TABLE faces
(date timestamp, time timestamp, date_time timestamp, numFaces INT)''')
c.execute('''CREATE TABLE visitorFace
(date_time timestamp, faceID INT)''')
except:
print('tables already exist')
#find the last visitor faceID to start counting from
df = pd.read_sql_query('select MAX(faceID) FROM visitorFace', conn)
lastFid = df.iloc[0]['MAX(faceID)']
if lastFid is None:
fid=1
else:
fid = 1 + int(df.iloc[0]['MAX(faceID)'])
try:
# Checks to see if OpenCV can be found
ocv = os.getenv("OPENCV_DIR")
print(ocv)
except KeyError:
print('Cannot find OpenCV')
# Setup Classifiers
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
#draw rectangles around faces
def draw_detections(img, rects, thickness = 2):
for x, y, w, h in rects:
pad_w, pad_h = int(0.15*w), int(0.05*h)
cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), thickness)
#track visitor faces based on their location
def face_tracking(rects):
global fid
global facesArray
numFaces= len(rects)
#age faces out when no faces are detected
if len(rects) == 0:
for index, f in enumerate(facesArray):
if((datetime.now()-f.getLastSeen()).seconds > 2):
print("[INFO] face removed " + str(f.getId()))
facesArray.pop(index)
for x, y, w, h in rects:
new = True
xCenter = x + w/2
yCenter = y + h/2
for index, f in enumerate(facesArray):
#age the face out
if((datetime.now()-f.getLastSeen()).seconds > 2):
print("[INFO] face removed " + str(f.getId()))
facesArray.pop(index)
new = False
break
dist = math.sqrt((xCenter - f.getX())**2 + (yCenter - f.getY())**2)
#found an update to existing face
if dist <= w/4 and dist <= h/4:
new = False
f.updateCoords(xCenter,yCenter,datetime.now())
c.execute("INSERT INTO visitorFace VALUES (strftime('%Y-%m-%d %H:%M:%f'),"+ str(f.getId())+")")
break
#add a new face
if new == True:
print("[INFO] new face " + str(fid))
f = Face.Face(fid, xCenter, yCenter, datetime.now())
facesArray.append(f)
fid += 1
c.execute("INSERT INTO visitorFace VALUES (strftime('%Y-%m-%d %H:%M:%f'),"+ str(f.getId()) +")")
print(len(facesArray))
c.execute("INSERT INTO faces VALUES (DATE('now'),strftime('%H:%M:%f'),strftime('%Y-%m-%d %H:%M:%f'), "+ str(len(facesArray))+")")
try:
# Initialize Default Camera
webcam = cv2.VideoCapture(0)
# Check if Camera initialized correctly
success = webcam.isOpened()
if success == True:
print('Grabbing Camera ..')
elif success == False:
print('Error: Camera could not be opened')
while(True):
# Read each frame in video stream
ret, frame = webcam.read()
# Perform operations on the frame here
# First convert to Grayscale
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# Next run filters
gray = cv2.equalizeHist(gray)
faces = face_cascade.detectMultiScale(gray, 1.3, 5, minSize=(30, 30), flags=cv2.CASCADE_SCALE_IMAGE)
print('Number of faces detected: ' + str(len(faces)))
#send the faces to be tracked for visitorFace faces
face_tracking(faces)
#draw on the cv viewing window
out = frame.copy()
draw_detections(out,faces)
cv2.imshow('Facetracker', out)
#commit the sqlite insertions to the database
conn.commit()
# Wait for Esc Key to quit
if cv2.waitKey(5) == 27:
break
# Release all resources used
webcam.release()
cv2.destroyAllWindows()
except cv2.error as e:
print('Please correct OpenCV Error')Code 1: faceCounting.py code for detecting faces
Below is the Face class for each visitor face in the facesArray.
class Face:
path = []
def __init__(self, id, x, y, lastSeen):
self.id = id
self.x = x
self.y = y
self.lastSeen = lastSeen
def getId(self):
return self.id
def getX(self):
return self.x
def getY(self):
return self.y
def getLastSeen(self):
return self.lastSeen
def updateCoords(self, newX, newY, newLastSeen):
self.x = newX
self.y = newY
self.lastSeen = newLastSeenCode 2: face.py for Face Class
Processing and Graphing
To visualize the data after doing some data processing on it, Dash will be used to create a graph that will update itself as new data comes in. The graph will also be in a web application running at localhost:8050. This way insights and information can be seen directly in the fog.
In addition to graphing the number of faces data, the data will be smoothed using Pandas exponentially weighted moving average (EWMA) method to create a less jagged line.
#query the datetime and the number of faces seen at that time
df = pd.read_sql_query('SELECT numFaces, date_time from faces;'
, con)
#do a EWMA mean to smooth the data
df['EWMA']= df['numFaces'].tail(tail+100).ewm(span=200).mean()Code 3: Code snippet to query the number of faces and do EWMA smoothing
Figure 3: Face Counting Data graph and EWMA smoothed data
From the visitorFace table, the face IDs and timestamps will be used to calculate the time someone looked at the display, sometimes referred to as dwelling time. This type of data could be monetized to drive revenue of ads at peak traffic times and report ad impact/reach. The dwelling time is calculated by getting the first and last timestamp for each face ID and then finding the difference between them.
#query the datetime and the number of faces seen at that time, we need this for the next query
dfFaces = pd.read_sql_query('SELECT numFaces, date_time from faces;'
, con)
#calculate the viewing time of each visitor face
#query the first date time and last date time that a visitor face was seen from the tail of the dfFaces data
#taking the data from the tail will make the Viewing Time Data graph and the Face Counting Data Graph line up
df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen
from (select * from visitorFace Where date_time > "%s") group by faceID;""" %(min(dfFaces['date_time'].tail(tail)))
, con)
#calculate the seconds a visitor face viewed the display
df['VD']= (pd.to_datetime(df['Last_Seen']) - pd.to_datetime(df['First_Seen'])).dt.secondsCode 4: Code snippet to query the viewing time for each visitor face
Figure 4: Scatter plot of each visitor face viewing time
The Dash graph application itself has a few key pieces: the page layout, the callback for the update interval, and the graph components. The layout for this application has four main graphs with four headers: Face Counting Data, Viewing Time Data, Daily Face Count, and Hourly Face Count. The interval will call the callback every second to pull the last 1000 data entries from the sqlite database to graph, hence the graph will update itself as new data is added to database. For the graph components, each line or scatter dot is called a trace and the overall graph’s axis’s values are defined.
from __future__ import print_function
import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output, State, Event
import plotly.plotly as py
from plotly.graph_objs import *
from flask import Flask
import numpy as np
import pandas as pd
import os
import sqlite3
import datetime as dt
from datetime import timedelta
#number of most recent datapoints to grab
tail = 1000
app = dash.Dash()
#dash page layout
app.layout = html.Div([
html.Div([
html.H2("Face Counting Data")
]),
html.Div([
html.Div([
dcc.Graph(id='num-faces'),
]),
dcc.Interval(id='face-count-update', interval=1000),
]),
html.Div([
html.H2("Viewing Time Data")
]),
html.Div([
html.Div([
dcc.Graph(id='viewing-data'),
]),
dcc.Interval(id='viewing-data-update', interval=1000),
]),
html.Div([
html.H2("Daily and Hourly Face Count")
]),
html.Div([
html.Div([
dcc.Graph(id='num-faces-daily'),
]),
dcc.Interval(id='face-count-daily-update', interval=1000),
]),
html.Div([
html.H2("Hourly Face Count")
]),
html.Div([
html.Div([
dcc.Graph(id='num-faces-hourly'),
]),
dcc.Interval(id='face-count-hourly-update', interval=1000),
]),
], style={'padding': '0px 10px 15px 10px','marginLeft': 'auto', 'marginRight': 'auto', "width": "1200px",'boxShadow': '0px 0px 5px 5px rgba(204,204,204,0.4)'})
#callback method for face counting data graph interval
@app.callback(Output('num-faces', 'figure'), [],
[],
[Event('face-count-update', 'interval')])
def gen_num_faces():
con = sqlite3.connect("./Data/faces.db")
#query the datetime and the number of faces seen at that time
df = pd.read_sql_query('SELECT numFaces, date_time from faces;'
, con)
#do a EWMA mean to smooth the data
df['EWMA']= df['numFaces'].tail(tail+100).ewm(span=200).mean()
trace = Scatter(
x=df['date_time'].tail(tail),
y=df['numFaces'].tail(tail),
line=Line(
color='#42C4F7'
),
hoverinfo='x+y',
mode='lines',
showlegend=False,
name= '# Faces'
)
trace2 = Scatter(
x=df['date_time'].tail(tail),
y=df['EWMA'].tail(tail),
line=Line(
color='#FF4500'
),
hoverinfo='x+y',
mode='lines',
showlegend=False,
name= 'EWMA'
)
layout = Layout(
height=600,
xaxis=dict(
range=[min(df['date_time'].tail(tail)),
max(df['date_time'].tail(tail))],
showgrid=True,
showline=True,
zeroline=False,
fixedrange=True,
nticks=max(8,10),
title='Date Time'
),
yaxis=dict(
range=[0,
max(df['numFaces'].tail(tail))],
showline=True,
fixedrange=True,
zeroline=False,
nticks=max(2,max(df['numFaces'].tail(tail))),
title='Faces'
),
margin=Margin(
t=45,
l=50,
r=50,
b=100
)
)
return Figure(data=[trace,trace2], layout=layout)
#callback method for viewing time data graph interval
@app.callback(Output('viewing-data', 'figure'), [],
[],
[Event('viewing-data-update', 'interval')])
def gen_viewing_data():
con = sqlite3.connect("./Data/faces.db")
#query the datetime and the number of faces seen at that time, we need this for the next query
dfFaces = pd.read_sql_query('SELECT numFaces, date_time from faces;'
, con)
#calculate the viewing time of each visitor face
#query the first date time and last date time that a visitor face was seen from the tail of the dfFaces data
#taking the data from the tail will make the Viewing Time Data graph and the Face Counting Data Graph line up
df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen
from (select * from visitorFace Where date_time > "%s") group by faceID;""" %(min(dfFaces['date_time'].tail(tail)))
, con)
#calculate the seconds a visitor face viewed the display
df['VD']= (pd.to_datetime(df['Last_Seen']) - pd.to_datetime(df['First_Seen'])).dt.seconds
#declare the traces array to hold a trace for each visitor face
traces = []
for i in range(len(df.index)):
traces.append(Scatter(
x= [df['First_Seen'][i],df['Last_Seen'][i]],
y= [i],
hoverinfo='name',
mode='markers',
opacity=0.7,
marker={'size':df['VD'][i],'line': {'width':0.5,'color':'white'}
},
showlegend=False,
name= df['VD'][i]
))
layout = Layout(
height=600,
xaxis=dict(
range=[min(dfFaces['date_time'].tail(tail)),
max(dfFaces['date_time'].tail(tail))],
showgrid=True,
showline=True,
zeroline=False,
fixedrange=True,
nticks=max(8,10),
title='Date Time'
),
yaxis=dict(
range=[0,len(df.index)],
showline=True,
fixedrange=True,
zeroline=False,
nticks=max(2,len(df.index)/4),
title='Viewer'
),
margin=Margin(
t=45,
l=50,
r=50,
b=100
)
)
return Figure(data=traces, layout=layout)
#callback method for daily face count graph interval
@app.callback(Output('num-faces-daily', 'figure'), [],
[],
[Event('face-count-daily-update', 'interval')])
def gen_num_faces_daily():
weekAgo = dt.datetime.today() - timedelta(days=6)
con = sqlite3.connect("./Data/faces.db")
#query the first date time and last date time that a visitor face was seen
df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen
from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
#reset the index to do a groupby
df= df.set_index(['First_Seen'])
df.index = pd.to_datetime(df.index)
#group the visitor faces by day and record the number of faces per day
dfDaily = df.groupby(pd.TimeGrouper('D')).size().reset_index(name='counts')
trace = Scatter(
x=dfDaily['First_Seen'],
y=dfDaily['counts'],
line=Line(
color='#42C4F7'
),
hoverinfo='x+y',
mode='lines',
name= 'Daily'
)
layout = Layout(
height=600,
xaxis=dict(
range=[min(dfDaily['First_Seen']),
max(dfDaily['First_Seen'])],
showgrid=True,
showline=True,
zeroline=False,
fixedrange=True,
nticks=max(2,len(dfDaily.index)),
title='Date Time'
),
yaxis=dict(
range=[min(0, min(dfDaily['counts'])),
max(dfDaily['counts'])],
showline=True,
fixedrange=True,
zeroline=False,
nticks=max(4,len(dfDaily.index)),
title='Faces'
),
margin=Margin(
t=45,
l=50,
r=50,
b=100
)
)
return Figure(data=[trace], layout=layout)
#callback method for hourly face count graph interval
@app.callback(Output('num-faces-hourly', 'figure'), [],
[],
[Event('face-count-hourly-update', 'interval')])
def gen_num_faces_hourly():
weekAgo = dt.datetime.today() - timedelta(days=6)
con = sqlite3.connect("./Data/faces.db")
#query the first date time and last date time that a visitor face was seen
df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen
from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
#reset the index to do a groupby
df= df.set_index(['First_Seen'])
df.index = pd.to_datetime(df.index)
#group the visitor faces by day and record the number of faces per hour
dfHourly = df.groupby(pd.TimeGrouper('H')).size().reset_index(name='counts')
trace = Scatter(
x=dfHourly['First_Seen'],
y=dfHourly['counts'],
line=Line(
color='#42C4F7'
),
hoverinfo='x+y',
mode='lines'
)
layout = Layout(
height=600,
xaxis=dict(
range=[min(dfHourly['First_Seen']),
max(dfHourly['First_Seen'])],
showgrid=True,
showline=True,
zeroline=False,
fixedrange=True,
nticks=14,
title='Date Time'
),
yaxis=dict(
range=[min(0, min(dfHourly['counts'])),
max(dfHourly['counts'])],
showline=True,
fixedrange=True,
zeroline=False,
nticks=max(2,max(dfHourly['counts'])/10),
title='Faces'
),
margin=Margin(
t=45,
l=50,
r=50,
b=100
)
)
return Figure(data=[trace], layout=layout)
external_css = ["https://cdnjs.cloudflare.com/ajax/libs/skeleton/2.0.4/skeleton.min.css","https://fonts.googleapis.com/css?family=Raleway:400,400i,700,700i","https://fonts.googleapis.com/css?family=Product+Sans:400,400i,700,700i"]
for css in external_css:
app.css.append_css({"external_url": css})
if __name__ == '__main__':
app.run_server()Code 5: app.py code for Dash by Plotly web graph
The above graphs give a very close up view of the traffic in front of the display. To get a higher level picture of what is going on, let’s arrange the data and look at how many visitor faces per day and per hour from the last week.
weekAgo = dt.datetime.today() - timedelta(days=6)
con = sqlite3.connect("./Data/faces.db")
#query the first date time and last date time that a visitor face was seen
df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen
from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
#reset the index to do a groupby
df= df.set_index(['First_Seen'])
df.index = pd.to_datetime(df.index)
#group the visitor faces by day and record the number of faces per day
dfDaily = df.groupby(pd.TimeGrouper('D')).size().reset_index(name='counts')
dfHourly = df.groupby(pd.TimeGrouper('H')).size().reset_index(name='counts')Code 6: Code snippet to query face count daily and hourly
Figure 5: Daily Face Count graph
Figure 6: Hourly Face Count Graph
Summary
So concludes this POC of gathering and analyzing data in the fog. Running the face tracking application and the Dash application at the same time will show data graphing in real time. From here any of the processed data is ready to send to the cloud to store for long term.
About the author
Whitney Foster is a software engineer at Intel in the Software Solutions Group working on scale enabling projects for Internet of Things.
Notices
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.
This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.
The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request.
Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.
Intel, the Intel logo, and Intel RealSense are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others
© 2017 Intel Corporation.