The Learning Home

CST 499

Joe Sarabia

October 17, 2017

 2

Executive Summary

This project enhances the usability of home automation by correlating user location and
user activity within a home to build behavioral models of individual users which can then used to
automatically apply state based on detection of a user and other environmental factors. This
project affects all home automation users. This project delivers a functional prototype
demonstrating how machine learning and home automation could be combined to achieve
enhanced usability.
 3

Executive Summary 2

Overview 5

Project Description 5
The Issue in Technology 6
Solution to the Issue 6

Project Goals and Objectives 7

Phase I: Research-based 7
Phase II: Project-based 8
Objective 1: Follow Me 8
Objective 2: Apply a Learned Behavior 9

Community and Stakeholders 10

Evidence that this Project is Needed 11

Who Benefits 12
Why Is It Important 12
Background Information 12

Feasibility Discussion 13
Paper 1: Bluetooth Low Energy for Smart Home Energy Management 13
Paper 1: Options and Justifications 14
Paper 2: Machine Learning On Real-Time Data to Enhance Home Automation 15
Paper 2: Options and Justifications 15

Design Requirements 16
Functional Decomposition 16
Selection of Design Criterion 18
Final Deliverables 18
Methodology 18
Milestone 1: Market Analysis and Feasibility Assessment 19
Milestone 2: Procure Equipment 19
Milestone 3: Prototype Implementation 20
Milestone 4: Prepare Demonstration 20
Ethical Considerations 20
Ethical Concerns 20
Negative Impacts 23
 4

Environmental Impacts 24

Timeline and Budget 24

Milestone 1 24
Milestone 2 25
Milestone 3 25
Milestone 4 26

Usability Testing and Evaluation 27

Target Audience 27
Project Testers 27
Testing Observations 28
Tester Feedback & Improvements 30
Improvements Based on Feedback 30
Short Term Improvements 30
Long Term Improvements 31

Final Implementation 31
Beacon Devices 31
Home Assistant and Raspberry Pi 32
iOS Application 34
Estimote Cloud Manager 35
Estimote Proximity Manager 37

Conclusion 40
Vision of Projects Future 41

References 42

Appendix 43
Home Assistant Configuration 43
MQTT for Room Presence Detection Configuration 44
Detecting Lights On after Entering Room 44
Event Samples for State Changes 46
 5

Overview

This capstone project is a hybrid of a research-based and a project-based capstone. The

first part of the the project was research-based and incorporated a survey of the industry in order

to hone in on a specific area of focus for the second part of the capstone. The second part was

project-based where the focus shifted to implementing a prototype as informed by the

discoveries made during the first part of the project.

Project Description

While substantial progress has been made towards delivering on the promise of the smart

home of tomorrow, even the modest smart home capabilities of today generally require a

prohibitive level of expertise or experience to implement and utilize on a consistent basis,

thereby excluding vast portions of this technologys potential user base. This research

investigated the extent to which todays technology can be used to create a learning home by

implementing precise indoor location tracking combined with a state machine tracking device

usage from which an artificial intelligence (AI) could be derived which would be able to learn

the behavior of its occupants and derive the desired state of the home at any given time based

upon a combination of parameters such as time of day, biometrics, weather conditions and the

specific location of a particular occupant or combination of occupants within the home. The

ultimate goal is for this state engine to be able to control common Internet of Things (IoT)

devices such as lighting, music speakers, electrical outlets, appliances, and televisions. The

research delivers a functional prototype of this state engine demonstrating control of a subset of

these devices based on a subset of the aforementioned conditions.

 6

The Issue in Technology

Can todays existing technology be used or augmented to build an AI to create a learning

home? This research-based portion of the capstone project sought to answer whether particular

hardware such as IoT devices and presence detection devices can be used to accurately and

precisely track the location of specific individuals within a home. Furthermore, this research

sought to answer whether a software implementation can be created to detect and track the

change of state of IoT devices within the home and correlate that to the location of a specific

individual within the home. Finally, this research sought to answer whether an AI can be built to

learn the behaviors of a homes occupants and automatically apply desired state based upon

applicable parameters such as time of day, weather conditions and other factors which influence

a particular behavior.

Solution to the Issue

Given the findings from the survey of the current state of the industry, this capstone then

shifted toward answering a second question, specifically, how well can todays existing

technology be used to create an early, functional prototype of a learning home? Based upon the

findings from the research-based portion, the project-based portion of this capstone implemented

a prototype which is capable of precisely tracking indoor location of occupants within a home,

observing state changes of IoT devices within that home and then correlating these data together

to produce a recommendation for desired state of a home based upon the location of a particular

occupant and at least one other desirable parameter such as time of day.

The final culmination of this capstone project was an early prototype that effectively

demonstrates the ideas embodied within this research as outlined above. While the machine
 7

learning and AI aspects of this project will be left for further study, the prototype does enable a

home to exhibit the capability to accurately respond to particular combinations of inputs based

upon previous learned behaviors and detection of occupants. The idea is to facilitate the

transition from quasi-smart homes to truly smart homes.

Project Goals and Objectives

The three most important criteria needed to demonstrate the ideas investigated in this

research are that a change of state occurs based upon a combination of the following factors: a

particular users location within a home, some other pre-condition such as time of day and that it

exhibits control over more than one class of devices and preferably in combination with each

other.

Phase I: Research-based

During the research based portion of the this capstone, a survey of the industry was

conducted in order to identify suitable technologies and their capabilities to fulfill the above use

cases. For purposes of precise location tracking, the objective was to identify 1-2 technologies

which are able to determine specific location within a room. For example, detecting an occupant

in the living room may not be sufficient: the technology should be able to detect the difference

between sitting on the couch in the living room and just walking through it. A requirement for

these technologies is that they have a public API which can be used for integration in the broader

solution. For purposes of detecting behaviors and ultimately automatically applying state or at

least generating a recommended state, an objective of this portion of the research was to identify

2-3 IoT devices with APIs that can be used to read and detect state changes.
 8

This research investigated products from both Pixie Technologies which use UWB

technology for RTLS and products from Estimote using a combination of iBeacon technology,

UWB technology and their own technologies for accurately tracking indoor location. While the

initial review of Pixies tracking devices looked promising due to an apparently high level of

precision, their SDK was not extensively documented, and was thus unclear how that capability

could be rapidly converted to a prototype to correlate indoor movement to rooms within a home.

Additionally, the Pixie devices non-replaceable batteries were exhausted after less than 6 weeks,

despite their promise to last one year. Pixie customer support did an excellent job of replacing

the devices, however, such delays posed a major encumbrance to a this projects relatively short

timeline. Ultimately after evaluating both Pixies and Estimotes devices, Estimote was chosen

because of the breadth of its product line, including 4 different types of beacons, along with a

mature SDK, and active developers community, and various sample applications to demonstrate

a variety of product capabilities. Additionally, Estimote has been producing beacons and SDKs

for beacon based applications since the release of iBeacon by Apple and appeared to have the

most mature, most active and most well defined approach to this space.

Phase II: Project-based

There are two sample use cases that I explored as early prototypes based upon findings in

the research-based portion of the project. The ultimate goal with these was to investigate

mechanisms to enhance the usability of home automation.

Objective 1: Follow Me

The objective with this use case was to demonstrate the feasibility and a prototype of some

desired state that follows an occupants traversal through various parts of a home. Similar to how
 9

light sensors work, this can be thought of a sensor with persona, that is, the sensors know the

difference between multiple occupants such that they only activate for particular occupants under

particular circumstances. One interesting use case to demonstrate here is follow-me audio, where

a multi-speaker, multi-room audio system is able to continuously adapt its state based upon an

occupants location. An example of this could be a playlist that starts playing on a speaker when

an occupant walks into the bathroom to get ready for the day, then switches from the bathroom

speaker to the bedroom speaker while the occupants finishes getting ready and then switches to

the living room speaker when the occupant moves to couch to prepare a to-do list for the day. Its

not necessary and indeed inefficient for speakers to be playing in a room the occupant doesnt

occupy, and this use case is intended to demonstrate the ability for a home to apply a state, in this

case a playlist, and have it follow that user around the home.

Objective 2: Apply a Learned Behavior

A behavior, in the context of this research, is some combination or sequence of states

across multiple IoT devices or systems applied by a particular occupant in a discernible and

predictable manner. The objective of this use case was to demonstrate the ability to learn and

appropriately apply such a behavior. An example use case of this is a breakfast making routine

where every weekday morning between 8:15a and 8:45a an occupant enters the kitchen, turns on

two specific lights, requests to hear the news from a home assistance device, and then plays a

playlist consisting of 3-4 songs while making and eating breakfast. Before leaving the kitchen,

the occupant turns off the playlist and turns off the lights while heading out the door for work.

The objective of this use case was to learn this behavior and then appropriately apply it if the

occupant is present during the learned timelines. Importantly, this use case should also
 10

demonstrate that state changes dont occur for that same occupant in that location at times

previously unobserved, that is, it shouldnt apply the breakfast behavior during dinner, and it

should only apply the behavior for intended occupants and state should be unaffected by the

presence of some other occupant. For example a second occupant passing through the kitchen

during the same time should have no impact on the state of the home.

Community and Stakeholders

As previously mentioned, the stakeholders and community for this project are any current

or future home automation user. What this group of individuals stands to gain is an improved

realization of home automation benefits, many of which could be construed as comfort level

benefits. Nonetheless, home automation promises advances in many aspects of ones home

including security and energy efficiency. One common IoT device with interesting capabilities is

the smart camera which can be used to detect activity within zones, and more recently, human

presence in a specified area. This activity detection can then be used to trigger desired home

automation actions. While useful, today these are not personalized. For example, the cameras can

detect a person but not a specific person. This paper asserts that the ability to detect particular

people in particular spaces coupled with behavioral modeling will add greater persona and

reduced rigidity to the usability of home automation. On the other end of the spectrum, another

consideration for the home automation user community is one of privacy. There are some

prospective users of home automation who may have no interest in having cameras storing

images of their activity in some external storage service, or there might be objections to the use

of cameras in particular parts of the home, such as bedrooms and bathrooms. The lack of

cameras in these areas otherwise reduces home automation capabilities which might otherwise be
 11

achieved. This paper seeks to overcome such privacy objections by researching feasible options

to trigger actions based on the presence of a particular person in a particular position within a

space without a camera while also providing greater location precision.

One particular area of interest that this research sought investigate is to what extent the

techniques discussed herein might benefit disabled people using home automation. As an

example, great strides have been made in speech based home assistant type devices, but an

individual with a speech impediment may not benefit from some of the capabilities such devices

provide to control their home. Additionally, in the case of some disability that encumbers

interaction with physical devices such as lights and doors, this technology could be used to learn

the persons desired behavioral patterns in a home, perhaps by enlisting help from an assistant,

and then once learned, the behaviors could be applied automatically as the person moves around

the house, assuming that mobility is not a restriction, thus proving that person greater autonomy.

Lastly, anybody new to home automation or perhaps intimidated by the technology such

as elderly people would benefit from this research.

Evidence that this Project is Needed

Given the slow rate of adoption of smart home technology and its current barriers to

adoption, this research seeks to reduce or neutralize some of these impediments which inhibit

adopters from fully realizing its potential benefits. It also provides a prototype that establishes a

pattern for other consumer and commercial IoT applications where the application of learned

behavioral profiles and precise indoor location tracking would be useful.

 12

Who Benefits

Potentially any current or future user of home automation based technologies will benefit

from this research. In particular, classes of users who are otherwise excluded from or

substantially disadvantaged in leveraging home automation and its capabilities today would

benefit.

Why Is It Important

This research is important on two levels. Firstly, software is becoming increasingly

ubiquitous and its capabilities are increasingly impressive. There is frequent conversation in the

national dialogue today amongst policymakers and others about the role of automation and

artificial intelligence in our future economy. On one level, this research is important to

demonstrate the power of artificial intelligence in a tangible way that is readily accessible and to

close the gap between the sci-fi based depictions of a smart home, such as those depicted in Back

to the Future and The Jetsons, with the real capabilities of todays technology. On another level,

I think the realization of these capabilities opens up more possibilities for societal and individual

benefit when our surrounding environments can intuitively adapt to our presence.

Background Information

Primarily my theories are based upon my observations in implementing and maximizing

smart home technologies in my home over the last 3.5 years. Initially, my attempts at

implementing home automation yielded a very low wife acceptance factor, and I realized that not

everyone would find it appealing to control their home from their phone, or even have a phone

from which they could control their home. Ive observed numerous deficiencies in existing

products such as geofences that are orders of magnitude larger than my home and which dont
 13

intelligently account for the presence of multiple people within a home. More than once, I might

leave to go to the store and all the lights in the house turned off as I drove away while my wife

was sitting on the couch watching television. This is where the initial ideas behind occupant

detection came into play.

In addition, I eventually came to a realization that the cumbersome nature of switching

between multiple apps on my phone to yield a desired state in my home could be improved. I

realized that I was typically taking the same set of actions from the same location within the

home, and that the this sequence of steps along with precise occupant detection should be

sufficient to create a home that can learn and apply my desired behaviors and truly be called a

smart home.

Feasibility Discussion

A brief review of existing research yielded two results which have some similar concepts

yet not entirely overlapping concepts and proposals. The two papers reviewed are covered

below.

Paper 1: Bluetooth Low Energy for Smart Home Energy Management

The paper proposes an interesting approach to home automation whereby home

appliances communicate to an energy management unit (EMU), and some intermediary system

between the appliances and the user of the appliance is able to make a recommendations about

the use of the appliance when the user attempts to use it, generally in an effort to reduce peak

load demand and consumptions charges. For example, the EMU might suggest that the electric

clothes dryer be used at a later time when the usage fee for electricity is lower. While this does

involve a use case where home automation interacts with the user which is something The
 14

Learning Home project proposes, in this case it occurs through an intermediary system and is not

completed automated based on behavior. The Learning Home proposes that similar activity with

appliances be applied by presence detection. Nonetheless, there could certainly be use cases

where both applications might exist, or where the data collection by the EMU provides an input

into The Learning Homes state engine as a parameter to consider in determining whether some

automated state should be applied.

This paper proposes BLE to coordinate communication between home appliances as a

mechanism for home energy management (HEM). The choice of wireless protocol is critical

according to this paper. BLE connection setup and data transfer occur in a range from 3ms to

1.28s. Tens or even hundreds of milliseconds to transfer data would likely be sufficient to satisfy

the perception of near real-time reaction in the cases proposed by The Learning Home. The

authors point out the substantial BLE penetration in the industry as a major benefit, for example,

its already technology included in commonly owned devices such as smart phones, smart

watches, and tablets. They also point out that general reliability is key to achieving accurate

automation functionality. The paper asserts that BLE is the best choice in applications of home

automation.

Paper 1: Options and Justifications

Given the maturity of BLE relative to UWB and the broader availability of BLE based

devices in the industry, this research elected to adopt devices that at least had BLE as one of its

available protocols. To the papers point about ubiquity, BLE is a required component for

iBeacon technology to function and is usually implemented alongside other protocols such as

Googles Eddystone and vendors custom protocols such as Estimotes Monitoring. One early
 15

area of examination conducted during this research was whether BLEs data transfer rates are

sufficient to achieve the perception of near real-time reaction. At least in iOS, there are non

user-modifiable constraints on how often and iOS device polls for bluetooth devices. This

research investigated various options for combining iOS polling constraints with configurable

packet broadcasting time in order to achieve a balance between quasi-real-time applications and

battery life of the beacon devices themselves. The outcome of this research will be discussed in a

later section.

Paper 2: Machine Learning On Real-Time Data to Enhance Home Automation

This particular project sounds similar in that their goal is to use ML to generate

personalized and time-variant home automation plans. It posits a solution to the same problem

that The Learning Home seeks to improve upon, namely the rigidity of usability. This paper also

proposes a system that will mimic a users actions based upon previously detected behavioral

patterns.

While these concepts have some overlap with concepts that The Learning Home also

seeks to investigate, this paper is two pages long and is primarily conceptual in its nature with no

indication of whether a prototype was attempted or implemented. Additionally, there is no

reference to the user differentiation detection capabilities or precise real-time location services

that The Learning Home proposes as part of its solution.

Paper 2: Options and Justifications

This research validate the concepts introduced in this paper and seeks to pickup where

this research left off by implementing a prototype embodying and demonstrating some of its

ideas.
 16

Design Requirements

The following section of the document provides detailed specifications which served as a

basis for the project and guided the entire development process.

Functional Decomposition

The diagram below is a depiction of the major components of the project and how the

flow of activities and data works to achieve the desired outcomes. From a hardware perspective,

the major components of the design are a home outfitted with home automation devices, a

raspberry pi device to host open source software components, beacons used to track indoor

location and mobile devices which are used by occupants to detect signals from the beacons. On

the software side there are three primary components. The first component is an open source

software package called Home Assistant. This package has three major architectural components

of interest for this project, namely, a state machine which tracks the state of all devices it knows

about over time, and event bus which listens to events to be fired such as state changes and a rule

engine which can be used to listen to the event bus, evaluate the event againsts a set of rules and

then change desired state. Using Home Assistant APIs, a software package built as part of this

project extracts data for examination and to perform behavioral pattern detection. Additionally, a

software package built as part of this project tracks an occupants location within a home and

sends messages to the Home Assistant platform when the occupant enters or leaves a particular
 17

room.

Figure 1. High Level Solution Architecture

While the original goal of this research was to apply machine learning algorithms to the

datasets generated in order to create recommendations for state changes to perform based on an

individual occupants location, the mix of research and implementation in this project posed a

time constraint such that this final architecture was achieved about 75% of the way through the

available timeline. This did not leave adequate time to generate a sufficient amount of data from

which to train models, so in lieu of that, SQL queries of the Home Assistant databases state

table were used to derive a first pass of automation rules that combined presence detection and
 18

state changes. The rules were then added to Home Assistant such that after placement of the

beacons and running the iOS app, the home is able to automatically apply state changes at

appropriate times based on upon user occupancy.

Selection of Design Criterion

The primary factors that influenced choosing the appropriate hardware to support this

project were the cost, reliability and ease of use. On the cost side, the goal was to develop a

solution where equipment needed for indoor location tracking was no more than $20 per room

and would be easy to install. Additionally, it was important to select a vendor with strong

expertise in the area of beacon technology which also had a mature and well documented API.

The clear winner after testing devices from two vendors was Estimote who offers a broad

portfolio of hardware products and has been working with beacon technology since its inception.

Final Deliverables

The final deliverables for this project will include an iOS mobile application used to track

and record indoor location along with the software packages discussed above which are used to

extract data from Home Assistant, correlate it with location data, produce state recommendations

and finally create rulesets and apply them to Home Assistant. Due to the nature of this project,

the final deliverables will also include a video demonstrating the capabilities of this project.

Methodology

The methodology I followed in completing this project was Agile, using iterative and

incremental development techniques and fast feedback loops to adjust as the project went on. I

used real-world consumer devices combined with software to implement prototypes of the topics
 19

discussed herein. I had identified four major milestones as part of this project at its outset, which

are discussed in the following subsections.

Milestone 1: Market Analysis and Feasibility Assessment

I researched various devices currently available in the consumer marketplace.

Specifically, I researched the smart cameras which are suitable candidates for real-time facial

detection and devices which can be used to precisely track real-time location of a user within a

home. As part of the solution intended to correlate the activity of devices with the location of

users in order to build behavioral models which could later apply state, I researched home

automation smart hubs which aggregate the control of multiple devices where some such model

can be applied. One important part of research that I completed in this milestone was the

assessment of APIs for all devices that I research, as that was important for later stages of

project. Ultimately during this phase, I selected the use of Estimote beacons for tracking location

and the Wink Hub for aggregated control of home automation devices.

Milestone 2: Procure Equipment

Based upon the previous milestone, I procured Estimote Nearables, Estimote Proximity

Beacons, Estimote Location Beacons and Estimote Location Beacons with UWB. Unfortunately,

the Location Beacons with UWB, which were the most promising piece of hardware for my

desired application, were received days before the end of research completion and as such were

not obtained in sufficient time to incorporate into this project. I already owned a Wink Hub

device, which moderately accelerates the development process and reduces the budget.
 20

Milestone 3: Prototype Implementation

I used the equipment previously procured or already owned to build a functional

prototype of the concepts discussed in this paper. Specifically, I built a prototype to demonstrate

the ability to correlate user activity with user location to build a behavior model. The camera

based approach quickly proved to be unreliable as the facial detection capabilities built into

commercial products were unreliable and yielded frequent false positives. Finally, I conducted

some initial research on machine learning, but descoped that element from this project due to

time constraints.

Milestone 4: Prepare Demonstration

Most of the testing was conducted in my home and I have devised ways to present a

scaled down version of this project on-campus. To that extent, during this portion of the project I

experimented with a scaled down version or other mechanisms to effectively demonstrate the

broader ideas behind the project. Finally, I opted to use Zoom as the platform to create my

demonstration as the most effective way to display this projects capabilities incorporated live

video while simultaneously capturing events in a browser and iOS application state.

Ethical Considerations

Given that this research deals with tracking user activity and location very precisely

within ones home, there are important ethical considerations to consider both in its design and

implementation.

Ethical Concerns

The primary ethical concerns associated with this capstone project relate to privacy. This

project sought to investigate two fundamental methods as mechanisms to track user behavior and
 21

build corresponding behavioral models using machine learning, each of which has a discrete set

of privacy considerations.

Location Services. While location tracking capabilities, which are commonly called

location services, provide substantial capabilities in a variety of mobile devices, they can be

viewed as useful or invasive. Mobile devices commonly leverage location services to provide

useful functions such as GPS based directions and geo-tagging of objects such as pictures.

Nevertheless, many device users are not comfortable with the notion of their every movement

and location being tracked and recorded. As a result of these concerns, some users opt to turn off

location services in an effort to reduce the recording of such data.

A common use of location services in home automation is for the control of lighting, for

example, to turn lights on or off when entering or exiting a region such as a geofence around

somebodys home. In the case of iOS devices, Apple states that Location Services uses GPS

and Bluetooth (where they're available), along with crowd-sourced Wi-Fi hotspots and cellular

towers to determine the approximate location of your device. One of the primary mechanisms

that this project uses for precisely tracking user location within a home is via Bluetooth

communications between a personal device such as a mobile phone (iPhone) or wearable device

(Apple Watch) and beacons. Any user who wishes to completely turn off all location services

would not be able to benefit from the techniques discussed in this paper.

Nevertheless, the opposition to tracking everywhere a person goes outside of their home

is expected to be more more objectionable than tracking where they are inside their home. For

example, the former may expose illicit or morally objectionable behavior that a device user may

wish to conceal, while in the case of the latter, a visit to the closet or the living room is less likely
 22

to be of substantial consequence or a target of desired concealment. This is an assumption that

merits validation with potential users now that this research has yielded some viable prototypes.

One method of mitigating this concern, should it become one, would be to use some

ancillary device other than mobile device as a tracking mechanism to represent the user, such as

a bluetooth beacon attached to a keychain or simply carried in ones pocket.

Cameras. The other primary method this research evaluated as part of its efforts to build

user behavioral models was the use of IoT cameras. As demonstrated by the Persirai IoT botnet

in May 2017, IoT cameras have proven to be a favorite target of malware authors and used as a

launching point for distributed denial of service (DDOS) attacks. Additionally, while users have

generally been accepting of storing streaming video of activity outside their homes with some

external service provider as a method of home security, it is expected that users would have

strong opposition to continuous video recording inside their home, particularly in certain rooms

within the home such as bedrooms and bathrooms where a generally high level of privacy should

rightfully be expected.

This research investigated the use of IoT cameras to perform facial recognition as a

means to discern and identify various actors within the home and potentially use those

recognition capabilities to apply learned behaviors. For example, this could provide for the

detection and application of different lighting based upon who enters the front door. This

research originally aimed to mitigate concerns about cameras within a home by positioning them

to cover main points of ingress and egress such as a front door or garage entry door and main

living spaces such as a family room and kitchen; however, due to aforementioned technical

constraints, this approach was not extensively pursued nor was it incorporated into the final
 23

prototype. This research did not consider the application of IoT devices in other parts of the

home, and therefore concluded that a hybrid approach between the bluetooth and camera based

methods could be used to achieve the desired effects, should the technical constraints be

mitigated in the future.

Negative Impacts

Home automation devices today are expensive compared to their traditional

non-automated counterparts, and thus the barrier for entry could be very high for lower income

individuals and families. According to Pew Research, while 77% of all Americans own a

smartphone, as many as 64% of low income owners also own smartphones. Bluetooth is a

common and fundamental technology on smartphones, so a majority of people should be able to

leverage at least some of techniques which are part of this research and nearly all smartphones

users should theoretically be able to, at least from a technology perspective, using capabilities

built-in to their phone. The bigger issue which may exclude low-income groups would be the

additional cost of the items required to track location inside a home such as beacons, IoT

cameras or home automation devices such as lighting.

A long term goal of this research is to determine whether it would be useful or feasible to

incorporate BLE chips in more commonly used devices such as light switches and power plugs

to facilitate some of the techniques that this research has examined. The ability to do so in a

viable manner may reduce the potential financial burdens associated with implementation of the

technology.
 24

Environmental Impacts

One of the primary motivations behind this project and indeed home automation in

general is to have a positive environmental impact by enabling people to use their homes more

efficiently. Automation of lighting is a common entry point into home automation and this

research intends to build techniques that will make the use of homes and devices beyond just

their lighting more efficient.

Timeline and Budget

During different milestones of this project, it was at times ahead of schedule, at times

behind schedule, at times way over budget and at times right on budget. Overall, this project

primarily ran slightly behind its timeline and was slightly over budget.

Milestone Timeline Budget

Milestone 1: Ahead of schedule On budget

Market Analysis and
Feasibility Assessment

Milestone 2: Behind schedule On budget

Procure Equipment

Milestone 3: Behind schedule Over budget

Prototype Implementation

Milestone 4: On schedule On Budget

Prepare Demonstration
Figure 2. Project Performance Summary

Milestone 1

This milestone ran ahead of time and ultimately on budget. I had procured some

equipment in advance of the start of class which enabled this milestone to be achieved more

quickly. Additionally, I had been following Estimotes technology for some time and had an idea
 25

that it would be the more suitable option for rapid prototyping, but to avoid confirmation bias, I

ensured that I tested another vendors technology.

Milestone 2

This milestone ran behind schedule and on budget. As previously discussed, Estimotes

Location Beacons with UWB sounded promising for this research and provided all the

capabilities of its other beacons combined into a single form factor. Unfortunately, these devices

were backordered and then encountered additional delays during the backorder and were not

incorporated into this project. I ultimately ended up procuring every type of beacon that Estimote

produces and after testing returned the ones that were not useful for this research.

Milestone 3

This milestone ran behind schedule and over budget. The implementation of the

prototype took much longer than initially anticipated. There were several factors which

contributed to this delay.

Firstly, once it was known that this researchs first choice of hardware would likely not

be available in sufficient time for incorporation into the final project, alternative approaches had

to be evaluated and implemented. Secondly, one of the final deliverables was on a platform and

in a language which I had never worked with before, specifically Swift on iOS. There was a

significant learning curve to overcome. During the initial implementation of the prototype, I

encountered a few iBeacon protocol limitations that were substantial encumbrances, specifically,

I found that it was difficult to control distance in software and that the reachability of a beacon

from a mobile device was mostly achieved through physical placement of the device and by

adjusting signal strength. Additionally, I discovered that iBeacon imposes a 30 second wait for
 26

exit events. This was problematic because as it created false positives in terms of an occupants

actual location, that is, it made for the appearance of an occupant being in two rooms

simultaneously which was undesirable from a data correlation perspective. Finally, I

underestimated the time that the Machine Learning component would take and I wound up with

insufficient data and time to train models. I discovered that the general recommendation for

training is to apply a 80%/20% rule, that is train with 80% of the data and then validate it with

the remaining 20%. In this case, because I know that I have variations of activity throughout a

week, it would have been useful for this project to treat a week as an atomic unit and have 4

weeks of data to train on and then 1 week of data to validate on. Unfortunately, with the solution

architecture finalized and a prototype completed around the 6.5 week mark, there wasnt time to

complete all of the machine learning milestone and it had to be descoped from the project and

left for future research.

This milestone ended up being over budget because during the course of implementing

the prototype, I came across the excellent Home Assistant platform which required some sort of

device to run on to collect state data. I purchased a Canakit Raspberry Pi to run Home Assistant

as a Docker container, which worked out well.

Milestone 4

This milestone was on schedule and on budget. I completed a viable demonstration

shortly after the prototype was completed. There was no budget I anticipated being required for

this milestone which aligned to the realities upon its completion.

 27

Usability Testing and Evaluation

Target Audience

My target audience is all ages, but Im initially targeting digital natives or generally any

person who was born after 1980, i.e. 18-37 year olds. I am initially targeting this audience as

familiarity with technology, mobile devices and and mobile apps is a prerequisite in the early

prototype stages. Other age groups are also reasonable targets if they satisfy these criteria.

Additionally, I am targeting home automation users and general technology enthusiasts

during the prototyping phase. Longer term, my project will target people with disabilities or

mobility restrictions that could benefit from this project in additional to non-technology focused

people who might appreciate its benefits but dont have the interest or knowledge to configure

complex automation rulesets for themselves.

Project Testers

As this project is currently in an early prototype phase and involves a significant

hardware component to it, there were substantial limitations on the ability to test the project.

Testing of the project was constrained to the occupants of the home where the prototype has been

implemented.

The names of the testers were Joe and Kasie. I fit the target audience as I am a digital

native who is familiar with technology and I am a home automation enthusiast. My wife Kasie

fits my target audience as she is also a digital native but part of the later target group of people

who appreciates home automation but dont want to hassle with it and doesnt have the time or

expertise to configure complex rulesets for herself.

 28

Primary Testing Tasks

Testing was rather straightforward. It involved simply behaving in the manner in which

one normally would in the home, but while carrying a mobile device running the prototype

application with them.

The specific tasks are:

1. Install The Learning Home iOS application on the device via XCode.

2. Run application.

3. Visit at least three distinct rooms in the home, such as the Living Room, Kitchen

and Master Bedroom.

4. Remain in each room for at least 30 seconds.

5. In 2 of the 3 rooms, manually interact with at least one class of device such as a

light switch.

6. In 1 of these 2 rooms interact with more than one class of device such as a light

switch and a music speaker, optionally using an existing application to manage

the device interaction.

Testing Observations

There are two parts of this testing that were important to analyze. The first was end-user

behaviour, which was quite straightforward: start the mobile app, put the phone in your pocket,

and then proceed with your typical routine. In the case of both testers, the tasks were completed

in an unremarkable fashion.
 29

The second part of the testing analysis had more to do with the data generated by the

application and how well the data generated correlates to each testers activities. On one hand,

testing demonstrated the ability to accurately detect and capture the state of the home, as

depicted in this graph of activity in the living room:

Figure 3. Room Activity

The testing also demonstrated the ability to detect the testers location in particular rooms

as show in the following screen capture:

Figure 4. Room Presence Detection

A primary issuing uncovered during testing, however, related to the reliability of room

detection. There are frequent false positive enter and exit events reported by the application

where the tester and its nearest beacon were stationary and had been stationary for over 10

seconds, but where an exit and then enter beacon region events would be reported. Additionally,
 30

testing uncovered that the detection of a room enter event usually lags physical entrance by

approximately 2-5 seconds. These creates a perceptible delay that for certain tasks, such as

turning on lights when entering a dark room, was too lengthy and was viewed as an encumbrance

during testing.

Tester Feedback & Improvements

Testers were able to complete all of the tasks, as the prototype required them to launch an

application on their phone and then use the home in the manner they normally would.

Improvements Based on Feedback

As mentioned above, the most immediate short term need is to improve the timing of

activities. Currently, there are is a delay between physically entering a room and a room entrance

event being detected in software. Additionally, exit events for leaving beacon range are software

controlled via iOS to occur 30 seconds after leaving the region which creates circumstances

where the data represents the tester as being in two locations at same time even after having

recently left one of the rooms. This creates issues for data correlation down the road.

Short Term Improvements

Estimote recently released a new version of its SDK using their own protocol which

purports to overcome some of these limitations of iBeacon and allows for greater precision

around measuring entry and exit events that may address some of these shortcomings. In the

short-term, I plan to use this new Proximity SDK for room detection to see if its possible to

achieve results that are closer to real-time.

 31

Long Term Improvements

The long term plan will be to use UWB based technologies, which I received too late in

this course to incorporate into the current prototype, to further investigate higher precision and

more real-time presence detection.

Final Implementation

The final implementation of this project comprises a substantial portion of the originally

envisioned functionality. The prototype that was built is able accurately track the presence of a

homes occupant in various rooms throughout the house. Additionally, the prototype

implementation tracks state changes to devices within the home. All of this data is tracked in the

same database and was used to derive automation rules which are then applied based on user

presence.

Beacon Devices

The final implementation uses Estimotes Proximity Beacons, with one placed in the

kitchen, living room and master bedroom in my home. After encountering some of the issues

with the iBeacon protocol discussed earlier in this paper, Estimotes proprietary Monitoring
 32

protocol was used in the final prototype to overcome some of these constraints. Note that in the

configuration screens above that the iBeacon protocol has been disabled and the Estimote

Monitoring protocol is enabled. This is a limitation of the Estimote Proximity Beacon: it can

only broadcast one protocol at at time. Estimote Location Beacons are able to simultaneously

broadcast multiple packet types.

Home Assistant and Raspberry Pi

Home Assistant is an open source application that provides a very extensible and robust

platform for home automation with a copious number of plugins and configuration options.

Included in the Home Assistant platform and of interest for this project were a state machine to
 33

track changes to entities over time, as well as an event bus that other components in the platform

are able to plug into. One of these key components is an automation rule engine which can

monitor events on the bus, validate rules, and then apply state changes.

In order to have dedicated hardware to run software packages that would enable type of

data collection that I desired for this project, I obtained a CanaKit Raspberry Pi 3 Complete

Starter Kit - 32 GB Edition and installed Raspbian Jessie. Initially, I used the Raspberry Pi to

host software packages that provided Apple HomeKit support, anticipating that the integration of

HomeKit would be desirable for an iOS application, and a software package for text-to-speech

(TTS) support via the Sonos API, which enabled me to create announcement messages on

speakers during early prototyping. Ultimately, upon discovering Home Assistant, I abandoned

the use of these other software packages, installed Docker, and then installed hass.io, the
 34

dockerized version of Home Assistant. After installation, I configured Home Assistant with most

of my devices, which was facilitated greatly by Home Assistant's Wink component. Finally, I

configured some rooms in Home Assitant to group devices together that I wanted to test for this

project.

iOS Application

An iOS application using Swift was implemented to read Estimote Monitoring packets

and send notification messages to the Home Assistant platform using the MQTT protocol.

Figure 7. iOS Application

The implementation of this prototype included two critical pieces, an MQTT client for

iOS that could send notifications to Home Assistant, along with an extension of an application

template that would allow reading beacon information from the Estimote Cloud.
 35

Estimote Cloud Manager

An application template provided by Estimote provides a basic blueprint for retrieving

beacon information from the Estimote Cloud where various configuration data lives.

Unfortunately, the provided template only fetches the beacons identifier which looks something

like this: 5a1b4d9ad7fe92f5adc70c9f74894f3e. It was clear that the sending such an identifier

to Home Assistant would only obfuscate later efforts to correlate user position to other activities,

so I wrote an extension to Cloud Manager that fetches the name of the beacons, which Ive

named to correlate to their physical positions.

This implementation involved three components. The first was to add a new function to

fetch the names of the beacons:

func fetchNames(beaconIdentifiers: [String], completion: @escaping

((RequestResult<[String: String]>) -> ())) {

let request = ESTRequestGetBeaconsDetails(macAddresses:

beaconIdentifiers, andFields: ESTBeaconDetailsFields.fieldName)

request.sendRequest(completion: { (beaconInfos, error) in

guard error == nil, let beaconInfos = beaconInfos as?
[ESTBeaconVO] else {
completion(.error)
return
}

var nameByIdentifier = [String: String]()

for beaconInfo in beaconInfos {
let name = beaconInfo.name
nameByIdentifier[beaconInfo.macAddress] = name
}
completion(.success(nameByIdentifier))
})
}
 36

The second part was to call the new function as part of a pre-fetch routine so that it could

be used by later UI routines:

self.cloudManager.fetchNames(beaconIdentifiers:
self.beaconIdentifiers) { (result) in

switch result {
case .error:
self.presentFetchingNamesFailedAlert()
return

case .success(let nameByIdentifier):

self.addBeaconZoneViews(colorByBeaconIdentifier:
colorByIdentifier, nameByBeaconIdentifier: nameByIdentifier)

// Step 2: Start monitoring proximity of your beacons

self.proximityManager.delegate = self
self.proximityManager.startMonitoringProximity(identifiers:
self.beaconIdentifiers)
}
}
The final part was to update the UI of the application with the name of the beacon that is

in range:

let identifierLabel = UILabel()

identifierLabel.translatesAutoresizingMaskIntoConstraints = false
identifierLabel.textAlignment = .center
identifierLabel.font = UIFont.boldSystemFont(ofSize: 18.0)
identifierLabel.numberOfLines = 0
identifierLabel.textColor = UIColor.white
if let name = nameByBeaconIdentifier[identifier] {
identifierLabel.text = "You are in or near:\n\(name)"
}
zoneView.addSubview(identifierLabel)
 37

Estimote Proximity Manager

The piece which tied everything together was the implementation of an MQTT client for

iOS. Home Assistant provides an MQTT broker along with an MQTT Room presence detection

capability in a class of entities that it refers to as sensors. By defining an MQTT topic associated

with a particular device id, the Home Assistant platform allows publishing messages to a

subtopic where the subtopic represents a room. If the payload of the message published to a

subtopic contains a device id which matches the configuration of the sensor, then Home

Assistant fires a state change indicating that the device now occupies whatever subtopic, or

room, that the message was published to.

Fortunately, Estimote provides a ProximityManagerDelegate in its API which is used to

track when beacons come in and out of range. Two of the primary advantages of using Estimote

Monitoring packets over iBeacon packets was the elimination of 30 second exit delays and,

perhaps more importantly, the ability to define desired distances in software that would

constitute the beacons perimeter.

The first part of this implementation and perhaps the trickiest for somebody new to iOS

development like I was, was using CocoaPods to properly import the Moscapsule library, a

commonly used MQTT client for iOS. Once this was done, the next step involved setting up the

MQTT client in the ProximityManager:

// create MQTT Client Configuration

let mqttConfig = MQTTConfig(clientId: "joephone", host: "raspberrypi",
port: 1883, keepAlive: 60, protocolVersion: MQTTProtocol.v3_1_1)

var mqttClient: MQTTClient?

let cloudManager = CloudManager()

 38

var namesByBeaconIdentifier = [String: String]()

func startMonitoringProximity(identifiers: [String]) {

self.monitoringManager.startMonitoring(forIdentifiers:
identifiers)

// MQTT auth config

mqttConfig.mqttAuthOpts = MQTTAuthOpts(username: "homeassistant",
password: "xxxx")

// create new MQTT Connection

mqttClient = MQTT.newConnection(mqttConfig, connectImmediately:
true)

self.cloudManager.fetchNames(beaconIdentifiers: identifiers) {
(result) in

s witch result {
case .error:
print("error attempting to retrieve beacon names")
return

c ase .success(let nameByIdentifier):

self.namesByBeaconIdentifier = nameByIdentifier
}
}
}

There are two items worth noting in the above code. The first is the protocol version. If

unspecified, this defaults to MQTT version 3.1; however, the broker that ships with Home

Assistant 0.55.2 expects MQTT version 3.1.1. The second item worth noting is the reuse of the

fetchNames() function previously created for the UI updates. This turns out to be very useful for

the message publishing.

 39

The next step in updating the ProximityManager was to expand the capabilities of the

ESTMonitoringV2ManagerDelegate protocol. The Estimote SDK uses a monitoring manager to

fire events whenever the initial state is observed, when a beacon enters the desired range and

when a beacon exits the desired range. In each of these cases, the ProximityManager class calls a

function named updateBeaconsInRange(), which was updated as follows:

fileprivate func updateBeaconsInRange() {

let monitoredIdentifiers =
Array<String>(self.monitoringManager.monitoredIdentifiers)
let beaconIdentifiersInRange = monitoredIdentifiers.filter { [weak
self] (identifier) -> Bool in
return self?.monitoringManager.stateForBeacon(withIdentifier:
identifier) == .insideZone
}

if beaconIdentifiersInRange.count == 0 {
self.sendAwayMessage()
}
else {
for identifier in beaconIdentifiersInRange {
self.sendEntryMessageForIdentifier(identifier: identifier)
}
}

self.delegate?.proximityManager(self, didUpdateBeaconsInRange:
beaconIdentifiersInRange)
}

Once a filtered array of beacons is stored in beaconIdentifiersInRange, one of two things

happens: either an away message is sent when the array of in range beacons is empty or an entry

message is sent for the identifier in range. These two functions are what ultimately publish the

MQTT messages and tie the presence detection back to Home Assistant. The entry message

function is relatively straightforward:

 40

fileprivate func sendEntryMessageForIdentifier(identifier: String) {

l et i d = self.mqttConfig.clientId
l et n ame = self.mqttConfig.clientId
l et d istance = Parameters.desiredDistance

let msg: String = "{\"id\": \"\(id)\", \"name\": \"\(name)\",

\"distance\": \(distance)}"

if let room = self.namesByBeaconIdentifier[identifier] {

print("Publishing Message: \(msg) to subtopic: \(room)")
mqttClient?.publish(string: msg, topic:
"room_presence/\(room)", qos: 2, retain: false)
}

Its worth noting again that the clientId specified in the original configuration of the

MQTT client above should match the sensor configuration in Home Assistant, otherwise the

message will not have the desired effect. The function publishes a message containing the client

id, the client name, and a distance. Also note that the message is published to a subtopic that

matches the name of beacon.

The away message function operates in a similar fashion; however it publishes to the

room_presence/away subtopic to indicate that the occupant is not in a monitored room within

the home.

Conclusion

This project serves as a proof of concept and validates some of the core capabilities

around improving home automation via presence detection.

 41

Vision of Projects Future

The most immediate work to carry forward with this project will be the implementation

of the machine learning and AI features which were descoped from the project. Firstly, this will

involve collecting sufficient data over the next four weeks to adequately train and validate

machine learning models. Part of this process will involve data extraction and data cleansing.

Once the models are trained and generating reliable output, a software package will be

created that uses the output from machine learning to construct the rules, represented as YAML

configuration files, and then uploads the rules back to Home Assistant such that over time after

placement of the beacons and running the iOS app, the home is able to automatically apply state

changes at appropriate times based on upon user occupancy with little to no additional

intervention by the home automation user.

 42

References

Apple Inc. (2017, February 24). Turn Location Services and GPS on or off on your iPhone, iPad,

or iPod touch. Retrieved August 22, 2017 from

https://support.apple.com/en-us/HT207092

Collotta, M., & Pau, G. (2015). A Solution Based on Bluetooth Low Energy for Smart Home

Energy Management. Energies, 8(10), 11916-11938. https://doi.org/10.3390/en81011916

Shaikh, S., Attar, A., Pathan, S., & Shaikh, R. (2016). Machine Learning On Real-Time Data to

Enhance Home Automation. International Journal of Engineering Trends and Technology

(IJETT), 42(1), 6-8. https://doi.org/10.14445/22315381/IJETT-V42P202

Smith, A. (2017, January 12). Record shares of Americans now own smartphones, have home

broadband. Pew Research Center. Retrieved from

http://www.pewresearch.org/fact-tank/2017/01/12/evolution-of-technology/
 43

Appendix

Home Assistant Configuration

Rooms:
rooms:
name: Rooms
view: yes
icon: mdi:home
entities:
- group.kitchen
- group.living_room
- group.master_bedroom

k itchen:
name: Kitchen
entities:
- light.kitchen
- light.kitchen_island
- switch.kitchen_cabinet_underlights
- media_player.kitchen

l iving_room:
name: Living Room
entities:
- light.living_room
- light.floor_lamp

m aster_bedroom:
name: Master Bedroom
entities:
- light.joes_nightstand
- light.kasies_nightstand
- light.master_bedroom
- light.master_hanging_lamp
- media_player.master_bedroom
 44

MQTT for Room Presence Detection Configuration

MQTT topic: room_presence

Sample iOS Application Output:

Publishing Message: {"id": "joephone", "name": "

joephone", " distance":
0.0} to subtopic: away
Publishing Message: {"id": "joephone", "name": " joephone", " distance":
1.5} to subtopic: living room (blueberry)

Sample MQTT message:

{
"id": "joephone",
"name": "joephone",
"distance": 1.5
}

Sample MQTT sensor configuration:

sensor:
- platform: mqtt_room
device_id: joephone
name: 'Joe Room Detection'
state_topic: 'room_presence'
timeout: 3
away_timeout: 36000

Detecting Lights On after Entering Room

Database output of entry event:

"13279" "13279" "sensor" "sensor.joe_room_detection" "kitchen

(ice)" "{""distance"": 1.5, ""friendly_name"": ""Joe Room
Detection""}" "2017-10-16 00:17:54.007371" "2017-10-16
00:17:54.007371" "2017-10-16 00:17:54.030369"
 45

Sample SQL query to find events related to room entry:

select *
from states
where entity_id like '%kitchen%'
and
last_changed >= datetime (
(
select last_changed
from states
where state_id = 13279
), '-15 seconds')
and
last_changed <= datetime (
(
select last_changed
from states
where state_id = 13279
), '+15 seconds')

Database output of related results (lights turning on):

"13281" "light" "light.kitchen_island" "on"

"{""brightness"": 99, ""min_mireds"": 154, ""max_mireds"": 500,
""manufacturer_device_model"": ""lutron_p_pkg1_w_wh_d"",
""device_manufacturer"": ""lutron"", ""model_name"": ""Caseta Wireless
Dimmer & Pico"", ""friendly_name"": ""Kitchen Island"",
""supported_features"": 19}" "2017-10-16 00:17:59.030521"
"2017-10-16 00:17:59.030521" "2017-10-16 00:17:59.069588"
"13282" "group" "group.kitchen" "on" "{""entity_id"":
[""light.kitchen"", ""light.kitchen_island"",
""switch.kitchen_cabinet_underlights"", ""media_player.kitchen""],
""order"": 1, ""friendly_name"": ""Kitchen"", ""assumed_state"":
false}" "2017-10-16 00:17:59.057770" "2017-10-16
00:17:59.057770" "2017-10-16 00:17:59.129638"

config.yaml file snippet of automation derived from above query:

automation:
- alias: 'Kitchen Lights On for Joe'
 46

initial_state: 'on'
trigger:
- platform: state
entity_id: sensor.joe_room_detection
to: 'kitchen (ice)'
action:
- service: light.turn_on
entity_id: light.kitchen_island
data:
brightness: 100

Event Samples for State Changes

Event Sample of location state change (entering Kitchen):

{
"entity_id":"sensor.joe_room_detection",
"old_state":{
"entity_id":"sensor.joe_room_detection",
"state":"away",
"attributes":{
"distance":0.0,
"friendly_name":"Joe Room Detection"
},
"last_changed":"2017-10-17T05:01:41.425214+00:00",
"last_updated":"2017-10-17T05:01:41.425214+00:00"
},
"new_state":{
"entity_id":"sensor.joe_room_detection",
"state":"kitchen (ice)",
"attributes":{
"distance":1.5,
"friendly_name":"Joe Room Detection"
},
"last_changed":"2017-10-17T05:01:53.556113+00:00",
"last_updated":"2017-10-17T05:01:53.556113+00:00"
}
}
 47

Event Sample of location state change (exit Kitchen):

{
"entity_id":"sensor.joe_room_detection",
"old_state":{
"entity_id":"sensor.joe_room_detection",
"state":"kitchen (ice)",
"attributes":{
"distance":1.5,
"friendly_name":"Joe Room Detection"
},
"last_changed":"2017-10-17T05:01:53.556113+00:00",
"last_updated":"2017-10-17T05:01:53.556113+00:00"
},
"new_state":{
"entity_id":"sensor.joe_room_detection",
"state":"away",
"attributes":{
"distance":0.0,
"friendly_name":"Joe Room Detection"
},
"last_changed":"2017-10-17T05:02:24.752570+00:00",
"last_updated":"2017-10-17T05:02:24.752570+00:00"
}
}

Event Sample of device state change (turning kitchen island light on):

{
"entity_id":"light.kitchen_island",
"old_state":{
"entity_id":"light.kitchen_island",
"state":"off",
"attributes":{
"manufacturer_device_model":"lutron_p_pkg1_w_wh_d",
"device_manufacturer":"lutron",
"model_name":"Caseta Wireless Dimmer & Pico",
 48

"friendly_name":"Kitchen Island",
"supported_features":19
},
"last_changed":"2017-10-17T05:02:30.919073+00:00",
"last_updated":"2017-10-17T05:02:30.919073+00:00"
},
"new_state":{
"entity_id":"light.kitchen_island",
"state":"on",
"attributes":{
"brightness":255,
"min_mireds":154,
"max_mireds":500,
"manufacturer_device_model":"lutron_p_pkg1_w_wh_d",
"device_manufacturer":"lutron",
"model_name":"Caseta Wireless Dimmer & Pico",
"friendly_name":"Kitchen Island",
"supported_features":19
},
"last_changed":"2017-10-17T13:51:38.853085+00:00",
"last_updated":"2017-10-17T13:51:38.853085+00:00"
}
}