Computer Science - Story Archives: Henry Kautz Joins Computer Science Faculty

Periodically, the Science, Technology, Engineering, & Mathematics Portal showcases new Computer Science discoveries. As we post new material to the Computer Science Portal, we move other Computer Science stories to the archives.

Return to Computer Science Story Archives | Return to Computer Science Portal

Artificial Intelligence to Help People
    with Cognitive Disabilities

    (Professor Henry Kautz,)
    Computer Science)

     by Lois H. Gresh

February 21, 2007: Professor Henry Kautz of the University of Rochester Computer Science Department is an expert at devising ways to help people with cognitive impairments travel alone and accomplish their daily tasks. The technology he and his students are developing could help people with Alzheimer's Disease age gracefully in their own homes, help people with mental retardation travel safely to work, and help people with traumatic brain injury learn to monitor their own behavior.

Professor Kautz' research combines the fields of ubiquitous computing, which explores new kinds of distributed sensing and computing devices; artificial intelligence, which has devised probabilistic reasoning algorithms that can be used to model and infer human behavior from sensor data; human-computer interaction, which studies ways to make computing devices more natural and easier to use; and assistive technology, which creates systems and devices to help people with disabilities.

The interdisciplinary nature of his work is reflected in the wide variety of journals and conferences in which his work is published, ranging from the ACM SIGACCESS Conference on Computers and Accessibility to the International Joint Conference on Artificial Intelligence.

Professor Kautz' interest in artificial intelligence began in graduate school, which he began at the University of Toronto and ended at the University of Rochester, where he earned his doctorate in 1987. "My work was mainly theoretical," Kautz says, "until my father developed early-onset Alzheimer's Disease. He was able to live at home and perform many activities for years because my mother did such a good job of monitoring his behavior and providing simple prompts when he forgot what he was doing or became confused. When he finally did need to move into a nursing home, I saw that many of the other patients were much more capable than he was at that point, but had been institutionalized because they had no one who could provide the kind of monitoring and prompting my father had received. I began to think about how one could create a kind of cognitive assistant using AI technology."

The U.S. Census Bureau predicts that, by 2050, the United States will have approximately 15 million male and 20 million female citizens over age 80. In addition, says Kautz, "The number of people with Alzheimer's Disease is rapidly growing. In 1950, at most 200,000 people in the U.S. had Alzheimer's disease. This number increased to 500,000 by 1975 and stands at 4 million today. By 2050, the number of Alzheimer's patients in the U.S. is expected to be 15 million, out of a world total of 80 million."

Image -- Population Pyramid for the Year 2050

Professor Kautz defines the goal of assisted cognition as "developing computer systems that improve the independence and safety of people suffering from cognitive limitations by understanding human behavior from sensor data, actively prompting, warning, and advising; and alerting caregivers as necessary."

An AI Wayfinding Assistant

In one of his projects, Professor Kautz gives GPS-enabled cell phones the intelligence necessary to tell when a user is lost and needs help getting back on the right path. The prototype system he and his students have built can automatically learn the transportation plans a user commonly performs, and then notice when the user's routine is changed in a way that might indicate an error.

Image -- The Activity Compass Prototype Wayfinding System
The system has inferred that the user is about to begin a transportation plan and presents a menu of the most likely destinations.

This type of system could be useful for people suffering from mental retardation, traumatic brain injury, or early stage Alzheimer's Disease. Many people with these conditions become socially isolated, because they cannot drive and public transportation is cognitively challenging for them. They find it difficult to learn bus routes and numbers, make transfers between vehicles, and recover from their own mistakes, such as taking a wrong turn or getting on the wrong bus.

The system being developed, called the Activity Compass, can infer the user's location and mode of transportation, predict the user's destination, and detect errors along the way. The device begins with a general model of transportation plans that a person can change from walking to riding at a bus stop (to name one example) and includes the locations of streets and bus stops. The user trains the system simply by carrying it for a few days, which allows the system to learn the user's typical destinations (such as home, work, or shopping) and the ways the user travels between the destinations. The user model is probabilistic, so that at any point in time the system can use its current GPS data to infer the user's most likely destination and the actions the user should take to get there (such as turning left at the next street corner, or getting off a bus at a particular stop).

Video -- Predicting Destinations and Trip Segments
Blue circles are predicted destinations based on the user's current position and movement. The user travels toward the first predicted destination (his home), but then he turns left, so the system calculates a new predicted destination (his friend's house). Note that this video illustrates the Activity Compass' reasoning process, but it is not shown to the user. (Please be patient, as this video may take a couple of minutes to load on a low-speed connection.)

When the Activity Compass predicts that the user is about to begin a trip, it presents the user with photographs of the most likely destinations. If the user selects one (for example, the user's home), then the system can detect if the user deviates from the expected plan, and if so, proactively offer step-by-step guidance. Even if the user does not explicitly choose a destination, the system can still determine that the user may need help if the user's movements do not correspond to a path to any likely destination.

Video -- Error Detection
This video illustrates the Activity Compass tracking a user who is taking a bus to his home. When the user fails to get off at the expected stop, the system determines that a user error is likely. Note that this video illustrates the Activity Compass' reasoning process, but it is not shown to the user. (Please be patient, as this video may take a minute to load on a low-speed connection.)

Helping the Elderly with Daily Tasks

A person with Alzheimer's Disease may have to go to a nursing home long before he really needs to move. Many sufferers do not have a family member home 24 hours a day to help monitor activities. Furthermore, family caregivers often become ill themselves from the stress of constant vigilance.

In the near future, home sensors and AI technology will be commonly used to monitor and assist with the daily activities of people with early-stage Alzheimer's Disease. The system will determine, for example, if the user is making coffee, setting the table, or eating breakfast. If the user makes an error -- for example, forgetting to turn on the kettle when making tea -- the system will provide a helpful prompt. Furthermore, if the user's pattern of activities over time shows a turn for the worse -- for example, skipping meals or sleeping irregularly -- the system will notify family and medical caregivers.

The hardware for Professor Kautz' system is mainly off-the-shelf and available now. For several years he has worked with Intel researchers who are extending the state of the art of Radio Frequency Identification (RFID) technology. RFID chips are about the size of a postage stamp, and they contain no batteries. A wireless RFID reader in the vicinity of the chips simply reads identification numbers off them and sends the numbers to a computer. In the scenario imagined by Professor Kautz, objects in a person's home are labeled with tiny, inexpensive, nearly-indestructible RFID tags. The user wears a bracelet or watch that contains an RFID tag reader that can sense when the user touches a tagged object. The stream of object "touches" lets the system infer the user's activities.

Image -- An RFID Tag Reader in the Form of a Bracelet
This working prototype, developed by Intel Research, is not yet commercially available.

Knowledge of how activities are performed is represented in the system by a Hidden Markov Model (HMM). In the model, a hidden variable represents the activity the user is performing, while an observed variable represents the object touches received from the RFID tag reader. The system can then infer the probability that each activity will occur over designated time period. Unlike much of the previous activity recognition research that designed systems to recognize a single kind of activity, the RFID tag-based method works for essentially all daily activities identified in the health care literature.

Image -- The General Form of an HMM
This HMM represents daily activities (left) and a sample output of the system (right) as the user perform various morning chores.

Future research will delve into other artificial intelligence applications that affect health care. For example, heart rate, respiration, and temperature can all be monitored with sensors. An AI system could tell patients what to do if heart rate escalates, for example, or respiration becomes labored. Professor Kautz is working closely with the University of Rochester's Center for Future Health, including Professor James Allen and Research Scientist George Ferguson of the Department of Computer Science.

Professor Kautz' Assisted Cognition Project is supported by grants from the National Science Foundation, Intel, the National Institute on Disability and Rehabilitation Research, and the Department of Defense.

Additional Details

For further details, see:

Matthai Philipose, Kenneth P. Fishkin, Mike Perkowitz, Donald J. Patterson, Dirk Hahnel, Dieter Fox, and Henry Kautz, "Inferring ADLs from Interactions with Objects," IEEE Pervasive Computing, Volume 3, Number 4, Pages 50-56, 2004.

Donald J. Patterson, Lin Liao, Krzysztof Gajos, Michael Collier, Nik Livic, Katherine Olson, Shiaokai Wang, Dieter Fox, and Henry Kautz, "Opportunity Knocks: a System to Provide Cognitive Assistance with Transportation Services," Sixth International Conference on Ubiquitous Computing (UBICOMP 2004), Nottingham, England, 2004.

Alan L. Liu, Harlan Hile, Henry Kautz, Gaetano Borriello, Pat A. Brown, Mark Harniss, and Kurt Johnson, "Indoor Wayfinding: Developing a Functional Interface for Individuals with Cognitive Impairments," Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2006), Portland, OR, 2006.

For more information, please contact:
   Professor Henry Kautz
   Email: kautz@cs.rochester.edu
   Faculty Webpage: http://www.cs.rochester.edu/u/kautz/

Return to Top

Return to Computer Science Story Archives | Return to Computer Science Portal

Return to Science, Technology, Engineering, & Math Portal Homepage