Emotion Analysis#
Introduction#
In this tutorial, we present how to use PyTorch
models in EvaDB to detect faces and classify their emotions. In particular, we focus on detecting faces in a person’s video and classifying their emotions. EvaDB makes it easy to do face detection and emotion classification using its built-in support for PyTorch
models.
In this tutorial, we will showcase an usecase where we chain
the outputs of two AI models in a single query. After detecting faces, we will crop
the bounding box of the detected face and send it to an EmotionDetector
function.
Prerequisites#
To follow along, you will need to set up a local instance of EvaDB via pip.
Connect to EvaDB#
After installing EvaDB, use the following Python code to establish a connection and obtain a cursor
for running EvaQL
queries.
import evadb
cursor = evadb.connect().cursor()
We will assume that the input defhappy
video is loaded into EvaDB
. To download the video and load it into EvaDB
, see the complete emotion analysis notebook on Colab.
Create Face and Emotion Detection Functions#
To create custom FaceDetector
and EmotionDetector
functions, use the CREATE FUNCTION
statement. In these queries, we leverage EvaDB’s built-in support for custom models. We will assume that the files containing these functions are downloaded and stored locally. Now, run the following queries to register these functions:
CREATE UDF IF NOT EXISTS FaceDetector
INPUT (frame NDARRAY UINT8(3, ANYDIM, ANYDIM))
OUTPUT (bboxes NDARRAY FLOAT32(ANYDIM, 4),
scores NDARRAY FLOAT32(ANYDIM))
TYPE FaceDetection
IMPL 'face_detector.py';
CREATE FUNCTION IF NOT EXISTS EmotionDetector
INPUT (frame NDARRAY UINT8(3, ANYDIM, ANYDIM))
OUTPUT (labels NDARRAY STR(ANYDIM),
scores NDARRAY FLOAT32(ANYDIM))
TYPE Classification
IMPL 'emotion_detector.py';
The FaceDetector
function takes a frame
as input (NDARRAY
type) and returns bounding boxes (bboxes
) of detected faces along with corresponding confidence scores (scores
).
The EmotionDetector
function takes a frame
as input (NDARRAY
type) and returns a label along with the corresponding confidence score.
Emotion Analysis Queries#
After the functions are registered in EvaDB
, you can use them in subsequent SQL queries in different ways.
In the following query, we call the FaceDetector
function on every image in the video. The output of the function is stored in the bboxes
and scores
columns of the output DataFrame
.
SELECT id, FaceDetector(data)
FROM HAPPY
WHERE id < 10;
This query returns the faces detected in the first ten frames of the video:
+----------+---------------------+-------------------------+
| happy.id | facedetector.bboxes | facedetector.scores |
+----------+---------------------+-------------------------+
| 0 | [[499 94 761 435] | [0.9999081 0.791269 ] |
| | [237 296 326 399]] | |
| 1 | [[501 95 763 436] | [0.9999021 0.8090978] |
| | [238 296 325 399]] | |
| 2 | [[503 96 767 439]] | [0.9998859] |
| 3 | [[499 90 776 447]] | [0.99995923] |
| 4 | [[498 100 768 444]] | [0.99999774] |
| 5 | [[504 89 778 439] | [0.9999716 0.8267602] |
| | [236 305 324 406]] | |
| 6 | [[500 88 779 451]] | [0.9999136] |
| 7 | [[502 88 780 450]] | [0.99990165] |
| 8 | [[504 90 783 450] | [0.9999368 0.72843623] |
| | [235 308 325 411]] | |
| 9 | [[508 90 785 448] | [0.99992466 0.7014416 ] |
| | [235 309 325 412]] | |
+----------+---------------------+-------------------------+
Chaining Functions in a Single AI Query#
In the following query, we use the output of the FaceDetector
to crop the detected face from the frame and send it to the EmotionDetector
to identify the emotion in that person’s face. Here, Crop
is a built-in function in EvaDB that is used for cropping the given bounding box (bbox
) from the given frame (data
).
We use LATERAL JOIN
clause in the query to map the output of the FaceDetector
to each frame in the HAPPY
video table.
SELECT id, bbox, EmotionDetector(Crop(data, bbox))
FROM HAPPY
JOIN LATERAL UNNEST(FaceDetector(data)) AS Face(bbox, conf)
WHERE id < 15;
Now, the DataFrame
only contains the emotions of the detected faces:
+----------+-------------------+------------------------+------------------------+
| happy.id | Face.bbox | emotiondetector.labels | emotiondetector.scores |
+----------+-------------------+------------------------+------------------------+
| 0 | [499 94 761 435] | happy | 0.9996277093887329 |
| 0 | [237 296 326 399] | neutral | 0.8095587491989136 |
| 1 | [501 95 763 436] | happy | 0.9996333122253418 |
| 1 | [238 296 325 399] | neutral | 0.5969993472099304 |
| 2 | [503 96 767 439] | happy | 0.999674916267395 |
| 3 | [499 90 776 447] | happy | 0.9996751546859741 |
| 4 | [498 100 768 444] | happy | 0.9996353387832642 |
| 5 | [504 89 778 439] | happy | 0.9996645450592041 |
| 5 | [236 305 324 406] | neutral | 0.7656209468841553 |
| 6 | [500 88 779 451] | happy | 0.999667763710022 |
| 7 | [502 88 780 450] | happy | 0.9996945858001709 |
| 8 | [504 90 783 450] | happy | 0.9996901750564575 |
| 8 | [235 308 325 411] | neutral | 0.6952931880950928 |
| 9 | [508 90 785 448] | happy | 0.9996936321258545 |
| 9 | [235 309 325 412] | neutral | 0.6790497899055481 |
| 10 | [508 89 788 449] | happy | 0.9997019171714783 |
| 10 | [235 311 323 413] | neutral | 0.7205461263656616 |
| 11 | [514 87 789 454] | happy | 0.9997119307518005 |
| 12 | [513 87 789 454] | happy | 0.9997120499610901 |
| 13 | [513 87 789 456] | happy | 0.9997060894966125 |
| 14 | [515 88 790 454] | happy | 0.9997135996818542 |
+----------+-------------------+------------------------+------------------------+
What’s Next?#
👋 EvaDB’s vision is to bring AI inside your database system and make it easy to build fast AI-powered apps. If you liked this tutorial and are excited about our vision, show some ❤️ by:
🐙 giving a ⭐ for the EvaDB repository on Github: https://github.com/georgia-tech-db/evadb
📟 engaging with the EvaDB community on Slack to ask questions and share your ideas and thoughts: https://evadb.ai/community
🎉 contributing to EvaDB by developing cool applications/integrations: https://github.com/georgia-tech-db/evadb/issues
🐦 following us on Twitter: https://twitter.com/evadb_ai
📝 following us on Medium: https://medium.com/evadb-blog