Intruder Detection System
I have recently come up with an idea to develop an affordable intruder detection system because my deliveries have been stolen from my doorstep over and over again and I wanna find out who did this. To do this, I have chosen ESP32-Cam as the hardware platform for its attractive price (around $6 with delivery fee) and IoT capability.
ESP32 Cam
ESP32-Cam is an affordable and fun IoT hardware to play around with, the official arduino example on this Camera module allows taking still picture or live streaming video on your browser, what’s even better is that it also comes with facial detection and recognition which is the least thing you would expect on this tiny cheap microcontroller that has already spared a lot of on-board resources for WiFi connection as well as taking and processing the image.
Getting Started with the hardware
If you are new to this hardware, make sure to check out randomnerdtutorials for a better understanding of this device and to get started.
Motion Detection Implementation
The built-in facial recognition feature has s lot of limitations. From my testing, the object’s face must be in close proximity to the camera ( about less than 1 meter) and that is not gonna work when porch pirate usually hide their face and move very swiftly, thus this feature is not suitable for this task.
Some makers has created motion detection system using an additional PIR sensor, this works but it definitely increase the cost and size which make it easier to be spotted.
After some digging on the internet, I found this blog post that use downscaled grayscale image to detect if " considerable portion of the image changed from one frame to the next ". This is what the camera sees and process,
FABULOUS! Looks like I have found the missing piece to my project… or have I?
After testing the algorithm provided, I realized this also has its own limitations… It can only take grayscale image at 320x240 resolution, this might be good for processing as fewer pixels needed to be processed, but image at this quality is simply too bad to identify the porch pirate.
Wait… can’t we switch to the colored JPEG format as the camera output?
After consulting the ESP32’s official documentation and the IDF’s source code on Github, I have managed to configure the camera and let it output 2 types of images for both motion detection algorithm and human viewing, here is a sample of the 320x240 resolution JPEG format image taken by the camera,
Not bad right?
Now it’s time to complete the rest and make it an IoT project.
IoT and the Alert System
Since ESP32 cam has WiFi connection, I must not let it go wasted. Here I am using EMAIL to upload the JPEG image taken by the camera whenever there is motion detected in front of the camera. To do this, we have to make use of the onboard SPI Flash memory, File System and the SMTP (Simple Message Transfer Protocol) library to store the image temporarily on Flash memory and send it out as attachment using email. To learn how to use Flash memory and email on ESP32, check out this link.
Source Code
//
// Note this code currently only support ESP32 Cam from AI-Thinker
//
// Credits to randomnerdtutorials and SIMONE from eloquentarduino
// for their idea and code on email and motion detection algorithm
//
#define CAMERA_MODEL_AI_THINKER
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27
#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22
#include "esp_camera.h"
#include "SPI.h"
#include "soc/soc.h" // Disable brownour problems
#include "soc/rtc_cntl_reg.h" // Disable brownour problems
#include "ESP32_MailClient.h"
#include <FS.h>
#include <SPIFFS.h>
#include <WiFi.h>
#define FRAME_SIZE FRAMESIZE_QVGA
#define WIDTH 320
#define HEIGHT 240
#define BLOCK_SIZE 10
#define W (WIDTH / BLOCK_SIZE)
#define H (HEIGHT / BLOCK_SIZE)
#define BLOCK_DIFF_THRESHOLD 0.2
#define IMAGE_DIFF_THRESHOLD 0.2
#define DEBUG 1
uint16_t prev_frame[H][W] = { 0 };
uint16_t current_frame[H][W] = { 0 };
bool setup_camera(framesize_t);
bool capture_still();
bool motion_detect();
void update_frame();
void print_frame(uint16_t frame[H][W]);
const char* ssid = "xxxxxxxxx";
const char* password = "xxxxxxxxxxx";
//Note this part of code is optional, if you face issue connecting to wifi, feel free to delete this part to get dynamic IP address
// Set your Static IP address
IPAddress local_IP(192, 168, 10, 184);
// Set your Gateway IP address
IPAddress gateway(192, 168, 10, 1);
IPAddress subnet(255, 255, 0, 0);
IPAddress primaryDNS(8, 8, 8, 8); //optional
IPAddress secondaryDNS(8, 8, 4, 4); //optional
// To send Email using Gmail use port 465 (SSL) and SMTP Server smtp.gmail.com
// YOU MUST ENABLE less secure app option <https://myaccount.google.com/lesssecureapps?pli=1>
#define emailSenderAccount "xxxxxxxxxxxx"
#define emailSenderPassword "xxxxxxxxxxxx"
#define smtpServer "smtp.gmail.com"
#define smtpServerPort 465
#define emailSubject "ESP32-CAM Photo Captured"
#define emailRecipient "xxxxxxxxxxxxx"
// The Email Sending data object contains config and data to send
SMTPData smtpData;
// Photo File Name to save in SPIFFS
#define FILE_PHOTO "/photo.jpg"
/**
*
*/
void setup() {
WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 0); //disable brownout detector
Serial.begin(115200);
Serial.println(setup_camera(FRAME_SIZE, true) ? "--GRAYSCALE Camera Setup OK--" : "--ERR CAM INIT--");
if (!SPIFFS.begin(true)) {
Serial.println("An Error has occurred while mounting SPIFFS");
ESP.restart();
}
else {
delay(500);
Serial.println("SPIFFS mounted successfully");
}
if(!WiFi.config(local_IP, gateway, subnet, primaryDNS, secondaryDNS)) {
Serial.println("STA Failed to configure");
}
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
}
/**
*
*/
void loop() {
if (!capture_still()) {
Serial.println("Failed capture");
delay(3000);
return;
}
//delay(200);
if (motion_detect()) {
Serial.println("## Motion detected ##");
Serial.println(esp_camera_deinit()? "Deinit success" : "Deinit failed");
// Re-configure the camera to take JPEG photo
Serial.println(setup_camera(FRAME_SIZE, false) ? "--JPEG Camera Setup OK--" : "--ERR CAM INIT--");
capturePhotoSaveSpiffs();
sendPhoto();
Serial.println(esp_camera_deinit()? "Deinit success" : "Deinit failed");
// set CAM back to grayscale for motion detection
Serial.println(setup_camera(FRAME_SIZE, true) ? "--GRAYSCALE Camera Setup OK--" : "--ERR CAM INIT--");
}
update_frame();
Serial.println("=================");
//delay(200);
}
/**
*
*/
bool setup_camera(framesize_t frameSize, bool forMotion) {
camera_config_t config;
config.ledc_channel = LEDC_CHANNEL_0;
config.ledc_timer = LEDC_TIMER_0;
config.pin_d0 = Y2_GPIO_NUM;
config.pin_d1 = Y3_GPIO_NUM;
config.pin_d2 = Y4_GPIO_NUM;
config.pin_d3 = Y5_GPIO_NUM;
config.pin_d4 = Y6_GPIO_NUM;
config.pin_d5 = Y7_GPIO_NUM;
config.pin_d6 = Y8_GPIO_NUM;
config.pin_d7 = Y9_GPIO_NUM;
config.pin_xclk = XCLK_GPIO_NUM;
config.pin_pclk = PCLK_GPIO_NUM;
config.pin_vsync = VSYNC_GPIO_NUM;
config.pin_href = HREF_GPIO_NUM;
config.pin_sscb_sda = SIOD_GPIO_NUM;
config.pin_sscb_scl = SIOC_GPIO_NUM;
config.pin_pwdn = PWDN_GPIO_NUM;
config.pin_reset = RESET_GPIO_NUM;
config.xclk_freq_hz = 20000000;
// if for motion detection, then set it to grayscale, else JPEG
if(forMotion){
config.pixel_format = PIXFORMAT_GRAYSCALE;
} else {
config.pixel_format = PIXFORMAT_JPEG;
}
config.frame_size = frameSize;
config.jpeg_quality = 12;
config.fb_count = 1;
bool ok = esp_camera_init(&config) == ESP_OK;
sensor_t *sensor = esp_camera_sensor_get();
sensor->set_framesize(sensor, frameSize);
return ok;
}
/**
* Capture image and do down-sampling
*/
bool capture_still() {
camera_fb_t *frame_buffer = esp_camera_fb_get();
if (!frame_buffer)
return false;
// set all 0s in current frame
for (int y = 0; y < H; y++)
for (int x = 0; x < W; x++)
current_frame[y][x] = 0;
// down-sample image in blocks
for (uint32_t i = 0; i < WIDTH * HEIGHT; i++) {
const uint16_t x = i % WIDTH;
const uint16_t y = floor(i / WIDTH);
const uint8_t block_x = floor(x / BLOCK_SIZE);
const uint8_t block_y = floor(y / BLOCK_SIZE);
const uint8_t pixel = frame_buffer->buf[i];
const uint16_t current = current_frame[block_y][block_x];
// average pixels in block (accumulate)
current_frame[block_y][block_x] += pixel;
}
// average pixels in block (rescale)
for (int y = 0; y < H; y++)
for (int x = 0; x < W; x++)
current_frame[y][x] /= BLOCK_SIZE * BLOCK_SIZE;
#if DEBUG
Serial.println("Current frame:");
print_frame(current_frame);
Serial.println("---------------");
#endif
return true;
}
/**
* Compute the number of different blocks
* If there are enough, then motion happened
*/
bool motion_detect() {
uint16_t changes = 0;
const uint16_t blocks = (WIDTH * HEIGHT) / (BLOCK_SIZE * BLOCK_SIZE);
for (int y = 0; y < H; y++) {
for (int x = 0; x < W; x++) {
float current = current_frame[y][x];
float prev = prev_frame[y][x];
float delta = abs(current - prev) / prev;
if (delta >= BLOCK_DIFF_THRESHOLD) {
#if DEBUG
Serial.print("diff\\t");
Serial.print(y);
Serial.print('\\t');
Serial.println(x);
#endif
changes += 1;
}
}
}
Serial.print("Changed ");
Serial.print(changes);
Serial.print(" out of ");
Serial.println(blocks);
return (1.0 * changes / blocks) > IMAGE_DIFF_THRESHOLD;
}
/**
* Copy current frame to previous
*/
void update_frame() {
for (int y = 0; y < H; y++) {
for (int x = 0; x < W; x++) {
prev_frame[y][x] = current_frame[y][x];
}
}
}
/**
* For serial debugging
* @param frame
*/
void print_frame(uint16_t frame[H][W]) {
for (int y = 0; y < H; y++) {
for (int x = 0; x < W; x++) {
Serial.print(frame[y][x]);
Serial.print('\\t');
}
Serial.println();
}
}
// Check if photo capture was successful
bool checkPhoto( fs::FS &fs ) {
File f_pic = fs.open( FILE_PHOTO );
unsigned int pic_sz = f_pic.size();
return ( pic_sz > 100 );
}
// Capture Photo and Save it to SPIFFS
void capturePhotoSaveSpiffs( void ) {
camera_fb_t * fb = NULL; // pointer
bool ok = 0; // Boolean indicating if the picture has been taken correctly
do {
// Take a photo with the camera
Serial.println("Taking a photo...");
fb = esp_camera_fb_get();
if (!fb) {
Serial.println("Camera capture failed");
return;
}
// Photo file name
Serial.printf("Picture file name: %s\\n", FILE_PHOTO);
File file = SPIFFS.open(FILE_PHOTO, FILE_WRITE);
// Insert the data in the photo file
if (!file) {
Serial.println("Failed to open file in writing mode");
}
else {
file.write(fb->buf, fb->len); // payload (image), payload length
Serial.print("The picture has been saved in ");
Serial.print(FILE_PHOTO);
Serial.print(" - Size: ");
Serial.print(file.size());
Serial.println(" bytes");
}
// Close the file
file.close();
esp_camera_fb_return(fb);
// check if file has been correctly saved in SPIFFS
ok = checkPhoto(SPIFFS);
} while ( !ok );
}
void sendPhoto( void ) {
// Preparing email
Serial.println("Sending email...");
// Set the SMTP Server Email host, port, account and password
smtpData.setLogin(smtpServer, smtpServerPort, emailSenderAccount, emailSenderPassword);
// Set the sender name and Email
smtpData.setSender("ESP32-CAM", emailSenderAccount);
// Set Email priority or importance High, Normal, Low or 1 to 5 (1 is highest)
smtpData.setPriority("High");
// Set the subject
smtpData.setSubject(emailSubject);
// Set the email message in HTML format
smtpData.setMessage("<h2>Photo captured with ESP32-CAM and attached in this email.</h2>", true);
// Set the email message in text format
//smtpData.setMessage("Photo captured with ESP32-CAM and attached in this email.", false);
// Add recipients, can add more than one recipient
smtpData.addRecipient(emailRecipient);
//smtpData.addRecipient(emailRecipient2);
// Add attach files from SPIFFS
smtpData.addAttachFile(FILE_PHOTO, "image/jpg");
// Set the storage type to attach files in your email (SPIFFS)
smtpData.setFileStorageType(MailClientStorageType::SPIFFS);
smtpData.setSendCallback(sendCallback);
// Start sending Email, can be set callback function to track the status
if (!MailClient.sendMail(smtpData))
Serial.println("Error sending Email, " + MailClient.smtpErrorReason());
// Clear all data from Email object to free memory
smtpData.empty();
}
// Callback function to get the Email sending status
void sendCallback(SendStatus msg) {
//Print the current status
Serial.println(msg.info());
}
Results
With the source code provided, simply change the wifi information to yours and set up the sender and recipient email (the sender email has to be set to allow “less secure application”). Once uploaded to ESP32 using Arduino IDE, wait until it connects to your WiFi, and then you can start playing around with it, you should get a lot of email at the start because the camera captures motion and keep spamming emails, but once the ESP32 Cam is settled and face still background, you will stop getting the email in a while.
