diff --git a/.gitignore b/.gitignore index 05a8e1cf..3a9ad0b8 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,11 @@ node_modules .rpt2_cache .env* + tmp proto weights_uncompressed -weights_unused \ No newline at end of file +weights_unused +docs +out +build \ No newline at end of file diff --git a/.npmignore b/.npmignore index 69b21017..48c03f67 100644 --- a/.npmignore +++ b/.npmignore @@ -6,5 +6,7 @@ proto weights weights_uncompressed weights_unused +src test -tools \ No newline at end of file +tools +docs \ No newline at end of file diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 00000000..70b26bae --- /dev/null +++ b/.travis.yml @@ -0,0 +1,25 @@ +sudo: required +language: node_js +node_js: + #- "node" + - "12" + - "11" + - "10" + - "8" + # node 6 is not compatible with tfjs-node + # - "6" +services: + - xvfb +env: + global: + - BACKEND_CPU=true EXCLUDE_UNCOMPRESSED=true + matrix: + - ENV=browser + - ENV=node +addons: + chrome: stable +install: npm install +script: + - if [ $ENV == 'browser' ]; then npm run test-browser; fi + - if [ $ENV == 'node' ]; then npm run test-node; fi + - npm run build \ No newline at end of file diff --git a/README.md b/README.md index 8d3ea09a..df98850e 100644 --- a/README.md +++ b/README.md @@ -1,177 +1,204 @@ # face-api.js +[![Build Status](https://travis-ci.org/justadudewhohacks/face-api.js.svg?branch=master)](https://travis-ci.org/justadudewhohacks/face-api.js) [![Slack](https://slack.bri.im/badge.svg)](https://slack.bri.im) -**JavaScript API for face detection and face recognition in the browser implemented on top of the tensorflow.js core API ([tensorflow/tfjs-core](https://github.com/tensorflow/tfjs-core))** +**JavaScript face recognition API for the browser and nodejs implemented on top of tensorflow.js core ([tensorflow/tfjs-core](https://github.com/tensorflow/tfjs))** -Check out my face-api.js tutorials: +![faceapi](https://user-images.githubusercontent.com/31125521/57224752-ad3dc080-700a-11e9-85b9-1357b9f9bca4.gif) + +## **[Click me for Live Demos!](https://justadudewhohacks.github.io/face-api.js/)** + +## Tutorials * **[face-api.js — JavaScript API for Face Recognition in the Browser with tensorflow.js](https://itnext.io/face-api-js-javascript-api-for-face-recognition-in-the-browser-with-tensorflow-js-bcc2a6c4cf07)** * **[Realtime JavaScript Face Tracking and Face Recognition using face-api.js’ MTCNN Face Detector](https://itnext.io/realtime-javascript-face-tracking-and-face-recognition-using-face-api-js-mtcnn-face-detector-d924dd8b5740)** +* **[Realtime Webcam Face Detection And Emotion Recognition - Video](https://youtu.be/CVClHLwv-4I)** +* **[Easy Face Recognition Tutorial With JavaScript - Video](https://youtu.be/AZ4PdALMqx0)** +* **[Using face-api.js with Vue.js and Electron](https://medium.com/@andreas.schallwig/do-not-laugh-a-simple-ai-powered-game-3e22ad0f8166)** +* **[Add Masks to People - Gant Laborde on Learn with Jason](https://www.learnwithjason.dev/fun-with-machine-learning-pt-2)** -**Check out the live demos [here](https://justadudewhohacks.github.io/face-api.js/)!** - -Table of Contents: +## Table of Contents +* **[Features](#features)** * **[Running the Examples](#running-the-examples)** -* **[About the Package](#about-the-package)** - * **[Face Detection - SSD Mobilenet v1](#about-face-detection-ssd)** - * **[Face Detection - Tiny Yolo v2](#about-face-detection-yolo)** - * **[Face Detection & 5 Point Face Landmarks - MTCNN](#about-face-detection-mtcnn)** - * **[Face Recognition](#about-face-recognition)** - * **[68 Point Face Landmark Detection](#about-face-landmark-detection)** -* **[Usage](#usage)** - * **[Loading the Models](#usage-load-models)** - * **[Face Detection - SSD Mobilenet v1](#usage-face-detection-ssd)** - * **[Face Detection - Tiny Yolo v2](#usage-face-detection-yolo)** - * **[Face Detection & 5 Point Face Landmarks - MTCNN](#usage-face-detection-mtcnn)** - * **[Face Recognition](#usage-face-recognition)** - * **[68 Point Face Landmark Detection](#usage-face-landmark-detection)** - * **[Shortcut Functions for Full Face Description](#shortcut-functions)** - -## Examples +* **[face-api.js for the Browser](#face-api.js-for-the-browser)** +* **[face-api.js for Nodejs](#face-api.js-for-nodejs)** +* **[Usage](#getting-started)** + * **[Loading the Models](#getting-started-loading-models)** + * **[High Level API](#high-level-api)** + * **[Displaying Detection Results](#getting-started-displaying-detection-results)** + * **[Face Detection Options](#getting-started-face-detection-options)** + * **[Utility Classes](#getting-started-utility-classes)** + * **[Other Useful Utility](#other-useful-utility)** +* **[Available Models](#models)** + * **[Face Detection](#models-face-detection)** + * **[Face Landmark Detection](#models-face-landmark-detection)** + * **[Face Recognition](#models-face-recognition)** + * **[Face Expression Recognition](#models-face-expression-recognition)** + * **[Age Estimation and Gender Recognition](#models-age-and-gender-recognition)** +* **[API Documentation](https://justadudewhohacks.github.io/face-api.js/docs/globals.html)** -### Face Recognition +# Features -![preview_face-detection-and-recognition](https://user-images.githubusercontent.com/31125521/41526995-1a90e4e6-72e6-11e8-96d4-8b2ccdee5f79.gif) +## Face Recognition -![preview_face-recognition_gif](https://user-images.githubusercontent.com/31125521/40313021-c3afdfec-5d14-11e8-86df-cf89a00668e2.gif) +![face-recognition](https://user-images.githubusercontent.com/31125521/57297377-bfcdfd80-70cf-11e9-8afa-2044cb167bff.gif) -### Face Similarity +## Face Landmark Detection -![preview_face-similarity](https://user-images.githubusercontent.com/31125521/40316573-0a1190c0-5d1f-11e8-8797-f6deaa344523.gif) +![face_landmark_detection](https://user-images.githubusercontent.com/31125521/57297731-b1ccac80-70d0-11e9-9bd7-59d77f180322.jpg) -### Face Landmarks +## Face Expression Recognition -![face_landmarks_boxes_1](https://user-images.githubusercontent.com/31125521/46063403-fff9f480-c16c-11e8-900f-e4b7a3828d1d.jpg) -![face_landmarks_boxes_2](https://user-images.githubusercontent.com/31125521/46063404-00928b00-c16d-11e8-8f29-e9c50afd2bc8.jpg) +![preview_face-expression-recognition](https://user-images.githubusercontent.com/31125521/50575270-f501d080-0dfb-11e9-9676-8f419efdade4.png) -![preview_face_landmarks](https://user-images.githubusercontent.com/31125521/41507950-e121b05e-723c-11e8-89f2-d8f9348a8e86.png) +## Age Estimation & Gender Recognition -### Live Face Detection +![age_gender_recognition](https://user-images.githubusercontent.com/31125521/57297736-b5603380-70d0-11e9-873d-8b6c7243eb64.jpg) -**SSD Mobilenet v1** - -![preview_video-facedetection](https://user-images.githubusercontent.com/31125521/41238649-bbf10046-6d96-11e8-9041-1de46c6adccd.jpg) + -**MTCNN** +# Running the Examples -![mtcnn-preview](https://user-images.githubusercontent.com/31125521/42756818-0a41edaa-88fe-11e8-9033-8cd141b0fa09.gif) +Clone the repository: - +``` bash +git clone https://github.com/justadudewhohacks/face-api.js.git +``` -## Running the Examples +## Running the Browser Examples ``` bash -cd examples +cd face-api.js/examples/examples-browser npm i npm start ``` Browse to http://localhost:3000/. - - -## About the Package +## Running the Nodejs Examples - +``` bash +cd face-api.js/examples/examples-nodejs +npm i +``` -### Face Detection - SSD Mobilenet v1 +Now run one of the examples using ts-node: -For face detection, this project implements a SSD (Single Shot Multibox Detector) based on MobileNetV1. The neural net will compute the locations of each face in an image and will return the bounding boxes together with it's probability for each face. This face detector is aiming towards obtaining high accuracy in detecting face bounding boxes instead of low inference time. +``` bash +ts-node faceDetection.ts +``` -The face detection model has been trained on the [WIDERFACE dataset](http://mmlab.ie.cuhk.edu.hk/projects/WIDERFace/) and the weights are provided by [yeephycho](https://github.com/yeephycho) in [this](https://github.com/yeephycho/tensorflow-face-detection) repo. +Or simply compile and run them with node: - +``` bash +tsc faceDetection.ts +node faceDetection.js +``` -### Face Detection - Tiny Yolo v2 + -The Tiny Yolo v2 implementation is a very performant face detector, which can easily adapt to different input image sizes, thus can be used as an alternative to SSD Mobilenet v1 to trade off accuracy for performance (inference time). In general the models ability to locate smaller face bounding boxes is not as accurate as SSD Mobilenet v1. +# face-api.js for the Browser -The face detector has been trained on a custom dataset of ~10K images labeled with bounding boxes and uses depthwise separable convolutions instead of regular convolutions, which ensures very fast inference and allows to have a quantized model size of only 1.7MB making the model extremely mobile and web friendly. Thus, the Tiny Yolo v2 face detector should be your GO-TO face detector on mobile devices. +Simply include the latest script from [dist/face-api.js](https://github.com/justadudewhohacks/face-api.js/tree/master/dist). - +Or install it via npm: -### Face Detection & 5 Point Face Landmarks - MTCNN +``` bash +npm i face-api.js +``` -MTCNN (Multi-task Cascaded Convolutional Neural Networks) represents an alternative face detector to SSD Mobilenet v1 and Tiny Yolo v2, which offers much more room for configuration. By tuning the input parameters, MTCNN is able to detect a wide range of face bounding box sizes. MTCNN is a 3 stage cascaded CNN, which simultaneously returns 5 face landmark points along with the bounding boxes and scores for each face. By limiting the minimum size of faces expected in an image, MTCNN allows you to process frames from your webcam in realtime. Additionally with the model size is only 2MB. + -MTCNN has been presented in the paper [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://kpzhang93.github.io/MTCNN_face_detection_alignment/paper/spl.pdf) by Zhang et al. and the model weights are provided in the official [repo](https://github.com/kpzhang93/MTCNN_face_detection_alignment) of the MTCNN implementation. +# face-api.js for Nodejs - +We can use the equivalent API in a nodejs environment by polyfilling some browser specifics, such as HTMLImageElement, HTMLCanvasElement and ImageData. The easiest way to do so is by installing the node-canvas package. -### Face Recognition +Alternatively you can simply construct your own tensors from image data and pass tensors as inputs to the API. -For face recognition, a ResNet-34 like architecture is implemented to compute a face descriptor (a feature vector with 128 values) from any given face image, which is used to describe the characteristics of a persons face. The model is **not** limited to the set of faces used for training, meaning you can use it for face recognition of any person, for example yourself. You can determine the similarity of two arbitrary faces by comparing their face descriptors, for example by computing the euclidean distance or using any other classifier of your choice. +Furthermore you want to install @tensorflow/tfjs-node (not required, but highly recommended), which speeds things up drastically by compiling and binding to the native Tensorflow C++ library: -The neural net is equivalent to the **FaceRecognizerNet** used in [face-recognition.js](https://github.com/justadudewhohacks/face-recognition.js) and the net used in the [dlib](https://github.com/davisking/dlib/blob/master/examples/dnn_face_recognition_ex.cpp) face recognition example. The weights have been trained by [davisking](https://github.com/davisking) and the model achieves a prediction accuracy of 99.38% on the LFW (Labeled Faces in the Wild) benchmark for face recognition. +``` bash +npm i face-api.js canvas @tensorflow/tfjs-node +``` - +Now we simply monkey patch the environment to use the polyfills: -### 68 Point Face Landmark Detection +``` javascript +// import nodejs bindings to native tensorflow, +// not required, but will speed up things drastically (python required) +import '@tensorflow/tfjs-node'; -This package implements a very lightweight and fast, yet accurate 68 point face landmark detector. The default model has a size of only 350kb and the tiny model is only 80kb. Both models employ the ideas of depthwise separable convolutions as well as densely connected blocks. The models have been trained on a dataset of ~35k face images labeled with 68 face landmark points. +// implements nodejs wrappers for HTMLCanvasElement, HTMLImageElement, ImageData +import * as canvas from 'canvas'; - +import * as faceapi from 'face-api.js'; -## Usage +// patch nodejs environment, we need to provide an implementation of +// HTMLCanvasElement and HTMLImageElement +const { Canvas, Image, ImageData } = canvas +faceapi.env.monkeyPatch({ Canvas, Image, ImageData }) +``` -Get the latest build from dist/face-api.js or dist/face-api.min.js and include the script: + -``` html - -``` +# Getting Started -Or install the package: + -``` bash -npm i face-api.js -``` +## Loading the Models - +All global neural network instances are exported via faceapi.nets: -### Loading the Models +``` javascript +console.log(faceapi.nets) +// ageGenderNet +// faceExpressionNet +// faceLandmark68Net +// faceLandmark68TinyNet +// faceRecognitionNet +// ssdMobilenetv1 +// tinyFaceDetector +// tinyYolov2 +``` -To load a model, you have provide the corresponding manifest.json file as well as the model weight files (shards) as assets. Simply copy them to your public or assets folder. The manifest.json and shard files of a model have to be located in the same directory / accessible under the same route. +To load a model, you have to provide the corresponding manifest.json file as well as the model weight files (shards) as assets. Simply copy them to your public or assets folder. The manifest.json and shard files of a model have to be located in the same directory / accessible under the same route. Assuming the models reside in **public/models**: ``` javascript -await faceapi.loadFaceDetectionModel('/models') +await faceapi.nets.ssdMobilenetv1.loadFromUri('/models') // accordingly for the other models: -// await faceapi.loadFaceLandmarkModel('/models') -// await faceapi.loadFaceLandmarkTinyModel('/models') -// await faceapi.loadFaceRecognitionModel('/models') -// await faceapi.loadMtcnnModel('/models') -// await faceapi.loadTinyYolov2Model('/models') +// await faceapi.nets.faceLandmark68Net.loadFromUri('/models') +// await faceapi.nets.faceRecognitionNet.loadFromUri('/models') +// ... ``` -As an alternative, you can also create instance of the neural nets: +In a nodejs environment you can furthermore load the models directly from disk: ``` javascript -const net = new faceapi.FaceDetectionNet() -// accordingly for the other models: -// const net = new faceapi.FaceLandmark68Net() -// const net = new faceapi.FaceLandmark68TinyNet() -// const net = new faceapi.FaceRecognitionNet() -// const net = new faceapi.Mtcnn() -// const net = new faceapi.TinyYolov2() +await faceapi.nets.ssdMobilenetv1.loadFromDisk('./models') +``` + +You can also load the model from a tf.NamedTensorMap: + +``` javascript +await faceapi.nets.ssdMobilenetv1.loadFromWeightMap(weightMap) +``` -await net.load('/models/face_detection_model-weights_manifest.json') -// await net.load('/models/face_landmark_68_model-weights_manifest.json') -// await net.load('/models/face_landmark_68_tiny_model-weights_manifest.json') -// await net.load('/models/face_recognition_model-weights_manifest.json') -// await net.load('/models/mtcnn_model-weights_manifest.json') -// await net.load('/models/tiny_yolov2_separable_conv_model-weights_manifest.json') +Alternatively, you can also create own instances of the neural nets: + +``` javascript +const net = new faceapi.SsdMobilenetv1() +await net.loadFromUri('/models') ``` -Using instances, you can also load the weights as a Float32Array (in case you want to use the uncompressed models): +You can also load the weights as a Float32Array (in case you want to use the uncompressed models): ``` javascript // using fetch -const res = await fetch('/models/face_detection_model.weights') -const weights = new Float32Array(await res.arrayBuffer()) -net.load(weights) +net.load(await faceapi.fetchNetWeights('/models/face_detection_model.weights')) // using axios const res = await axios.get('/models/face_detection_model.weights', { responseType: 'arraybuffer' }) @@ -179,166 +206,521 @@ const weights = new Float32Array(res.data) net.load(weights) ``` - + + +## High Level API -### Face Detection - SSD Mobilenet v1 +In the following **input** can be an HTML img, video or canvas element or the id of that element. -Detect faces and get the bounding boxes and scores: +``` html + +