general information
Nowadays we see cute and funny face stickers everywhere. They are used not only in camera apps but also in social media and entertainment apps. In this article, I will show you how to create 2D stickers using the HUAWEI ML Kit tool. We'll also be covering the 3D sticker development process soon, so stay tuned!
Scripts
Photo capture and editing apps such as selfie cameras and social media (TikTok, Weibo, WeChat, etc.) often offer a set of stickers for customizing images. With these stickers, users can create and share engaging and vibrant content.
Training
Add Maven Huawei repository to project level file build.gradle
Open the build.gradle file in the root directory of your Android Studio project .
Add the Maven repository address.
buildscript {
{
maven {url 'http://developer.huawei.com/repo/'}
}
}
allprojects {
repositories {
maven { url 'http://developer.huawei.com/repo/'}
}
}
Add SDK dependencies to app-level file build.gradle
// Face detection SDK.
implementation 'com.huawei.hms:ml-computer-vision-face:2.0.1.300'
// Face detection model.
implementation 'com.huawei.hms:ml-computer-vision-face-shape-point-model:2.0.1.300'
Request permissions for camera, network and memory in AndroidManifest.xml file
<!--Camera permission-->
<uses-feature android:name="android.hardware.camera" />
<uses-permission android:name="android.permission.CAMERA" />
<!--Write permission-->
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<!--Read permission-->
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
Code development
Set up your face analyzer
MLFaceAnalyzerSetting detectorOptions;
detectorOptions = new MLFaceAnalyzerSetting.Factory()
.setFeatureType(MLFaceAnalyzerSetting.TYPE_UNSUPPORT_FEATURES)
.setShapeType(MLFaceAnalyzerSetting.TYPE_SHAPES)
.allowTracing(MLFaceAnalyzerSetting.MODE_TRACING_FAST)
.create();
detector = MLAnalyzerFactory.getInstance().getFaceAnalyzer(detectorOptions);
Get face contour points and pass them to FacePointEngine
Use the camera callback to get the camera data and then call the face analyzer to get the face contour points and pass those points to the FacePointEngine . The sticker filter will be able to use them later.
@Override
public void onPreviewFrame(final byte[] imgData, final Camera camera) {
int width = mPreviewWidth;
int height = mPreviewHeight;
long startTime = System.currentTimeMillis();
// Set the shooting directions of the front and rear cameras to be the same.
if (isFrontCamera()){
mOrientation = 0;
}else {
mOrientation = 2;
}
MLFrame.Property property =
new MLFrame.Property.Creator()
.setFormatType(ImageFormat.NV21)
.setWidth(width)
.setHeight(height)
.setQuadrant(mOrientation)
.create();
ByteBuffer data = ByteBuffer.wrap(imgData);
// Call the face analyzer API.
SparseArray<MLFace> faces = detector.analyseFrame(MLFrame.fromByteBuffer(data,property));
// Determine whether face information is obtained.
if(faces.size()>0){
MLFace mLFace = faces.get(0);
EGLFace EGLFace = FacePointEngine.getInstance().getOneFace(0);
EGLFace.pitch = mLFace.getRotationAngleX();
EGLFace.yaw = mLFace.getRotationAngleY();
EGLFace.roll = mLFace.getRotationAngleZ() - 90;
if (isFrontCamera())
EGLFace.roll = -EGLFace.roll;
if (EGLFace.vertexPoints == null) {
EGLFace.vertexPoints = new PointF[131];
}
int index = 0;
// Obtain the coordinates of a user's face contour points and convert them to the floating point numbers in normalized coordinate system of OpenGL.
for (MLFaceShape contour : mLFace.getFaceShapeList()) {
if (contour == null) {
continue;
}
List<MLPosition> points = contour.getPoints();
for (int i = 0; i < points.size(); i++) {
MLPosition point = points.get(i);
float x = ( point.getY() / height) * 2 - 1;
float y = ( point.getX() / width ) * 2 - 1;
if (isFrontCamera())
x = -x;
PointF Point = new PointF(x,y);
EGLFace.vertexPoints[index] = Point;
index++;
}
}
// Insert a face object.
FacePointEngine.getInstance().putOneFace(0, EGLFace);
// Set the number of faces.
FacePointEngine.getInstance().setFaceSize(faces!= null ? faces.size() : 0);
}else{
FacePointEngine.getInstance().clearAll();
}
long endTime = System.currentTimeMillis();
Log.d("TAG","Face detect time: " + String.valueOf(endTime - startTime));
}
The image below shows how face contour points are returned using the ML Kit API.
Defining sticker data in JSON format
public class FaceStickerJson {
public int[] centerIndexList; // Center coordinate index list. If the list contains multiple indexes, these indexes are used to calculate the central point.
public float offsetX; // X-axis offset relative to the center coordinate of the sticker, in pixels.
public float offsetY; // Y-axis offset relative to the center coordinate of the sticker, in pixels.
public float baseScale; // Base scale factor of the sticker.
public int startIndex; // Face start index, which is used for computing the face width.
public int endIndex; // Face end index, which is used for computing the face width.
public int width; // Sticker width.
public int height; // Sticker height.
public int frames; // Number of sticker frames.
public int action; // Action. 0 indicates default display. This is used for processing the sticker action.
public String stickerName; // Sticker name, which is used for marking the folder or PNG file path.
public int duration; // Sticker frame displays interval.
public boolean stickerLooping; // Indicates whether to perform rendering in loops for the sticker.
public int maxCount; // Maximum number of rendering times.
...
}
Make a sticker with a picture of a cat
Create a JSON file for the cat sticker and define the center point between the eyebrows (84) and the tip of the nose (85) using the face index. Paste the cat ears and nose, and then place the JSON file and image in the assets folder .
{
"stickerList": [{
"type": "sticker",
"centerIndexList": [84],
"offsetX": 0.0,
"offsetY": 0.0,
"baseScale": 1.3024,
"startIndex": 11,
"endIndex": 28,
"width": 495,
"height": 120,
"frames": 2,
"action": 0,
"stickerName": "nose",
"duration": 100,
"stickerLooping": 1,
"maxcount": 5
}, {
"type": "sticker",
"centerIndexList": [83],
"offsetX": 0.0,
"offsetY": -1.1834,
"baseScale": 1.3453,
"startIndex": 11,
"endIndex": 28,
"width": 454,
"height": 150,
"frames": 2,
"action": 0,
"stickerName": "ear",
"duration": 100,
"stickerLooping": 1,
"maxcount": 5
}]
}
Rendering a sticker to a texture
We render the sticker to a texture using the GLSurfaceView class - it's simpler than the TextureView . Instantiate the sticker filter in onSurfaceChanged , pass the sticker path, and start the camera.
@Override
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
GLES30.glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
mTextures = new int[1];
mTextures[0] = OpenGLUtils.createOESTexture();
mSurfaceTexture = new SurfaceTexture(mTextures[0]);
mSurfaceTexture.setOnFrameAvailableListener(this);
// Pass the samplerExternalOES into the texture.
cameraFilter = new CameraFilter(this.context);
// Set the face sticker path in the assets directory.
String folderPath ="cat";
stickerFilter = new FaceStickerFilter(this.context,folderPath);
// Create a screen filter object.
screenFilter = new BaseFilter(this.context);
facePointsFilter = new FacePointsFilter(this.context);
mEGLCamera.openCamera();
}
Initialize the sticker filter to onSurfaceChanged
@Override
public void onSurfaceChanged(GL10 gl, int width, int height) {
Log.d(TAG, "onSurfaceChanged. width: " + width + ", height: " + height);
int previewWidth = mEGLCamera.getPreviewWidth();
int previewHeight = mEGLCamera.getPreviewHeight();
if (width > height) {
setAspectRatio(previewWidth, previewHeight);
} else {
setAspectRatio(previewHeight, previewWidth);
}
// Set the image size, create a FrameBuffer, and set the display size.
cameraFilter.onInputSizeChanged(previewWidth, previewHeight);
cameraFilter.initFrameBuffer(previewWidth, previewHeight);
cameraFilter.onDisplaySizeChanged(width, height);
stickerFilter.onInputSizeChanged(previewHeight, previewWidth);
stickerFilter.initFrameBuffer(previewHeight, previewWidth);
stickerFilter.onDisplaySizeChanged(width, height);
screenFilter.onInputSizeChanged(previewWidth, previewHeight);
screenFilter.initFrameBuffer(previewWidth, previewHeight);
screenFilter.onDisplaySizeChanged(width, height);
facePointsFilter.onInputSizeChanged(previewHeight, previewWidth);
facePointsFilter.onDisplaySizeChanged(width, height);
mEGLCamera.startPreview(mSurfaceTexture);
}
Draw a sticker on the screen using onDrawFrame
@Override
public void onDrawFrame(GL10 gl) {
int textureId;
// Clear the screen and depth buffer.
GLES30.glClear(GLES30.GL_COLOR_BUFFER_BIT | GLES30.GL_DEPTH_BUFFER_BIT);
// Update a texture image.
mSurfaceTexture.updateTexImage();
// Obtain the SurfaceTexture transform matrix.
mSurfaceTexture.getTransformMatrix(mMatrix);
// Set the camera display transform matrix.
cameraFilter.setTextureTransformMatrix(mMatrix);
// Draw the camera texture.
textureId = cameraFilter.drawFrameBuffer(mTextures[0],mVertexBuffer,mTextureBuffer);
// Draw the sticker texture.
textureId = stickerFilter.drawFrameBuffer(textureId,mVertexBuffer,mTextureBuffer);
// Draw on the screen.
screenFilter.drawFrame(textureId , mDisplayVertexBuffer, mDisplayTextureBuffer);
if(drawFacePoints){
facePointsFilter.drawFrame(textureId, mDisplayVertexBuffer, mDisplayTextureBuffer);
}
}
Happened! Your face sticker is ready.
Let's see it in action!
Please visit our official website for details .
You can also see the sample code .