Camera2 custom camera development process detailed explanation

[[432612]]

Preface

Today I will introduce the detailed explanation of Camera2 camera development process to you guys;

Android 5.0 has completely redesigned the camera API and added a newly designed Camera 2 API. These APIs not only greatly improve the camera function of the Android system, but also support RAW photo output and even allow programs to adjust the camera's focus mode, exposure mode, shutter, etc.

1. Camera2 Architecture Introduction

Camera2 uses the concept of pipeline to connect Android devices and cameras. The system sends a Capture request to the camera, and the camera returns CameraMetadata. All this is built in a session called CameraCaptureSession;

The core participating roles of the architecture are: CameraManager, CameraDevice, CameraCharacteristics, CameraRequest and CameraRequest.Builder, CameraCaptureSession and CaptureResult;

1. CameraManager

Located under android.hardware.camera2.CameraManager, it was also added in Android 21 (5.0). Like other system services, it is initialized through Context.getSystemService(CameraManager.class) or Context.getSystemService(Context.CAMERA_SERVICE). It is mainly used to manage the system camera:

• Get the camera list of the Android device through the getCameraIdList() method;
• getCameraCharacteristics(String cameraId) gets the detailed parameters and supported functions of the camera;

2. CameraDevice

• CameraDevice is an object abstracted from Camera2, directly connected to the system hardware camera;
• That is, the camera functions can be divided into two levels: limit and full. When the camera is at the limited level, the functions of Camera2 and the earlier Camera are similar. In addition, in the Camera2 architecture, CameraDevice also undertakes other important tasks:
• Monitor the camera status through CameraDevice.StateCallback (mainly including onOpened, onClosed, onDisconnected, onErro);
• Manage CameraCaptureSession, - create sessions through the createCaptureSession() method and the createReprocessableCaptureSession() method. Usually, the corresponding methods are called in CameraDevice.StateCallback to create a preview session;
• Manage CaptureRequest, mainly including creating capture requests through createCaptureRequest(int templateType). When you need to preview, take pictures, and preview again, you need to create requests to complete them;

3. CameraCaptureSession

• The system sends a Capture request to the camera, and the camera returns CameraMetadata. All this is done in the CameraCaptureSession session created by the corresponding CameraDevice. When the program needs to preview, take pictures, and preview again, it must first go through the session;
• CameraCaptureSession has many important callbacks:
• Manages the CameraCaptureSession.StateCallback state callback, a callback object used to receive updates about the CameraCaptureSession state;
• Manages the CameraCaptureSession.CaptureCallback capture callback, which is used to receive the callback of the capture request status. The corresponding method of the callback will be triggered when the request triggers the capture to start; when the capture is completed; when an error occurs when capturing an image;
• Submitting a capture image request by calling the capture() method is to take a photo. The request defines all the parameters for capturing a single image, including the sensor, lens, flash, and post-processing parameters. The result of each request will generate a CaptureResult, which can generate a new frame for one or more Surfaces, and then attach it to the corresponding Surface for display through the addTarget(Surface) method of CaptureRequest.Builder;
• Calling the method setRepeatingRequest() to request repeated image capture, that is, to achieve preview;
• Call stopRepeating() to stop capturing images, that is, stop previewing.

4. Camera Characteristics

An object describing the properties of a Camera device, which can be queried using CameraManager via getCameraCharacteristics(String cameraId);

5. CameraRequest and CameraRequest.Builder

CameraRequest represents a capture request, and CameraRequest.Builder is used to describe various parameter settings for capturing images, including capture hardware (sensor, lens, flash memory), focus mode, exposure mode, processing pipeline, control algorithm and output buffer configuration, which are then passed to the corresponding session for setting. CameraRequest.Builder is responsible for generating the CameraRequest object. When the program calls the setRepeatingRequest() method for preview, or calls the capture() method for taking pictures, the CameraRequest parameter needs to be passed in. CameraRequest can be initialized through CameraRequest.Builder and obtained by calling createCaptureRequest;

6. CaptureResult

A CaptureRequest is an object that describes a subset of the results of capturing a single image from an image sensor and is generated by a CameraDevice after the CaptureRequest is processed;

7. Camera2: Connections between the main characters

CameraManager is at the top management position and is responsible for detecting and obtaining all cameras and their characteristics and passing in the specified CameraDevice.StateCallback callback to open the specified camera. CameraDevice is responsible for managing abstract objects, including monitoring the Camera state callback CameraDevice.StateCallback, creating CameraCaptureSession and CameraRequest;

2. Detailed explanation of Camera2 camera development process

1. Apply for permissions

 <uses-permission android: name = "android.permission.CAMERA" />
 <uses-permission android: name = "android.permission.WRITE_EXTERNAL_STORAGE" /> 

2. Define a TextureView in the xml layout file

 <TextureView
   android:id= "@+id/textureView"  
   android:layout_width= "match_parent"  
   android:layout_height= "match_parent" />

3. Create a CameraHelper class and add a callback function to the TextureView object

 class Camera2Helper(val mActivity: Activity, private val mTextureView: TextureView) {
 companion object {
 const val PREVIEW_WIDTH = 720 //Preview width
 const val PREVIEW_HEIGHT = 1280 //Preview height
 const val SAVE_WIDTH = 720 //Save the image width
 const val SAVE_HEIGHT = 1280 //Save the image height
 }
    private lateinit var mCameraManager: CameraManager
    private var mImageReader: ImageReader? = null  
    private var mCameraDevice: CameraDevice? = null  
    private var mCameraCaptureSession: CameraCaptureSession? = null  
    private var mCameraId = "0"  
    private lateinit var mCameraCharacteristics: CameraCharacteristics
 private var mCameraSensorOrientation = 0 //Camera orientation
 private var mCameraFacing = CameraCharacteristics.LENS_FACING_BACK //Use the rear camera by default
 private val mDisplayRotation = mActivity.windowManager.defaultDisplay.rotation //Phone direction
 private var canTakePic = true //Can I take a photo?
 private var canExchangeCamera = false //Can I switch the camera?
    private var mCameraHandler: Handler
    private val handlerThread = HandlerThread( "CameraThread" )
 private var mPreviewSize = Size (PREVIEW_WIDTH, PREVIEW_HEIGHT) //Preview size
 private var mSavePicSize = Size (SAVE_WIDTH, SAVE_HEIGHT) //Save picture size
 init {
 handlerThread.start()
        mCameraHandler = Handler(handlerThread.looper)
        mTextureView.surfaceTextureListener = object : TextureView.SurfaceTextureListener {
            override fun onSurfaceTextureSizeChanged(surface: SurfaceTexture?, width: Int , height: Int ) {
 }
            override fun onSurfaceTextureUpdated(surface: SurfaceTexture?) {
 }
            override fun onSurfaceTextureDestroyed(surface: SurfaceTexture?): Boolean {
                releaseCamera()
 return   true  
 }
            override fun onSurfaceTextureAvailable(surface: SurfaceTexture?, width: Int , height: Int ) {
                initCameraInfo()
 }
 }
 }
 }

• Opening the camera and creating a session are time-consuming operations, so we start a HandlerThread to handle them in the child thread;
• There are two variables about size, one is the preview size (displayed on the screen), and the other is the size of the saved image (the size of the image saved to the SD card);
• There are two directions, one is the phone direction (if it is a vertical screen application, this direction is 0), and the other is the camera direction (generally speaking, the front camera direction is 270, and the rear camera direction is 90);

4. Initialize related parameters

 /**
 * Initialization
 */
    private fun initCameraInfo() {
        mCameraManager = mActivity.getSystemService(Context.CAMERA_SERVICE) as CameraManager
        val cameraIdList = mCameraManager.cameraIdList
        if (cameraIdList.isEmpty()) {
 mActivity.toast( "No camera available" )
 return  
 }
 for (id in cameraIdList) {
            val cameraCharacteristics = mCameraManager.getCameraCharacteristics(id)
            val facing = cameraCharacteristics.get(CameraCharacteristics.LENS_FACING)
            if (facing == mCameraFacing) {
                mCameraId = id
                mCameraCharacteristics = cameraCharacteristics
 }
 log( "camera $id in device" )
 }
        val supportLevel = mCameraCharacteristics.get(CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL)
        if (supportLevel == CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL_LEGACY) {
 mActivity.toast( "Camera hardware does not support new features" )
 }
 //Get the camera direction
        mCameraSensorOrientation = mCameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)
 //Get StreamConfigurationMap, which manages all output formats and sizes supported by the camera
        val configurationMap = mCameraCharacteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)
 val savePicSize = configurationMap.getOutputSizes(ImageFormat.JPEG) //Save photo size
 val previewSize = configurationMap.getOutputSizes(SurfaceTexture::class.java) //Preview size
        val exchange = exchangeWidthAndHeight(mDisplayRotation, mCameraSensorOrientation)
        mSavePicSize = getBestSize(
                if (exchange) mSavePicSize.height else mSavePicSize.width,
                if (exchange) mSavePicSize.width else mSavePicSize.height,
                if (exchange) mSavePicSize.height else mSavePicSize.width,
                if (exchange) mSavePicSize.width else mSavePicSize.height,
                savePicSize.toList())
        mPreviewSize = getBestSize(
                if (exchange) mPreviewSize.height else mPreviewSize.width,
                if (exchange) mPreviewSize.width else mPreviewSize.height,
                if (exchange) mTextureView.height else mTextureView.width,
                if (exchange) mTextureView.width else mTextureView.height,
                previewSize.toList())
        mTextureView.surfaceTexture.setDefaultBufferSize(mPreviewSize.width, mPreviewSize.height)
 log( "Optimal preview size: ${mPreviewSize.width} * ${mPreviewSize.height}, ratio ${mPreviewSize.width.toFloat() / mPreviewSize.height}" )
 log( "Optimal size for saving pictures: ${mSavePicSize.width} * ${mSavePicSize.height}, ratio ${mSavePicSize.width.toFloat() / mSavePicSize.height}" )
 //Adjust the size of TextureView according to the preview size to ensure that the image is not stretched
        val orientation = mActivity.resources.configuration.orientation
        if (orientation == Configuration.ORIENTATION_LANDSCAPE)
            mTextureView.setAspectRatio(mPreviewSize.width, mPreviewSize.height)
 else  
            mTextureView.setAspectRatio(mPreviewSize.height, mPreviewSize.width)
        mImageReader = ImageReader.newInstance(mPreviewSize.width, mPreviewSize.height, ImageFormat.JPEG, 1)
        mImageReader?.setOnImageAvailableListener(onImageAvailableListener, mCameraHandler)
 if (openFaceDetect)
 initFaceDetect()
 openCamera()
 }
 /**
 * Returns whether the width and height need to be swapped based on the provided screen orientation [displayRotation] and camera orientation [sensorOrientation]
 */
    private fun exchangeWidthAndHeight(displayRotation: Int , sensorOrientation: Int ): Boolean {
 var exchange = false  
 when (displayRotation) {
            Surface.ROTATION_0, Surface.ROTATION_180 ->
                if (sensorOrientation == 90 || sensorOrientation == 270) {
                    exchange = true  
 }
            Surface.ROTATION_90, Surface.ROTATION_270 ->
                if (sensorOrientation == 0 || sensorOrientation == 180) {
                    exchange = true  
 }
 else -> log( "Display rotation is invalid: $displayRotation" )
 }
 log( "screen direction $displayRotation" )
 log( "Camera orientation $sensorOrientation" )
 Return exchange
 }
 /**
 *
 * Returns the size that is equal to or closest to the specified width and height according to the provided parameter values
 *
 * @param targetWidth target width
 * @param targetHeight target height
 * @param maxWidth Maximum width (ie, the width of TextureView)
 * @param maxHeight Maximum height (ie the height of TextureView)
 * @param sizeList supported Size list
 *
 * @return Returns the size that is equal to or closest to the specified width and height
 *
 */
    private fun getBestSize(targetWidth: Int , targetHeight: Int , maxWidth: Int , maxHeight: Int , sizeList: List< Size >): Size {
 val bigEnough = ArrayList< Size >() // Size list larger than the specified width and height
 val notBigEnough = ArrayList< Size >() // Size list smaller than the specified width and height
 for ( size   in sizeList) {
 //width <= maximum width && height <= maximum height && aspect ratio == target aspect ratio
            if ( size .width <= maxWidth && size .height <= maxHeight
                    && size .width == size .height * targetWidth / targetHeight) {
                if ( size .width >= targetWidth && size .height >= targetHeight)
                    bigEnough.add ( size )
 else  
                    notBigEnough. add ( size )
 }
 log( "System supported sizes: ${size.width} * ${size.height}, ratio: ${size.width.toFloat() / size.height}" )
 }
 log( "Maximum size: $maxWidth * $maxHeight, ratio: ${targetWidth.toFloat() / targetHeight}" )
 log( "target size: $targetWidth * $targetHeight, ratio: ${targetWidth.toFloat() / targetHeight}" )
 //Select the smallest value in bigEnough or the largest value in notBigEnough
 return   when {
            bigEnough. size > 0 -> Collections. min (bigEnough, CompareSizesByArea())
            notBigEnough. size > 0 -> Collections. max (notBigEnough, CompareSizesByArea())
 else -> sizeList[0]
 }
 }

• First, get the CameraManager instance through mActivity.getSystemService(Context.CAMERA_SERVICE) as CameraManager;
• By looping through the available cameras in the device, various information about the camera can be obtained through mCameraManager.getCameraCharacteristics(id);
• mCameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION) gets the direction of the camera sensor;
• Get the preview size and saved image size supported by the camera through configurationMap.getOutputSizes(ImageFormat.JPEG) and configurationMap.getOutputSizes(SurfaceTexture::class.java);
• The function of the exchangeWidthAndHeight(displayRotation: Int, sensorOrientation: Int) method is to determine whether the width and height need to be exchanged based on the screen orientation and camera orientation;
• Get the optimal width and height through the getBestSize() method. According to the input target width and height values, the maximum width and height values ​​(i.e. the screen size) and the size list supported by the camera, get an optimal value from the size list supported by the camera;
• Use the mTextureView.surfaceTexture.setDefaultBufferSize() method to set the preview size of TextureView;
• Create an ImageReader object and set the callback function;
• Process the obtained image data in onImageAvailableListener;

5. Open the camera

 /**
 * Open the camera
 */
    private fun openCamera() {
        if (ContextCompat.checkSelfPermission(mActivity, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED) {
 mActivity.toast( "No camera permission!" )
 return  
 }
        mCameraManager.openCamera(mCameraId, object : CameraDevice.StateCallback() {
            override fun onOpened(camera: CameraDevice) {
                log( "onOpened" )
                mCameraDevice = camera
                createCaptureSession(camera)
 }
            override fun onDisconnected(camera: CameraDevice) {
                log( "onDisconnected" )
 }
            override fun onError(camera: CameraDevice, error: Int ) {
                log( "onError $error" )
 mActivity.toast( "Failed to open camera! $error" )
 }
 }, mCameraHandler)
 }

6. Create a preview session

 /**
 * Create a preview session
 */
    private fun createCaptureSession(cameraDevice: CameraDevice) {
        val captureRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
        val surface = Surface(mTextureView.surfaceTexture)
 captureRequestBuilder.addTarget(surface) // Bind the CaptureRequest builder to the Surface object
        captureRequestBuilder.set ( CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_ON_AUTO_FLASH) // Flash
        captureRequestBuilder.set ( CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE) // Auto focus
 // Create a CameraCaptureSession object for camera preview
        cameraDevice.createCaptureSession(arrayListOf(surface, mImageReader?.surface), object : CameraCaptureSession.StateCallback() {
            override fun onConfigureFailed(session: CameraCaptureSession?) {
 mActivity.toast( "Failed to open preview session!" )
 }
            override fun onConfigured(session: CameraCaptureSession) {
                mCameraCaptureSession = session
                session.setRepeatingRequest(captureRequestBuilder.build(), mCaptureCallBack, mCameraHandler)
 }
 }, mCameraHandler)
 }
    private val mCaptureCallBack = object : CameraCaptureSession.CaptureCallback() {
        override fun onCaptureCompleted(session: CameraCaptureSession, request: CaptureRequest?, result: TotalCaptureResult) {
            super.onCaptureCompleted(session, request, result)
            canExchangeCamera = true  
 canTakePic = true  
 }
        override fun onCaptureFailed(session: CameraCaptureSession?, request: CaptureRequest?, failure: CaptureFailure?) {
            super.onCaptureFailed(session, request, failure)
            log( "onCaptureFailed" )
 mActivity.toast( "Failed to open preview!" )
 }
 }

• Create a Builder object for preview through cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
• Add a Surface object to the Builder object and set various related parameters;
• Create a session through cameraDevice.createCaptureSession, and pass a surface and mImageReader?.surface in the first parameter. This indicates that the image data of this session is output to these two objects;
• When the session is successfully created, a preview request is initiated through session.setRepeatingRequest(captureRequestBuilder.build(), mCaptureCallBack, mCameraHandler);

7. Take photos and save them

 /**
     * Photograph
 */
 fun takePic() {
        if (mCameraDevice == null || !mTextureView.isAvailable || !canTakePic) return  
        mCameraDevice?.apply {
            val captureRequestBuilder = createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE)
            captureRequestBuilder.addTarget(mImageReader?.surface)
            captureRequestBuilder.set ( CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE) // Auto focus
            captureRequestBuilder.set ( CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_ON_AUTO_FLASH) // Flash
 captureRequestBuilder.set ( CaptureRequest.JPEG_ORIENTATION, mCameraSensorOrientation) //Rotate the saved photo according to the camera direction to make it "natural direction"  
            mCameraCaptureSession?.capture(captureRequestBuilder.build(), null , mCameraHandler)
 ?: mActivity.toast( "Photo taking abnormality!" )
 }
 }
    private val onImageAvailableListener = OnImageAvailableListener {
        val image = it.acquireNextImage()
        val byteBuffer = image.planes[0].buffer
        val byteArray = ByteArray(byteBuffer.remaining())
        byteBuffer.get(byteArray)
 it.close ()
        BitmapUtils.savePic(byteArray, mCameraSensorOrientation == 270, { savedPath, time ->
            mActivity.runOnUiThread {
 mActivity.toast( "Image saved successfully! Saved path: $savedPath Time consumed: $time" )
 }
 }, { msg ->
            mActivity.runOnUiThread {
 mActivity.toast( "Image saving failed! $msg" )
 }
 })
 }

• Create a Builder object for a photo request through createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
• Then set various parameters. Note that captureRequestBuilder.set(CaptureRequest.JPEG_ORIENTATION, mCameraSensorOrientation) is used to set the rotation direction of the saved photo. If it is not set, the saved photo will not be in the "natural direction";
• The result of taking a photo is obtained in the OnImageAvailableListener object;
• Get an Image object through acquireNextImage() method, then get byteBuffer through image.planes[0].buffer, and convert this byteBuffer into byteArray;
• This byteArray is the image data obtained by taking a photo. Then you can save this byteArray as a picture in the phone storage;

8. Release the camera and threads

 fun releaseCamera() {
        mCameraCaptureSession?. close ()
        mCameraCaptureSession = null  
        mCameraDevice? .close ()
 mCameraDevice = null  
        mImageReader?. close ()
 mImageReader = null  
        canExchangeCamera = false  
 }
 fun releaseThread() {
        handlerThread.quitSafely()
 }

Summarize

Camera2 API is very different from the old Camera API. You may not be used to it at first, but Camera2 has many advantages and provides a lot of parameters for us to control;

Let’s study together and come on, my friends;