Audio and video development journey (4) Camera video capture

Audio and video development journey (4) Camera video capture

table of Contents

Camera basics

Video capture process

Problems encountered and common pits (emphasis)

reward

1. Camera basic knowledge Camera has several important basic concepts.

The direction of the facing camera, generally the rear camera and the front camera.

Orientation: The angle at which the camera collects pictures. The sensor of the camera is horizontal in the mobile phone. When previewing, rotate the corresponding angle clockwise according to the preview direction of the Camera. If it is not set correctly, it will cause problems such as upside down and mirroring during preview. To save the previewed picture as a photo album, you also need to set the direction separately. Note that this direction is not related to the preview direction.

The size of the preview picture The size of the preview container and the picture size of the picture preview supported by the camera. If you set a preview size that is not supported by the Camera, it will cause a black screen.

You can set frame callbacks and then perform business processing in each frame, such as face recognition and other functions

Camera preview image formats include NV21 YUV420sp, etc.

Camera needs a container to display the Surface on the screen, generally SurfaceView, TextureView, etc.

2. The video capture process gets the SurfaceHolder through SurfaceView, and then sets the addCallback callback. When the Surface is created, destroyed, or changed, the corresponding callback is triggered, in which Camera initialization and parameter settings can be performed

Generate an object by new Camera (cameralId). Then Camera.getParams obtains the relevant parameters, and can print out important or comparative parmas, such as how many cameras are supported, the size of the supported preview pictures, and the direction of each camera. You can set the corresponding parameters as needed, such as the format of the picture, the preview size of the picture, and so on. Of course, there is a must to set the camera preview display direction, otherwise the previewed picture is inconsistent with the normal direction.

You can set Camera's setPreviewCallback to get the callback of each frame, set processing as needed, and start previewing startPreview and frame callback processing

Camera switching

If you start the camera switch, you need to release the previous camera, regenerate and set the relationship between the camera switch preview and the Activity life cycle. This is determined by the SurfaceView. It is created or recreated when the page is visible (onResume), and the page is not visible. (OnPause) destroy and release

The specific implementation is as follows

SurfaceView settings

private SurfaceHolder mSurfaceHolder;

private void initSurfaceView() {
    mSurfaceHolder = surfaceview.getHolder();
    mSurfaceHolder.addCallback(new SurfaceHolder.Callback() {
        @Override
        public void surfaceCreated(SurfaceHolder holder) {
            Log.d(TAG, "surfaceCreated: ");
            handleSurfaceCreated();
        }

        @Override
        public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
        }

        @Override
        public void surfaceDestroyed(SurfaceHolder holder) {
            Log.d(TAG, "surfaceDestroyed: ");
            handleSurfaceDestroyed();
        }
    });
}

private void handleSurfaceDestroyed() {
    releaseCamera();
    mSurfaceHolder = null;
    Log.i(TAG, "handleSurfaceDestroyed: ");
}

private void handleSurfaceCreated() {
    Log.i(TAG, "handleSurfaceCreated: start");
    if (mSurfaceHolder == null) {
        mSurfaceHolder = surfaceview.getHolder();
    }
    if (mCamera == null) {
        initCamera(curCameraId);
    }
    try {
       //2 SurfaceView 
       //Camera is being used after Camera.release() was called
       //surfaceDestroyed Camera release  null,
       //-- Camera 
       //onResume-- surfaceCreated
       //onPause-- surfaceDestroyed
        mCamera.setPreviewDisplay(mSurfaceHolder);
    } catch (IOException e) {
        e.printStackTrace();
        Log.e(TAG, "handleSurfaceCreated: " + e.getMessage());
    }
    startPreview();
    Log.i(TAG, "handleSurfaceCreated: end");
}

private void startPreview() {
 

//mCamera.setPreviewCallback(new Camera.PreviewCallback() {//@Override//public void onPreviewFrame(byte[] data, Camera camera) {//Log.i(TAG, "onPreviewFrame: setPreviewCallback");//}//});//Question: In many cases, it is not only necessary to preview, but when previewing the video, I hope to do some detection, such as face detection. This requires obtaining the preview frame video, what should I do? mCamera.setOneShotPreviewCallback(new Camera.PreviewCallback() {@Override public void onPreviewFrame(byte[] data, Camera camera) {Log.i(TAG, "onPreviewFrame: setOneShotPreviewCallback"); Camera.Size previewSize = mCamera.getParameters(). getPreviewSize(); YuvImage yuvImage = new YuvImage(data, ImageFormat.NV21, previewSize.width, previewSize.height, null); ByteArrayOutputStream os = new ByteArrayOutputStream(data.length); if(!yuvImage.compressToJpeg(new Rect(0, 0,previewSize.width,previewSize.height),100,

           //bitmap 
           //Camera 
            if(curCameraId == Camera.CameraInfo.CAMERA_FACING_BACK){
                bitmap = BitmapUtils.rotate(bitmap,90);
            }else {
                bitmap = BitmapUtils.mirror(BitmapUtils.rotate(bitmap,270));
            }
            FileUtils.saveBitmapToFile(bitmap,"oneShot.jpg");
        }
    });
 

//mCamera.setPreviewCallbackWithBuffer(new Camera.PreviewCallback() {//@Override//public void onPreviewFrame(byte[] data, Camera camera) {//Log.i(TAG, "onPreviewFrame: setPreviewCallbackWithBuffer");//}//}); mCamera.startPreview();

}
 

2: Camera initialization and Params settings

private void initCamera(int cameraId) {curCameraId = cameraId; mCamera = Camera.open(curCameraId); Log.d(TAG, "initCamera: Camera Open ");

    setCamerDisplayOrientation(this, curCameraId, mCamera);

    if (!hadPrinted) {
        printCameraInfo();
        hadPrinted = true;
    }
    Camera.Parameters parameters = mCamera.getParameters();
    Camera.Size closelyPreSize = CameraUtil.getCloselyPreSize(true, SystemUtils.getDisplayWidth(), SystemUtils.getDisplayHeight(), parameters.getSupportedPreviewSizes());
    Log.i(TAG, "initCamera: closelyPreSizeW="+closelyPreSize.width+" closelyPreSizeH="+closelyPreSize.height);
    parameters.setPreviewSize(closelyPreSize.width, closelyPreSize.height);
    mCamera.setParameters(parameters);
}

private void printCameraInfo() {
   //1.  getParameters Parameters
    Camera.Parameters parameters = mCamera.getParameters();

   //2.  Camera ( NV21 YUV420sp)
    int previewFormat = parameters.getPreviewFormat();
    Log.d(TAG, "initCamera: previewFormat=" + previewFormat);//NV21

   //3.  Camera W H 
   // Camera W H camera Camera 
   // Camera 
   // WH 
   //Camera  W>H
    List<Camera.Size> supportedPreviewSizes = parameters.getSupportedPreviewSizes();
    for (Camera.Size item : supportedPreviewSizes
    ) {
        Log.d(TAG, "initCamera: supportedPreviewSizes w= " + item.width + " h=" + item.height);
    }

   //Camera H>W Camera W>H
    Camera.Size previewSize = parameters.getPreviewSize();
    int[] physicalSS = SystemUtils.getPhysicalSS(this);
    Log.i(TAG, "initCamera: w=" + previewSize.width + " h=" + previewSize.height
            + " screenW=" + SystemUtils.getDisplayWidth() + " screenH=" + SystemUtils.getDisplayHeight()
            + " physicalW=" + physicalSS[0] + " physicalH=" + physicalSS[1]);

   //4.  Camera   10 30
    List<Integer> supportedPreviewFrameRates = parameters.getSupportedPreviewFrameRates();
    for (Integer item : supportedPreviewFrameRates
    ) {
        Log.i(TAG, "initCamera: supportedPreviewFrameRates frameRate=" + item);
    }

   //5.  Camera Camera orientation Camera orientation Camera DisplayOrientation 
    int numberOfCameras = Camera.getNumberOfCameras();
    Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
    for (int i = 0; i < numberOfCameras; i++) {
        Camera.getCameraInfo(i, cameraInfo);
        Log.i(TAG, "initCamera: facing=" + cameraInfo.facing
                + " orientation=" + cameraInfo.orientation);
    }
}

/**
 *
 * @param activity
 * @param cameraId
 * @param camera
 */
public static void setCamerDisplayOrientation(Activity activity, int cameraId, Camera camera) {
    Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
    Camera.getCameraInfo(cameraId, cameraInfo);
    int rotation = activity.getWindowManager().getDefaultDisplay().getRotation();
    Log.i(TAG, "setCamerDisplayOrientation: rotation=" + rotation + " cameraId=" + cameraId);
    int degress = 0;

    switch (rotation) {
        case Surface.ROTATION_0:
            degress = 0;
            break;
        case Surface.ROTATION_90:
            degress = 90;
            break;
        case Surface.ROTATION_180:
            degress = 180;
            break;
        case Surface.ROTATION_270:
            degress = 270;
            break;
    }
    int result = 0;
    if (cameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
        result = (cameraInfo.orientation + degress) % 360;
        result = (360 - result) % 360;

    } else if (cameraInfo.facing == Camera.CameraInfo.CAMERA_FACING_BACK) {
        result = (cameraInfo.orientation - degress + 360) % 360;
    }
    Log.i(TAG, "setCamerDisplayOrientation: result=" + result + " cameraId=" + cameraId + " facing=" + cameraInfo.facing + " cameraInfo.orientation=" + cameraInfo.orientation);

    camera.setDisplayOrientation(result);
}
 

Camera preview, frame callback processing, save picture rotation and mirroring processing

private void startPreview() {

   //
    mCamera.setOneShotPreviewCallback(new Camera.PreviewCallback() {
        @Override
        public void onPreviewFrame(byte[] data, Camera camera) {
            Log.i(TAG, "onPreviewFrame: setOneShotPreviewCallback");
            Camera.Size previewSize = mCamera.getParameters().getPreviewSize();
            YuvImage yuvImage = new YuvImage(data, ImageFormat.NV21, previewSize.width, previewSize.height, null);
            ByteArrayOutputStream os = new ByteArrayOutputStream(data.length);
            if(!yuvImage.compressToJpeg(new Rect(0,0,previewSize.width,previewSize.height),100,os)){
                Log.e(TAG, "onPreviewFrame: compressToJpeg error" );
                return;
            }
            byte[] bytes = os.toByteArray();
            Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);

           //bitmap 
           //Camera 
            if(curCameraId == Camera.CameraInfo.CAMERA_FACING_BACK){
                bitmap = rotate(bitmap,90);
            }else {
                bitmap = mirror(rotate(bitmap,270));
            }
            saveBitmapToFile(bitmap,"oneShot.jpg");
        }
    });


    mCamera.startPreview();

}
 

void saveBitmapToFile(Bitmap bitmap, String fileName) {File file = new File(MyApplication.getContext().getExternalFilesDir(Environment.DIRECTORY_PICTURES), fileName); if (file.exists()) {file.delete();} try { file.createNewFile(); FileOutputStream fos = new FileOutputStream(file); bitmap.compress(Bitmap.CompressFormat.JPEG, 100, fos); fos.flush(); fos.close();} catch (IOException e) {e .printStackTrace();}}

//Horizontal mirror flip public Bitmap mirror(Bitmap rawBitmap) {Matrix matrix = new Matrix(); matrix.postScale(-1f, 1f); return Bitmap.createBitmap(rawBitmap, 0, 0, rawBitmap.getWidth(), rawBitmap. getHeight(), matrix, true);}

//
public Bitmap rotate(Bitmap rawBitmap, float degree) {
    Matrix matrix = new Matrix();
    matrix.postRotate(degree);
    return Bitmap.createBitmap(rawBitmap, 0, 0, rawBitmap.getWidth(), rawBitmap.getHeight(), matrix, true);
}
 

Camera switching

private void switchCamera() {if (mCamera != null) {releaseCamera(); initCamera((curCameraId == Camera.CameraInfo.CAMERA_FACING_FRONT)? Camera.CameraInfo.CAMERA_FACING_BACK: Camera.CameraInfo.CAMERA_FACING_FRONT); try {mCamera.setPreviewDisplay( mSurfaceHolder);} catch (IOException e) {e.printStackTrace();} startPreview();}}

private void releaseCamera() {
    if (mCamera != null) {
        mCamera.setPreviewCallback(null);
        mCamera.stopPreview();
        mCamera.release();
        mCamera = null;
    }
}
 

3. Problems encountered (page stuck, black screen, upside down, etc.) Problem 1: The picture is stuck after switching the camera. Solution: You need to close the Camera to release the resources, then reopen the switched Camera, reset the PreviewDisplay, and start previewing

Problem 2: After the page is reopened (press the Home button on the preview page to push to the background, and then back to the foreground). The content of the SurfaceView is black. Solution: Check the log and see an exception message: "Camera is being used after Camera.release( ) was called It turned out that the release of Camera was called when surfaceDestroyed, but it was not set to null. In surfaceCreated, it is judged whether it needs to be reinitialized according to whether the camera is empty.

Problem 3: The front camera preview appears inverted and is in a mirrored state

public static void setCamerDisplayOrientation(Activity activity, int cameraId, Camera camera) {Camera.CameraInfo cameraInfo = new Camera.CameraInfo(); Camera.getCameraInfo(cameraId, cameraInfo); int result = 0; if (cameraInfo.facing == Camera. CameraInfo.CAMERA_FACING_FRONT) {result = (cameraInfo.orientation)% 360;} camera.setDisplayOrientation(result); Solution:

`public static void setCamerDisplayOrientation(Activity activity, int cameraId, Camera camera) {Camera.CameraInfo cameraInfo = new Camera.CameraInfo(); Camera.getCameraInfo(cameraId, cameraInfo); int result = 0; if (cameraInfo.facing == Camera .CameraInfo.CAMERA_FACING_FRONT) {result = (cameraInfo.orientation)% 360; result = (360-result)% 360;} camera.setDisplayOrientation(result);

Picture from [Android: Camera Development Detailed Explanation (Part 1)]

Question 4: In many cases, it is not only necessary to preview, but when previewing the video, I hope to do some detection, such as face detection. This requires obtaining the preview frame video, what should I do? Camera provides three methods: setPreviewCallback, setOneShotPreviewCallback, and setPreviewCallbackWithBuffer for users to process frame callbacks. For example, in the following processing, a bitmap of one frame is obtained through setOneShotPreviewCallback, and then saved to a file

mCamera.setOneShotPreviewCallback(new Camera.PreviewCallback() {@Override public void onPreviewFrame(byte[] data, Camera camera) {Log.i(TAG, "onPreviewFrame: setOneShotPreviewCallback"); Camera.Size previewSize = mCamera.getParameters(). getPreviewSize(); YuvImage yuvImage = new YuvImage(data, ImageFormat.NV21, previewSize.width, previewSize.height, null); ByteArrayOutputStream os = new ByteArrayOutputStream(data.length); if(!yuvImage.compressToJpeg(new Rect(0, 0,previewSize.width,previewSize.height),100,os)){ Log.e(TAG, "onPreviewFrame: compressToJpeg error" ); return;} byte[] bytes = os.toByteArray(); Bitmap bitmap = BitmapFactory. decodeByteArray(bytes, 0, bytes.length);

       //bitmap 
      FileUtils.saveBitmapToFile(bitmap,"oneShot.jpg");
    }
});
 

Problem 5: It is found that the orientation of the saved picture and the previewed picture are inconsistent. Solution: The preview is processed by the camera's setDisplay Orientation according to the angle of the front and rear cameras to be rotated, but saving as a picture is not related to the setting of the preview. Separate processing

222.png

Picture from [Android: Camera Development Detailed Explanation (Part 1)]

//We can find out this picture that the direction of the picture saved in the preview is wrong, or the original direction of the Camera, which needs to be rotated by a certain angle.

            if(curCameraId == Camera.CameraInfo.CAMERA_FACING_BACK){
                bitmap = BitmapUtils.rotate(bitmap,90);
            }else {
                bitmap = BitmapUtils.mirror(BitmapUtils.rotate(bitmap,270));
            }
 

public class BitmapUtils {//Horizontal mirror flip public static Bitmap mirror(Bitmap rawBitmap) {Matrix matrix = new Matrix(); matrix.postScale(-1f, 1f); return Bitmap.createBitmap(rawBitmap, 0, 0, rawBitmap.getWidth (), rawBitmap.getHeight(), matrix, true);}

//
public static Bitmap rotate(Bitmap rawBitmap, float degree) {
    Matrix matrix = new Matrix();
    matrix.postRotate(degree);
    return Bitmap.createBitmap(rawBitmap, 0, 0, rawBitmap.getWidth(), rawBitmap.getHeight(), matrix, true);
}
 

} Reference materials "Advanced Audio and Video Development Guide" [Android: Camera Camera Development Detailed Explanation (Part 1)] Android camera2 realizes camera preview and obtains preview frame data stream When does the View start drawing after Activity is started (onCreate or after onResume?)

Gain the practice and understanding of basic knowledge of Camera's Facing, Orientation, Size, PreviewCallback, etc.

Familiar with the camera preview process

Analysis and treatment of black screen, stuck, inverted, mirrored and other issues

Thank you for reading.

In the next article, let s learn and practice MediaExtractor and MediaMuxer to parse and encapsulate MP4 files. Welcome to pay attention and grow together.

Original link