Camera capture session android

Understanding Android camera capture sessions and requests

Following up on the previous blog post about camera enumeration, let’s take a look at two major components of the Android camera framework: capture sessions and capture requests.

One CameraDevice, multiple streams

A single Android device can have multiple cameras. Each camera is a CameraDevice, and a CameraDevice can output multiple streams simultaneously. Why would we do that? Well, one stream might be optimized for a specific use-case, such as displaying a viewfinder, while others might be used to take a photo or make a video recording. We can think of the streams as parallel pipelines that process raw frames coming out of the camera, one frame at a time:

Parallel processing implies that there could be performance limits depending on the available processing power from the CPU, GPU, or something else. If a pipeline can’t keep up with the incoming frames it starts dropping them.

Note that each pipeline has its own output format. The raw data coming in is automatically transformed into the appropriate output format by implicit logic associated with each pipeline. More about this later.

The CameraDevice can be used to create a CameraCaptureSession, which will be specific to that CameraDevice. A CameraDevice must receive a frame configuration for each output raw frame via the CameraCaptureSession. One configuration in, one raw frame out. The configuration specifies camera attributes such as autofocus, aperture, effects, and exposure. Due to hardware constraints, only a single configuration can be active in the camera sensor at any given time; this is called the active configuration.

A CameraCaptureSession describes all the possible pipelines available to the CameraDevice. Once a session is created, you cannot add or remove pipelines. The CameraCaptureSession maintains a queue of CaptureRequests which become the active configuration.

A CaptureRequest adds a configuration to the queue and selects one or more (or all) of the available pipelines to receive a frame from the CameraDevice. You can send many capture requests over the life of a capture session. Each request can change the active configuration and set of output pipelines that will receive the raw image.

Creating a CameraCaptureSession

To create a camera session, we need to provide it with one or more buffers where output frames can be written. Each buffer represents a pipeline. This must be done before you start using the camera so the framework can configure the device’s internal pipelines and allocate memory buffers for sending frames to the desired output targets.

Here is how we could prepare a camera session with two output buffers, one belonging to a SurfaceView and another to an ImageReader:

Note that at this point we have not defined the camera’s active configuration. Once the session has been configured, we can create and dispatch capture requests to do that.

Remember that we said above that each pipeline knows how to transform the input written to its buffer into the appropriate format? That transformation is determined by the type of each target, which must be a Surface. The Android framework knows how to convert a raw image in the active configuration into a format appropriate for each target. The conversion is controlled by the pixel format and size of the particular Surface. The framework tries to do its best, but a Surface may have a configuration that won’t work, in which case some bad things might happen: the session can’t be created, you’ll get a runtime error when you dispatch a request, or performance might degrade. The framework provides guarantees for specific combinations of device, surface, and request parameters. We’ll discuss all this in a future post. (In the meantime, if you’re curious, read the documentation for createCaptureSession.)

One-shot CaptureRequests

The configuration used for each frame is encoded in a CaptureRequest, which is sent to the camera. To create a capture request, we can use one of the predefined templates (or we can use TEMPLATE_MANUAL for full control). Once we have chosen a template, we need to provide one or more target output buffers to be used with the request. We can only use buffers that were already defined on the capture session we intend to use.

Capture requests use a builder pattern and give developers the opportunity to set many different options including auto-exposure, auto-focus and lens aperture. Before setting a field, make sure that the specific option is available for the device by calling CameraCharacteristics.getAvailableCaptureRequestKeys() and that the desired value is supported by checking the appropriate camera characteristic — for example, available auto-exposure modes.

To create a simple capture request for our SurfaceView using the template designed for video preview without any modifications, use CameraDevice.TEMPLATE_PREVIEW:

Armed with a capture request, we can finally dispatch it to the camera session:

When an output frame is put into the target buffer(s), a capture callback is triggered. In many cases additional callbacks may be triggered as well once the frame it contains has been processed, for example ImageReader.OnImageAvailableListener. It is at this point that we can retrieve image data out of the target buffer.

Repeating CaptureRequests

One-shot camera requests are easy to do, but for displaying a live preview they aren’t very useful. In that case, we would like to receive a continuous stream of frames, not just a single one. Luckily, there is a way to set a repeating request to the session:

A repeating capture request will make the camera device continually capture images using the settings in the provided CaptureRequest, at the maximum rate possible.

Interleaving CaptureRequests

To make things a bit more complicated and closer to a real-world scenario… What if we wanted to send a second capture request while the repeating capture request is active? That’s what an app needs to do to display a viewfinder and let users capture a photo. In that case, we don’t need to stop the ongoing repeating request, we can simply issue a non-repeating capture request — just remember that any output target buffer being used needs to be configured as part of the camera session when the session is first created. It is important to note that repeating requests have lower priority than one-shot or burst requests, which lets us do something like this:

However, there is a big gotcha with this approach: we don’t know exactly when the single request will occur. In the figure below, if A is the repeating capture request and B is the one-shot capture request, this is how the session would process the request queue:

There are no guarantees for the latency between the last repeating request from A before request B kicks in and the next time that A is being used again, so we may experience some skipped frames. There are a few things we can do to mitigate this problem:

1. Add the output targets from request A also to request B. That way, when B’s frame is ready, it will be copied into A’s output targets. This is essential, for example, when doing video snapshots to maintain a steady frame rate. In the code above, we would simply add singleRequest.addTarget(previewSurface) before building the request.
2. Use a combination of templates designed to work for this particular scenario, such as zero-shutter-lag (often abbreviated to ZSL).
3. If the configuration of the single capture request is the same as the repeating request, there is no need for multiple requests at all. Set a single repeating request to output to both the target of the repeating request as well as the target of the single request. Even though they must have the same capture request settings, they can still be of different sizes and formats.

Lesson learned

It’s important to understand camera sessions and capture requests when creating applications that use the Android camera API. In this blog post, we have covered:

1. How camera sessions work
2. How capture requests work, including one-shot and repeating capture requests
3. How capture requests are added to a queue, and different techniques for dispatching them without disrupting ongoing streams

In a future post, we will learn how to handle the complexities of requests with more than one target as part of one session and how to configure the camera pipelines to get the most out of a device’s capabilities.

Источник

CameraCaptureSession

java.lang.Object
↳	android.hardware.camera2.CameraCaptureSession

Known Direct Subclasses CameraConstrainedHighSpeedCaptureSession A constrained high speed capture session for a CameraDevice , used for capturing high speed images from the CameraDevice for high speed video recording use case. Class Overview A configured capture session for a CameraDevice , used for capturing images from the camera or reprocessing images captured from the camera in the same session previously. A CameraCaptureSession is created by providing a set of target output surfaces to , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createCaptureSession , or by providing an InputConfiguration and a set of target output surfaces to , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createReprocessableCaptureSession for a reprocessable capture session. Once created, the session is active until a new session is created by the camera device, or the camera device is closed. All capture sessions can be used for capturing images from the camera but only reprocessable capture sessions can reprocess images captured from the camera in the same session previously. Creating a session is an expensive operation and can take several hundred milliseconds, since it requires configuring the camera device’s internal pipelines and allocating memory buffers for sending images to the desired targets. Therefore the setup is done asynchronously, and , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createCaptureSession and , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createReprocessableCaptureSession will send the ready-to-use CameraCaptureSession to the provided listener’s onConfigured callback. If configuration cannot be completed, then the onConfigureFailed is called, and the session will not become active. onConfigureFailed is called, all queued capture requests are discarded. If a new session is created by the camera device, then the previous session is closed, and its associated onClosed callback will be invoked. All of the session methods will throw an IllegalStateException if called once the session is closed. A closed session clears any repeating requests (as if stopRepeating() had been called), but will still complete all of its in-progress capture requests as normal, before a newly created session takes over and reconfigures the camera device. Summary A callback object for tracking the progress of a CaptureRequest submitted to the camera device.

CameraCaptureSession.StateCallback

A callback object for receiving updates about the state of a camera capture session.

Public Constructors

Submit a request for an image to be captured by the camera device.

Pre-allocate all buffers for an output Surface.

Request endlessly repeating capture of a sequence of images by this capture session.

Public Constructors

public CameraCaptureSession ()

Public Methods

public abstract void abortCaptures ()

Discard all captures currently pending and in-progress as fast as possible.

The camera device will discard all of its current work as fast as possible. Some in-flight captures may complete successfully and call onCaptureCompleted(CameraCaptureSession, CaptureRequest, TotalCaptureResult) , while others will trigger their onCaptureFailed(CameraCaptureSession, CaptureRequest, CaptureFailure) callbacks. If a repeating request or a repeating burst is set, it will be cleared.

This method is the fastest way to switch the camera device to a new session with , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createCaptureSession(List , CameraCaptureSession.StateCallback, Handler) or , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createReprocessableCaptureSession(InputConfiguration, List , CameraCaptureSession.StateCallback, Handler) , at the cost of discarding in-progress work. It must be called before the new session is created. Once all pending requests are either completed or thrown away, the onReady(CameraCaptureSession) callback will be called, if the session has not been closed. Otherwise, the onClosed(CameraCaptureSession) callback will be fired when a new session is created by the camera device.

Cancelling will introduce at least a brief pause in the stream of data from the camera device, since once the camera device is emptied, the first new request has to make it through the entire camera pipeline before new output buffers are produced.

This means that using abortCaptures() to simply remove pending requests is not recommended; it’s best used for quickly switching output configurations, or for cancelling long in-progress requests (such as a multi-second capture).

Throws

Public Methods

CameraAccessException	if the camera device is no longer connected or has encountered a fatal error
IllegalStateException	if this session is no longer active, either because the session was explicitly closed, a new session has been created or the camera device has been closed.

See Also

public abstract int capture (CaptureRequest request, CameraCaptureSession.CaptureCallback listener, Handler handler)

Submit a request for an image to be captured by the camera device.

The request defines all the parameters for capturing the single image, including sensor, lens, flash, and post-processing settings.

Each request will produce one CaptureResult and produce new frames for one or more target Surfaces, set with the CaptureRequest builder’s addTarget(Surface) method. The target surfaces (set with addTarget(Surface) ) must be a subset of the surfaces provided when this capture session was created.

Multiple regular and reprocess requests can be in progress at once. If there are only regular requests or reprocess requests in progress, they are processed in first-in, first-out order. If there are both regular and reprocess requests in progress, regular requests are processed in first-in, first-out order and reprocess requests are processed in first-in, first-out order, respectively. However, the processing order of a regular request and a reprocess request in progress is not specified. In other words, a regular request will always be processed before regular requets that are submitted later. A reprocess request will always be processed before reprocess requests that are submitted later. However, a regular request may not be processed before reprocess requests that are submitted later.

All capture sessions can be used for capturing images from the camera but only capture sessions created by , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createReprocessableCaptureSession can submit reprocess capture requests. Submitting a reprocess request to a regular capture session will result in an IllegalArgumentException .

Parameters

request	the settings for this capture
listener	The callback object to notify once this request has been processed. If null, no metadata will be produced for this capture, although image data will still be produced.
handler	the handler on which the listener should be invoked, or null to use the current thread’s looper .

Returns

• int A unique capture sequence ID used by onCaptureSequenceCompleted(CameraCaptureSession, int, long) .

Throws

CameraAccessException	if the camera device is no longer connected or has encountered a fatal error
IllegalStateException	if this session is no longer active, either because the session was explicitly closed, a new session has been created or the camera device has been closed.
IllegalArgumentException	if the request targets no Surfaces or Surfaces that are not configured as outputs for this session; or the request targets a set of Surfaces that cannot be submitted simultaneously in a reprocessable capture session; or a reprocess capture request is submitted in a non-reprocessable capture session; or the reprocess capture request was created with a TotalCaptureResult from a different session; or the capture targets a Surface in the middle of being prepared ; or the handler is null, the listener is not null, and the calling thread has no looper.

See Also

public abstract int captureBurst (List requests, CameraCaptureSession.CaptureCallback listener, Handler handler)

Submit a list of requests to be captured in sequence as a burst. The burst will be captured in the minimum amount of time possible, and will not be interleaved with requests submitted by other capture or repeat calls.

Regular and reprocess requests can be mixed together in a single burst. Regular requests will be captured in order and reprocess requests will be processed in order, respectively. However, the processing order between a regular request and a reprocess request is not specified. Each capture produces one CaptureResult and image buffers for one or more target surfaces . The target surfaces (set with addTarget(Surface) ) must be a subset of the surfaces provided when this capture session was created.

The main difference between this method and simply calling capture(CaptureRequest, CameraCaptureSession.CaptureCallback, Handler) repeatedly is that this method guarantees that no other requests will be interspersed with the burst.

All capture sessions can be used for capturing images from the camera but only capture sessions created by , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createReprocessableCaptureSession can submit reprocess capture requests. Submitting a reprocess request to a regular capture session will result in an IllegalArgumentException .

Parameters

requests	the list of settings for this burst capture
listener	The callback object to notify each time one of the requests in the burst has been processed. If null, no metadata will be produced for any requests in this burst, although image data will still be produced.
handler	the handler on which the listener should be invoked, or null to use the current thread’s looper .

Returns

• int A unique capture sequence ID used by onCaptureSequenceCompleted(CameraCaptureSession, int, long) .

Throws

CameraAccessException	if the camera device is no longer connected or has encountered a fatal error
IllegalStateException	if this session is no longer active, either because the session was explicitly closed, a new session has been created or the camera device has been closed.
IllegalArgumentException	If the requests target no Surfaces, or the requests target Surfaces not currently configured as outputs; or one of the requests targets a set of Surfaces that cannot be submitted simultaneously in a reprocessable capture session; or a reprocess capture request is submitted in a non-reprocessable capture session; or one of the reprocess capture requests was created with a TotalCaptureResult from a different session; or one of the captures targets a Surface in the middle of being prepared ; or if the handler is null, the listener is not null, and the calling thread has no looper.

See Also

public abstract void close ()

Close this capture session asynchronously.

Closing a session frees up the target output Surfaces of the session for reuse with either a new session, or to other APIs that can draw to Surfaces.

Note that creating a new capture session with , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createCaptureSession(List , CameraCaptureSession.StateCallback, Handler) will close any existing capture session automatically, and call the older session listener’s onClosed(CameraCaptureSession) callback. Using , android.hardware.camera2.CameraCaptureSession.StateCallback, android.os.Handler)»>createCaptureSession(List , CameraCaptureSession.StateCallback, Handler) directly without closing is the recommended approach for quickly switching to a new session, since unchanged target outputs can be reused more efficiently.

Once a session is closed, all methods on it will throw an IllegalStateException, and any repeating requests or bursts are stopped (as if stopRepeating() was called). However, any in-progress capture requests submitted to the session will be completed as normal; once all captures have completed and the session has been torn down, onClosed(CameraCaptureSession) will be called.

Closing a session is idempotent; closing more than once has no effect.

public abstract CameraDevice getDevice ()

Get the camera device that this session is created for.

public abstract Surface getInputSurface ()

Get the input Surface associated with a reprocessable capture session.

Each reprocessable capture session has an input Surface where the reprocess capture requests get the input images from, rather than the camera device. The application can create a ImageWriter with this input Surface and use it to provide input images for reprocess capture requests. When the reprocessable capture session is closed, the input Surface is abandoned and becomes invalid.

Returns

• The Surface where reprocessing capture requests get the input images from. If this is not a reprocess capture session, null will be returned.

See Also

public abstract boolean isReprocessable ()

Return if the application can submit reprocess capture requests with this camera capture session.

Returns

• true if the application can submit reprocess capture requests with this camera capture session. false otherwise.

See Also

public abstract void prepare (Surface surface)

Pre-allocate all buffers for an output Surface.

Normally, the image buffers for a given output Surface are allocated on-demand, to minimize startup latency and memory overhead.

However, in some cases, it may be desirable for the buffers to be allocated before any requests targeting the Surface are actually submitted to the device. Large buffers may take some time to allocate, which can result in delays in submitting requests until sufficient buffers are allocated to reach steady-state behavior. Such delays can cause bursts to take longer than desired, or cause skips or stutters in preview output.

The prepare() method can be used to perform this preallocation. It may only be called for a given output Surface before that Surface is used as a target for a request. The number of buffers allocated is the sum of the count needed by the consumer providing the output Surface, and the maximum number needed by the camera device to fill its pipeline. Since this may be a larger number than what is actually required for steady-state operation, using prepare may result in higher memory consumption than the normal on-demand behavior results in. Prepare() will also delay the time to first output to a given Surface, in exchange for smoother frame rate once the allocation is complete.

For example, an application that creates an ImageReader with a maxImages argument of 10, but only uses 3 simultaneous Images at once would normally only cause those 3 images to be allocated (plus what is needed by the camera device for smooth operation). But using prepare() on the ImageReader Surface will result in all 10 Images being allocated. So applications using this method should take care to request only the number of buffers actually necessary for their application.

If the same output Surface is used in consecutive sessions (without closing the first session explicitly), then its already-allocated buffers are carried over, and if it was used as a target of a capture request in the first session, prepare cannot be called on it in the second session.

Once allocation is complete, onSurfacePrepared(CameraCaptureSession, Surface) will be invoked with the Surface provided to this method. Between the prepare call and the onSurfacePrepared call, the Surface provided to prepare must not be used as a target of a CaptureRequest submitted to this session.

LEGACY devices cannot pre-allocate output buffers; for those devices, onSurfacePrepared(CameraCaptureSession, Surface) will be immediately called, and no preallocation is done.

Parameters

surface	the output Surface for which buffers should be pre-allocated. Must be one of the output Surfaces used to create this session.

Throws

CameraAccessException	if the camera device is no longer connected or has encountered a fatal error
IllegalStateException	if this session is no longer active, either because the session was explicitly closed, a new session has been created or the camera device has been closed.
IllegalArgumentException	if the Surface is invalid, not part of this Session, or has already been used as a target of a CaptureRequest in this session or immediately prior sessions.

See Also

public abstract int setRepeatingBurst (List requests, CameraCaptureSession.CaptureCallback listener, Handler handler)

Request endlessly repeating capture of a sequence of images by this capture session.

With this method, the camera device will continually capture images, cycling through the settings in the provided list of CaptureRequests , at the maximum rate possible.

If a request is submitted through capture(CaptureRequest, CameraCaptureSession.CaptureCallback, Handler) or , android.hardware.camera2.CameraCaptureSession.CaptureCallback, android.os.Handler)»>captureBurst(List , CameraCaptureSession.CaptureCallback, Handler) , the current repetition of the request list will be completed before the higher-priority request is handled. This guarantees that the application always receives a complete repeat burst captured in minimal time, instead of bursts interleaved with higher-priority captures, or incomplete captures.

Repeating burst requests are a simple way for an application to maintain a preview or other continuous stream of frames where each request is different in a predicatable way, without having to continually submit requests through , android.hardware.camera2.CameraCaptureSession.CaptureCallback, android.os.Handler)»>captureBurst(List , CameraCaptureSession.CaptureCallback, Handler) .

To stop the repeating capture, call stopRepeating() . Any ongoing burst will still be completed, however. Calling abortCaptures() will also clear the request.

Calling this method will replace a previously-set repeating request or burst set up by this method or setRepeatingRequest(CaptureRequest, CameraCaptureSession.CaptureCallback, Handler) , although any in-progress burst will be completed before the new repeat burst will be used.

This method does not support reprocess capture requests because each reprocess CaptureRequest must be created from the TotalCaptureResult that matches the input image to be reprocessed. This is either the TotalCaptureResult of capture that is sent for reprocessing, or one of the TotalCaptureResults of a set of captures, when data from the whole set is combined by the application into a single reprocess input image. The request must be capturing images from the camera. If a reprocess capture request is submitted, this method will throw IllegalArgumentException.

Источник