OH-v3.0-LTS Camera相机驱动框架(L2)解析5_图像采集原创精华

NL_AIDC_Ryanzx

发布于 2022-2-23 10:47

浏览

1收藏

OpenHarmory-v3.0-LTS Camera相机驱动框架(L2)解析5_图像采集

新大陆自动识别：郑曦

应用层代码

mainDemo->CaptureON(STREAM_ID_PREVIEW, CAPTURE_ID_PREVIEW, CAPTURE_PREVIEW)；

RetCode Hos3516Demo::CaptureON(const int streamId, const int captureId, CaptureMode mode)
{
    CAMERA_LOGD("demo test: CaptureON enter mode == %{public}d", mode);

    std::shared_ptr<Camera::CaptureInfo> captureInfo = std::make_shared<Camera::CaptureInfo>();
    captureInfo->streamIds_ = {streamId};
    captureInfo->captureSetting_ = ability_;
    captureInfo->enableShutterCallback_ = false;

    int rc = streamOperator_->Capture(captureId, captureInfo, true);
    ......
    CAMERA_LOGD("demo test: CaptureON exit");

    return RC_OK;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.

1. StreamOperator::Capture

创建一个CaptureRequest请求，调用AddRequest将request添加到对应的Stream里

//drivers/peripheral/camera/hal/hdi_impl/src/stream_operator/stream_operator.cpp
CamRetCode StreamOperator::Capture(int captureId, const std::shared_ptr<CaptureInfo>& captureInfo, bool isStreaming)
{
...
    CaptureSetting setting = captureInfo->captureSetting_;
    auto request =
        std::make_shared<CaptureRequest>(captureId, captureInfo->streamIds_.size(), setting,
                                         captureInfo->enableShutterCallback_, isStreaming);
    for (auto id : captureInfo->streamIds_) {
        RetCode rc = streamMap_[id]->AddRequest(request);
        if (rc != RC_OK) {
            return DEVICE_ERROR;
        }
    }
...
    return NO_ERROR;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.

StreamBase::AddRequest() 在第一次请求时先调用StartStream()来启动“流”

//drivers/peripheral/camera/hal/hdi_impl/src/stream_operator/stream_base.cpp
RetCode StreamBase::AddRequest(std::shared_ptr<CaptureRequest>& request)
{
    ......
    if (isFirstRequest) {
        ......
        RetCode rc = StartStream();
        if (rc != RC_OK) {
            CAMERA_LOGE("start stream [id:%{public}d] failed", streamId_);
            return RC_ERROR;
        }
        request->SetFirstRequest(true);
        isFirstRequest = false;
    }

    {
        std::unique_lock<std::mutex> l(wtLock_);
        waitingList_.emplace_back(request);
        cv_.notify_one();
    }

    return RC_OK;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.

1、pipeline_->Prepare 和 Start 会分别调用前面创建好的流节点列表中每个Node的init和Start接口
2、创建线程处理HandleRequest()

//drivers/peripheral/camera/hal/hdi_impl/src/stream_operator/stream_base.cpp
RetCode StreamBase::StartStream()
{
    ...

    RetCode rc = pipeline_->Prepare({streamId_});
    ...

    state_ = STREAM_STATE_BUSY;
    std::string threadName =
        g_avaliableStreamType[static_cast<StreamIntent>(streamType_)] + "#" + std::to_string(streamId_);

        handler_ = std::make_unique<std::thread>([this, &threadName] {
        prctl(PR_SET_NAME, threadName.c_str());
        while (state_ == STREAM_STATE_BUSY) {
            HandleRequest();
        }
    });

    rc = pipeline_->Start({streamId_});
    ...

    return RC_OK;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.

StartSteam()在完成了必要的启动流程后会通过HandleRequest()的线程，继续处理当前Request的Process()函数。
CaptureRequest的Process启动的是下面的StramBase的Capture()
1、pipeline_->Config 和 Capture 会分别调用前面创建好的流节点列表中每个Node的Config和Capture接口
2、SendMessage负责向应用层上送start的message
3、最后启动buffer分发函数DeliverBuffer() 开始图像数据的传递。

//drivers/peripheral/camera/hal/hdi_impl/src/stream_operator/stream_base.cpp
RetCode StreamBase::Capture(const std::shared_ptr<CaptureRequest>& request)
{
    ...
    rc = pipeline_->Config({streamId_}, request->GetCaptureSetting());
    ...
    rc = pipeline_->Capture({streamId_}, request->GetCaptureId());
    ...
    if (request->IsFirstOne()) {
        if (messenger_ == nullptr) {
            CAMERA_LOGE("stream [id:%{public}d] can't send message, messenger_ is null", streamId_);
            return RC_ERROR;
        }
        std::shared_ptr<ICaptureMessage> startMessage = std::make_shared<CaptureStartedMessage>(
            streamId_, request->GetCaptureId(), request->GetBeginTime(), request->GetOwnerCount());
        messenger_->SendMessage(startMessage);
        request->SetFirstRequest(false);
    }
    // DeliverBuffer must be called after Capture, or this capture request will miss a buffer.
    do {
        rc = DeliverBuffer();
    } while (rc != RC_OK && state_ == STREAM_STATE_BUSY);

    return RC_OK;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.

到此Capture的请求就处理完成，剩下就是Stream负责让数据流在Node中轮转并通过Surface图形缓冲区接口将数据传递到应用层。
整个的数据轮转涉及到多个线程，下面会把关键的代码整理出来。细节的部分有兴趣的同学可以自己深入去看。

2. Buffer的申请和下送 StreamBase::DeliverBuffer()

tunnel_->GetBuffer() 会向生产型Surface 要一个Buffer,同时创建一个CameraBuffer(IBuffer类型)和SurfaceBuffer 对应。
取到的IBuffer添加到bufferPool缓冲池来管理。

RetCode StreamBase::DeliverBuffer()
{
    CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel_, RC_ERROR);
    CHECK_IF_PTR_NULL_RETURN_VALUE(bufferPool_, RC_ERROR);

    std::shared_ptr<IBuffer> buffer = tunnel_->GetBuffer();
    CHECK_IF_PTR_NULL_RETURN_VALUE(buffer, RC_ERROR);

    buffer->SetEncodeType(streamConfig_.encodeType);
    buffer->SetStreamId(streamId_);
    bufferPool_->AddBuffer(buffer);
    CAMERA_LOGI("stream [id:%{public}d] enqueue buffer index:%{public}d", streamId_, buffer->GetIndex());
    return RC_OK;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

bufferPool里有有效的buffer后，会将IBuffer转换成frameSpec格式，下送给SDKnode->ProvideBuffers(frameSpec)
同样是在Vpss这个Node上实现！

//drivers/peripheral/camera/hal/pipeline_core/nodes/src/source_node/source_node.cpp
void SourceNode::PortHandler::CollectBuffers()
{
    CHECK_IF_PTR_NULL_RETURN_VOID(pool);
    std::shared_ptr<IBuffer> buffer = pool->AcquireBuffer(-1);
    CHECK_IF_PTR_NULL_RETURN_VOID(buffer);

    PortFormat format = {};
    port->GetFormat(format);
    std::shared_ptr<FrameSpec> frameSpec = std::make_shared<FrameSpec>();
    frameSpec->bufferPoolId_ = format.bufferPoolId_;
    frameSpec->bufferCount_ = format.bufferCount_;
    frameSpec->buffer_ = buffer;

    auto node = port->GetNode();
    CHECK_IF_PTR_NULL_RETURN_VOID(node);
    RetCode rc = node->ProvideBuffers(frameSpec);
    if (rc == RC_ERROR) {
        CAMERA_LOGE("provide buffer failed.");
    }
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.

//drivers/peripheral/camera/hal/adapter/chipset/hispark_taurus/src/pipeline_core/nodes/vpss_node/vpss_node.cpp
RetCode VpssNode::ProvideBuffers(std::shared_ptr<FrameSpec> frameSpec)
{
    if (deviceManager_->SendFrameBuffer(frameSpec) == RC_OK) {
        return RC_OK;
    }
    CAMERA_LOGE("provide buffer failed.");
    return RC_ERROR;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.

3. 两个重要的Node回调接口

3.1 VpssNode::SetBufferCallback()

在前面pipeline_->Start()中重要的一个节点是VpssNode (VpssNode is-a SourceNode)
这里分别启动了另外两个线程负责收集Buffer和分发Buffer
另外要看下SetBufferCallback() 给到Hi3516底层的SDK设置重要的回调接口。接口的最终调用是SourceNode::PortHandler::OnBuffer函数。

//drivers/peripheral/camera/hal/pipeline_core/nodes/src/source_node/source_node.cpp
RetCode SourceNode::Start(const int32_t streamId)
{
    ...
    SetBufferCallback();
    ...
    RetCode rc = handler_[streamId]->StartCollectBuffers();
    CHECK_IF_NOT_EQUAL_RETURN_VALUE(rc, RC_OK, RC_ERROR);

    rc = handler_[streamId]->StartDistributeBuffers();
    CHECK_IF_NOT_EQUAL_RETURN_VALUE(rc, RC_OK, RC_ERROR);

    return RC_OK;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

//drivers/peripheral/camera/hal/adapter/chipset/hispark_taurus/src/pipeline_core/nodes/vpss_node/vpss_node.cpp
void VpssNode::SetBufferCallback()
{
    deviceManager_->SetNodeCallBack([&](std::shared_ptr<FrameSpec> frameSpec) {
            OnPackBuffer(frameSpec);
    });
    return;
}
1.
2.
3.
4.
5.
6.
7.
8.

3.2 SinkNode::SetCallBack()

另外一个重要的Node是ViNode（ViNode is-a SinkNode）
这个回调的设置要追溯下前几章的代码
从下面的代码片段可以看到SinkNode的CallBack() 调用的是HandleResult() 这个函数。

//drivers\peripheral\camera\hal\hdi_impl\src\stream_operator\stream_base.cpp
RetCode StreamBase::CommitStream()
{
    ...
    RetCode rc = hostStreamMgr_->CreateHostStream(info, [this](std::shared_ptr<IBuffer> buffer) {
        HandleResult(buffer);
        return;
    });
    ...
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

std::shared_ptr<Pipeline> StreamPipelineBuilder::Build(const std::shared_ptr<PipelineSpec>& pipelineSpec)
{
...
    std::optional<int32_t> typeId = GetTypeId(it.type_, G_STREAM_TABLE_PTR, G_STREAM_TABLE_SIZE);
    if (typeId) {
        newNode->SetCallBack(hostStreamMgr_->GetBufferCb(it.streamId_));
    }
...
}
1.
2.
3.
4.
5.
6.
7.
8.
9.

4. camera图像上送

当camera采集到一帧数据，SDK会通过VpssNode的回调通知Camera HDI层。这个时候就来到了上面VpssNode::SetBufferCallback()设置的
SourceNode::PortHandler::OnBuffer()函数。

OnBuffer通知分发线程将接收到的buffers数据分发给实现了DeliverBuffer()接口的Node。本示例代码中SinkNode的DeliverBuffer()

//drivers/peripheral/camera/hal/pipeline_core/nodes/src/sink_node/sink_node.cpp
void SinkNode::DeliverBuffer(std::shared_ptr<IBuffer>& buffer)
{
    cb_(buffer);
    return;
}
1.
2.
3.
4.
5.
6.

SinkNode的回调StreamBase::HandleResult()最终会来到StreamBase::OnFrame()调用StreamBase::ReceiveBuffer()这个函数;
ReceiveBuffer先把这个buffer归还给bufferPool缓冲池
然后调用StreamTunnel::PutBuffer

//drivers/peripheral/camera/hal/hdi_impl/src/stream_operator/stream_base.cpp
RetCode StreamBase::ReceiveBuffer(std::shared_ptr<IBuffer>& buffer)
{
    CHECK_IF_PTR_NULL_RETURN_VALUE(buffer, RC_ERROR);
    CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel_, RC_ERROR);
    CHECK_IF_PTR_NULL_RETURN_VALUE(bufferPool_, RC_ERROR);

   
    bufferPool_->ReturnBuffer(buffer);
    tunnel_->PutBuffer(buffer);
    return RC_OK;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

StreamTunnel会调用Surface的FlushBuffer接口归还一个生产好的SurfaceBuffer对象并携带一些信息。
到此处上层应用的消费型Surface就可以得到这个图像的数据。


RetCode StreamTunnel::PutBuffer(const std::shared_ptr<IBuffer>& buffer)
{
    ...
    if (buffer->GetBufferStatus() == CAMERA_BUFFER_STATUS_OK) {
        int32_t fence = 0;
        EsFrmaeInfo esInfo = buffer->GetEsFrameInfo();
        if (esInfo.size != -1 && esInfo.timestamp != -1) {
            sb->ExtraSet("dataSize", esInfo.size);
            sb->ExtraSet("isKeyFrame", esInfo.isKey);
            sb->ExtraSet("timeStamp", esInfo.timestamp);
            sb->ExtraSet("frameNum", esInfo.frameNum);
        }
        bufferQueue_->FlushBuffer(sb, fence, flushConfig_);
        frameCount_++;
    } else {
        bufferQueue_->CancelBuffer(sb);
    }

    ...
    return RC_OK;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.

5.小结

整个取图上送的代码涉及到很多个线程之间的协同，代码阅读难度相对较大，需要多点耐心
抓住几个重要的Node来梳理代码。
一个有效的pipeLine 应该必须包含一个SourceNode（VpssNode)和一个SinkNode(VoNode)
本章结束后cameraHDI层的总体框架和工作流程介绍完毕。感谢耐心看完系列章节的同学。个人理解有误的地方还请评论指出。

分类

标签

1条回复

按时间正序

按时间倒序

红叶亦知秋

赶快结合大佬之前的文章一起跑一遍

2022-2-23 11:07:41

51CTO

51CTO博客

51CTO学堂

OH-v3.0-LTS Camera相机驱动框架(L2)解析5_图像采集原创精华

OpenHarmory-v3.0-LTS Camera相机驱动框架(L2)解析5_图像采集

应用层代码

1. StreamOperator::Capture

2. Buffer的申请和下送 StreamBase::DeliverBuffer()

3. 两个重要的Node回调接口

3.1 VpssNode::SetBufferCallback()

3.2 SinkNode::SetCallBack()

4. camera图像上送

5.小结

目录

订阅鸿蒙技术特刊，精选内容抢先看

51CTO

51CTO博客

51CTO学堂

OH-v3.0-LTS Camera相机驱动框架(L2)解析5_图像采集 原创 精华

OpenHarmory-v3.0-LTS Camera相机驱动框架(L2)解析5_图像采集

应用层代码

1. StreamOperator::Capture

2. Buffer的申请和下送 StreamBase::DeliverBuffer()

3. 两个重要的Node回调接口

3.1 VpssNode::SetBufferCallback()

3.2 SinkNode::SetCallBack()

4. camera图像上送

5.小结

目录

订阅鸿蒙技术特刊，精选内容抢先看

OH-v3.0-LTS Camera相机驱动框架(L2)解析5_图像采集原创精华