OpenHarmony相机用户态驱动框架原创精华

润和软件HiHope

发布于 2022-4-18 21:30

2.2w浏览

2收藏

作者：润和软件郭新星
相机作为智能手机上少有的成长空间不错的，能够做出差异化的功能，每年都能成为各大Android手机厂商争相宣传的亮点。众所周知Android采用Linux 作为其内核，而Linux采用的开源协议具有传染性^[1]，导致Android HAL^[2]成为了手机厂商们竞争的重要战场。随着OpenHarmony 3.1^[3]的发布，相机模块也逐渐完善起来，目前提供了基础预览和拍照的能力。OpenHarmony中，相机用户态驱动框架承担了和Android Camera HAL一样的角色，这部分位于OpenHarmony的HDF^[4]中，对上实现相机HDI^[5]接口，对下实现相机Pipeline模型，管理相机各个硬件设备。
OpenHarmony相机用户态驱动框架-鸿蒙开发者社区
相机用户态驱动框架（下图的CameraHost 部分）总体可以分为三层，HDI实现层，实现相机标准南向接口；框架层，对接HDI实现层的控制、流的转发，实现数据通路的搭建、管理相机各个硬件设备等功能；适配层，屏蔽底层芯片和OS差异，支持多平台适配。
OpenHarmony相机用户态驱动框架-鸿蒙开发者社区
•模块介绍

HDI Implementation：对上实现HDI接口，向下调用框架层的接口，完成HDI接口任务的转发。

Buffer Manager ：屏蔽不同内存管理的差异，为子系统提供统一的操作接口，同时提供buffer轮转的功能。

Pipeline Core ：解析HCS配置完成pipeline的搭建，调度pipeline中的各个node完成流的处理

Device Manager：通过调用底层硬件适配层接口，实现查询控制底层设备、枚举监听底层设备的功能

Platform Adaption ：屏蔽硬件差异，为Device Manager提供统一的操作底层硬件的能力
•目录结构

Shell
drivers/peripheral/camera

|-- README_zh.md

|-- bundle.json

|-- figures

|   `-- logic-view-of-modules-related-to-this-repository_zh.png

|-- hal

|   |-- BUILD.gn

|   |-- adapter

|   |-- buffer_manager

|   |-- camera.gni

|   |-- device_manager

|   |-- hdi_impl

|   |-- include

|   |-- init

|   |-- pipeline_core

|   |-- test

|   `-- utils

|-- hal_c

|   |-- BUILD.gn

|   |-- camera.gni

|   |-- hdi_cif

|   `-- include

`-- interfaces

    |-- hdi_ipc

    |-- hdi_passthrough

    `-- include

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.

•HDI Implementation中的预览流程

接下来我们通过已经发布的OpenHarmony 3.1开源代码，来看看预览是怎么完成的吧

drivers/peripheral/camera/hal/test/v4l2/src /preview_test.cpp存放了针对v4l2的预览测试代码，入口如下

C++
TEST_F(UtestPreviewTest, camera_preview_0001)

{

    std::cout << "==========[test log] Preview stream, expected success." << std::endl;

    // Get the stream manager

    display_->AchieveStreamOperator(); // 获取stream operator

    // start stream

    display_->intents = {Camera::PREVIEW}; // 预览流

    display_->StartStream(display_->intents); // 起流

    // Get preview

    display_->StartCapture(display_->streamId_preview, display_->captureId_preview, false, true);

    // release stream

    display_->captureIds = {display_->captureId_preview};

    display_->streamIds = {display_->streamId_preview};

    display_->StopStream(display_->captureIds, display_->streamIds);

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.

先获取stream operator实例

C++
void testdisplay::achievestreamoperator()

{

    // create and get streamoperator information

    std::shared_ptr<ohos::camera::istreamoperatorcallback> streamoperatorcallback =

        std::make_shared<ohos::camera::istreamoperatorcallback>();

    rc = cameradevice->getstreamoperator(streamoperatorcallback, streamoperator);

					// ........

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.

通过前文的streamOperator创建流

C++
void TestDisplay::StartStream(std::vector<OHOS::Camera::StreamIntent> intents)

{

    // ..............................

    for (auto& intent : intents) {

        if (intent == 0) {

            std::shared_ptr<IBufferProducer> producer = IBufferProducer::CreateBufferQueue();

            producer->SetQueueSize(8); // 创建buffer的生产端，并和相应的流进行绑定

            auto callback = [this](std::shared_ptr<SurfaceBuffer> Prebuffer) {

                BufferCallback(Prebuffer, preview_mode);

                return;

            };

            producer->SetCallback(callback);

            streamInfo->streamId_ = streamId_preview;

            streamInfo->width_ = 640; // 640:picture width

            streamInfo->height_ = 480; // 480:picture height

            streamInfo->format_ = CAMERA_FORMAT_YUYV_422_PKG;

            streamInfo->datasapce_ = 8; // 8:picture datasapce

            streamInfo->intent_ = intent;

            streamInfo->tunneledMode_ = 5; // 5:tunnel mode

            streamInfo->bufferQueue_ = producer;

            streamInfos.push_back(streamInfo);

        } else if (intent == 1) {

      // .......................

    }

    rc = streamOperator->CreateStreams(streamInfos); // 创建流

    // ................................

    rc = streamOperator->CommitStreams(Camera::NORMAL, ability); // 提交流

    // .................................

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.

下面我们正式进入到hal的源代码中看看是怎么创建流的吧

C++
CamRetCode StreamOperator::CreateStreams(const std::vector<std::shared_ptr<StreamInfo>>& streamInfos)

{

 // .....

    for (auto it : streamInfos) {

//....

        std::shared_ptr<IStream> stream = StreamFactory::Instance().CreateShared(

            IStream::g_availableStreamType[it->intent_], it->streamId_, it->intent_, pipelineCore_, messenger_); // 创建流实例

// ...

        StreamConfiguration scg;

        scg.id = it->streamId_;

        scg.type = it->intent_;

        scg.width = it->width_;

        scg.height = it->height_;

        PixelFormat pf = static_cast<PixelFormat>(it->format_);

        scg.format = BufferAdapter::PixelFormatToCameraFormat(pf);

        scg.dataspace = it->datasapce_;

        scg.tunnelMode = it->tunneledMode_;

        scg.minFrameDuration = it->minFrameDuration_;

        scg.encodeType = it->encodeType_;



        RetCode rc = stream->ConfigStream(scg); // 依据上文的流信息配置流

// ...

        if (it->bufferQueue_ != nullptr) {  // 绑定前文的生产端

            auto tunnel = std::make_shared<StreamTunnel>();

            CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel, INSUFFICIENT_RESOURCES);

            RetCode rc = tunnel->AttachBufferQueue(it->bufferQueue_);

            CHECK_IF_NOT_EQUAL_RETURN_VALUE(rc, RC_OK, INVALID_ARGUMENT);

            if (stream->AttachStreamTunnel(tunnel) != RC_OK) {

                CAMERA_LOGE("attach buffer queue to stream [id = %{public}d] failed", it->streamId_);

                return INVALID_ARGUMENT;

            }

        }

        {

            std::lock_guard<std::mutex> l(streamLock_);

            streamMap_[stream->GetStreamId()] = stream; // 保存流实例

        }

// ...

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.

从上面可以看出，消费端传递到了hal，那必然是由hal从bufferproducer获取buffer，并触发预览的启动流程。那看看AttachStreamTunnel 的实现吧

C++
RetCode StreamBase::AttachStreamTunnel(std::shared_ptr<StreamTunnel>& tunnel)

{

    if (state_ == STREAM_STATE_BUSY || state_ == STREAM_STATE_OFFLINE) {

        return RC_ERROR;

    }



    tunnel_ = tunnel; // 绑定生产端

    CHECK_IF_PTR_NULL_RETURN_VALUE(tunnel_, RC_ERROR);

    tunnel_->SetBufferCount(GetBufferCount()); // 配置轮转的buffer个数

    TunnelConfig config = {(uint32_t)streamConfig_.width, (uint32_t)streamConfig_.height,

        (uint32_t)streamConfig_.format, streamConfig_.usage};

    tunnel_->Config(config);



    streamConfig_.tunnelMode = true;

    return RC_OK;

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.

CreateStream之后便是CommitStream，这里的CommitStream 做了些什么事情呢，我们接着往下看

C++
CamRetCode StreamOperator::CommitStreams(OperationMode mode,

                                         const std::shared_ptr<CameraStandard::CameraMetadata>& modeSetting)

{

// ......

    std::vector<StreamConfiguration> configs = {};

    {

        std::lock_guard<std::mutex> l(streamLock_);

        for (auto it : streamMap_) { // 获取流的配置，前文CreateStrea时保存的流

            configs.emplace_back(it.second->GetStreamAttribute());

        }

    }

	// 检查流是否被支持

    DynamicStreamSwitchMode method = streamPipeline_->CheckStreamsSupported(mode, modeSetting, configs);

    if (method == DYNAMIC_STREAM_SWITCH_NOT_SUPPORT) {

        return INVALID_ARGUMENT;

    }

    if (method == DYNAMIC_STREAM_SWITCH_NEED_INNER_RESTART) {

        std::lock_guard<std::mutex> l(streamLock_);

        for (auto it : streamMap_) {

            it.second->StopStream();// 如果流被支持，但需要内部重启，这里先停流

        }

    }

    {

        std::lock_guard<std::mutex> l(streamLock_);

        for (auto it : streamMap_) {

            if (it.second->CommitStream() != RC_OK) { // 真正的 CommitStream，下面再细说

                CAMERA_LOGE("commit stream [id = %{public}d] failed.", it.first);

                return DEVICE_ERROR;

            }

        }

    }

    RetCode rc = streamPipeline_->PreConfig(modeSetting); // 把模式传入进行预配置

    if (rc != RC_OK) {

        CAMERA_LOGE("prepare mode settings failed");

        return DEVICE_ERROR;

    }

    rc = streamPipeline_->CreatePipeline(mode);// 创建pipeline

    if (rc != RC_OK) {

        CAMERA_LOGE("create pipeline failed.");

        return INVALID_ARGUMENT;

    }



    DFX_LOCAL_HITRACE_END;

    return NO_ERROR;

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
90.
91.

C++
RetCode StreamBase::CommitStream()

{

// ...

    hostStreamMgr_ = pipelineCore_->GetHostStreamMgr(); //从pipelinecore获取hoststreamanager

    CHECK_IF_PTR_NULL_RETURN_VALUE(hostStreamMgr_, RC_ERROR);

// ...

        info.bufferPoolId_ = poolId_;

        info.bufferCount_ = GetBufferCount();

    //  初始化 bufferpool

        RetCode rc = bufferPool_->Init(streamConfig_.width, streamConfig_.height, streamConfig_.usage,

                                       streamConfig_.format, GetBufferCount(), CAMERA_BUFFER_SOURCE_TYPE_EXTERNAL);

        if (rc != RC_OK) {

            CAMERA_LOGE("stream [id:%{public}d] initialize buffer pool failed.", streamId_);

            return RC_ERROR;

        }

    }

// stream传递到pipelinecore 并进行绑定

    RetCode rc = hostStreamMgr_->CreateHostStream(info, [this](std::shared_ptr<IBuffer> buffer) {

        HandleResult(buffer);

        return;

    });

// ....

    return RC_OK;

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.

CreateStream 和CommitStream结束之后便是Capture，这里包含了起流的动作，关键实现如下

C++
CamRetCode StreamOperator::Capture(int captureId, const std::shared_ptr<CaptureInfo>& captureInfo, bool isStreaming)

{

// ...

//  captureId 捕获请求的id; captureInfo 预览/拍照/录像的参数；isStreaming 连续捕获还是单次捕获（拍照）

    CaptureSetting setting = captureInfo->captureSetting_;

    auto request =

        std::make_shared<CaptureRequest>(captureId, captureInfo->streamIds_.size(), setting,

                                         captureInfo->enableShutterCallback_, isStreaming);

    for (auto id : captureInfo->streamIds_) {

        // 创建捕获请求，并传递给前文创建的流

        RetCode rc = streamMap_[id]->AddRequest(request);

        if (rc != RC_OK) {

            return DEVICE_ERROR;

        }

    }

// ...

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.

从上面的代码可知预览、拍照、录像都是通过捕获请求触发，单次拍照则为单次捕获请求，预览和录像则是连续捕获请求。

C++
RetCode StreamBase::AddRequest(std::shared_ptr<CaptureRequest>& request)

{

    CHECK_IF_PTR_NULL_RETURN_VALUE(request, RC_ERROR);

    request->AddOwner(shared_from_this());



    request->SetFirstRequest(false);

    if (isFirstRequest) {

        RetCode rc = StartStream(); // 起流

        if (rc != RC_OK) {

            CAMERA_LOGE("start stream [id:%{public}d] failed", streamId_);

            return RC_ERROR;

        }

        request->SetFirstRequest(true);

        isFirstRequest = false;

    }



    {

        std::unique_lock<std::mutex> l(wtLock_);

        waitingList_.emplace_back(request); // 捕获请求添加到waitingList

        cv_.notify_one();

    }



    return RC_OK;

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.

看看StreamStream是怎么实现的吧

C++
RetCode StreamBase::StartStream()

{

// ...

    RetCode rc = pipeline_->Prepare({streamId_}); //  pipeline先完成一些准备工作

// ...



    state_ = STREAM_STATE_BUSY;

    std::string threadName =

        g_availableStreamType[static_cast<StreamIntent>(streamType_)] + "#" + std::to_string(streamId_);

    handler_ = std::make_unique<std::thread>([this, &threadName] {// 创建轮转线程

        prctl(PR_SET_NAME, threadName.c_str());

        while (state_ == STREAM_STATE_BUSY) {

            HandleRequest(); // 处理捕获请求

        }

    });

// ...

    rc = pipeline_->Start({streamId_}); // 通知pipeline和底层硬件可以开始出帧了

// ...

    return RC_OK;

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.

C++
void StreamBase::HandleRequest()

{

    // 如果有 捕获请求下发，则退出等待状态

    if (waitingList_.empty()) {

        std::unique_lock<std::mutex> l(wtLock_);

        if (waitingList_.empty()) {

            cv_.wait(l, [this] { return !(state_ == STREAM_STATE_BUSY && waitingList_.empty()); });

        }

    }

// ...

        request = waitingList_.front();

        CHECK_IF_PTR_NULL_RETURN_VOID(request);

        if (!request->IsContinous()) { // 如果是连续捕获，则保留一份拷贝在waitinglist

            waitingList_.pop_front();

        }

    }

// 处理捕获请求

    request->Process(streamId_);// 最终调用下面的Capture接口



    return;

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.

C++
RetCode StreamBase::Capture(const std::shared_ptr<CaptureRequest>& request)

{

    CHECK_IF_PTR_NULL_RETURN_VALUE(request, RC_ERROR);

    CHECK_IF_PTR_NULL_RETURN_VALUE(pipeline_, RC_ERROR);



    RetCode rc = RC_ERROR;

    if (request->IsFirstOne() && !request->IsContinous()) {

        uint32_t n = GetBufferCount();

        for (uint32_t i = 0; i < n; i++) {

            DeliverBuffer();// 单次捕获一次性下发所有的buffer

        }

    } else {

        do {

            rc = DeliverBuffer();// 连续捕获每次下发一个buffer

        } while (rc != RC_OK && state_ == STREAM_STATE_BUSY);

    }



    if (request->NeedCancel()) {// 被取消的捕获则退出

        CAMERA_LOGE("StreamBase::Capture stream [id:%{public}d] request->NeedCancel", streamId_);

        return RC_OK;

    }



    rc = pipeline_->Config({streamId_}, request->GetCaptureSetting());// 通知pipeline配置

    if (rc != RC_OK) {

        CAMERA_LOGE("stream [id:%{public}d] config pipeline failed.", streamId_);

        return RC_ERROR;

    }



    rc = pipeline_->Capture({streamId_}, request->GetCaptureId());//  这里的capture指的是pipeline中的source node开始回buffer



    {

        std::unique_lock<std::mutex> l(tsLock_);

        inTransitList_.emplace_back(request);// 处理过的捕获请求存放在inTransitList

    }



    return RC_OK;

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.

到这起流的流程就结束了，pipeline回上来的帧通过OnFrame接口处理

C++
RetCode StreamBase::OnFrame(const std::shared_ptr<CaptureRequest>& request)

{

// ...



    bool isEnded = false;

    if (!request->IsContinous()) {

        isEnded = true;

    } else if (request->NeedCancel()) {

        isEnded = true;

    }



    {

        // inTransitList_ may has multiple copies of continious-capture request, we just need erase one of them.

        std::unique_lock<std::mutex> l(tsLock_);

        for (auto it = inTransitList_.begin(); it != inTransitList_.end(); it++) {

            if ((*it) == request) {

                inTransitList_.erase(it);// 已经回帧的请求，从inTransitList删除

                break;

            }

        }



        if (isEnded) {

            // if this is the last request of capture, send CaptureEndedMessage.

            auto it = std::find(inTransitList_.begin(), inTransitList_.end(), request);

            if (it == inTransitList_.end()) {

                std::shared_ptr<ICaptureMessage> endMessage =

                    std::make_shared<CaptureEndedMessage>(streamId_, request->GetCaptureId(), request->GetEndTime(),

                                                          request->GetOwnerCount(), tunnel_->GetFrameCount());

                CAMERA_LOGV("end of stream [%d], ready to send end message, capture id = %d",

                    streamId_, request->GetCaptureId());

                messenger_->SendMessage(endMessage);

                pipeline_->CancelCapture({streamId_});// 如果此次捕获结束，则取消捕获

            }

        }

    }
    ReceiveBuffer(buffer);// 底层返回的buffer送还到生产端，最终帧数据送到消费端

    return RC_OK;

}

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.

附录

linux和Android的关系 - 知乎 (zhihu.com) ↩︎
HAL Subsystem | Android Open Source Project (google.cn) ↩︎
zh-cn/release-notes/OpenHarmony-v3.1-release.md · OpenHarmony/docs - Gitee.com ↩︎
OpenHarmony HDF 驱动框架介绍和驱动加载过程分析-OpenHarmony技术社区-51CTO.COM ↩︎
OpenHarmony HDF HDI基础能力分析与使用-51CTO.COM ↩︎