Rendering offscreen Edge (WebView2) to D3D11 texture for overlays (C++/win32)

Edge WebView2 on D3D11.md

Using Edge WebView2 is really straightforward, but using the Composition controller is poorly documented and only discussed on few issues in the WebView2 repo.

The browser itself is rendered on a separate process using DirectComposition API wich does not directly interoperate with DirectX, but can be readed using GraphicsCaptureItem WinRT apis. This means that a capture session must be setup into the host visual root, and it will provide accelerated GPU texture capture of the browser (with transparency)

There are few extra steps to setup this pipeline under win32 apps, apart of usual WebView2 setup:

1. A DispatcherQueueController need to be created in the hosting thread in order to be able to use the Visual interfaces
2. Use CreateCoreWebView2CompositionController instead CreateCoreWebView2Controller or its WithConfig equivalent.
3. Create a WinRT DirectComposition IContainerVisual and IVisual as in the example
4. Create a WinRT Direct3D11CaptureFramePool and start the session using GraphicsCaptureItem::CreateFromVisual
5. Handle input mouse and keyboard events (in example)
6. Handle resizing window, will resize frame capture session, buffers... etc...
7. Rendering the captured texture to D3D11 swapchain

The following example can be embedded both in the usual hwnd way and rendering to D3D11 texture, and contains a bunch of useless code that shows how to use different WebView2 APIs

main.cpp

////////////////////////////////////
// Choose impl: 
//  - offscreen  = render to d3d11 texture (hwnd is source of input events)
//  - !offscreen = render to hwnd
////////////////////////////////////
bool OFFSCREEN_RENDERING = true;
////////////////////////////////////
#include <stdio.h>
#include "framework.h"
#include "Browser.h"
#include <shellapi.h>
#include <Commctrl.h>
#pragma comment(lib, "Comctl32")
#include <dwmapi.h>
#pragma comment(lib, "dwmapi")
#include <uxtheme.h>
#pragma comment(lib, "uxtheme")
#define interface  struct
#include "webview2/include/WebView2.h"
#pragma comment(lib, "webview2/x64/WebView2LoaderStatic.lib")
#ifdef _DEBUG
#include <dxgidebug.h>
#pragma comment(lib, "dxguid")
#endif
#define safe_release(x) if(x) { x->Release(); x = nullptr; }
#pragma comment(lib,"windowsapp")
#pragma comment(lib,"d3d11")
#include <winrt/base.h>
#include <winrt/windows.ui.composition.desktop.h>
#include <winrt/windows.ui.composition.h>
#include <winrt/windows.ui.h>
#include <winrt/Windows.System.h>
#include <ShellScalingAPI.h>
#include <DispatcherQueue.h>
#include <windows.ui.composition.interop.h>
#include <d3d11.h>
#include <Unknwn.h>
#include <winrt/Windows.Foundation.h>
#include <winrt/Windows.Storage.Streams.h>
#include <winrt/Windows.System.h>
#include <winrt/Windows.UI.h>
#include <winrt/Windows.UI.Composition.h>
#include <winrt/Windows.UI.Composition.Desktop.h>
#include <winrt/Windows.UI.Popups.h>
#include <winrt/Windows.Graphics.Capture.h>
#include <winrt/Windows.Graphics.DirectX.h>
#include <winrt/Windows.Graphics.DirectX.Direct3d11.h>
#include <winrt/Windows.Graphics.DirectX.Direct3D11.h>
#include <DispatcherQueue.h>
#include <shobjidl_core.h>
#include <windows.graphics.capture.interop.h>
#include <Windows.Graphics.DirectX.Direct3D11.interop.h>
#include <wrl.h>
#include <objidl.h>
#include <gdiplus.h>
#pragma comment (lib,"Gdiplus.lib")

namespace winrt {
    using namespace std::literals;
    using namespace Windows::System;
    using namespace Windows::Graphics;
    using namespace Windows::Graphics::Capture;
    using namespace Windows::Graphics::DirectX;
    using namespace Windows::Graphics::DirectX::Direct3D11;
    using namespace Windows::UI::Composition;
}
namespace rt {
    using namespace winrt::Windows::Foundation;
    using namespace ABI::Windows::Graphics::Capture;
}
namespace wrl {
    using namespace Microsoft::WRL;
}

#define MAX_LOADSTRING 100

class edge_browser;

Gdiplus::GdiplusStartupInput        gdip_input_start{ nullptr };
ULONG_PTR                           gdip_token{};
HINSTANCE                           hinst{ nullptr };
ID3D11DeviceContext*                context{ nullptr };
IDXGISwapChain*                     swapchain{ nullptr };
edge_browser*                       browser{ nullptr };

struct edge_config
{
    HWND                            hwnd;
    BOOL                            offscreen; 
    union
    {
        struct
        {
            ID3D11Device*           device{ nullptr };
        } 
        edge_d3d11;

        struct
        {
        }
        edge_hwnd;
    };

    // add other properties...
} ;


class edge_browser : 
    public ICoreWebView2CreateCoreWebView2EnvironmentCompletedHandler,
    public ICoreWebView2CreateCoreWebView2ControllerCompletedHandler,
    public ICoreWebView2WebMessageReceivedEventHandler,
    public ICoreWebView2ProcessFailedEventHandler,
    public ICoreWebView2ExecuteScriptCompletedHandler,
    public ICoreWebView2ScriptDialogOpeningEventHandler,
    public ICoreWebView2DocumentTitleChangedEventHandler,
    public ICoreWebView2CreateCoreWebView2CompositionControllerCompletedHandler
{
private:

    void handle_failed_edge_load()
    {
        //"https://go.microsoft.com/fwlink/p/?LinkId=2124703" // direct download
        int result;
        if (SUCCEEDED(TaskDialog(NULL, hinst, L"Browser", 0, L"Edge runtime was not found. Do you want to download the runtime now?", TDCBF_YES_BUTTON | TDCBF_NO_BUTTON, 0, &result)) && IDYES == result)
            ShellExecuteA(0, NULL, "https://developer.microsoft.com/microsoft-edge/webview2", NULL, NULL, SW_SHOWDEFAULT);
        exit(0);
    }

public:

    edge_browser(edge_config* _edge) : edge(_edge)
    {

        // setup com and winrt for calling thread
        CoInitializeEx(nullptr, COINIT_APARTMENTTHREADED);
        winrt::init_apartment(winrt::apartment_type::single_threaded);

        if (edge->offscreen)
        {
            // a win32 app needs a DispatcherQueueController running to use winrt ui Compositor (to use Visual)
            DispatcherQueueOptions qcoptions{ sizeof(DispatcherQueueOptions), DQTYPE_THREAD_CURRENT, DQTAT_COM_STA };
            winrt::check_hresult(CreateDispatcherQueueController(qcoptions, reinterpret_cast<ABI::Windows::System::IDispatcherQueueController**>(winrt::put_abi(queue_controller))));
        }

        // run edge api entry-point. we dont use any special setting so no need to call CreateCoreWebView2EnvironmentWithOptions
        if (FAILED(CreateCoreWebView2Environment(this))) handle_failed_edge_load();
    }

private:

    inline void release_resources()
    {
        if (controller) controller->Close();

        safe_release(comp);
        safe_release(webview);
        safe_release(settings);
        safe_release(controller);
        safe_release(env);
        
        if (m_session)   m_session.Close();
        if (m_framePool) m_framePool.Close();
        m_framePool = nullptr;
        m_session = nullptr;
        m_item = nullptr;
    }

    ~edge_browser() 
    {
        if (webview) webview->remove_WebMessageReceived(CoreWebView2WebMessageReceivedEventRegistrationToken);
        if (webview) webview->remove_ProcessFailed(CoreWebView2ProcessFailedEventRegistrationToken);
        if (webview) webview->remove_ScriptDialogOpening(CoreWebView2ScriptDialogOpeningEventRegistrationToken);
        if (webview) webview->remove_DocumentTitleChanged(CoreWebView2DocumentTitleChangedEventRegistrationToken);
        if (webview) webview->remove_FaviconChanged(CoreWebView2FaviconChangedEventRegistrationToken);
        if (webview) webview->remove_PermissionRequested(CoreWebView2PermissionRequestedEventRegistrationToken);
        if (comp)    comp->remove_CursorChanged(CoreWebView2CursorChangedEventRegistrationToken);

        release_resources();

        if (queue_controller) queue_controller.ShutdownQueueAsync();

        winrt::uninit_apartment();
        CoUninitialize();
    }

public:

    bool translate_msg_proc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
    {
        if (!comp) return false;
        if (GetKeyState(VK_F12) & 0x8000) { webview->OpenDevToolsWindow();    return true; }
        if (GetKeyState(VK_F11) & 0x8000) { webview->Print(print_settings, 0);return true; }
        if (GetKeyState(VK_F10) & 0x8000) { webview->OpenTaskManagerWindow(); return true; }
        track_mouse();
        if (message != WM_MOUSEWHEEL) point = { LOWORD(lParam), HIWORD(lParam) };
        int flag = COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE;
        if (GetKeyState(VK_SHIFT)   & 0x8000) flag |= COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_SHIFT;
        if (GetKeyState(VK_CONTROL) & 0x8000) flag |= COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_CONTROL;
        #define VIRT_FLAG(x) (COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS)((last_button = x + set_capture_mouse(x != 0)) | flag)
        switch (message)
        {
        case WM_MOUSEMOVE:   comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_MOVE,              VIRT_FLAG(last_button), 0, point); break;
        case WM_LBUTTONDOWN: comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_DOWN,  VIRT_FLAG(COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_LEFT_BUTTON),  0, point); break;
        case WM_LBUTTONUP:   comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEFT_BUTTON_UP,    VIRT_FLAG(COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE),         0, point); break;
        case WM_RBUTTONDOWN: comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_RIGHT_BUTTON_DOWN, VIRT_FLAG(COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_RIGHT_BUTTON), 0, point); break;
        case WM_RBUTTONUP:   comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_RIGHT_BUTTON_UP,   VIRT_FLAG(COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE),         0, point); break;
        case WM_MBUTTONDOWN: comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_MIDDLE_BUTTON_DOWN,VIRT_FLAG(COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_MIDDLE_BUTTON),0, point); break;
        case WM_MBUTTONUP:   comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_MIDDLE_BUTTON_UP,  VIRT_FLAG(COREWEBVIEW2_MOUSE_EVENT_VIRTUAL_KEYS_NONE),         0, point); break;
        case WM_MOUSEWHEEL:  comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_WHEEL,             VIRT_FLAG(last_button), static_cast<unsigned>(GET_WHEEL_DELTA_WPARAM(wParam)), point); break;
        case WM_MOUSELEAVE:  comp->SendMouseInput(COREWEBVIEW2_MOUSE_EVENT_KIND_LEAVE,             VIRT_FLAG(last_button), 0, point); tracking_on = false;  break;
        case WM_SETCURSOR:   { HCURSOR cur=0; if (SUCCEEDED(comp->get_Cursor(&cur)) && cur) SetCursor(cur); } break;
        default:             return false;
        }
        return true;
    }
    
    void resize(int w, int h)
    {
        if (controller && IsWindow(edge->hwnd))
        {
            if (w <= 0 || h <= 0) return;
            controller->put_Bounds({ 0, 0, w, h });
            update_browser_window();
            if (edge->offscreen && m_framePool) m_framePool.Recreate(m_device, m_pixelFormat, 2, { w, h });
            if (swapchain) swapchain->ResizeBuffers(2, w, h, DXGI_FORMAT_B8G8R8A8_UNORM, 0); // todo, this sould not be here...
        }
    }

    ID3D11Texture2D* texture()
    {
        if (!m_framePool) return nullptr;
        ID3D11Texture2D* tex = nullptr;
        auto frame = m_framePool.TryGetNextFrame();
        if (!frame) return nullptr;
        auto access = frame.Surface().as<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();
        if (!access) return nullptr;
        winrt::check_hresult(access->GetInterface(winrt::guid_of<ID3D11Texture2D>(), (void**)&tex));
        return tex; // dont forget to release!
    }
    
    // IUnknown
    virtual HRESULT STDMETHODCALLTYPE QueryInterface(REFIID riid, void **ppvObject) override
    {
        return E_NOINTERFACE;
    }

    virtual ULONG STDMETHODCALLTYPE AddRef(void) override
    {
        return InterlockedIncrement(&reference_count);
    }

    virtual ULONG STDMETHODCALLTYPE Release(void) override
    {
        auto mc = InterlockedDecrement(&reference_count);
        if (mc == 0) 
            delete this;
        return mc;
    }

    // helper: notify repositioning
    void update_browser_window() { if (controller) controller->NotifyParentWindowPositionChanged(); }

private:

    // ICoreWebView2CreateCoreWebView2EnvironmentCompletedHandler
    virtual HRESULT STDMETHODCALLTYPE Invoke(HRESULT errorCode, ICoreWebView2Environment *_env) override
    {
        if (SUCCEEDED(errorCode))
        {
            _env->QueryInterface(&env);
            safe_release(_env);

            if (!edge->offscreen) env->CreateCoreWebView2Controller(edge->hwnd, this);
            else       env->CreateCoreWebView2CompositionController(edge->hwnd, this);

            //env->CreateSharedBuffer(0xFFFF, &shared_buffer);
            env->CreatePrintSettings(&print_settings);
            print_settings->put_ShouldPrintHeaderAndFooter(FALSE);
        }
        else 
        {
            handle_failed_edge_load();
        }
        return errorCode;
    }
   
    // ICoreWebView2CreateCoreWebView2CompositionControllerCompletedHandler
    HRESULT __stdcall Invoke(HRESULT errorCode, ICoreWebView2CompositionController* _comp) override
    {
        if (FAILED(errorCode))
        {
            handle_failed_edge_load();
            return errorCode;
        }

        _comp->QueryInterface(&comp);

        if (edge->offscreen) 
        { 
            // setup d3d11 capture visual
            auto _compositor = winrt::Windows::UI::Composition::Compositor();
            auto compositor = _compositor.try_as<ABI::Windows::UI::Composition::ICompositor>();

            winrt::com_ptr<ABI::Windows::UI::Composition::IContainerVisual> root;
            winrt::check_hresult(compositor->CreateContainerVisual(root.put()));

            auto surface = root.try_as<ABI::Windows::UI::Composition::IVisual>();
            assert(surface);

            surface->put_Size({ 1, 1 });
            winrt::check_hresult(surface->put_IsVisible(true));

            winrt::com_ptr<ABI::Windows::UI::Composition::IVisual> webview_visual;
            winrt::check_hresult(compositor->CreateContainerVisual(reinterpret_cast<ABI::Windows::UI::Composition::IContainerVisual**>(webview_visual.put())));

            auto webview_visual2 = webview_visual.try_as<ABI::Windows::UI::Composition::IVisual2>();
            if (webview_visual2) webview_visual2->put_RelativeSizeAdjustment({ 1.0f, 1.0f });

            winrt::com_ptr<ABI::Windows::UI::Composition::IVisualCollection> children;
            winrt::check_hresult(root->get_Children(children.put()));
            winrt::check_hresult(children->InsertAtTop(webview_visual.get()));
            winrt::check_hresult(_comp->put_RootVisualTarget(webview_visual.get()));
            auto rt_visual = surface.try_as<winrt::Visual>();
            assert(rt_visual);

            m_item = winrt::GraphicsCaptureItem::CreateFromVisual(rt_visual);
            IDXGIDevice* dxgi_device = nullptr;
            winrt::check_hresult(edge->edge_d3d11.device->QueryInterface(&dxgi_device));
        
            winrt::com_ptr<IInspectable> d3d_device;
            winrt::check_hresult(CreateDirect3D11DeviceFromDXGIDevice(dxgi_device, d3d_device.put()));
            m_device = d3d_device.as<winrt::IDirect3DDevice>();
            safe_release(dxgi_device);

            m_pixelFormat = winrt::DirectXPixelFormat::B8G8R8A8UIntNormalized;
            m_framePool = winrt::Direct3D11CaptureFramePool::Create(m_device, m_pixelFormat, 2, m_item.Size());
            m_session = m_framePool.CreateCaptureSession(m_item);
            if (m_session) m_session.StartCapture();

            update_cursor_mouse();
            winrt::check_hresult(comp->add_CursorChanged(wrl::Callback<ICoreWebView2CursorChangedEventHandler>(
                [&](ICoreWebView2CompositionController* sender, IUnknown* args) -> HRESULT {
                    update_cursor_mouse();
                    return S_OK;
            }).Get(), &CoreWebView2CursorChangedEventRegistrationToken));
        }

        // trigger ICoreWebView2CreateCoreWebView2ControllerCompletedHandler
        ICoreWebView2Controller* _controller = 0;
        _comp->QueryInterface(&_controller);
        auto hr = Invoke(errorCode, _controller);
        safe_release(_controller);

        return hr;
    }

    // ICoreWebView2CreateCoreWebView2ControllerCompletedHandler
    virtual HRESULT STDMETHODCALLTYPE Invoke(HRESULT errorCode, ICoreWebView2Controller *_controller) override
    {
        if (FAILED(errorCode))
        {
            handle_failed_edge_load();
            return errorCode;
        }

        winrt::check_hresult(_controller->QueryInterface(&controller));
        ICoreWebView2* _webview2 = 0;
        winrt::check_hresult(controller->get_CoreWebView2(&_webview2));
        winrt::check_hresult(_webview2->QueryInterface(&webview));
        safe_release(_webview2);

        ICoreWebView2Settings* _settings = 0;
        winrt::check_hresult(webview->get_Settings(&_settings));
        winrt::check_hresult(_settings->QueryInterface(__uuidof(ICoreWebView2Settings7), (void**)&settings));
        safe_release(_settings);

        // configure browser
        winrt::check_hresult(controller->put_ShouldDetectMonitorScaleChanges(false));
        winrt::check_hresult(controller->put_DefaultBackgroundColor(COREWEBVIEW2_COLOR{0}));
        winrt::check_hresult(controller->put_BoundsMode(COREWEBVIEW2_BOUNDS_MODE_USE_RAW_PIXELS));
        winrt::check_hresult(controller->put_RasterizationScale(1.0));
        winrt::check_hresult(controller->put_IsVisible(true));
        
        winrt::check_hresult(settings->put_AreHostObjectsAllowed(TRUE));
        winrt::check_hresult(settings->put_IsScriptEnabled(TRUE));
        winrt::check_hresult(settings->put_IsWebMessageEnabled(TRUE));
        winrt::check_hresult(settings->put_IsStatusBarEnabled(FALSE));
        winrt::check_hresult(settings->put_IsBuiltInErrorPageEnabled(FALSE));
        winrt::check_hresult(settings->put_AreDefaultContextMenusEnabled(FALSE));
        //winrt::check_hresult(settings->put_AreBrowserAcceleratorKeysEnabled(FALSE));
        winrt::check_hresult(settings->put_AreDefaultScriptDialogsEnabled(FALSE));

        winrt::check_hresult(webview->add_WebMessageReceived(this, &CoreWebView2WebMessageReceivedEventRegistrationToken));
        winrt::check_hresult(webview->add_ProcessFailed(this, &CoreWebView2ProcessFailedEventRegistrationToken));
        winrt::check_hresult(webview->add_ScriptDialogOpening(this, &CoreWebView2ScriptDialogOpeningEventRegistrationToken));
        winrt::check_hresult(webview->add_DocumentTitleChanged(this, &CoreWebView2DocumentTitleChangedEventRegistrationToken));
        winrt::check_hresult(webview->AddScriptToExecuteOnDocumentCreated(L"(function() { console.log('loaded document', window.location.href) })();", NULL));
        winrt::check_hresult(webview->Navigate(L"https://google.es"));
        //webview->NavigateToString(html);

        // execute some code example
        winrt::check_hresult(webview->ExecuteScript(L"window.chrome.webview.postMessage('example'); return 666;",
            wrl::Callback<ICoreWebView2ExecuteScriptCompletedHandler>(
            [&](HRESULT errorCode, LPCWSTR resultObjectAsJson) -> HRESULT {
                // do something with the result
                return errorCode;
        }).Get()));

        // automatically handle permission requests
        winrt::check_hresult(webview->add_PermissionRequested(wrl::Callback<ICoreWebView2PermissionRequestedEventHandler>(
            [&](ICoreWebView2* sender, ICoreWebView2PermissionRequestedEventArgs* args) -> HRESULT {
                COREWEBVIEW2_PERMISSION_KIND kind;
                if (SUCCEEDED(args->get_PermissionKind(&kind)))
                {
                    switch (kind)
                    {
                    case COREWEBVIEW2_PERMISSION_KIND_CLIPBOARD_READ:
                    case COREWEBVIEW2_PERMISSION_KIND_FILE_READ_WRITE:
                    case COREWEBVIEW2_PERMISSION_KIND_AUTOPLAY:
                    case COREWEBVIEW2_PERMISSION_KIND_LOCAL_FONTS:
                    case COREWEBVIEW2_PERMISSION_KIND_OTHER_SENSORS:
                    case COREWEBVIEW2_PERMISSION_KIND_CAMERA:
                    case COREWEBVIEW2_PERMISSION_KIND_MICROPHONE: {
                             args->put_State(COREWEBVIEW2_PERMISSION_STATE_ALLOW);
                    } break;
                    default: args->put_State(COREWEBVIEW2_PERMISSION_STATE_DENY);
                    }
                }
                return S_OK;
        }).Get(), &CoreWebView2PermissionRequestedEventRegistrationToken));
        
        // set window icon as the current favicon
        winrt::check_hresult(webview->add_FaviconChanged(wrl::Callback<ICoreWebView2FaviconChangedEventHandler>(
            [&](ICoreWebView2* sender, IUnknown* args) -> HRESULT {
                webview->GetFavicon(
                    COREWEBVIEW2_FAVICON_IMAGE_FORMAT_PNG,
                    wrl::Callback<ICoreWebView2GetFaviconCompletedHandler>(
                        [&](HRESULT errorCode, IStream* iconStream) -> HRESULT
                        {
                            if (FAILED(errorCode)) return S_OK;
                            HICON icon = 0; Gdiplus::Bitmap iconBitmap(iconStream);
                            if (iconBitmap.GetHICON(&icon) == Gdiplus::Status::Ok)  
                                 { SendMessage(edge->hwnd, WM_SETICON, ICON_SMALL, (LPARAM)icon); }
                            else { SendMessage(edge->hwnd, WM_SETICON, ICON_SMALL, (LPARAM)IDC_ICON); }
                            return S_OK;
                        })
                    .Get());
                return S_OK; 
        }).Get(), &CoreWebView2FaviconChangedEventRegistrationToken));

        RECT rc;
        GetClientRect(edge->hwnd, &rc);
        browser->resize(rc.right, rc.bottom);

        return errorCode;
    }

    // ICoreWebView2WebMessageReceivedEventHandler
    virtual HRESULT STDMETHODCALLTYPE Invoke(ICoreWebView2 *sender, ICoreWebView2WebMessageReceivedEventArgs *args) override
    {
        // executed when webmessage is posted
        LPWSTR s;
        if (SUCCEEDED(args->TryGetWebMessageAsString(&s))) wprintf(L"Message: %s\n", s);
        return S_OK;
    }

    // ICoreWebView2ProcessFailedEventHandler
    virtual HRESULT STDMETHODCALLTYPE Invoke(ICoreWebView2 *sender, ICoreWebView2ProcessFailedEventArgs *args) override
    {
        // executed when a browser child process exits unexpectedly
        COREWEBVIEW2_PROCESS_FAILED_KIND fail;
        if (SUCCEEDED(args->get_ProcessFailedKind(&fail)) && fail == COREWEBVIEW2_PROCESS_FAILED_KIND_RENDER_PROCESS_EXITED)
        {
            // can we recover?
            release_resources();
            if (FAILED(CreateCoreWebView2Environment(this))) exit(-2);
        }
        return S_OK;
    }

    // ICoreWebView2ExecuteScriptCompletedHandler
    virtual HRESULT STDMETHODCALLTYPE Invoke(HRESULT errorCode, LPCWSTR resultObjectAsJson) override
    {
        wprintf(L"Message: %s\n", resultObjectAsJson);
        return S_OK;
    }

    // ICoreWebView2ScriptDialogOpeningEventHandler
    HRESULT STDMETHODCALLTYPE Invoke(ICoreWebView2* sender, ICoreWebView2ScriptDialogOpeningEventArgs* args) override
    {
        // handles js dialogs
        COREWEBVIEW2_SCRIPT_DIALOG_KIND type;
        if (SUCCEEDED(args->get_Kind(&type)))
        {
            switch (type)
            {
            case COREWEBVIEW2_SCRIPT_DIALOG_KIND_ALERT:
            {
                LPWSTR msg; LPWSTR str; int result;
                if (SUCCEEDED(args->get_Message(&msg)) && webview->get_DocumentTitle(&str))
                    return TaskDialog(edge->hwnd, hinst, str, 0, msg, TDCBF_OK_BUTTON, 0, &result);
            } 
            break;
            case COREWEBVIEW2_SCRIPT_DIALOG_KIND_CONFIRM:
            {
                LPWSTR msg; int result; LPWSTR str;
                if (SUCCEEDED(webview->get_DocumentTitle(&str)) && SUCCEEDED(args->get_Message(&msg)))
                    if (SUCCEEDED(TaskDialog(edge->hwnd, hinst, str, 0, msg, TDCBF_YES_BUTTON | TDCBF_NO_BUTTON, 0, &result)) && IDYES == result)
                        return args->Accept();
            } 
            break;
            }
        }
        return S_OK;
    }

    // ICoreWebView2DocumentTitleChangedEventHandler
    HRESULT STDMETHODCALLTYPE Invoke(ICoreWebView2* sender, IUnknown* args) override
    {
        // set window title as current document title
        LPWSTR str;
        if (SUCCEEDED(webview->get_DocumentTitle(&str))) SetWindowText(edge->hwnd, str);
        return S_OK;
    }

    // helper: enable reciving mouse events while mouse its outside (offscreen)
    inline int set_capture_mouse(bool enable)
    {
        if (enable) { if (!capture_on) SetCapture(edge->hwnd); capture_on = true; }
        else { if (capture_on) ReleaseCapture();       capture_on = false; }
        return 0;
    }

    // helper: enable reciving events for mouse leave (offscreen)
    inline void track_mouse()
    {
        if (tracking_on) return;
        TRACKMOUSEEVENT tme;
        tme.cbSize = sizeof(TRACKMOUSEEVENT);
        tme.dwFlags = TME_LEAVE;
        tme.hwndTrack = edge->hwnd;
        tracking_on = TrackMouseEvent(&tme);
    }


    // helper: send WM_SETCURSOR to window
    inline void update_cursor_mouse() { PostMessage(edge->hwnd, WM_SETCURSOR, 0, 0); }


    // downgrade all the interfaces as possible to add version support
    edge_config*                edge{ nullptr };
    //ICoreWebView2Environment13*env{ nullptr };
    ICoreWebView2Environment6*  env{ nullptr };
    //ICoreWebView2Controller4* controller{ nullptr };
    ICoreWebView2Controller3*   controller{ nullptr };
    //ICoreWebView2_22*         webview{ nullptr };
    ICoreWebView2_16*           webview{ nullptr };
    //ICoreWebView2Settings7*   settings{ nullptr };
    ICoreWebView2Settings*      settings{ nullptr };
    //ICoreWebView2CompositionController4* comp{ nullptr };
    ICoreWebView2CompositionController* comp{ nullptr };
    //ICoreWebView2SharedBuffer*  shared_buffer{ nullptr };
    ICoreWebView2PrintSettings* print_settings{ nullptr };

    EventRegistrationToken      CoreWebView2WebMessageReceivedEventRegistrationToken{ 0 };
    EventRegistrationToken      CoreWebView2ProcessFailedEventRegistrationToken{ 0 };
    EventRegistrationToken      CoreWebView2ScriptDialogOpeningEventRegistrationToken{ 0 };
    EventRegistrationToken      CoreWebView2DocumentTitleChangedEventRegistrationToken{ 0 };
    EventRegistrationToken      CoreWebView2CursorChangedEventRegistrationToken{ 0 };
    EventRegistrationToken      CoreWebView2FaviconChangedEventRegistrationToken{ 0 };
    EventRegistrationToken      CoreWebView2PermissionRequestedEventRegistrationToken{ 0 };

    int                         last_button = 0;
    bool                        capture_on = false;
    bool                        tracking_on = false;
    uint64_t                    reference_count = 1;
    POINT                       point{ 0,0 };

    winrt::GraphicsCaptureItem          m_item{ nullptr };
    winrt::IDirect3DDevice              m_device{ nullptr };
    winrt::DirectXPixelFormat           m_pixelFormat;
    winrt::Direct3D11CaptureFramePool   m_framePool{ nullptr };
    winrt::GraphicsCaptureSession       m_session{ nullptr };
    winrt::DispatcherQueueController    queue_controller{ nullptr };
};

// impl

int APIENTRY wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPWSTR lpCmdLine, int nCmdShow)
{
    WCHAR                            sz_title[MAX_LOADSTRING];
    WCHAR                            sz_class[MAX_LOADSTRING];
    static edge_config               edge{};
    edge.offscreen                   = OFFSCREEN_RENDERING;
    hinst                            = hInstance;
    Gdiplus::GdiplusStartup(&gdip_token, &gdip_input_start, NULL);
#ifdef CONSOLE
    AllocConsole();
    freopen("CONIN$", "r", stdin);
    freopen("CONOUT$", "w", stdout);
#endif
    DwmEnableMMCSS(true);
    InitCommonControls();
    LoadStringW(hInstance, IDS_APP_TITLE, sz_title, MAX_LOADSTRING);
    LoadStringW(hInstance, IDC_BROWSER, sz_class, MAX_LOADSTRING);

    // init scope
    {
        WNDCLASSEXW wcex;
        wcex.cbSize = sizeof(WNDCLASSEX);
        wcex.style = CS_HREDRAW | CS_VREDRAW;
        wcex.lpfnWndProc = [](HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam) -> LRESULT CALLBACK
        {
            // pass messages to browser
            if (browser && edge.offscreen && browser->translate_msg_proc(hWnd, message, wParam, lParam)) return 0;
            switch (message)
            {
                case WM_MOVE:    if (browser) browser->update_browser_window(); break;
                case WM_SIZE:    if (browser) browser->resize(LOWORD(lParam), HIWORD(lParam)); break;
                case WM_DESTROY: PostQuitMessage(0); break;
                default:         return DefWindowProc(hWnd, message, wParam, lParam);
            }
        };
        wcex.cbClsExtra     = 0;
        wcex.cbWndExtra     = 0;
        wcex.hInstance      = hInstance;
        wcex.hIcon          = LoadIcon(hInstance, MAKEINTRESOURCE(IDI_BROWSER));
        wcex.hCursor        = LoadCursor(nullptr, IDC_ARROW);
        wcex.hbrBackground  = (HBRUSH)(COLOR_WINDOW + 1);
        wcex.lpszMenuName   = NULL;
        wcex.lpszClassName  = sz_class;
        wcex.hIconSm        = LoadIcon(wcex.hInstance, MAKEINTRESOURCE(IDI_SMALL));
        RegisterClassExW(&wcex);

        edge.hwnd  = CreateWindowW(sz_class, sz_title, WS_OVERLAPPEDWINDOW, CW_USEDEFAULT, 0, 1280, 720, nullptr, nullptr, hInstance, nullptr);
        if (!edge.hwnd) return FALSE;

        // set dark theme
        BOOL dmOn = true;
        DwmSetWindowAttribute(edge.hwnd, 20, &dmOn, sizeof(dmOn));
        SetWindowTheme(edge.hwnd, L"DarkMode_Explorer", nullptr);

        ShowWindow(edge.hwnd, SW_SHOW);
        UpdateWindow(edge.hwnd);

        if (OFFSCREEN_RENDERING)
        {
            // create d3d11 device
            constexpr UINT creation_flags {
                D3D11_CREATE_DEVICE_VIDEO_SUPPORT |
                D3D11_CREATE_DEVICE_BGRA_SUPPORT |
                D3D11_CREATE_DEVICE_PREVENT_INTERNAL_THREADING_OPTIMIZATIONS
                #ifdef _DEBUG
                | D3D11_CREATE_DEVICE_DEBUG
                #endif 
            };
            constexpr D3D_FEATURE_LEVEL feature_levels[] = {
                D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0, D3D_FEATURE_LEVEL_10_1,
                D3D_FEATURE_LEVEL_10_0, D3D_FEATURE_LEVEL_9_3,  D3D_FEATURE_LEVEL_9_2,
                D3D_FEATURE_LEVEL_9_1
            };
            D3D_FEATURE_LEVEL* ftret = 0;
            winrt::check_hresult(D3D11CreateDevice(nullptr, D3D_DRIVER_TYPE_HARDWARE, 0, creation_flags, feature_levels, 7, D3D11_SDK_VERSION, &edge.edge_d3d11.device, ftret, &context));
            ID3D10Multithread* pMultithread = nullptr;
            winrt::check_hresult(edge.edge_d3d11.device->QueryInterface(IID_PPV_ARGS(&pMultithread)));
            winrt::check_hresult(pMultithread->SetMultithreadProtected(TRUE));
            safe_release(pMultithread);

            IDXGIDevice* dxgi_device = nullptr;
            winrt::check_hresult(edge.edge_d3d11.device->QueryInterface(__uuidof(IDXGIDevice), (void**)&dxgi_device));
            IDXGIAdapter* dxgi_adapter = nullptr;
            winrt::check_hresult(dxgi_device->GetParent(__uuidof(IDXGIAdapter), (void**)&dxgi_adapter));
            IDXGIFactory* dxgi_factory = nullptr;
            winrt::check_hresult(dxgi_adapter->GetParent(__uuidof(IDXGIFactory), (void**)&dxgi_factory));

            // create d3d11 swapchain on window
            DXGI_SWAP_CHAIN_DESC sd{};
            sd.BufferCount = 2;
            sd.BufferDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
            sd.BufferDesc.Width = 1280;
            sd.BufferDesc.Height = 720;
            sd.BufferDesc.RefreshRate.Numerator = 0;
            sd.BufferDesc.RefreshRate.Denominator = 0;
            sd.BufferDesc.Scaling = DXGI_MODE_SCALING_UNSPECIFIED;
            sd.BufferDesc.ScanlineOrdering = DXGI_MODE_SCANLINE_ORDER_UNSPECIFIED;
            sd.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT;
            sd.Flags = DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH;
            sd.OutputWindow = edge.hwnd;
            sd.SampleDesc.Count = 1;
            sd.SampleDesc.Quality = 0;
            sd.SwapEffect = DXGI_SWAP_EFFECT_FLIP_DISCARD;
            sd.Windowed = true;
            winrt::check_hresult(dxgi_factory->CreateSwapChain(edge.edge_d3d11.device, &sd, &swapchain));
            winrt::check_hresult(dxgi_factory->MakeWindowAssociation(edge.hwnd, 0));

            safe_release(dxgi_factory);
            safe_release(dxgi_adapter);
            safe_release(dxgi_device);
        }
        
        // create a browser
        browser = new edge_browser(&edge);
    }


    HACCEL accel_table = LoadAccelerators(hInstance, MAKEINTRESOURCE(IDC_BROWSER));
    MSG msg;

    while (true)
    {
        // copy texture on rendertarget texture for easy rendering... normally more complete rendering pipeline has to be done
        if (OFFSCREEN_RENDERING && !IsIconic(edge.hwnd))
        {
            if (browser)
            {
                ID3D11Texture2D* tex = browser->texture();
                if (tex)
                {
                    ID3D11Texture2D* dst = nullptr;
                    if (SUCCEEDED(swapchain->GetBuffer(0, __uuidof(ID3D11Texture2D), (void**)&dst)))
                    {
                        context->CopySubresourceRegion(dst, 0, 0, 0, 0, tex, 0, 0);
                        safe_release(dst);
                    }
                    safe_release(tex);
                }
            }

            // present at vsync
            swapchain->Present(1, 0);
        }

        // message loop
        while (PeekMessage(&msg, NULL, NULL, NULL, PM_REMOVE))
        {
            if (!TranslateAccelerator(msg.hwnd, accel_table, &msg))
            {
                TranslateMessage(&msg);
                DispatchMessage(&msg);
            }
        }

        // add some idle time and check if need to exit
        SleepEx(10, false);
        if (msg.message == WM_QUIT) break;
    }

    // dispose everithing...
    safe_release(browser);

    if (OFFSCREEN_RENDERING)
    {
        safe_release(edge.edge_d3d11.device);
        safe_release(context);
        safe_release(swapchain);
    }

    Gdiplus::GdiplusShutdown(gdip_token);

#ifdef _DEBUG
    IDXGIDebug* debug = nullptr;
    if (SUCCEEDED(((HRESULT(WINAPI*)(REFIID riid, void * ppDebug))
        GetProcAddress(LoadLibraryA("DXGIDebug.dll"), "DXGIGetDebugInterface"))
        (IID_PPV_ARGS(&debug))))
    {
        OutputDebugStringA("*** start DXGI live objects ***\n");
        debug->ReportLiveObjects(DXGI_DEBUG_ALL, DXGI_DEBUG_RLO_DETAIL);
        OutputDebugStringA("*** end   DXGI live objects ***\n");
        debug->Release();
    }
#endif

    return 0;
}

// use always the dedicated hardware when there are integrated graphics
extern "C"
{
    __declspec(dllexport) DWORD NvOptimusEnablement = 0x00000001;
    __declspec(dllexport) int AmdPowerXpressRequestHighPerformance = 1;
}

#pragma comment(linker,"\"/manifestdependency:type='win32' \
name='Microsoft.Windows.Common-Controls' version='6.0.0.0' \
processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'\"")

Hi, I'm trying to run your program but can't because I don't know what's in the framework.h and Browser.h files. Are these libraries I need to download somewhere or self written? Thanks in advance

@badduck32 the only thing you need to run is the webview library, you can obtain the webview2 nuget package

what i do is to unzip the nupkg file and get the files needed from build/native

Thanks for the quick response :) I was already doing what you are describing. I looked through the files of the nuget package and couldn't find any framework.h or Browser.h files. This is the error I get when I use your exact code as stated above:

main.cpp(9,10): error C1083: Cannot open include file: 'framework.h': No such file or directory

Commenting out #include "framework.h" and #include "Browser.h" I get a lot of strange errors in shellapi.h, which I assume are happening because there are missing dependencies. Here are some examples of these errors:

shellapi.h(82,1): error C2146: syntax error: missing ';' before identifier 'DECLSPEC_IMPORT'
shellapi.h(84,1): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
shellapi.h(84,1): error C2086: 'int EXTERN_C': redefinition

eh @badduck32 that files are just usual that comes up with visual studio when creating a window app project normally contains some defines and #define WIN32_LEAN_AND_MEAN

@pabloko Which Visual Studio project template did you use? I'm using Visual Studio 17.12.3 with C++ and WinRT. I'm very interested in your solution because I am working on a project with OpenXR as the rendering engine. I'm hung in getting visual root hooked to the OpenXR rendering engine. I too can't seem to find the Framwork.h and Browser.h either. (A full path might be helpful). Thanks in advance!
I agree the WebView2 is "not well documented".

hello @laultman theres nothing special about the rest of the files, you may be missing to use c++ 17, if you're getting compilation issues (required by winrt atrocious sdk).

im not sure about OpenXR, ive googled it and only you pop up on stackoverflow talking about it 😅

Edge renders on directcomposition that seems to not have full directx interop, but rather using winrt apis, you can create a directx "capture" queue of frames, in this case, from the root visual of the browser. I guess those d3d11 texture2d frames can be rendered on OpenXR by using the correct interop.

Edge does not allow to control the framerate of the composition, so youre stuck at the vsync selected on the host machine, also audio mirroring is not possible without windows 11 hackish hook, so if you need either audio or controlling the framerate you will prefer to use CEF wich seems this year has managed to finally fix the accelerated textures on official builds.

@pabloko WebView2 is anything but straight forward outside Win32, IMHO. The Unity team has it working in their engine, but it is my guess they are not using OpenXR to actually render the WebView2. We do a lot of work for the U.S. Government and getting third-party software authorized is next to impossible to certify on their networks. I would use a different tool if it were up to me. I will have to conquer the audio as well. I would appreciate any insight into the audio hacks.
A little more on the choices of tools. WebView2 is now standard issue on Windows 11 and managed by Microsoft. Because it is native to the OS, we don't have to certify it, Microsoft is responsible for that problem. When a government worker wants to implement our solution, that problem is avoided, and the customer has one less thing to deal with. Anytime you deal with the government the last thing you want to do is have the gov employee have to do something out of the ordinary. If you are interested SBOM.

hello @laultman

ah the wonderful world of regulations, I develop medical software and equipment in the EU (they love regulations) here being part of the OS does not mean anything and you have to certify the complete process anyway (wich technically means pay an insane amount of money to a nepotic and obscure company for a paper)

About the audio "hacks" i revisited the msdn documentation and they now state the API is present on Windows 10 build 20348 or later a year ago they only mentioned x build of win11. Basically you have to establish a WASAPI loopback capture, but activate the interface with new parameter AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK and then you will be able to control what process (pids) capture. there is some example available. I've tested this feature with MiniAudio library when it was on development and i can confirm that latency is tolerable.

About framerate, youre locked on machine LUID vsync. if the framerate match your xr setup probably you will not have tearing but that depends greatly on the goggles. ive used the oculus a long ago to try design an ocular keratorefractometer (regular test involves operator to measure the ocular aberration parameters, on a xr based test, you would tweak a knob to align green and red figures displayed on each eye a couple of times, resulting on the same aberration ocular metric values) at that time ive stopped since it was very cluttery and expensive.

If your goggles are the kind of oculus where you submit frames it can be challenging, if theyre the kind of HDMI/video output, then youre fine.

About how to integrate on OpenXR, seems not really hard looking at the documentation just use proper flags and perform GPU texture copies on synchronization events from the textures obtained from the CaptureQueue. In my sample here i just lazily copy it to the swapchain's backbuffer to be presented on screen (see how no pipeline/shaders are setup)

In this sample ive observed tearing on testufo, the way to solve it is present inmediately and lock to vsync with something like DwmFlush();

Apart from all of this, Edge contains "experimental" APIs on its "prerelease" channel, it contains a TextureStream API wich basically allows you to pass D3D11 textures to the engine and be displayed as video element or used in webgl/canvas contexts with javascript.

You may also find useful to share memory arraybuffers with the script to pass things like rotation vectors/matrix etc to properly render your webview without needing to pass json or serializing each frame.

By the way, I do this kind of stuff professionally, so if you want someone to take care of this for you, send me an email

Hello again. I've spent the past year trying to get this work in the way I want it. I've managed to render to an offscreen image buffer, but have a lot of issues pertaining focus management. I have a window that I'm embedding webview2 into, let's call it the "main window". Correct click and keyboard input while keeping the main window focussed seems to be impossible, and also an open issue. I wanted to ask, do you have any fix? Any way of correct input and focus management without losing focus to the main window? You seem to have a lot of experience with this, a lot more than I do, so if there are things that seem promising to you or approaches you have already tried, please do tell me.

@badduck32 i'm not really understanding what you mean with focus issues on the input events. When using edge with the composition controller, all input events including focus are signaled by API, this gist does a very basic proof-of-concept event management where translate_msg_proc will "steal" message proc events from the main filter of the window and send them to edge via API.

If you have complex window composition where you're rendering edge you need to setup a proper message filter that pass to edge API the intended user events, and to seamslessly keep your controls updated, but this is totally unrelated to edge itself. If you have a gist or a repo i can take a look anyway.

Sorry I should have been clearer. Taking left mouse click as an example, when calling comp->SendMouseInput on the webview, the webview will steal the OS level focus, and the app that I'm trying to embed this webview into loses said focus, and thus cannot receive keyboard events anymore for example. Is this intended behaviour without a fix, or is there something wrong with my implementation?

Also, my code is a mess now, but I could send you a minimal version with the most important parts, and also a video to illustrate the issue if that is needed. Thanks for the quick answer!

@badduck32 so if i understand some internal window get activated once API is called, i've took a look with Spy++ and i see a window of class Chrome_WidgetWin_0, is that the offending one?

In that case you might try a dirty trick from win32 to prevent it to get activated:

Inside ICoreWebView2CreateCoreWebView2ControllerCompletedHandler::Invoke

HWND wnd = FindWindow(L"Chrome_WidgetWin_0", NULL);
if (wnd)
{
    SetWindowLong(wnd, GWL_EXSTYLE, WS_EX_NOACTIVATE);
    SetWindowLong(wnd, GWL_STYLE, WS_EX_NOACTIVATE);
}

If not, i still can take a look at your proj

Hmmm I have tried what you have suggested but no luck, but I imagine there is also just too little information here for you to help me (and maybe my implementation is incorrect somewhere else). I will also look at this Spy++ program you've mentioned and see what I can do on my own, and I will provide a minimal implementation showcasing my issue soon, because the code now is a mess, and it's compiled as a DLL and loaded into a program made in the LÖVE framework, which is a bit too much of a hassle I imagine. I have an exam soon but after that I have the time to work on that.

Have you yourself been able to create a working offscreen webview implementation with WebView2? I'm not sure for what use-case you created this gist, but for me personally I want to create a library that exposes a simple API (moveMouse, pressMouse, pressKey, ...) to embed a webview into applications easily. It's 80% of the way there, the only issues left being focus handling issues. If you have any guidance, or perhaps already attempted this I would be glad to hear any advice. I do want to thank you already for the help you've already given me.

Hello, I'm currently using cef to render ui in my c++ win32 game, but for some users cef crashing even if using cpu based renderer instead of accelerated graphics.
So i have a question, is it worth it to rewrite ui in my game to WebView2 with this implementation for better performance and stability?