Integrating with GPU

• Introduction

• Architecture and Infrastructure

• The EGL Implementation for MiniGUI

  • New APIs for GPU integration

  • Specification of EGL implementation for MiniGUI

• Using MiniGUI EGL for OpenGL and OpenGL ES

  • A simple EGLUT implementation

• Cairo and MiniGUI

Introduction

The graphics stack on Linux has had a long time evolution.

Initially, graphics applications on Linux mainly used the old SVGALib; the Linux kernel did not provide any driver for graphics or GPU. In 2000s, the Linux kernel introduced the frame buffer driver to support the various graphics devices. However, the frame buffer driver did not provide a good implementation to support modern GPUs. If a software want to use the powerful GPU functions to render 3D objects, it has to write a lot of code for a specific GPU in application space. For desktop systems, it is not a problem, because the XFree86 project provided a complete graphics stack for 2D/3D rendering. But it is a nightmare for embedded systems.

Around 2010, the Free Desktop project introduced a new graphics stack for Linux system called DRI (Direct Rendering Infrastructure). As the name suggests, DRI provides applications with the ability to directly access the GPU for 2D/3D rendering. With or without X Window, applications can get the direct GPU rendering capabilities through DRI. This greatly improves the performance and user experience of Linux desktop systems.

After more than ten years of development, DRI technology has matured. Now, Linux-based desktop systems have switched from the traditional frame buffer driver to DRI. And the Linux-based embedded systems are switching from frame buffer to DRI.

Therefore, we introduced the support DRI in MiniGUI version 4.0.4, and developed the EGL implementation for MiniGUI based on Mesa, also the MiniGUI backend for Cairo.

Now, it is very easy to integrate MiniGUI with your GPU. Your MiniGUI app can exploit the GPU accelerated functions to render 2D/3D objects.

Architecture and Infrastructure

In practice, MiniGUI and the software which are used to integrated with GPU constitute the graphics stack of HybridOS.

HybridOS is a totally new open source operating system designed for smart IoT devices and cloud computing environment. FMSoft Technologies, the developer of MiniGUI, initiated HybridOS project in 2018.

HybridOS uses MiniGUI as the underlying windowing system, and the members of HybridOS project are now maintaining the whole graphics stack.

The following chart shows the graphics stack of HybridOS:

  -----------------------------------------------
 |           MiniGUI/HybridOS Apps               |
 |-----------------------------------------------|
 |           |         (Graphics Stack)          |
 |           |              ---------------------|
 |           |              | hiMesa             |
 |           | hiCairo      |  ------------------|
 |           | MiniGUI      |  | EGL for MiniGUI |
 | C++ libs  | hiDRMDrivers |  | GL, GLES, VG    |
 | C libs    | hiDRM        |  | GPU drivers     |
 |-----------------------------------------------|
 |  Linux Kernel                                 |
 |            -----------------------------------|
 |           |        DRI and DRI Drivers        |
  -----------------------------------------------

As shown in the chart above, the HybridOS graphics stack consists of the following software:

• hiDRM is the LibDRM derivative for HybridOS.

• hiDRMDrivers contains the drivers for MiniGUI DRM engine. The drivers implement the basic hardware acclerated graphics operations of various GPUs for MiniGUI.

• hiMesa is the Mesa derivative for HybridOS, while Mesa is the open source implementation of OpenGL and other graphics APIs, including OpenGL ES (versions 1, 2, 3), OpenCL, OpenMAX, and Vulkan. It contains the following components:

  1. The implementatin of OpenGL, OpenGL ES (v1, 2, 3), and other graphics APIs.

  2. The EGL implementation for MiniGUI platform.

  3. The graphics drivers for various GPUs and a software driver called swrast.

• hiCairo is the Cairo derivative for HybridOS. Cairo is a 2D vector graphics library for Gtk. We provide support for MiniGUI backend in hiCairo.

You can use the following script to fetch the source code of above software:

#!/bin/bash

# Use this if you want to visit GitHub via HTTPS
REPO_URL=https://github.com/FMSoftCN

# Use this one if you can visit GitHub via SSH
#REPO_URL=[e-mail]:FMSoftCN

# Use this one if you are a developer of MiniGUI/HybridOS
#REPO_URL=[e-mail]:hybridos

git clone $REPO_URL/hidrm -b hybridos
git clone $REPO_URL/hidrmdrivers
git clone $REPO_URL/himesa -b minigui-backend
git clone $REPO_URL/hicairo -b minigui-backend

[NOTE] The above fetching script may changed in the future.

The software all ship with the GNU autotools building scripts or the meson building scripts. You can refer to the README file for the instructions to build and install the software to your system.

Please note that the installation order of the software:

1. hiDRM.

2. MiniGUI with DRM engine enabled.

3. hiDRMDrivers.

4. hiMesa with support for MiniGUI platform enabled.

5. hiCairo with MiniGUI backend enabled.

The EGL Implementation for MiniGUI

The following words give the official definition for EGL:

EGL™ is an interface between Khronos rendering APIs such as OpenGL ES or OpenVG and the underlying native platform window system. It handles graphics context management, surface/buffer binding, and rendering synchronization and enables high-performance, accelerated, mixed-mode 2D and 3D rendering using other Khronos APIs. EGL also provides interop capability between Khronos to enable efficient transfer of data between APIs – for example between a video subsystem running OpenMAX AL and a GPU running OpenGL ES.

Obviously, to integrate OpenGL and other graphics APIs with MiniGUI, we must implement EGL for MiniGUI.

As mentioned before, Mesa uses DRI to drive various GPUs and implement the graphics APIs. The EGL implementation for MiniGUI in Mesa is a sub driver of egl_dri2, which supports many platforms including x11, wayland, drm, and surfaceless.

Basically, the EGL implementation for MiniGUI depends on the DRM engine of MiniGUI. On the other hand, if one MiniGUI instance was not using DRM engine, the EGL implementation for MiniGUI can still use the software driver in Mesa to render the graphics objects.

For more information about EGL, please refer to:

• EGL Specification: https://www.khronos.org/egl/

• EGL in Mesa: https://mesa3d.org/egl.html

New APIs for GPU integration

MiniGUI introduced some new APIs for GPU integration:

• GetVideoHandle gets the handle of the video engine which corresponds to the given device context. By using the video engine handle returned by this function, you can call drmGetDeviceFD to get the DRI device file descriptor opened by MiniGUI DRM engine.

• drmGetDeviceFD returns the DRI device file descriptor opened by MiniGUI DRM engine.

• drmGetSurfaceInfo returns the DRM surface information from a specific device context.

• drmCreateDCFromName creates a memory DC with a DRM surface which is created by a foreign process and identified by a global name handle.

• drmCreateDCFromPrimeFd creates a memory DC with a DRM surface which is created by a foreign process and identified by a PRIME file descriptor.

• drmCreateDCFromHandle creates a memory DC with a DRM surface which is created by a foreign graphics component.

Note that all functions prefixed by drm is only available when support for Linux DRM NEWGAL engine (_MGGAL_DRM) is enabled.

Specification of EGL implementation for MiniGUI

The following text gives the specification of EGL implementation for MiniGUI:

Name

    EXT_platform_minigui

Name Strings

    EGL_EXT_platform_minigui

Contributors

    Vincent Wei <[e-mail]>

Contacts

    Vincent Wei <[e-mail]>

Status

    Beta

Version

    Version 1, 2019-12-10

Number

    EGL Extension <N/A>

Extension Type

    EGL client extension

Dependencies

    Requires EGL_EXT_client_extensions to query its existence without
    a display.

    Requires EGL_EXT_platform_base.

    This extension is written against the wording of version 7 of the
    EGL_EXT_platform_base specification.

Overview

    This extension defines how to create EGL resources from native MiniGUI
    resources using the functions defined by EGL_EXT_platform_base.

New Types

    None

New Procedures and Functions

    None

New Tokens

    Accepted as the <platform> argument of eglGetPlatformDisplayEXT:

        EGL_PLATFORM_MINIGUI_EXT                0x34A0

Additions to the EGL Specification

    None.

New Behavior

    To determine if the EGL implementation supports this extension, clients
    should query the EGL_EXTENSIONS string of EGL_NO_DISPLAY.

    To obtain an EGLDisplay backed by a MiniGUI video, call
    eglGetPlatformDisplayEXT with <platform> set to EGL_PLATFORM_MINIGUI_EXT.
    The <native_display> parameter specifies the MiniGUI video to use and must
    either a handle to video (returned by `GetVideoHandle()` or be EGL_DEFAULT_DISPLAY.
    If <native_display> is EGL_DEFAULT_DISPLAY, then EGL will use the video
    handle returned by `GetVideoHandle(HDC_SCREEN)`.

    To obtain an on-screen rendering surface from a MiniGUI window, call
    eglCreatePlatformWindowSurfaceEXT with a <dpy> that belongs to MiniGUI and
    a <native_window> that is a window handle.

    It is not valid to call eglCreatePlatformPixmapSurfaceEXT with a <dpy>
    that belongs to MiniGUI. Any such call fails and generates
    EGL_BAD_PARAMETER.

Issues

    1. Should this extension permit EGL_DEFAULT_DISPLAY as input to
       eglGetPlatformDisplayEXT()?

       Yes. When given EGL_DEFAULT_DISPLAY, eglGetPlatformDisplayEXT
       returns a display backed by the default MiniGUI video engine.

    2. Should this extension support creation EGLPixmap resources from MiniGUI
       memory DC?

       No. The implementation has no pixmap type.

Revision History

    Version 1, 2019-12-10 (Vincent)
        - Initial draft

Using MiniGUI EGL for OpenGL and OpenGL ES

HybridOS project gives some samples to use MiniGUI EGL for OpenGL and OpenGL ES in the directory /device-side/samples/himesa of the following repo:

https://github.com/FMSoftCN/hybridos

To check whether the EGL implementation contains the support for MiniGUI platform, you can use the following code:

#include <EGL/egl.h>
#include <EGL/eglext.h>

static EGLDisplay get_default_minigui_display (void)
{
    const char *extensions;
    extensions = eglQueryString(EGL_NO_DISPLAY, EGL_EXTENSIONS);

    if (extensions) {
        if (strstr (extensions, "EGL_EXT_platform_base")) {

            PFNEGLGETPLATFORMDISPLAYEXTPROC getPlatformDisplay =
                (PFNEGLGETPLATFORMDISPLAYEXTPROC)
                eglGetProcAddress ("eglGetPlatformDisplayEXT");

           if (strstr (extensions, "EGL_EXT_platform_minigui"))
                return getPlatformDisplay (EGL_PLATFORM_MINIGUI_EXT,
                                          EGL_DEFAULT_DISPLAY, NULL);
    }

    return EGL_NO_DISPLAY;
}

The function get_default_minigui_display checks and returns the default MiniGUI EGLDisplay. If failed, it returns EGL_NO_DISPLAY.

Before calling this function, you should call InitGUI to initialize MiniGUI. Note that if you use MiniGUIMain instead of main, this function will be called automatically.

Generally, you call the following code to initialize MiniGUI and MiniGUI EGL in your main function:

#include <minigui/common.h>
#include <minigui/minigui.h>
#include <minigui/gdi.h>
#include <minigui/window.h>

#include <EGL/egl.h>
#include <EGL/eglext.h>

static void _fatal(const char* msg)
{
    fprintf (stderr, msg);
    fprintf (stderr, "\n");
    exit (1);
}

int init_minigui_and_egl(int argc, char* argv[])
{
    EGLNativeDisplayType native_dpy;
    EGLDispaly dpy;
    EGLint major, minor;

    if (InitGUI (argc, (const char**)argv) != 0)
        _fatal("failed to initialize native display");

    native_dpy = (EGLNativeDisplayType)GetVideoHandle (HDC_SCREEN);
    dpy = eglGetDisplay (native_dpy);

    if (!eglInitialize (dpy, &major, &minor))
      _fatal("failed to initialize EGL display");

    return 0;
}

After initialized MiniGUI and EGL, you call the following EGL function on a MiniGUI window to create a window surface:

// Make sure to call eglChooseConfig to choose config before calling
// this function.
static EGLSurface create_window_surface (EGLDisplay dpy, EGLConfig config, HWND hwnd)
{
    EGLContext context;
    EGLSurface surface;
    EGLint context_attribs[4];
    EGLint api, i;

    i = 0;
    context_attribs[i] = EGL_NONE;

    // use OpenGL ES v2
    api = EGL_OPENGL_ES_API;
    context_attribs[i++] = EGL_CONTEXT_CLIENT_VERSION;
    context_attribs[i++] = 2;
    context_attribs[i] = EGL_NONE;

    eglBindAPI (api);

    context = eglCreateContext (dpy,
         config, EGL_NO_CONTEXT, context_attribs);
    if (!context)
        _fatal ("failed to create context");

    surface = eglCreateWindowSurface(dpy,
            config, (EGLNativeWindowType)hwnd, NULL);
    if (surface == EGL_NO_SURFACE)
      _fatal ("failed to create surface");

    if (!eglMakeCurrent(dpy, surface, surface, context))
      _fatal ("failed to make window current");

    // use additional data to store the surface
    SetWindowAdditionalData(hwnd, (DWORD)surface);

    return surface;
}

[TIP] You can use the extended style WS_EX_USEPRIVATECDC when creating MiniGUI main window or control for EGL window surface. This extended style can provide a performance improvement.

After we created a window surface and make the context as the current context, we can call OpenGL or OpenGL ES APIs to draw our graphics object. When we finished the rendering, we call eglSwapBuffers to swap the content in the window surface to the MiniGUI window. This generally happens in the handler of MSG_PAINT:

    case MSG_PAINT: {
        EGLSurface surface =
            (EGLSurface)GetWindowAdditionalData(hWnd);

        HDC hdc = BeginPaint (hWnd);
        eglSwapBuffers (dpy, surface);
        EndPaint (hWnd, hdc);
        return 0;
    }

When you have done with the MiniGUI window, you can use the following code to destroy the surface, context, and terminate EGL:

   eglMakeCurrent(dpy, EGL_NO_SURFACE, EGL_NO_SURFACE, EGL_NO_CONTEXT);
   eglDestroySurface(dpy, surface);
   eglDestroyContext(dpy, context);

alt

Figure 1 The gears rendered on MiniGUI by using hiMesa.

A simple EGLUT implementation

In the HybridOS himesa sample, we implement a simple EGLUT, which provide an easy-to-use interface to show OpenGL, OpenGL ES, or OpenVG rendering content to a MiniGUI main window.

By using the simple EGLUT, our OpenGL app will look very simple. The below sample draws a triangle with OpenGL and EGLUT:

#include <math.h>
#include <stdlib.h>
#include <string.h>
#include <GL/gl.h>

static GLfloat view_rotx = 0.0, view_roty = 0.0, view_rotz = 0.0;

static void
draw(void)
{
   static const GLfloat verts[3][2] = {
      { -1, -1 },
      {  1, -1 },
      {  0,  1 }
   };
   static const GLfloat colors[3][3] = {
      { 1, 0, 0 },
      { 0, 1, 0 },
      { 0, 0, 1 }
   };

   glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

   glPushMatrix();
   glRotatef(view_rotx, 1, 0, 0);
   glRotatef(view_roty, 0, 1, 0);
   glRotatef(view_rotz, 0, 0, 1);

   {
      glVertexPointer(2, GL_FLOAT, 0, verts);
      glColorPointer(3, GL_FLOAT, 0, colors);
      glEnableClientState(GL_VERTEX_ARRAY);
      glEnableClientState(GL_COLOR_ARRAY);

      glDrawArrays(GL_TRIANGLES, 0, 3);

      glDisableClientState(GL_VERTEX_ARRAY);
      glDisableClientState(GL_COLOR_ARRAY);
   }

   glPopMatrix();
}


/* new window size or exposure */
static void
reshape(int width, int height)
{
   GLfloat ar = (GLfloat) width / (GLfloat) height;

   glViewport(0, 0, (GLint) width, (GLint) height);

   glMatrixMode(GL_PROJECTION);
   glLoadIdentity();
   glFrustum(-ar, ar, -1, 1, 5.0, 60.0);

   glMatrixMode(GL_MODELVIEW);
   glLoadIdentity();
   glTranslatef(0.0, 0.0, -10.0);
}


static void
init(void)
{
   glClearColor(0.4, 0.4, 0.4, 0.0);
}


static void
special_key(int special)
{
   switch (special) {
   case EGLUT_KEY_LEFT:
      view_roty += 5.0;
      break;
   case EGLUT_KEY_RIGHT:
      view_roty -= 5.0;
      break;
   case EGLUT_KEY_UP:
      view_rotx += 5.0;
      break;
   case EGLUT_KEY_DOWN:
      view_rotx -= 5.0;
      break;
   default:
      break;
   }
}

int
main(int argc, char *argv[])
{
   eglutInitWindowSize(300, 300);
   eglutInitAPIMask(EGLUT_OPENGL_BIT);
   eglutInit(argc, argv);

   eglutCreateWindow("egltri");

   eglutReshapeFunc(reshape);
   eglutDisplayFunc(draw);
   eglutSpecialFunc(special_key);

   init();

   eglutMainLoop();

   return 0;
}

We only need to implement the functions to draw and transfer the 3D object. For the details, please refer to other samples in /device-side/samples/himesa of HybridOS repo.

Cairo and MiniGUI

In hiCairo, we implement the MiniGUI backend for Cairo. When MiniGUI backend is enabled, you can use one of the following functions to create a cairo surface:

• cairo_minigui_surface_create creates a cairo surface that targets the given DC. If the given DC is not a memory DC or screen DC, this function will create a memory DC which is compatible to the DC first.

• cairo_minigui_surface_create_with_memdc creates a surface which is associated with a new memory DC.

• cairo_minigui_surface_create_with_memdc_similar creates a surface associated with a new memory DC which is compatible to the given DC but in the specified size.

After created MiniGUI surface, you can use the following code to render 2D vector graphics on the surface:

static int draw_rectangle (HDC target_dc, int width, int height)
{
    int ret = 0;

    cairo_surface_t* surface = cairo_minigui_surface_create_with_memdc (NULL,
                CAIRO_FORMAT_RGB24, width, height);
    if (surface == NULL) {
        _ERR_PRINTF("hicairo: failed to create minigui surface\n");
        ret = 1;
        goto FAIL;
    }

    cairo_t* cr = cairo_create (surface);
    if (cr == NULL) {
        _ERR_PRINTF("hicairo: failed to create cairo context\n");
        ret = 2;
        goto FAIL;
    }

    // call cairo functions to draw vector objects here
    cairo_set_source_rgb (cr, 1.0, 0, 0);
    cairo_rectangle (cr, width * 0.25, height * 0.25, width * 0.5, height * 0.5);
    cairo_set_line_width (cr, 2.0);
    cairo_stroke (cr);

    // swap the content on the cairo surface to MiniGUI DC (target_dc)
    HDC cairo_dc = cairo_minigui_surface_get_dc (surface);
    BitBlt(cairo_dc, 0, 0, width, height, target_hdc, 0, 0, 0);

FAIL:
    if (cr) {
        cairo_destroy(cr);
    }

    if (surface) {
        cairo_surface_finish(surface);
        cairo_surface_destroy(surface);
    }

    return ret;
}

alt

Figure 2 The tiger rendered on MiniGUI by using hiCairo.

For the detailed usage, please refer to the source files in /device-side/samples/hicairo of HybridOS repo.

---

<< Using mGPlus for Vector Graphics | Table of Contents | Using mGEff for Visual Effects and Animations >>