Principle If you want to learn more about OpenGL ES, please go to the OpenGL ES related articles directory The code used in this article is in the ARKit branch of https://github.com/SquarePants1991/OpenGLESLearn.git. iOS11 introduced a new framework ARKit, which makes it easy to create AR apps through ARKit and SceneKit. Apple also provides a basic AR application framework, so you can start developing your AR app directly from it. However, this series of articles will use OpenGL ES to provide rendering support for ARKit. Next, let’s first learn about the theoretical knowledge of ARKit. AR Basic Concepts The most basic concept of AR is the technology that combines virtual computer graphics with the real environment. There are many ways to implement this technology.
WorldTracking Tracking real-world feature points, calculating the real camera position, and applying it to the virtual camera in the 3D world is the most important part of AR implementation. The accuracy of the calculation directly affects the rendered result. ARKit uses ARSession to manage the entire AR processing flow, including the calculation of the camera position.
The way to use ARSession is very simple. Initialize and set the delegate. To start ARSession, you need to pass in a configuration ARWorldTrackingSessionConfiguration. ARWorldTrackingSessionConfiguration means that the AR system will track the feature points in the real world and calculate the camera position. Apple may also release configurations such as ARMarkerTrackingSessionConfiguration to identify tracking markers in the future. After ARSession is turned on, the camera will be started and the phone position will be sensed through sensors. Borrow a picture from WWDC. ARSession integrates the video stream and location information captured by the camera to generate a series of continuous ARFrames.
Each ARFrame contains the image captured by the camera, information about the camera position, etc. In this method, we need to draw the image captured by the camera, draw 3D objects based on the camera position and other information. Plane detection ARKit provides another cool feature, which is detecting real-world planes and providing an ARPlaneAnchor object that describes the plane's position, size, orientation, and other information.
The above config.planeDetection = ARPlaneDetectionHorizontal; sets the type of detected plane to horizontal. However, this is the only option available. If ARKit detects a plane, it will use the delegate method - (void)session:(ARSession *)session didAddAnchors:(NSArray Hit Test Hit Test allows you to easily place objects on detected planes. When you tap the screen, Hit Test can detect which planes are at the location you tapped, and provide ARAnchor for setting the location to place the object.
Using ARFrame's hitTest method, the first point passed in ranges from (0,0) to (1,1), and the second parameter represents which objects can be detected. The objects that can be detected are as follows.
If the detection is successful, NSArray* is returned. ARHitTestResult contains the detection type, the distance of the intersection point, and the ARAnchor of the plane. Note that ARAnchor will only be present when ARHitTestResultTypeExistingPlane and ARHitTestResultTypeExistingPlaneUsingExtent are detected. These four detection types can exist at the same time by using the | method, such as ARHitTestResultTypeEstimatedHorizontalPlane | ARHitTestResultTypeExistingPlane. Light intensity adjustment ARKit also provides a function to detect light intensity, mainly to make the lighting of the 3D model consistent with the lighting intensity of the environment. In ARFrame, there is a variable called lightEstimate. If the light intensity is detected successfully, there will be a value. The value type is ARLightEstimate, which contains only one variable, ambientIntensity. In the 3D lighting model, it corresponds to ambient light, and its value ranges from 0 to 2000. When rendering with OpenGL, you can use this value to adjust the ambient light intensity in the lighting model. This is almost the end of the theoretical knowledge of ARKit. The next article will introduce how to use OpenGL ES to render the content in ARFrame. Implementation The code used in this article is in the ARKit branch of https://github.com/SquarePants1991/OpenGLESLearn.git. The OpenGL basic code used in this article comes from the OpenGL ES series, which has basic functions such as rendering geometry and textures. The implementation details will not be repeated. The key code for integrating ARKit is in ARGLBaseViewController. Let's take a look at its code. Processing ARFrame
The above code shows how to process the ARFrame captured by ARKit. The capturedImage of ARFrame stores the image information captured by the camera, and its type is CVPixelBufferRef. By default, the image information is in YUV format, which is stored in two Planes, and can also be understood as two pictures. One is in Y (Luminance) format, which stores the brightness information, and the other is in UV (Chrominance, Chroma), which stores the chroma and concentration. We need to bind these two pictures to different textures, and then use the algorithm in the Shader to convert YUV to RGB. The following is a fragment shader that processes textures and uses formulas for color conversion.
After processing and binding the texture, in order to ensure that the texture is not stretched non-uniformly under different screen sizes, the viewport is recalculated [self setupViewport: CGSizeMake(imageHeight, imageWidth)];. Next, the camera transformation calculated by ARKit is assigned to self.cameraMatrix. Note that the image captured by ARKit needs to be rotated 90 degrees to be displayed normally, so the width and height are deliberately reversed when setting the Viewport, and the camera is rotated at the end. VideoPlane VideoPlane is the geometry written to display video, it can receive two textures, Y and UV.
The other functions are very simple, which is to draw a square and finally cooperate with the Shader to display the video to render the data in YUV format. Perspective projection matrix In ARFrame, you can get the texture and camera matrix required for rendering. In addition to these, the perspective projection matrix that matches the real camera is also required. It can make the perspective of the rendered 3D objects look natural.
The above code demonstrates how to obtain the 3D perspective projection matrix through ARKit. With the perspective projection matrix and the camera matrix, you can easily use OpenGL to render objects.
This article mainly introduces the basic idea of OpenGL ES rendering ARKit, without describing too many OpenGL ES technical details. If you are interested, you can directly clone the code on Github to learn more. |
<<: Using convolutional autoencoders to reduce noise in images
>>: The third round of the Aite Tribe Story Collection with prizes has begun~
Can the Internet business be the savior of mobile...
American telecommunications giant AT&T and Ti...
[[130714]] Recently, Cyanogen, which has develope...
The 2022 National College Entrance Examination is...
Why does black smoke appear when the oil pan is h...
Before writing about competing products, you shou...
Expert of this article: Wang Li, attending physic...
Apple recently updated iOS15.3.1 to fix security ...
With the rise of live streaming platforms and sho...
This article was reviewed by Chu Yuhao, PhD from ...
When many users choose a smartphone , program run...
Introduction There are many creatures in the ocea...
NetEase Auto reported on December 2 that the fami...
Apple has confirmed that iCloud has temporarily b...
A sentence like " He said that any beauty th...