Car stuff
Compiled by?|?James
Editor?|?Xiaohan
Apple is here to provide new ideas to the automobile industry .
Since June, Apple has obtained 20 new patents in the automotive field, covering three major aspects: autonomous driving, smart cockpits, and smart bodies.
In this newly exposed patent, Apple is also very creative.
It uses AI technology similar to Google AlphaGo to teach machines to drive, and uses visual sensors such as long-wave infrared cameras to improve the efficiency of lidar; turning the car's windshield into an AR-HUD screen, automatically number the buildings outside the car to facilitate navigation, introduced its own FaceTime video calling function, and even built a drowning escape system for the car...
As the world's most valuable technology company, as well as in consumer A highly influential IT giant in the electronics field, every time Apple enters a field, it brings a lot of amazing innovations and pushes product experience to new heights.
Although it has given up the Titan Project to build its own cars and switched to autonomous driving. However, Apple has been applying for various smart car-related patents in recent years. Through these documents, we can not only see the innovative technologies and various whimsical ideas that Apple has brought to the industry, but also see how powerful this technology giant’s technical reserves are.
Automobile patents obtained by Apple this year
In June this year, Chexixi systematically reported on dozens of automobile patents obtained by Apple in the first half of the year (please refer to "70 Patents Decrypted by Apple") Car! The brain is so big that even Tesla can’t catch up."), this article mainly explains the new patents obtained after June.
1. Use AI to teach machines to drive? Google Alpha? Go’s same technology
In about 5 months, Apple obtained 10 self-driving patents in one go , which involves the three major fields of sensors, communication, and control of autonomous driving.
Nowadays, autonomous driving has been developed for many years, and the computing power of computing equipment is greatly improved. However, how to control autonomous vehicles is still a big problem. The difficulty lies in how autonomous vehicles make timely and reasonable decisions. Moreover, the more vehicles on the road and the more complex the traffic environment, the more difficult it is to make decisions.
In Apple’s patent diagram, Apple describes that self-driving vehicles can complete driving actions such as automatically entering and exiting ramps, automatically changing lanes, and overtaking. In order to complete these actions, the autonomous driving computer needs to complete two actions:
1. Predict the upcoming traffic conditions and make various plans for the vehicle’s driving trajectory;
2. Choose a correct driving action from multiple plans.
This decision-making process is very similar to the decision-making process of playing chess, so Apple uses a machine learning method similar to Google AlphaGo, namely Monte Carlo tree search. This machine learning method has been widely used. For board games and open-ended games.
When a vehicle enters the autonomous driving state, it can plan vehicle actions within a certain time range based on traffic conditions, including acceleration, deceleration, lane change, etc. These actions can be generated through a Monte Carlo search tree.
Four steps of Monte Carlo tree search
A cycle of Monte Carlo tree search includes four steps, namely selection, expansion, simulation and backpropagation. Starting from the root node R, the child nodes are gradually deduced downward. Each child node is the state of the vehicle, and the edge connecting the child node to the parent node represents the vehicle's action.
For example, the vehicle can choose a decision from the root node R to the child node L. If successful, it can continue to expand downward from L to E. After the E node, the random selection method will be used to continue downward decision-making and enter the simulation process. After this autonomous driving process ends, the system will perform reverse deduction from the bottom child node to R, allowing the machine to continue learning and become more powerful.
Illustration of Monte Carlo tree search in patent
In actual application, Apple’s decision-making process appears more complicated. By providing recorded vehicle actions, actions from simulations, and benefits to different Monte Carlo tree search models, Monte Carlo tree search in reinforcement learning policy neural networks and supervised learning value neural networks can increase breadth and depth respectively.
In fact, the method of controlling autonomous vehicles is not only machine learning, but also stochastic technology and deterministic technology. What is ultimately input into the decision-making process is the calculation result of a mixture of the three.
To summarize, in the process of making motion decisions, autonomous vehicles may encounter random and dynamic environments, infinite space, potentially huge fields of view, and real-time constraints. These factors lead to automatic Driving a vehicle cannot drive at will on the road, but relies on the calculations of the self-driving computer.
2. Sensors are installed together? Lidar can see more clearly
Autonomous driving vehicles are also inseparable from various sensor equipment. For high-level autonomous driving, lidar is even more important. essential. Apple originally obtained several patents related to lidar in 2018. Early patents showed that Apple would use lidar to measure distance and use this data for autonomous driving.
Since June this year, Apple has once again obtained three lidar-related patents in less than half a year, further elaborating on how lidar should be used.
In a patent titled "Object Detection and Ranging Using Remote Sensing," Apple uses sensors to assist lidar to achieve more accurate ranging and identification.
In the process of autonomous driving, not all road traffic conditions require the vehicle's attention. Some important traffic conditions require more focused attention. Nowadays, vision sensors are spread all over the car body, with almost no blind spots. These sensors can detect the traffic conditions around the vehicle in real time.
Infrared camera-assisted lidar ranging
At this time, the visual sensor can determine a key area of ??focus, and adding lidar to the ranging can make the distance judgment more accurate.
In another patent titled "System and Method for Solving Hidden Objects in Field of View", Apple uses a long-wavelength infrared sensor (LWIR) to assist lidar, which can detect in extremely dark light environments Animals and pedestrians that may be present on the road.
With long-wavelength infrared rays, the thermal energy of objects can be detected with higher resolution in a certain area, detecting objects that may not be detected by other sensors.
Both patent information shows that lidar can be used in combination with other sensors. For example, lidar and infrared cameras can measure distance simultaneously, which can improve the accuracy of distance measurement. Long-wavelength infrared sensors can be used in combination with monochromatic sensors, RGB sensors, lidar and other sensors to enhance target detection effects.
At the same time, the patent document also shows other sensor components, including cameras, biological detection processors, lidar, vertical cavity surface-emitting laser pulse controllers, etc. These sensors can all participate in the autonomous driving process. Object detection.
What’s interesting is that the first Apple device equipped with lidar is not a car, but this year’s iPad Pro and iPhone 12 Pro series. The lidar module installed on mobile devices can scan The 3D information of a room can also improve the focusing speed under low light conditions.
The picture "seen" by iPhone lidar
Here you can reasonably imagine that Apple's current layout of lidar hardware may also be applied to autonomous driving .
3. The windshield becomes a super large screen? You can also FaceTime at will
In addition to a large number of self-driving patents, Apple has not slackened at all in terms of in-car technology, including mobile devices. of excellent solutions packed into cars.
AR augmented reality has always been one of Apple's key technologies. Now when browsing Apple's official website, its main products can be viewed through AR. You can get a more accurate picture of the size and color of the device before purchasing. know.
View iPhone with AR
At the same time, a large number of AR applications are provided in the Apple App Store, and there are even AR versions of Angry Birds and Minecraft.
In the automotive industry, there has been a lot of discussion about AR this year, mainly focusing on AR-HUD.
In September this year, the 11th generation S-Class sedan released by Mercedes-Benz was equipped with AR-HUD. Its display area is equivalent to a 77-inch monitor. According to Mercedes-Benz, this AR-HUD uses the imaging unit of a digital micromirror device used in theater projections. It is composed of 1.3 million microlenses and a high-efficiency light source for high-resolution imaging. Matrix composition.
Mercedes-Benz S-Class AR HUD display effect
While driving, virtual information such as animated arrows and vehicle distances can be superimposed on the real world within 10 meters, thereby realizing AR Effect.
Apple, which has been laying out its plans for many years, naturally hopes that AR technology will enter cars, and the main application is still AR-HUD. According to statistics, Apple has obtained four AR-HUD patents in one go.
In a patent obtained by Apple called "Augmented Reality Technology Display System", it describes in detail the functions Apple hopes AR-HUD can achieve.
1. Provide navigation information and translate road signs
The original intention of HUD is to allow drivers to focus more on driving and keep their eyes off the road as much as possible. AR-HUD is a major improvement on the basis of HUD, allowing the original small-area, fixed-content HUD to be expanded to a larger area, while providing an augmented reality experience based on the external environment.
Based on this goal, the easiest thing to think of is to move navigation information from the car or instrument to the windshield. Through specific signs, drivers can get better navigation services, and at the same time, more drivers can shut up the navigation completely.
AR-HUD can translate street signs
AR-HUD only provides navigation services, which is relatively simple, so Apple added the function of translating street signs. If you drive overseas, due to the inconsistent road signs in various countries and the language barrier, it may lead to serious safety hazards. If vehicles can recognize road signs in real time and convert the meaning and text information of street signs into easy-to-understand content, it will also be convenient for more drivers.
2. Make a FaceTime call on the windshield? The passenger can also watch videos
FaceTime call was defined by Steve Jobs as Apple once again defined the iPhone, so it is naturally indispensable in the car. FaceTime call.
With AR-HUD, drivers can talk directly on the windshield, and the size of the display screen can be the same as that of a real person, making video calls more immersive.
However, during driving, FaceTime calls will not block the driver's view. The co-pilot can see the complete picture, and the main driver can only make audio calls. When the vehicle speed decreases or stops, the FaceTime screen will be displayed on the driver's side.
FaceTime video calls can be made through the windshield
In fact, after FaceTime was introduced into the car, the windshield became a large screen. During the trip, the co-pilot can watch videos or even play a game through the windshield.
The application of AR-HUD is actually more than that. Apple describes another use of AR-HUD in another patent titled "System and Method for Recognizing Objects in Mixed Reality Interface" .
In a complicated city, when we walk to a block with many high-rise buildings, it has become a big problem for people to go out and not know the name of the high-rise buildings. We do not know how to describe them.
Apple marks high-rise buildings or street-facing shops in AR-HUD, selects a number on the car, and the vehicle will automatically drive to a specific location, making it easier for the occupants in the car to reach a specific location.
Four. All four windows are smart glass? There is also a drowning escape system
In addition to software and services, Apple has also put a lot of effort into hardware. In terms of body structure, Apple also hopes to be "fully armed" to bring the hardware industry to new heights.
According to statistics, Apple obtained 10 patents related to car bodies within half a year. It may be that Apple attaches great importance to privacy protection and has applied for many patents related to "color-changing windows". This year, it has obtained 7 patents. At the same time, Apple has always attached great importance to the safety of the car body. Even if the vehicle falls into the water, it can make correct operations to facilitate the evacuation of the people in the vehicle.
1. Can smart car windows be virtual or real? 5-layer components protect privacy
Apple hopes to design the car windows into multiple layers, and each layer uses a different structure to realize the control of the car windows. The adjustment of the overall color atmosphere and transparency can not only ensure that the occupants in the car are not harmed by harmful light, but also protect the privacy without applying a film.
In a patent titled "Window and System", Apple hopes to use electrically adjustable windows to control the absorption of light, thereby achieving the effect of controlling the light in the car.
This technology can be used not only on side windows, but also on windshields, sunroofs, and rear windows. The light transmittance of the entire car window can be adjusted.
According to the description of the patent document, the car window glass is actually very similar to the liquid crystal display component. While adjusting the color, it can also be used to display images, such as displaying navigation information, providing interior lighting and other practical functions. It can all be realized on the car window glass.
In another patent titled "Car Window with Heated Infrared Transparent Area," Apple added a filter layer to the car window that can block infrared light. This will prevent too much heat from building up in the car when exposed to the sun. In addition, ohmic wire will be added to the rear window of the vehicle, and when powered on, it will generate heat to defrost the window.
In Apple’s latest patent, “Systems and Methods for Dynamic Privacy Protection and Window Tinting,” the layering of car windows is described in more detail.
It can be found from the patent drawing that the car window is composed of a polycarbonate layer, a conductive layer, and a Guest-Host liquid crystal layer. The polycarbonate layer and the conductive layer wrap the Guest-Host liquid crystal layer inside. .
The car window component*** has 5 layers of display components
Through this component, the car window can dynamically adjust the window tint. If there is strong light coming in, the windows will adjust the dazzling light. If the outside visibility is low, the windows will be clearer and brighter. In addition, the occupants in the car can also manually adjust the window tinting to protect privacy in the car. The vehicle can even use the lidar device. If someone outside the car looks into the car, the vehicle will adjust the light transmission of the windows to prevent privacy leakage in the car.
At the same time, in order to prevent the invisible light from cameras, lidar, night vision devices and other components from being absorbed by the car window interlayer, Apple has reserved a certain area on the car window for these hardware to ensure that the invisible light can pass through smoothly. Look out the car window into the distance.
2. Easily escape after drowning? Seat belts, windows, and doors are all key
Apple also obtained a patent called "Dynamic Element Protection Measures". After drowning, you can respond correctly according to the drowning situation to help the vehicle occupants escape as soon as possible.
Vehicle drowning is very dangerous
When a vehicle is partially submerged in water, one or more computer systems in the vehicle begin to intervene and operate to close the windows, lock the doors, and unlock the safety Take measures, turn on the flotation device, send out a distress signal and adjust the seat position to facilitate the escape of the vehicle occupants. At the same time, a reminder is issued to the occupants in the car to prepare for escape.
The vehicle will conduct life detection in real time. Using high-resolution ultra-wideband radar equipment, it can detect possible vital signs in the vehicle, including detection of breathing and heartbeat, and even pets left in the vehicle. was discovered in time.
After it is determined that all the occupants in the car have escaped, the vehicle will enter a self-protection state and lock the doors and windows to avoid greater losses.
In addition to the window and safety patents, Apple also obtained a smart suspension patent to make the vehicle drive more smoothly. At the same time, the ventilation ducts can be hidden in the car body frame, making the space inside the car more abundant. It can be seen from these details that Apple is very careful in building cars.
Conclusion: “Apple Smart Car” is worth learning from the industry
The number of automobile patents obtained by Apple has reached more than 100. The patents obtained in the past six months show Apple’s commitment to autonomous driving. The level of attention is rapidly increasing, and Apple's autonomous driving route is gradually becoming clearer.
Today, in the smart car industry, new car-making forces such as Tesla are a beautiful sight in the automotive industry, whether in vehicle intelligence, autonomous driving or three-electric systems. Giants like Apple are The automotive industry is constantly moving, and it has also given some excellent lessons to the new forces that are building cars. How can we build cars like Apple? Can the car defined by Apple still be welcomed by the market?
This article comes from the author of Autohome Chejiahao and does not represent the views and positions of Autohome.