April 5, 2019
By Isi Caulder and Lawrence Yu
Patent filings for fundamental artificial intelligence (AI) technologies continue to rise. Led by a number of high profile technology companies, including IBM, Google, Amazon, Microsoft, Samsung, and AT&T, patent applications directed to fundamental AI technologies, such as machine learning, neural networks, natural language processing, speech processing, expert systems, robotic and machine vision, are being filed and issued in ever-increasing numbers. In turn, these fundamental AI technologies are being applied to address problems in industries such as healthcare, manufacturing, and transportation. A somewhat unexpected source of fundamental AI technology development has been occurring in the field of video games.
Traditional board games have long been a subject of study for AI research. In the 1990’s, IBM created an AI for playing chess, Deep Blue, which was able to defeat top-caliber human players using brute force algorithms. More recently, machine learning algorithms have been developed for more complex board games, which include a larger breadth of possible moves. For example, DeepMind (since acquired by Google), recently developed the first AI capable of defeating professional Go players, AlphaGo.
Video games have recently garnered the interest of researchers, due to their closer similarity to the “messiness” and “continuousness” of the real world. In contrast to board games, video games typically include a greater number of possible actions that must be determined in real-time, using imperfect information. Earlier this year, DeepMind announced the development of AlphaStar, an AI for playing StarCraft II, a real time strategy game. AlphaStar was able to convincingly defeat two top professional players. Earlier last year, OpenAI, a non-profit research organization, developed OpenAI Five, an AI for playing Dota 2, a multiplayer online battle arena game. OpenAI Five was able to defeat a team composed of players in the 99.95th percentile, but ultimately lost to a team of professional players.
OpenAI does not appear to be assigned any patents or published patent applications at the time of this article. (Notably, OpenAI has stated that its patents, if any, will be shared with the world.) However, several other high profile technology companies, including DeepMind, have filed and been granted patents directed toward fundamental AI technologies developed through video games.
One example of such a patent is U.S. Patent No. 9,679,258, entitled “Methods and Apparatus for Reinforcement Learning”, originally assigned to DeepMind (and currently assigned to Google). The patent is generally directed to improved methods of reinforcement learning (i.e., machine learning based on reward maximization or cost minimization). The patent describes how Atari 2600 console games, including Pong, Breakout, Space Invaders, Seaquest, and Beam Rider were used to test the disclosed methods.
Figure 1 of U.S. Patent No. 9,679,258
The patent discloses how a reinforcement learning agent is trained to perform a set of actions (e.g., joystick or button inputs) on a subject system (e.g., an Atari 2600 console) to cause the subject system to move from one state (e.g., a first image frame) to another state (e.g., a second image frame). Training data, including state data and action data, and a target output, generated from a first neural network, is used to train a second neural network to select actions to be performed by the reinforcement learning agent. Periodically during training, the parameters of the second neural network are used to update the parameters of the first neural network. Notably, the patent asserts that the disclosed methods were able to achieve a level of competency in some of the Atari 2600 console games that surpassed that of an expert human player.
Undoubtedly, many more patents will continue to be filed in the coming years, as the field of artificial intelligence continues to develop. It remains to be seen how video games, and the unique challenges they present, will play a role in these developments.
Information on this website is for information only. It is not, and should not be taken as, legal advice. You should not rely on, or take or not take any action, based upon this information. Professional legal advice should be promptly obtained. Bereskin & Parr LLP professionals will be pleased to advise you.