In January 2017, four world-class poker players engaged in a three-week battle of heads-up no-limit Texas hold ’em.

Libratus: The Superhuman AI for No-Limit Poker (Demonstration) Noam Brown Computer Science Department Carnegie Mellon University noamb@cs.cmu.edu Tuomas Sandholm Computer Science Department Carnegie Mellon University Strategic Machine, Inc. Sandholm@cs.cmu.edu Abstract No-limit Texas Hold’em is the most popular vari-ant of poker in the world. Dec 18, 2017 Poker players, by contrast, contend with hidden information — what cards their opponents hold and whether an opponent is bluffing,” according to an interesting account on the CMU website. The proof, of course, is in the winning; Libratus did this in spades at a 20-day, 120,000-hand competition last year at Rivers Casino, Pittsburgh. Libratus the best poker bot, an AI Computer Recently Busted out Online Poker Pro Human Players Heads up and Winning in Poker Tournament Action One on One Libratus Computer Best Poker Bot Wins Poker Tournament Beating Pro Online Players showing the need to own the Best Poker Hud Software Available.

They were not competing against each other. Instead, they were fighting against a common foe: an AI system called Libratus that was developed by Carnegie Mellon researchers Noam Brown and Tuomas Sandholm. The official competition between human and machine took place over three weeks, but it was clear that the computer was king after only a few days of play. Libratus eventually won^[1] by a staggering 14.7 big blinds per 100 hands, trouncing the world’s top poker professionals with 99.98% statistical significance.

This was the first AI agent to beat professional players in heads-up no-limit Texas hold ’em.

Libratus is not the only game-playing AI to make recent news headlines, but it is uniquely impressive.

In 2015, DeepMind’s Deep Q-network mastered^[2] a number of Atari games. A Deep Q-network learns how to play under the reinforcement learning framework, where a single agent interacts with a fixed environment, possibly with imperfect information.

Also in 2015, DeepMind's AlphaGo used similar deep reinforcement learning techniques to beat professionals at Go for the first time in history. Go is the opposite of Atari games to some extent: while the game has perfect information, the challenge comes from the strategic interaction of multiple agents.

Libratus, on the other hand, is designed to operate in a scenario where multiple decision makers compete under imperfect information. This makes it unique: poker is harder than games like chess and Go because of the imperfect information available. (On a chessboard, every piece is visible, but in poker no player can see another player’s cards.)

At the same time, it's harder than other imperfect information games, like Atari games, because of the complex strategic interactions involved in multi-agent competition. (In Atari games, there may be a fixed strategy to 'beat' the game, but as we'll discuss later, there is no fixed strategy to 'beat' an opponent at poker.)

This combined uncertainty in poker has historically been challenging for AI algorithms to deal with. That is, until Libratus came along. Libratus used a game-theoretic approach to deal with the unique combination of multiple agents and imperfect information, and it explicitly considers the fact that a poker game involves both parties trying to maximize their own interests.

The poker variant that Libratus can play, no-limit heads up Texas Hold'em poker, is an extensive-form imperfect-information zero-sum game. We will first briefly introduce these concepts from game theory.

A normal form game

For our purposes, we will start with the normal form definition of a game. In this two-player game, each player has a collection of available actions: $A_1$ for Player 1, and $A_2$ for Player 2. The two players simultaneously choose actions $a_1 in A_1$, $a_2 in A_2$ and receive rewards $r_1(a_1, a_2)$ and $r_2(a_1, a_2)$ respectively. The game concludes after a single turn. These games are called normal form because they only involve a single action. An extensive form game, like poker, consists of multiple turns. Before we delve into that, we need to first have a notion of a good strategy.

The Nash equilibrium

Multi-agent systems are far more complex than single-agent games. An optimal policy always exists in a single-agent decision process, but with multiple agents, optimal play depends on your opponent’s strategy. To account for this, mathematicians use the concept of the Nash equilibrium.

A Nash equilibrium is a scenario where none of the game participants can improve their outcome by changing only their own strategy. Formally, a Nash equilibrium consists of strategies $sigma_1$ and $sigma_2$ such that if Player 1 chooses another strategy $sigma_1'$, we have
$$
r_1(sigma_1, sigma_2) geq r_1(sigma_1', sigma_2)
$$

This is because a rational player will change their actions to maximize their own game outcome. When the strategies of the players are at a Nash equilibrium, none of them can improve by changing his own. Thus this is an equilibrium. When allowing for mixed strategies (where players can choose different moves with different probabilities), Nash proved that all normal form games with a finite number of actions have Nash equilibria, though these equilibria are not guaranteed to be unique or easy to find.

Zero-sum games

While the Nash equilibrium is an immensely important notion in game theory, it is not unique. Thus, is hard to say which one is the optimal. Many two player games that people play (including poker) have an additional feature that one player’s reward is the negative of their opponent’s. Such games are called zero-sum.

Importantly, the Nash equilibria of zero-sum games are computationally tractable and are guaranteed to have the same unique value.

Consider a zero-sum game. For all $a_1 in A_1$ and $a_2 in A_2$, we have:

$$
r_1(a_1, a_2) + r_2(a_1, a_2) = 0
$$
Zero-sum games are interesting since any Nash equilibrium can be computed efficiently using the minmax theorem.

We define the maxmin value for Player 1 to be the maximum payoff that Player 1 can guarantee regardless of what action Player 2 chooses:

$$
maxmin_1 = max_{a_1}(min_{a_{2}} r_1(a_1, a_{2}))
$$

The corresponding minmax value of Player 1 is then the minimum payoff that Player 2 can force Player 1 into taking, if Player 2 chooses their action first:
$$
minmax_1 = min_{a_2}(max_{a_{1}} r_1(a_1, a_{2}))
$$

The minmax theorem states that minmax and maxmin are equal for a zero-sum game (allowing for mixed strategies) and that Nash equilibria consist of both players playing maxmin strategies. As an important corollary, the Nash equilibrium of a zero-sum game is the optimal strategy. Crucially, the minmax strategies can be obtained by solving a linear program in only polynomial time.

More Complex Games - Extensive Form Games

While many simple games are normal form games, more complex games like tic-tac-toe, poker, and chess are not. In normal form games, two players each take one action simultaneously. Occasion and genre. In contrast, games like poker are usually studied as extensive form games, a more general formalism where multiple actions take place one after another.

See Figure 1 for an example. Player 1 first decides between $L$ and $R$. The action $R$ ends the game with payoff $(5,2)$, while $L$ continues the game, offering Player 2 choice between $l$ and $r$ (and the corresponding rewards). All the possible games states are specified in the game tree.

The good news about extensive form games is that they reduce to normal form games mathematically. Since poker is a zero-sum extensive form game, it satisfies the minmax theorem and can be solved in polynomial time. However, as the tree illustrates, the state space grows quickly as the game goes on. Even worse, while zero-sum games can be solved efficiently, a naive approach to extensive games is polynomial in the number of pure strategies and this number grows exponentially with the size of game tree. Thus, finding an efficient representation of an extensive form game is a big challenge for game-playing agents. AlphaGo^[3] famously used neural networks to represent the outcome of a subtree of Go. In a similar vein, many of Libratus’s innovations boil down to

Libratus Poker Reddit