AI Poker Bots: How Algorithms Learn to Bluff and Win

The world is full of tasks related to interactions between multiple agents. Historically, people were the main participants in these multi-agent situations, but with the development of AI, we have the opportunity to introduce algorithms into our daily lives as equal participants and agents with whom we can interact. Right now, similar computer agents solve many tasks: from simple and harmless ones like automatic telephone systems, to critical ones like security management and even autonomous transport management. This allows us to significantly automate many everyday processes, transferring decision-making to algorithms and thereby reducing the burden on humans. Games with asymmetry and incomplete information require significantly more complex approaches to decision-making compared to games of similar size with ideal information fully available at any time. The optimal solution at any given time depends on knowing the strategies of opponents, which depend on information hidden from us and available only to them, which can only be assessed by their past actions. However, their previous actions also depend on information hidden from them about our actions and how our actions revealed this information. This recursive process shows the main difficulty in building effective decision-making algorithms.

An agent is any autonomous participant in the process who makes decisions, whether a human or a computer. In a multi-agent environment, agents interact with each other and do not always know the strategies, goals, and capabilities of other agents. The optimal behavior of an agent maximizing its result in such an environment depends on the actions of other agents. To build an effective agent in a multi-agent environment, it is necessary to adapt to the actions of other agents, modeling their strategies and learning based on their behavior. To enable agents to adapt in real-time, they need to choose optimal actions as they achieve their results. If using approaches based on Reinforcement Learning, agents will accumulate rewards for their actions. Agents will also balance between following their planned behavior (exploitation) and experimental exploratory actions (exploration), trying to learn useful information about the strategies of other players. In addition to the already complex problem statement, agents will face other limitations associated with working in a multi-agent environment with incomplete information. These include limited observations, stochasticity of observations, incompleteness of observed information, and dynamic behavior.

Poker serves as a prime example of a complex environment exhibiting all the properties mentioned above. It involves incomplete information about cards, the strategies of participating players, an element of chance related to card dealing, and other difficulties encountered during the game. Furthermore, the number of possible game states characterizing game situations is enormous. The history of AI development for poker spans over 30 years, with the most significant achievements occurring in the last 3 years. The game's complexity arises from factors like incomplete information, strategic depth, and randomness, making it a challenging yet valuable domain for AI research.

Early poker AI programs emerged in the 1980s, but a paradigm shift occurred in the 2000s, moving towards modeling approaches specifically for poker. In 2015, the University of Alberta introduced Cepheus, which 'solved' limit Heads-up poker. This was a significant milestone as it's the only game with incomplete information to have a complete optimal solution. In 2017, the University of Alberta presented DeepStack for no-limit Heads-Up poker, and Carnegie Mellon University's Libratus defeated professional poker players, demonstrating the ability of AI to handle complex strategic games.

Despite the seemingly limited applicability of poker bots to real-world tasks, their development has yielded numerous methods that can be transferred from the card game to practice. The algorithms of modern poker bots that defeat the best human players are universal and generally aimed at training agents in environments with incomplete and asymmetric information. They can be transferred to many applications where decision-making is required in a similarly complex environment: from security to marketing, in which you can simulate bidding for an audience. In the banking sector, there are also many practical tasks where the algorithms behind advanced Poker bots would find application. Among such business tasks of Sberbank, it is worth noting primarily risk-return management and pricing in the market with many other banking objects. But the list of these applications can be easily extended to tasks such as Customer Value Management or Next Best Action.

The Sberbank Holdem Challenge is a hackathon aimed at fostering the development of machine learning and AI by challenging participants to create AI agents capable of making optimal decisions in uncertain conditions and modeling the behavior of other players in poker. The challenge focuses on No-limit Texas Hold’em, the most complex variant of poker, and involves developing agents that can play against other AI bots in a series of rounds. The competition consists of an online qualifying tournament and an offline hackathon for the top 100 finalists.

To create a poker bot, you need a programming language, a poker game simulator (such as PyPokerEngine), and the bot code itself. A simple bot can be implemented in Python, where it processes game events and declares actions. The development of game strategies is not limited to Python and can be done in any other programming language. The bot interacts with the simulator through stdin/stdout, receiving game events and responding with actions.

After each tournament, participants have access to archives with the game logs of all bots. This allows you to analyze the strategy of your opponents by watching their actions during the game. However, remember that other participants can also analyze your bot's playing style and set up an ambush for your bot in the next tournament. Game replays are recorded as a JSON object with fields such as rule, seats, and rounds.

The environment for running bots is a specially prepared docker image. The bot code, packed in a ZIP archive, must be sent to the verification system. The archive must contain a metadata.json file specifying the docker image and entry point. The executable command exchanges with the game simulator via stdin/stdout. The simulator transmits one event per line to stdin, in the format event_type<\t>data, where data is a JSON object with event parameters. In response to the declare_action event, the bot must respond in stdout within the allotted time with a line in the format: action<\t>amount.