# Solving Pole-Balancing Problem with POMDP Essay

993 Words 4 Pages
Abstract — Partially Observable Markov Decision Processes (POMDP) has been widely applied in fields including robot navigation, machine maintenance, marketing, Medical Diagnosis, and so on [1]. But its exact solution is inefficient in both space and time. This paper investigates Smooth Partially Observable Value Approximation (SPOVA) [2], which approximates belief values by a differentiable function and then use gradient descent to update belief values. This POMDP approximation algorithm is applied on pole-balancing problem with regulation. Simulation results turn out this regulated approach is capable of estimating state transition probabilities and improving its policy simultaneously.
Keywords – POMDP; SPOVA; Pole-balancing.
The approach, Smooth Partially Observable Value Approximation (SPOVA) proposed by R. Parr and S. Russel, uses a differentiable function to approximate the value function; then doing gradient descent to minimize the Bellman residual. Smooth Partially Observable Value Approximation with Reinforcement Learning (SPOVA-RL) is a variation of SPOVA, which focus on the belief state that is encountered in the environment, rather than computes on all possible belief states. [2] has shown that the SPOVA-RL algorithm works nicely in robot navigation problem on 4x4 and 4x3 maps. The appealing factor of this algorithm is that the time required to approach near-optimal level is much less than that done by traditional POMDP algorithms.
However, the SPOVA-RL approach was only applied on simple problems that only involve a few states. We herein investigate a larger problem that needs lots of states to represent its status. Pole-balancing problem has long served as a benchmark for testing automatic control algorithms. The pole-cart system is expected to find an optimal policy to balance the pole. In addition, we assume have no information on the state transition probabilities, which needs to be estimated while the system is operating. In our two test cases, we found the agent, equipped with SPOVA-RL algorithm under regulation, could rapidly improve its performance.
The reminder of this paper is organized as follows. Section II describes the POMDP framework.

## Related Documents

• ###### Essay Solving Personal Problems With Personal Problem

find a solution to a problem, nevertheless some people are better at solving non-personal problems than personal problem. When I evaluate my problem-solving ability I conclude that I am better at solving personal problems than non-personal problems. I believe its more difficult to solve personal problems, since those problems are personal and my feelings affect the solution, however I solve this kind of problem better because the solution will affect my life. Non-personal problems are easier to solve…

Words: 1693 - Pages:
• ###### Problem Solving Essay

Problem Solving and Judgment Selina Appel, Denise Locke, Christina Stamatien, and Amber Hamilton PSY 640 May 12, 2014 Dr. Brian Newbury Problem Solving and Judgment What is problem solving? Problem solving is a complex mental process that includes identifying, analyzing, discovering and solving problems. “Problem solving is described in terms of searching a problem space, which consists of various states of the problem” (Anderson, 2009, p. 212). Often times, problems can be difficult, and…

Words: 1504 - Pages: 7
• ###### Essay on Problem Solving Simulation

Problem Solving Simulation BEH/225 How did you interpret the problem? When I saw the problem, I first thought it was not going to be possible to solve. Meaning; that I would not solve this problem without more information. When I sent the cat over first the mouse and dog did not fight by their selves. Then whenever I sent the dog or mouse over, they would fight with the cat. I even thought about leaving the cat on the existing shore and trying to move the mouse and dog, and found that the…

Words: 672 - Pages: 3
• ###### Essay Problem Solving Throughout The Workday

Problem Solving Throughout the Workday Haeley Rimmer I currently work at The Carousel Center, a child advocacy center here in Wilmington, as the Financial Specialist. The biggest part of my job is billing for each service that is provided by one of the many therapists, medical team members, and forensic interviewers at the center. This may seem like a relatively simple task; however, a great deal of information is needed about each client, service, and case before getting a check from the insurance…

Words: 1172 - Pages:
• ###### Problem Solving Essay

Solving Personal Problems: Applying the Five-Step Model Kyla Hill HUM/115 March 7, 2016 Robin Schultz Solving Personal Problems: Applying the Five-Step Model Problems are an occurrence in everyday life. One of the things that we need to know how to do, as critical thinkers, is define problems and solve them. I am going to explain my problem to you and how I came up with a solution to solve it using the “Five-Step Problem Solving Method” (Chaffee, 2015). When using the five-step model,…

Words: 827 - Pages: 4
• ###### Problem Solving Essay

Problem Solving      There are many different strategies that good problem solvers use to solve a problem. Before using a strategy, you must remember a few things. First, take your time. Few good problem solvers solve problems fast. Second, don't give up. You will never solve a problem if you don't try. Last, be flexible. If a first you don't succeed, try another way. And if the second way doesn't work, try a third way.      There are a few steps…

Words: 656 - Pages: 3
• ###### Essay Thinking Critically And Problem Solving

Critically and Problem Solving When students are given the facts they require, they memorize those facts and use them to serve their short-term goals, which consists of passing tests and graduating to a higher class. However, when you give students critical thinking skills, they will be able to find the needed information for themselves. Students will be able to evaluate the qualities and consequences of that information. Students will also be able to utilize that information to solve any problems at hand…

Words: 1650 - Pages:
• ###### Essay Solving The Problem Solving Process

six-step problems solving process. Problem-solving is the process where we work to get a solution to a specific problem. Problem-solving by definition is the process of finding solutions to difficult or complex issues. It takes several steps to get to an acceptable solution of each problem. There are six steps on the problem-solving process from the article "The problem-solving process". This is a breakdown and review of the process to the second scenario. Step One: Define the problem & Step Two:…

Words: 1380 - Pages: 6
• ###### Essay Critical Thinking and Problem Solving

HUM 101 Critical Thinking and Problem Solving-B1 Midterm 11/4/2011 1. In your own words, explain the 3 stages of knowing and use a sample situation to show how people in each of the 3 stages would respond to it. Be sure that you both explain each stage in general terms and then explain how each example fits that stage. Do not say that kids are Stage 1, teens are Stage 2, and adults are Stage 3; that’s too simplistic and also inaccurate. The three stages of knowing are essentially the various…

Words: 2184 - Pages: 9
• ###### Essay Game As A Problem Solving Platform

success. Game as a problem solving platform In our modern education system, students can’t solve any problem with their knowledge. This is because the formers are being grounded on didactical and linear progressions. Teaching is all about facts, information, and passing test, and left no place to wandering and alternatives (Paraskeva 2009). Higher order thinking skills and real problem solving skills is what needed in the area of teaching. Gaming concepts such as well-order problem, or cycle of expertise…

Words: 1737 - Pages: