Decision Making Under Uncertainty – Kochenderfer

Duff discusses the formulation of model-based reinforcement learning as a Bayes- adaptive Markov decision process [12]. In general, solving such a belief-state formulation exactly is intractable. Strens applies the concept of Thompson sampling [14] to model- based reinforcement learning [13]. Variants of the online planning algorithms presented in the previous chapter have been extended to Bayesian model-based reinforcement learning, including sparse sampling [15] and Monte Carlo tree search [16], [17]. Model-free reinforcement learning algorithms are often used for situations in which it is not feasible to build an explicit representation of the transition and reward models.

Q-learning and Sarsa are two commonly used model-free techniques. Eligibility traces were proposed in the context of temporal difference learning by Sutton [18], and they were extended to Sarsa(λ) [19] and Q (λ) [20], [21]. Much of the ongoing work in the field of reinforcement learning is concerned with generalizing from limited experience. The recent book Reinforcement Learning and Dynamic Programming Using Function Approximators by Busoniu et al. surveys a variety of different local and global function approximation methods [22].

Several different abstraction methods have been proposed over the years [23]–[26]. Although not discussed in this chapter, there has been some work on Bayesian approaches to model-free reinforcement learning. One approach is to maintain a dis- tribution over state-action values [27], [28]. There are also Bayesian policy gradient methods that have been used with some success [29]. Multiagent reinforcement learning was also not discussed in this chapter, but Busoniu, Babuska, and De Schutter survey recent research in the area [30]. 1. J.C. Gittins, “Bandit Processes and Dynamic Allocation Indices,” Journal of the Royal Statistical Society.

Series B (Methodological), vol. 41, no. 2, pp. 148–177, 1979. 2. R.S. Sutton and A.G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998. 3. M. Wiering and M. van Otterlo, eds., Reinforcement Learning: State of the Art. New York: Springer, 2012. 4. T.

Kovacs and R. Egginton, “On the Analysis and Design of Software for Rein- forcement Learning, with a Survey of Existing Systems,” Machine Learning, vol. 84, no. 1-2, pp. 7–49, 2011. doi: 10.1007/s10994-011-5237-8. 5. J. Gittins, K. Glazebrook, and R. Weber, Multi-Armed Bandit Allocation Indices, 2nd ed. Hoboken, NJ: Wiley, 2011.  6. J. Niño-Mora, “A (2/3)n Fast-Pivoting Algorithm for the Gittins Index and Optimal Stopping of a Markov Chain,” INFORMS Journal on Computing, vol. 19, no. 4, pp. 596–606, 2007.

doi: 10.1287/ijoc.1060.0206. 7. I.M. Sonin, “A Generalized Gittins Index for a Markov Chain and Its Recursive Calculation,” Statistics and Probability Letters, vol. 78, no. 12, pp. 1526–1533, 2008. doi: 10.1016/j.spl.2008.01.049. 8. P.R.

Chaw-Bing Chang and Keh-Ping Dunn MIT Lincoln Laboratory is a federally funded research and development center that applies advanced technology to problems of national security. The books in the MIT Lincoln Laboratory Series cover a broad range of technology areas in which Lincoln Laboratory has made leading contributions. The books listed above and future volumes in this series renew the knowledge-sharing tradition established by the seminal MIT Radiation Laboratory Series published between 1947 and 1953. %FDJTJPO .BLJOH 6OEFS 6ODFSUBJOUZ 5IFPSZ BOE “QQMJDBUJPO .ZLFM + ,PDIFOEFSGFS XJUI DPOUSJCVUJPOT GSPN $ISJTUPQIFS “NBUP (JSJTI $IPXEIBSZ +POBUIBO 1 )PX )BZMFZ + %BWJTPO 3FZOPMET +BTPO 3 5IPSOUPO 1FESP “ 5PSSFT$BSSBTRVJMMP / ,FNBM ÃSF +PIO 7JBO ɨF .*5 1SFTT $BNCSJEHF .BTTBDIVTFUUT -POEPO &OHMBOE ª  .BTTBDIVTFUUT *OTUJUVUF PG 5FDIOPMPHZ “MM SJHIUT SFTFSWFE /P QBSU PG UIJT CPPL NBZ CF SFQSPEVDFE JO BOZ GPSN CZ BOZ FMFDUSPOJD PS NFDIBOJDBM NFBOT JODMVEJOH QIPUPDPQZJOH SFDPSEJOH PS JOGPSNBUJPO TUPSBHF BOE SFUSJFWBM XJUIPVU QFSNJTTJPO JO XSJUJOH GSPN UIF QVCMJTIFS .*5 1SFTT CPPLT NBZ CF QVSDIBTFE BU TQFDJBM RVBOUJUZ EJTDPVOUT GPS CVTJOFTT PS TBMFT QSPNPUJPOBM VTF ‘PS JOGPSNBUJPO QMFBTF FNBJM TQFDJBM@TBMFT!NJUQSFTTNJUFEV ɨJT CPPL XBT TFU JO “EPCF (BSBNPOE 1SP CZ UIF BVUIPS JO -“5&9 1SJOUFE BOE CPVOE JO UIF 6OJUFE 4UBUFT PG “NFSJDB -JCSBSZ PG $POHSFTT $BUBMPHJOHJO1VCMJDBUJPO %BUB ,PDIFOEFSGFS .ZLFM + o %FDJTJPO NBLJOH VOEFS VODFSUBJOUZ  UIFPSZ BOE BQQMJDBUJPO  .ZLFM + ,PDIFOEFSGFS XJUI $ISJTUPQIFS “NBUP (JSJTI $IPXEIBSZ +POBUIBO 1 )PX )BZMFZ + %BWJTPO 3FZOPMET +BTPO 3 ɨPSOUPO 1FESP “ 5PSSFT$BSSBTRVJMMP / ,FNBM ÃSF BOE +PIO 7JBO Q DN ‰ -JODPMO -BCPSBUPSZ TFSJFT *ODMVEFT CJCMJPHSBQIJDBM SFGFSFODFT BOE JOEFY *4#/  IBSEDPWFS  BML QBQFS  *OUFMMJHFOU DPOUSPM TZTUFNT  “VUPNBUJD NBDIJOFSZ  %FDJTJPO NBLJOH‰.BUIFNBUJDBM NPEFMT * 5JUMF 5+,  h‰ED            %FEJDBUJPO To my family.

This is a short excerpt from the opening of “” by Unknown, quoted for review and introduction purposes. All rights belong to the copyright holders.

Book Information

• Unique ID: 4dd3a189fdb3e319
• File Extension: .pdf
• File Size: 6,419,617 bytes (6.122 MB)
• Title: –
• Author: Unknown
• Pages: 351
• Language: English (en)

Reading & Word Statistics

• Estimated Reading Time: 544.14 minutes
• Total Words: 108,828
• Total Characters: 666,922
• Average Words per Page: 310.05
• Average Characters per Page: 1900.06

Most Frequent Words

state (399), decision (383), model (366), function (284), system (282), value (269), search (258), algorithm (240), policy (218), set (217), distribution (212), using (209), probability (208), one (192), information (191), uif (186), learning (178), given (177), aircraft (173), agent (172), used (168), states (168), action (167), process (162), belief (158), agents (155), systems (153), space (153), time (152), utility (152), section (151), ibqufs (150), problem (149), parameters (146), methods (141), use (139), example (137), bayesian (136), problems (135), figure (135), optimal (132), different (130), variables (126), vol (122), two (121), collision (121), approach (120), models (119), boe (118), data (116), based (115), network (115), new (114), support (114), markov (113), values (112), number (111), actions (110), dynamic (109), also (107), conference (103), chapter (102), uncertainty (101), speech (101), programming (101), observation (100), joint (100), observed (100), planning (99), policies (99), reward (99), algorithms (98), nodes (98), equation (96), possible (95), observations (95), many (94), doi (94), jhvsf (92), between (88), known (88), step (87), estimate (87), local (85), language (84), color (84), performance (83), shown (82), within (82), variable (80), associated (80), case (79), tcas (78), image (77), human (77), however (77), strategy (76), iteration (76), surveillance (75), represent (75).

PDF Download