Picked from the software, topic, and level of “Class Reinforcement Learning Robot Manipulator #4: Building the RL (Gymnasium) Environment for the Manipulator”.
At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
Here is the final result, a reinforcement learning environment for a robotic arm with gymnasium standard. The agent observes 12 numbers, outputs torque corrections, and receives larger rewards when the arm follows the path more accurately. Let's get started. Hello, welcome to episode four of the series on robotic manipulator control with reinforcement learning. Reinforcement learning learns through trial and error, the agent takes an action, the environment gives a reward, and the agent improves its policy. Gymnasium is a standard interface for environments used by almost all RL libraries. Today we are wrapping our robot into a gymnasium environment.
Step 1: Residual RL & Observations
Create a file called environment_underscore_rl.py We are using residual RL approach, the simple P controller still works, and the agent only adds torque corrections up to a maximum of 20 Nm. This approach is much safer and faster to train than an agent that controls from scratch. The observation function arranges the 12 numbers that the agent observes, joint angles and velocities, sine and cosine of angles, as well as reference velocities and accelerations.
Step 2: Reset, Step & Reward
The environment arm class defines the observation space and action space of 2 numbers between minus 1 and 1. The reset function starts a new episode with a random load of 0 to 2 kg. Random disturbances up to 8 Nm, and a random starting point on the path. This randomness makes the agent learn to handle various conditions, the core of adaptive control. The step function applies the P torque plus the agent's correction, simulates for one hundredth of a second, then gives a reward of minus thousand times the squared joint angle, minus a small penalty on the action to prevent over-correction. One episode is 4 seconds or 400 steps.
Step 3: Check Env & Test Episode
Create a file called test_underscore_environment_dot_py The check-in function of gymnasium checks whether our environment complies with the standards. Then we run one episode with action 0, which is just a PD controller, and one episode with random actions. Run Environment passes validation with 12 observations and 2-dimensional actions. The two policies received a total reward of about minus thousand, with an average joint error of about 0.05 radians. Random actions are not helpful at all, So the agent must learn.
Step 4: Initial Error Before Learning
Let's take a look at the graph. PD controller joint error and random action are both oscillate several degrees throughout the episode. This is the starting point of the agent. In the next episode, The agent will learn to correct the torque error that causes this.
Done!
Done. We have already built the gymnasium environment for the robotic arm, residual RL above paye, observation, action, reward function, Randomizing load and disturbances, and check it in. In the next episode, we will train the soft actor-critic agent. Working on a thesis, dissertation, or research in robotics and reinforcement learning? Cerdas Inovasi Digital helps with robot control implementation, from manipulator kinematics and dynamics, PID and competitors, gymnasium environments, up to soft actor-critic agent training, as well as opening Python programming courses for electrical and mechatronic engineering. Want to master the entire series? From robotic arm kinematics to adaptive reinforcement learning agent testing? All 6 episodes, complete with step-by-step PDF guides and source code, are available for premium tutorial subscribers. Subscribe starting at Rp69.000 per month, save 28% with the 12-month package, or one-time payment of Rp999.000 for lifetime access. Want just one class? Buy single classes with lifetime access. Subscribe now at the premium page cerdasinovasidigital.com. Contact us via WhatsApp, or visit cerdasinovasidigital.com. Thank you for watching, see you in the next episode.
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
53 video classes1,290 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
