Picked from the software, topic, and level of “Class Reinforcement Learning Robot Manipulator #5: Training a Soft Actor-Critic (SAC) Agent for Trajectory Tracking”.
At most one email a week. Unfollow anytime with one click.
Transcript
The full narration of this video. The current sentence is highlighted while the video plays; click any sentence to jump there.
Show transcript
Introduction
This is the final result, a soft actor-critic agent that learns on its own to correct the robot arm's torque, through tens of thousands of trials. Its learning curve shows the reward increasing sharply from around -900 to around -100. Let's start. Hello, welcome to episode five of the robot manipulator control series using reinforcement learning. Soft actor-critic, or SAC, is a modern RL algorithm for continuous actions like motor torques. SAC has an actor network that selects actions, a critic network that evaluates actions, and encourages exploration by maximizing entropy. Today, we are training it with stable baseline 3.
Step 1: Training Report Callback
Create a file called train underscore SAC.py The number of training steps can be provided via an argument. The default is 60,000 steps, equivalent to 150 episodes. The class reporter is a callback that is called every step. Every 5,000 steps, it prints the average reward of the last 10 episodes and records it for the learning curve.
Step 2: SAC Configuration
The environment is wrapped with a monitor to record the reward per episode. The SAC model uses a standard MLP policy with a learning rate of 3x10^-4, size B256, 2,000 initial steps to collect random experiences, and a discount factor of 0.98. After training, the model is saved to SAC underscore arm dot zip.
Step 3: Training 60,000 Steps
Finally, we plot the learning curve. Run it, training takes several minutes on the CPU. Notice the report, at the first 5,000 steps, the average reward is still around -900, equivalent to PD alone. After 10,000 steps, the reward jumps, then fluctuates while improving, until around -100 at the end, roughly 10 times better than at the start.
Step 4: Learning Curve
Let's look at the learning curve. The reward jumps sharply at the beginning when the agent finds the correct correction pattern, then fluctuates because each episode has a different load and random disturbance. Fluctuations like this are normal in reinforcement learning. The model is already saved and ready to be fairly tested in the next episode.
Done!
Done We have trained the soft actor-critic agent as a torque compensator for the robot arm monitored the learning curve with a callback, and saved the model In the final episode, we tested its robustness against load changes and disturbances Currently working on a thesis, dissertation, or research in robotics and reinforcement learning? Cerdas Inovasi Digital helps with robot control implementation from manipulator kinematics and dynamics, PID and compensators, gym environments up to training the soft actor-critic agent as well as offering Python programming courses for electrical and mechatronic engineering Want to master the entire series from robot arm kinematics to adaptive reinforcement learning agent testing? All six episodes, complete with PDF step-by-step guides and source code are available to premium tutorial subscribers Subscribe starting at Rp69.000 per month Save 28% with the 12-month package or one-time payment of Rp999.000 for lifetime access Want just one class? Buy a single class with lifetime access Subscribe now on the premium page at cerdasinovasidigital.com Contact us via WhatsApp or visit cerdasinovasidigital.com Thank you for watching See you in the next episode
Ask the class material
Type a question or keyword; we find the part of this episode that explains it and jump the video there.
My notes
Notes are saved with the current video time and are only visible to you.
Questions & answers
Stuck on a step? Ask here; the instructor and other learners can answer.
No questions yet.
Your instructor
Cerdas Inovasi Digital Team
Software developers, BIM & 3D engineers, and AI practitioners
53 video classes1,290 episodes
We build software, BIM models, games and AI automations for clients every day, and turn that hands-on work into step-by-step classes: every step is shown on screen and comes with a PDF guide you can keep.
