游戏网站建设需要多少钱,域名申请好后 如何建设网站,网站里的动态是如何制作,网上购物系统需求分析强化学习环境 - robogym - 学习 - 3 文章目录 强化学习环境 - robogym - 学习 - 3项目地址为什么选择 robogymObservation - 观测信息Action - 动作信息Initialization - 初始状态设置 项目地址
https://github.com/openai/robogym
为什么选择 robogym 自己的项目需要做一些机…强化学习环境 - robogym - 学习 - 3 文章目录 强化学习环境 - robogym - 学习 - 3项目地址为什么选择 robogymObservation - 观测信息Action - 动作信息Initialization - 初始状态设置 项目地址
https://github.com/openai/robogym
为什么选择 robogym 自己的项目需要做一些机械臂 table-top 级的多任务操作 robogym 基于 mujoco 搭建构建了一个仿真机械臂桌面物体操作pick-place、stack、rearrange场景 robogym 的例程效果看支持多个相机示教包括眼在手上和眼在手外可以获取多视角视觉信息 robogym 的物体支持 YCB 数据集格式
主要是这些原因当然看官方 readme.md 文档它还有其他不错的功能。
国内主流社区对 robogym 的介绍比较少所以选择写一些文档记录一下作为参考。
Observation - 观测信息
robogym 的观测一般通过 obs env.reset() 返回即可得到。爬源码可得到 obs 是一个字典。
把字典的键排序按照值的方法进行了简答的分类可以得到仿真环境的 obs 字典是通过self.mujoco_simulation 、 robot_obs 、 self._goal 、 self._goal_info_dict 和 np.array 四个变量得到的。
obs {# 读取 self.mujoco_simulation 内部的方法返回作为值obj_pos: self.mujoco_simulation.get_object_pos(),obj_rel_pos: self.mujoco_simulation.get_object_rel_pos(),obj_vel_pos: self.mujoco_simulation.get_object_vel_pos(),obj_rot: self.mujoco_simulation.get_object_rot(),obj_vel_rot: self.mujoco_simulation.get_object_vel_rot(),qpos: self.mujoco_simulation.qpos,obj_gripper_contact: self.mujoco_simulation.get_object_gripper_contact(),obj_bbox_size: self.mujoco_simulation.get_object_bounding_box_sizes(),obj_colors: self.mujoco_simulation.get_object_colors(),# 在代码上面实例化了# robot_obs self.mujoco_simulation.robot.observe()# 这个实例这部分的键对应的值就是 robot_obs 的方法robot_joint_pos: robot_obs.joint_positions(),gripper_pos: robot_obs.tcp_xyz(),gripper_velp: robot_obs.tcp_vel(),gripper_controls: robot_obs.gripper_controls(),gripper_qpos: robot_obs.gripper_qpos(),gripper_vel: robot_obs.gripper_vel(),tcp_force: robot_obs.tcp_force(),tcp_torque: robot_obs.tcp_torque(),# self._goal 从源码来看就是每个物体重排列的位置。qpos_goal: self._goal[qpos_goal].copy(),goal_obj_pos: self._goal[obj_pos].copy(),goal_obj_rot: self._goal[obj_rot].copy(),rel_goal_obj_pos: self._goal_info_dict[rel_goal_obj_pos].copy(),rel_goal_obj_rot: self._goal_info_dict[rel_goal_obj_rot].copy(),is_goal_achieved: np.array([self._is_goal_achieved], np.int32),safety_stop: np.array([robot_obs.is_in_safety_stop()]),}这里列出了每个键对应的含义。
observation 键名每个键的意义object_posGet position for all objects.object_rel_posGet position for all objects relative to the gripper position.object_vel_posGet position velocity for all objects relative to tooltip velocity.object_rotGet rotation in euler angles for all objects.object_vel_rotGet rotation velocity for all objects.robot_joint_posArray of joint angles (one for each joint).gripper_posTooltip position in the Cartesian coordinate space.gripper_velpTooltip velocity in the Cartesian coordinate space.gripper_controlsGripper’s linear target position.gripper_qposGripper joint positions.gripper_velGripper joint velocities.qposCopy of full sim qpos including 3D-position and 4D-quaternion.qpos_goalCopy of full sim goal qpos including 3D-position and 4D-quaternion.goal_obj_posGet current-goal positions for all objects.goal_obj_rotGet current-goal rotations in euler angles for all objects.is_goal_achievedReturn if current goal is achieved.rel_goal_obj_posGet current-goal positions for all objects relative to the gripper position.rel_goal_obj_rotGet current-goal rotations for all objects relative to the gripper position.obj_gripper_contactA numpy array of shape [num objects, len(other_geom_ids)], in which each value is binary, 1 meaning having contact and 0 no contact.obj_bbox_sizeReturns the bounding box for one objects as a tuple of (positive, half size), where both positive and half size are np.array of shape (3,).obj_colorsThis logic works, assuming only assign a single color to one object.safety_stopTrue if the arm is in a safety stop, False otherwise.tcp_forceTCP force in world coordinates.tcp_torqueTCP torque in world coordinates.
根据自己的项目选择
object_pos 、 object_rot 代表了每个物体的位置和姿态gripper_pos 、 gripper_controls 代表了机械臂的位置和张开闭合程度goal_obj_pos 、 goal_obj_rot 代表了每个物体的目标位置和目标姿态。
需要精简一下观测的信息有三种思路 爬源码把不必要的观测信息直接注释掉 【注意】 一些项目中会在 observation 生成后再对里面的键做处理这样做会导致一些bug 利用 Open AI Gym 的 FilterObservation() 这个类过滤掉不想要的键 自己写一个函数把不必要的键过滤掉 【注意】 .reset() 和 .step() 的返回都需要进行这样的操作
这里我选择自己写一个函数。
# create a small util to filter the observation
def filter_obs(raw_obs: dict, name_list: list) - dict:result {}for name in name_list:result[name] copy.copy(raw_obs[name])return result最后的代码如下。选择 pprint.pprint() 进行输出可以更加格式化。
import copy
import pprint
from robogym.envs.rearrange.ycb import make_env# create a small util to filter the observation
def filter_obs(raw_obs: dict, name_list: list) - dict:result {}for name in name_list:result[name] copy.copy(raw_obs[name])return result# Create an environment with the default number of objects: 5
env make_env(parameters{simulation_params: {num_objects: 3,max_num_objects: 8,}}
)# Reset to randomly generate an environment with num_objects: 3
obs env.reset()
obs filter_obs(obs, [obj_pos, obj_rot, gripper_pos, gripper_controls, goal_obj_pos, goal_obj_rot])
pprint.pprint(obs)while True:a env.action_space.sample()next_obs, reward, done, info env.step(a)next_obs filter_obs(next_obs, [obj_pos, obj_rot, gripper_pos, gripper_controls, goal_obj_pos, goal_obj_rot])pprint.pprint(next_obs)env.render()得到结果
{goal_obj_pos: array([[1.39363232, 0.86174547, 0.51221652],[1.57460708, 0.70375038, 0.50919097],[1.20793525, 0.8834796 , 0.49350575],[0. , 0. , 0. ],[0. , 0. , 0. ],[0. , 0. , 0. ],[0. , 0. , 0. ],[0. , 0. , 0. ]]),goal_obj_rot: array([[ 0. , 0. , -1.79725862],[ 0. , 0. , -1.13518178],[ 0. , 0. , -2.40479252],[ 0. , 0. , 0. ],[ 0. , 0. , 0. ],[ 0. , 0. , 0. ],[ 0. , 0. , 0. ],[ 0. , 0. , 0. ]]),gripper_controls: array([0.]),gripper_pos: array([1.23887261, 0.43994768, 0.68622718]),obj_pos: array([[1.59604171, 0.81327296, 0.51217642],[1.57460711, 0.41286039, 0.50922118],[1.40990736, 0.64130153, 0.49354594],[0. , 0. , 0. ],[0. , 0. , 0. ],[0. , 0. , 0. ],[0. , 0. , 0. ],[0. , 0. , 0. ]]),obj_rot: array([[-8.89659174e-05, -7.47313090e-05, -1.79530140e00],[-3.00692282e-06, 4.73572520e-06, -1.13518163e00],[-4.85122664e-02, -4.51887581e-02, -2.40575071e00],[ 0.00000000e00, 0.00000000e00, 0.00000000e00],[ 0.00000000e00, 0.00000000e00, 0.00000000e00],[ 0.00000000e00, 0.00000000e00, 0.00000000e00],[ 0.00000000e00, 0.00000000e00, 0.00000000e00],[ 0.00000000e00, 0.00000000e00, 0.00000000e00]])}为什么 goal_obj_rot 这部分只有第三个元素有数值前面两个没数值
原因是这是用 rpy 格式描述姿态的。这三个元素依次表示roll 、 pitch 和 yaw 角。如下图所示。 而 table-top 级的物体都是“平躺”的所以默认 yaw 角有姿态。
在上述打印出来的字典可以看到当忽略很小的小数-3.00692282e-06时目标姿态和当前物体姿态差别不大这说明在当前环境中只需要机械臂做细致的平移就行。
Action - 动作信息
robogym 的动作空间比较特殊它通过一层 wrapper 把原本 [ − 1 , 1 ] [-1,1] [−1,1] 数值的动作空间给离散化了在 ~/robogym/robogym/wrappers/utils.py 里面 DiscretizeActionWrapper 把奖励值做了封装通过离散数值索引一个列表 [-1. -0.8 -0.6 -0.4 -0.2 0. 0.2 0.4 0.6 0.8 1. ] 获得机械臂TCP末端的偏移量前面三维度分别是 xyz后面两个维度是姿态角最后一个维度是夹爪的开闭但是测试效果是夹爪开闭似乎无效可能是因为这是 rearrange 环境对物体的操作更多是“push”而不是“pick-and-place”。
【注意】在这样的默认包装下保持机械臂末端位姿不动的动作向量是a np.asarray([5, 5, 5, 5, 5, 5]) 可以设置一个全局参数保存这个动作向量。
【注意】a np.asarray([0, 0, 0, 0, 0, 0]) 不是静止的向量相反它是偏移最大的动作向量
Initialization - 初始状态设置
好的初始状态既能完成更好的实验也能在做成视频的时候更美观些。对于机械臂 rearrange 环境设置初始状态的函数是在 ~/robogym/robogym/envs/rearrange/common/base.py 文件里面 RearrangeEnv 类的 _randomize_robot_initial_position() 函数中。函数中是通过设置末端TCP的初始位姿来进行状态初始化的。
action self.action_space.sample()
if self.parameters.n_random_initial_steps 1:return
for _ in range(self.parameters.n_random_initial_steps):self._set_action(action)self.mujoco_simulation.step()
self._set_action(action * 0.0)
for _ in range(100):# calling set_action each tick is necessary # for the robot to reach stability with relative actionsself._set_action(action * 0.0)self.mujoco_simulation.step()print(self.mujoco_simulation.get_qpos(robot0:arm_joint_angles))里面用到了这个类中实例过的mujoco接口 mujoco_simulation 。这里的mujoco接口保留了大量 mujoco-py 的方法可以爬源码看到都有哪些函数方法可以调用。
class SimulationInterface:Base class for domain-specific simulation interfaces tied to particular XML.Goal is to transform code interfacing with generic MjSim that looks like that:hand_angles sim.data.qpos[hand_angle_idx]cube_pos sim.data.qpos[cube_pos_idx]sim.model.actuator_gainprm[actuator_idx] actuator_kpssim.model.actuator_biasprm[actuator_idx] actuator_kpsInto more high-level and domain-specific version:hand_angles sim.hand.get_angles()cube_pos sim.get_cube_pos()sim.set_actuator_kp(actuator_kps)Etc.This is a base class that just exposes a few generic utilities to help the subclassesimplement the abovementioned functionality. By convention, the subclasses should be namedSomethingSimulation.__slots__ [sim,qpos_idxs,qvel_idxs,synchronization_points,_mujoco_viewer,]def __init__(self, sim: MjSim):self.sim simself.qpos_idxs: Dict[str, List[int]] {}self.qvel_idxs: Dict[str, List[int]] {}self.synchronization_points [] # type: ignoreself._mujoco_viewer Nonepropertydef mj_sim(self): MuJoCo simulation object - alias to make it clearer return self.simpropertydef mujoco_viewer(self):Get a nicely-interactive version of the mujoco viewerif self._mujoco_viewer is None:# Inline import since this is only relevant on platforms# which have GLFW support.from mujoco_py.mjviewer import MjViewer # noqaself._mujoco_viewer MjViewer(self.sim)return self._mujoco_viewerdef enable_pid(self): Enable our custom PID controller code for the actuators with user type cymj.set_pid_control(self.sim.model, self.sim.data)######################################################################################### SUBCLASS REGISTRATIONdef register_joint_group(self, group_name, prefix): Finds and collect joint ids for given joint name prefix or a list of prefixes. if isinstance(prefix, str):self.qpos_idxs[group_name] joint_qpos_ids_from_prefix(self.sim.model, prefix)self.qvel_idxs[group_name] joint_qvel_ids_from_prefix(self.sim.model, prefix)elif isinstance(prefix, list):self.qpos_idxs[group_name] list(it.chain.from_iterable(joint_qpos_ids_from_prefix(self.sim.model, p) for p in prefix))self.qvel_idxs[group_name] list(it.chain.from_iterable(joint_qvel_ids_from_prefix(self.sim.model, p) for p in prefix))def register_joint_group_by_name(self, group_name, name): Finds and collect joint ids for given joint name or list of names. if isinstance(name, str):self.qpos_idxs[group_name] joint_qpos_ids(self.sim.model, name)self.qvel_idxs[group_name] joint_qvel_ids(self.sim.model, name)elif isinstance(name, list):self.qpos_idxs[group_name] list(it.chain.from_iterable(joint_qpos_ids(self.sim.model, n) for n in name))self.qvel_idxs[group_name] list(it.chain.from_iterable(joint_qvel_ids(self.sim.model, n) for n in name))######################################################################################### GET DATA OUT OF SIMdef get_qpos(self, group_name): Gets qpos for a particular group. return self.sim.data.qpos[self.qpos_idxs[group_name]]def get_qpos_dict(self, group_names): Gets qpos dictionary for multiple groups. return {k: self.get_qpos(k) for k in group_names}def get_qvel(self, group_name): Gets qvel for a particular group. return self.sim.data.qvel[self.qvel_idxs[group_name]]def get_qvel_dict(self, group_names): Gets qpos dictionary for multiple groups. return {k: self.get_qvel(k) for k in group_names}propertydef qpos(self): Returns. copy of full sim qpos. return self.sim.data.qpos.copy()propertydef qvel(self): Returns copy of full sim qvel. return self.sim.data.qvel.copy()def get_state(self) - MjSimState:return self.sim.get_state()######################################################################################### SET DATA IN SIMdef set_qpos(self, group_name, value): Sets qpos for a given group. self.sim.data.qpos[self.qpos_idxs[group_name]] valuedef set_qvel(self, group_name, value): Sets qpos for a given group. self.sim.data.qvel[self.qvel_idxs[group_name]] valuedef add_qpos(self, group_name, value): Sets qpos for a given group. self.sim.data.qpos[self.qpos_idxs[group_name]] valuedef set_state(self, state: MjSimState):self.sim.set_state(state)######################################################################################### INTERFACE TO UNDERLYING SIMdef step(self, with_uddTrue):Advances the simulation by calling mj_step.If qpos or qvel have been modified directly, the user is required to call:meth:.forward before :meth:.step if their udd_callback requires access to MuJoCostate set during the forward dynamics.self.sim.step(with_uddwith_udd)self.sim.forward()# To potentially communicate with other processesfor point in self.synchronization_points:point.synchronize()def reset(self):Resets the simulation data and clears buffers.self.sim.reset()def set_constants(self):Sets the derived constants of the mujoco simulation.self.sim.set_constants()def forward(self):Computes the forward kinematics. Calls mj_forward internally.self.sim.forward()def render(self,widthNone,heightNone,*,camera_nameNone,depthFalse,modeoffscreen,device_id-1):Renders view from a camera and returns image as an numpy.ndarray.Args:- width (int): desired image width.- height (int): desired image height.- camera_name (str): name of camera in model. If None, the freecamera will be used.- depth (bool): if True, also return depth buffer- device (int): device to use for rendering (only for GPU-backedrendering).Returns:- rgb (uint8 array): image buffer from camera- depth (float array): depth buffer from camera (only returnedif depthTrue)return self.sim.render(widthwidth,heightheight,camera_namecamera_name,depthdepth,modemode,device_iddevice_id,)######################################################################################### PROPERTIESpropertydef n_substeps(self): Number of substeps in the mujoco sim return self.sim.nsubsteps在这里我主要通过单步调试实现一个关节角的初始化。具体做法是注释掉上面初始化状态的代码写入自己的代码
from math import piprint(self.mujoco_simulation.qpos_idxs.keys())
self.mujoco_simulation.set_qpos(robot0:arm_joint_angles,np.asarray([1.5 * 0.5 * pi, -0.5 * pi, 1.5 * 0.5 * pi,-1.74529567, -4.18881842, 2.35619837]))self.mujoco_simulation.step()结果现实代码可以运行。效果如下
