opencda.core.sensing.perception package

Submodules

opencda.core.sensing.perception.o3d_lidar_libs module

Utility functions for 3d lidar visualization and processing by utilizing open3d.

opencda.core.sensing.perception.o3d_lidar_libs.o3d_camera_lidar_fusion(objects, yolo_bbx, lidar_3d, projected_lidar, lidar_sensor)

Utilize the 3D lidar points to extend the 2D bounding box from camera to 3D bounding box under world coordinates.

Parameters:

• objects (dict) – The dictionary contains all object detection results.

• yolo_bbx (torch.Tensor) – Object detection bounding box at current photo from yolov5, shape (n, 5)->(n, [x1, y1, x2, y2, label])

• lidar_3d (np.ndarray) – Raw 3D lidar points in lidar coordinate system.

• projected_lidar (np.ndarray) – 3D lidar points projected to the camera space.

• lidar_sensor (carla.sensor) – The lidar sensor.

Returns:

objects – The update object dictionary that contains 3d bounding boxes.

Return type:

dict

opencda.core.sensing.perception.o3d_lidar_libs.o3d_pointcloud_encode(raw_data, point_cloud)

Encode the raw point cloud(np.array) to Open3d PointCloud object.

Parameters:

• raw_data (np.ndarray) – Raw lidar points, (N, 4).

• point_cloud (o3d.PointCloud) – Open3d PointCloud.

opencda.core.sensing.perception.o3d_lidar_libs.o3d_visualizer_init(actor_id)

Initialize the visualizer.

Parameters:

actor_id (int) – Ego vehicle’s id.

Returns:

vis – Initialize open3d visualizer.

Return type:

o3d.visualizer

opencda.core.sensing.perception.o3d_lidar_libs.o3d_visualizer_show(vis, count, point_cloud, objects)

Visualize the point cloud at runtime.

Parameters:

• vis (o3d.Visualizer) – Visualization interface.

• count (int) – Current step since simulation started.

• point_cloud (o3d.PointCloud) – Open3d point cloud.

• objects (dict) – The dictionary containing objects.

opencda.core.sensing.perception.obstacle_vehicle module

Obstacle vehicle class to save object detection.

class opencda.core.sensing.perception.obstacle_vehicle.BoundingBox(corners)

Bases: object

Bounding box class for obstacle vehicle.

Params: -corners : nd.nparray

Eight corners of the bounding box. (shape:(8, 3))

Attributes: -location : carla.location

The location of the object.

-extentcarla.vector3D

The extent of the object.

class opencda.core.sensing.perception.obstacle_vehicle.ObstacleVehicle(corners, o3d_bbx, vehicle=None, lidar=None, sumo2carla_ids=None)

Bases: object

A class for obstacle vehicle. The attributes are designed to match with carla.Vehicle class.

Parameters:

• corners (nd.nparray) – Eight corners of the bounding box. shape:(8, 3).

• o3d_bbx (pen3d.AlignedBoundingBox) – The bounding box object in Open3d. This is mainly used for visualization

• vehicle (carla.Vehicle) – The carla.Vehicle object.

• lidar (carla.sensor.lidar) – The lidar sensor.

• sumo2carla_ids (dict) – Sumo to carla mapping dictionary, this is used only when co-simulation is activated. We need this since the speed info of vehicles that are controlled by sumo can not be read from carla server. We will need this dict to read vehicle speed from sumo api–traci.

bounding_box

Bounding box of the osbject vehicle.

Type:

BoundingBox

location

Location of the object.

Type:

carla.location

velocity

Velocity of the object vehicle.

Type:

carla.Vector3D

carla_id

The obstacle vehicle’s id. It should be the same with the corresponding carla.Vehicle’s id. If no carla vehicle is matched with the obstacle vehicle, it should be -1.

Type:

int

get_location()

Return the location of the object vehicle.

get_transform()

Return the transform of the object vehicle.

get_velocity()

Return the velocity of the object vehicle.

set_carla_id(id)

Set carla id according to the carla.vehicle.

Parameters:

id (int) – The id from the carla.vehicle.

set_vehicle(vehicle, lidar, sumo2carla_ids)

Assign the attributes from carla.Vehicle to ObstacleVehicle.

Parameters:

• vehicle (carla.Vehicle) – The carla.Vehicle object.

• lidar (carla.sensor.lidar) –

  The lidar sensor, it is used to project world coordinates to

  sensor coordinates.

• sumo2carla_ids (dict) – Sumo to carla mapping dictionary, this is used only when co-simulation is activated. We need this since the speed info of vehicles that are controlled by sumo can not be read from carla server. We will need this dict to read vehicle speed from sumo api–traci.

set_velocity(velocity)

Set the velocity of the vehicle.

Parameters:

velocity (carla.Vector3D) – The target velocity in 3d vector format.

opencda.core.sensing.perception.obstacle_vehicle.is_vehicle_cococlass(label)

Check whether the label belongs to the vehicle class according to coco dataset. :param -label: The lable of the detecting object. :type -label: int

Returns:

Whether this label belongs to the vehicle class

Return type:

-is_vehicle(bool)

opencda.core.sensing.perception.perception_manager module

Perception module base.

class opencda.core.sensing.perception.perception_manager.CameraSensor(vehicle, world, relative_position, global_position)

Bases: object

Camera manager for vehicle or infrastructure.

Parameters:

• vehicle (carla.Vehicle) – The carla.Vehicle, this is for cav.

• world (carla.World) – The carla world object, this is for rsu.

• global_position (list) – Global position of the infrastructure, [x, y, z]

• relative_position (str) – Indicates the sensor is a front or rear camera. option: front, left, right.

image

Current received rgb image.

Type:

np.ndarray

sensor

The carla sensor that mounts at the vehicle.

Type:

carla.sensor

static spawn_point_estimation(relative_position, global_position)

class opencda.core.sensing.perception.perception_manager.LidarSensor(vehicle, world, config_yaml, global_position)

Bases: object

Lidar sensor manager.

Parameters:

• vehicle (carla.Vehicle) – The carla.Vehicle, this is for cav.

• world (carla.World) – The carla world object, this is for rsu.

• config_yaml (dict) – Configuration dictionary for lidar.

• global_position (list) – Global position of the infrastructure, [x, y, z]

o3d_pointcloud

Received point cloud, saved in o3d.Pointcloud format.

Type:

03d object

sensor

Lidar sensor that will be attached to the vehicle.

Type:

carla.sensor

class opencda.core.sensing.perception.perception_manager.PerceptionManager(vehicle, config_yaml, cav_world, data_dump=False, carla_world=None, infra_id=None)

Bases: object

Default perception module. Currenly only used to detect vehicles.

Parameters:

• vehicle (carla.Vehicle) – carla Vehicle, we need this to spawn sensors.

• config_yaml (dict) – Configuration dictionary for perception.

• cav_world (opencda object) –

  CAV World object that saves all cav information, shared ML model,

  and sumo2carla id mapping dictionary.

• data_dump (bool) – Whether dumping data, if true, semantic lidar will be spawned.

• carla_world (carla.world) – CARLA world, used for rsu.

lidar

Lidar sensor manager.

Type:

opencda object

rgb_camera

RGB camera manager.

Type:

opencda object

o3d_vis

Open3d point cloud visualizer.

Type:

o3d object

activate_mode(objects)

Use Yolov5 + Lidar fusion to detect objects.

Parameters:

objects (dict) – The dictionary that contains all category of detected objects. The key is the object category name and value is its 3d coordinates and confidence.

Returns:

objects – Updated object dictionary.

Return type:

dict

deactivate_mode(objects)

Object detection using server information directly.

Parameters:

objects (dict) – The dictionary that contains all category of detected objects. The key is the object category name and value is its 3d coordinates and confidence.

Returns:

objects – Updated object dictionary.

Return type:

dict

destroy()

Destroy sensors.

detect(ego_pos)

Detect surrounding objects. Currently only vehicle detection supported.

Parameters:

ego_pos (carla.Transform) – Ego vehicle pose.

Returns:

objects – A list that contains all detected obstacle vehicles.

Return type:

list

dist(a)

A fast method to retrieve the obstacle distance the ego vehicle from the server directly.

Parameters:

a (carla.actor) – The obstacle vehicle.

Returns:

distance – The distance between ego and the target actor.

Return type:

float

filter_vehicle_out_sensor(vehicle_list)

By utilizing semantic lidar, we can retrieve the objects that are in the lidar detection range from the server. This function is important for collect training data for object detection as it can filter out the objects out of the senor range.

Parameters:

vehicle_list (list) – The list contains all vehicles information retrieves from the server.

Returns:

new_vehicle_list – The list that filters out the out of scope vehicles.

Return type:

list

retrieve_traffic_lights(objects)

Retrieve the traffic lights nearby from the server directly. Next version may consider add traffic light detection module.

Parameters:

objects (dict) – The dictionary that contains all objects.

Returns:

object – The updated dictionary.

Return type:

dict

speed_retrieve(objects)

We don’t implement any obstacle speed calculation algorithm. The speed will be retrieved from the server directly.

Parameters:

objects (dict) – The dictionary contains the objects.

visualize_3d_bbx_front_camera(objects, rgb_image, camera_index)

Visualize the 3d bounding box on frontal camera image.

Parameters:

• objects (dict) – The object dictionary.

• rgb_image (np.ndarray) – Received rgb image at current timestamp.

• camera_index (int) – Indicate the index of the current camera.

class opencda.core.sensing.perception.perception_manager.SemanticLidarSensor(vehicle, world, config_yaml, global_position)

Bases: object

Semantic lidar sensor manager. This class is used when data dumping is needed.

Parameters:

• vehicle (carla.Vehicle) – The carla.Vehicle, this is for cav.

• world (carla.World) – The carla world object, this is for rsu.

• config_yaml (dict) – Configuration dictionary for lidar.

• global_position (list) – Global position of the infrastructure, [x, y, z]

o3d_pointcloud

Received point cloud, saved in o3d.Pointcloud format.

Type:

03d object

sensor

Lidar sensor that will be attached to the vehicle.

Type:

carla.sensor

opencda.core.sensing.perception.sensor_transformation module

This script contains the transformations between world and different sensors.

opencda.core.sensing.perception.sensor_transformation.bbx_to_world(cords, vehicle)

Convert bounding box coordinate at vehicle reference to world reference.

Parameters:

• cords (np.ndarray) – Bounding box coordinates with 8 vertices, shape (8, 4)

• vehicle (opencda object) – Opencda ObstacleVehicle.

Returns:

bb_world_cords – Bounding box coordinates under world reference.

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.create_bb_points(vehicle)

Extract the eight vertices of the bounding box from the vehicle.

Parameters:

vehicle (opencda object) – Opencda ObstacleVehicle that has attributes.

Returns:

bbx – 3d bounding box, shape:(8, 4).

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.get_2d_bb(vehicle, sensor, senosr_transform)

Summarize 2D bounding box creation.

Parameters:

• vehicle (carla.Vehicle) – Ego vehicle.

• sensor (carla.sensor) – Carla sensor.

• senosr_transform (carla.Transform) – Sensor position.

Returns:

p2d_bb – 2D bounding box.

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.get_bounding_box(vehicle, camera, sensor_transform)

Get vehicle bounding box and project to sensor image.

Parameters:

• vehicle (carla.Vehicle) – Ego vehicle.

• camera (carla.sensor) – Carla rgb camera spawned at the vehicles.

• sensor_transform (carla.Transform) – Sensor position in the world.

Returns:

camera_bbx – Bounding box coordinates in sensor image.

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.get_camera_intrinsic(sensor)

Retrieve the camera intrinsic matrix.

Parameters:

sensor (carla.sensor) – Carla rgb camera object.

Returns:

matrix_x – The 2d intrinsic matrix.

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.p3d_to_p2d_bb(p3d_bb)

Draw 2d bounding box(4 vertices) from 3d bounding box(8 vertices). 2D bounding box is represented by two corner points.

Parameters:

p3d_bb (np.ndarray) – The 3d bounding box is going to project to 2d.

Returns:

p2d_bb – Projected 2d bounding box.

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.project_lidar_to_camera(lidar, camera, point_cloud, rgb_image)

Project lidar to camera space.

Parameters:

• lidar (carla.sensor) – Lidar sensor.

• camera (carla.sensor) – RGB camera.

• point_cloud (np.ndarray) – Cloud points, shape: (n, 4).

• rgb_image (np.ndarray) – RGB image from camera.

Returns:

• rgb_image (np.ndarray) – New rgb image with lidar points projected.

• points_2d (np.ndarrya) – Point cloud projected to camera space.

opencda.core.sensing.perception.sensor_transformation.sensor_to_world(cords, sensor_transform)

Project coordinates in sensor to world reference.

Parameters:

• cords (np.ndarray) – Coordinates under sensor reference.

• sensor_transform (carla.Transform) – Sensor position in the world.

Returns:

world_cords – Coordinates projected to world space.

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.vehicle_to_sensor(cords, vehicle, sensor_transform)

Transform coordinates from vehicle reference to sensor reference.

Parameters:

• cords (np.ndarray) – Coordinates under vehicle reference, shape (n, 4).

• vehicle (opencda object) – Carla ObstacleVehicle.

• sensor_transform (carla.Transform) – Sensor position in the world.

Returns:

sensor_cord – Coordinates in the sensor reference, shape(4, n)

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.world_to_sensor(cords, sensor_transform)

Transform coordinates from world reference to sensor reference.

Parameters:

• cords (np.ndarray) – Coordinates under world reference, shape: (4, n).

• sensor_transform (carla.Transform) – Sensor position in the world.

Returns:

sensor_cords – Coordinates in the sensor reference.

Return type:

np.ndarray

opencda.core.sensing.perception.sensor_transformation.x_to_world_transformation(transform)

Get the transformation matrix from x(it can be vehicle or sensor) coordinates to world coordinate.

Parameters:

transform (carla.Transform) – The transform that contains location and rotation

Returns:

matrix – The transformation matrx.

Return type:

np.ndarray

Module contents