
# Overview
Cepton’s lidar sensors send out two kinds of packet on a regular basis: Point Data Packet and INFO Data Packet. Please refer [Data Format](https://developer.cepton.com/doc/data_format) for detailed structure information. Chapter 2 in this document outlines the implementation of Precision Time Protocol (PTP) in Cepton’s products to illustrate the generation of critical PTP related values reported in those two kinds of packets. These values are mainly used for displaying real-time system information, or run-time self-monitoring and diagnostic. Chapter 3 gives an example of how to use those reported values to calculate universal PTP timestamp for each point. Chapter 2 is for your better understanding, but you can skip this Chapter and directly refer to chapter 3 for usage only. 

# Reported Value Generation 
##  Clock adjustment
Each slave clock maintains a local counter calculating the number of microseconds (us) since the sensor last booted up. When connection with PTP master clock becomes active, the clock adjustments shall perform synchronization and syntonization. Therefore, the universal PTP timestamp calculation is generally composed of three parts: 
<center>

$ptp\_timestamp = local\_counter + synchronization\_correction + syntonization\_correction$

</center>

Synchronization eliminates the offset between slave’s counter and master’s clock.  Syntonization matches the frequency of local clock to that of the Grandmaster Clock in the domain, which is measured by the slave's counter increases every drift interval, it is one nanosecond slower than the master's clock. Thus, the formula becomes more concrete: 
<center>

$ptp\_timestamp = local\_counter + master\_slave\_offset + \frac{local\_counter\_increase}{drift\_interval}$

</center>

Clock adjustments are carried out through a sequence of PTP timing messages exchange between upstream PTP Instance and local slave. The basic message exchange pattern is illustrated in IEEE Std 1588-2019 Figure 16 as shown below : 
<p align="center">
  <img width="460" src="/images/ptp/message_exchange.png">
</p> 

(a) The Master PTP Instance sends a Sync message to the Slave PTP Instance and notes the time `t1` at which it was sent. The timestamp `t1` can be embedded in the `Sync` message or in a `Follow_Up` message. 

(b) The Slave PTP Instance receives the `Sync` message and notes the time of reception `t2`.   

(c) The Slave PTP Instance sends a `Delay_Req` message to the Master PTP Instance and notes the time `t3` at which it was sent. 

(d) The Master PTP Instance receives the `Delay_Req` message and notes the time of reception `t4`. The Master PTP Instance conveys to the Slave PTP Instance the timestamp `t4` by embedding it in a `Delay_Resp` message or in a `Delay_Resp_Follow_Up` message.

The message exchange pattern illustrated in 2.1 is repeated contiguously to maintain latest `master_slave_offset`  and  `drift_interval`, which will be explained in 2.2 and 2.3 independently.

## Update `master_slave_offset` 
Say master sends a message to the slave at PTP time `m1`, after a communication link propagation delay `d`, the slave receives message at its local counter time `c1`, then the offset currently equals to: $Δ1 = c1 - (m1+d)$. The link propagation delay is required for updating the offset. Assuming symmetric link, that link delay is a mean of t<sub>ms</sub>
 and t<sub>sm</sub>. However, (t1, t4) and (t2, t3) follows different time metrics, therefore, the delay is calculated by:   
<center>

$d = \frac{{(t_2 - t_1) + (t_4 - t_3)}}{2} = \frac{{(t_2 - t_3) + (t_4 - t_1)}}{2}$

</center>

Initially, it’s set to zero.   

Since link delay can only be updated every time a complete group of $<t1, t2, t3, t4>$ is obtained, the newest offset is calculated based on the last group of $<t1, t2, t3, t4>$.

## Update `drift_interval`
Differentiate from offset, which depends only on the last group of message exchange, The drift_interval that will be reported is an accumulated value, which is updated every time a new pair of (t1, t2) is recorded. As mentioned in 2.1, the slave's counter increases every drift interval, it becomes one nanosecond slower or faster than the master's clock. According to this, what is being maintained is drift_interval, but what is being added to timestamp calculation is: $\frac{local\_counter\_increase}{drift\_interval}$.
The `drift_interval`  represents a time range of counter in nanosecond unit and is updated as follows: 
<center>

$drift\_interval = \frac{counter\_range}{master\_range - counter\_range}$

</center>

where:

<center>

$master\_range = current\_master\_arrival\_timestamp - last\_master\_arrival\_timestamp$

$counter\_range = current\_counter\_ns - last\_counter\_ns$

</center>

The drift_interval can be a positive value or negative value: 

 - when drift_interval is positive, it means counter timestamp is increasing slower than master clock; 

 - when drift_interval is negative, it means counter timestamp is increasing faster than master clock; 

 - when drift_interval is zero, which means counter timestamp is increasing in the same speed as master clock, syntonization_correction will also be zero.
  
## Report PTP relevant values 
Recall the PTP timestamp calculation formula:  

<center>

$ptp\_timestamp = local\_counter + master\_slave\_offset + \frac{local\_counter\_increase}{drift\_interval}$

</center>

We have explained the generation of master_slave_offset in 2.2 and the generation of drift_interval in 2.3. The local_counter is the slave’s internal non-decreasing counter timestamp (currently using emac system time, in us unit). The value counts the time interval between the moment of power-up (emac power-up) to the moment of INFO packet sent-out. 

Cepton’s lidar sensors send out two kinds of packet on a regular basis: Point Data Packet and INFO Data Packet.  

In INFO packet header, there are three fields containing PTP related information, which are filled in the local_counter value, master_slave_offset value, and drift_interval value correspondingly: 
| Field Name              | Unit   | 
|-------------------------|--------|
| time_since_powerup      | us     |
| time_offset_from_master | us     | 
| time_drift_correction   | per ns | 

In Point Data Packet’s payload, each point has a relative_timestamp field that holds the relative counter value from the time since last point in the packet. For the very first point, it is time since the packet header's reference time.     

# PTP Usage Example
The Cepton SDK usage is almost all about using Callback APIs, please refer [SDK Guide](https://developer.cepton.com/doc/sdk_guide) for more information. Here, only point’s PTP timestamp calculation relevant part will be explained. 

The master_slave_offset and drift_interval is only reported in INFO packet, therefore, to calculate following points’ timestamp, a local copy of previous INFO packet information is needed, the callback looks like this: 

```c
int64_t last_info_offset   = 0;
int64_t last_info_drift    = 0;  
int64_t last_info_counter  = 0;  

void infoCallback(CeptonSensorHandle handle, 
            const struct CeptonSensor *info, void *user_data) {
  last_info_counter = info->power_up_timestamp; 
  last_info_offset  = info->time_sync_offset; 
  last_info_drift   = info->time_sync_drift;  
} 
```
Since the calculation is based on previous INFO packet, the local counter increase in the formula becomes the time difference between each point’s counter and the counter timestamp when the previous INFO packet is sent out. Therefore, the power_up_timestamp in INFO packet must also be recorded. 

Each point’s counter timestamp calculation is a litter bit confusing. Recall that each point has a relative_timestamp field that holds the relative counter value from the time since last point in the packet. For the very first point, it is time since the packet header's reference time.  Here is how you can get each point’s counter value: 

```c
int64_t prev_point_counter = 0;  

void pointCallback(CeptonSensorHandle handle,
              int64_t start_timestamp, size_t n_points,
              size_t stride, const uint8_t *points,
              void *user_data){
    prev_point_counter = start_timestamp; 
    for (size_t i = 0; i < n_points; i++) { 
        struct CeptonPoint point = *(struct CeptonPoint const*)(points + i * stride); 
        int64_t point_counter = prev_point_counter + (int64_t)point.relative_timestamp;  
        prev_point_counter = point_counter; 
    } 
}
```

Based on above obtained value and formula, each point’s timestamp can be calculated as: 
```c
#include <stdio.h>
int64_t drift_correction = (last_info_drift==0) ? 0 :
 (point_counter - last_info_counter)*1000 / last_info_drift; 
printf("calculated point PTP timestamp(ns): %lld \n",  
  (last_info_counter - last_info_offset)*1000 + drift_correction  
); 
```
Pay attention to the unit conversion. 

Finnally, a complete sample code for your reference: 

```c
#include <stdio.h>
#include <stdlib.h>

#include "cepton_sdk2.h"

void check_api_error(int err, char const *api) {
  if (err != CEPTON_SUCCESS) {
    printf("API Error for %s: %s\n", api, CeptonGetErrorCodeName(err));
    exit(1);
  }
}

void sensorErrorCallback(CeptonSensorHandle handle, int error_code,
                         const char *error_msg, const void *error_data,
                         size_t error_data_size) {
  printf("Got error: %s\n", error_msg);
}

int n_info = 0;

//(per ns)
int64_t last_info_drift    = 0; 
//(1us)
int64_t last_info_counter  = 0; 
int64_t last_info_offset   = 0;
int64_t prev_point_counter = 0;

void infoCallback(CeptonSensorHandle handle, 
            const struct CeptonSensor *info, void *user_data) {
    printf("Got %d info:", ++n_info);
    printf("ts_info_counter: %lld ts_offset: %lld ts_drift: %lld\n",  
          info->power_up_timestamp, 
          info->time_sync_offset,
          info->time_sync_drift
    );
    last_info_counter = info->power_up_timestamp;
    last_info_offset  = info->time_sync_offset;
    last_info_drift = info->time_sync_drift;
  
}

void pointCallback(CeptonSensorHandle handle,
              int64_t start_timestamp, size_t n_points,
              size_t stride, const uint8_t *points,
              void *user_data){
    prev_point_counter = start_timestamp;
    for (size_t i = 0; i < n_points; i++) {
        struct CeptonPoint point = *(struct CeptonPoint const*)(points + i * stride);
        int64_t point_counter = prev_point_counter + (int64_t)point.relative_timestamp; 
        if (i == 0 || i == n_points-1) {
            printf("the raw counter(us) of point %d is: %lld\n", (int)i, point_counter);
            int64_t drift_correction = (last_info_drift==0) ? 0 : (point_counter - last_info_counter)*1000 / last_info_drift;
            printf("calculated point PTP timestamp(ns): %lld \n", 
                (last_info_counter - last_info_offset)*1000 + drift_correction 
            );//temporary - offset
            
        }
        prev_point_counter = point_counter;
    }
}

int main() {
  int ret;

  // Initialize
  ret = CeptonInitialize(CEPTON_API_VERSION, sensorErrorCallback);
  check_api_error(ret, "CeptonInitialize");

  ret = CeptonEnableLegacyTranslation();
  check_api_error(ret, "EnableLegacyTranslation");

  // Start networking listener thread
  ret = CeptonStartNetworking();
  check_api_error(ret, "CeptonStartNetworking");

  // Listen for points
  ret = CeptonListenPoints(pointCallback, 0);
  check_api_error(ret, "CeptonListenPoints");

  // Listen for info
  ret = CeptonListenSensorInfo(infoCallback, 0);
  check_api_error(ret, "CeptonListenSensorInfo");
  
  // Sleep
  while (n_info < 10)
    ;

  // Deinitialize
  ret = CeptonDeinitialize();
  check_api_error(ret, "CeptonDeinitialize");
  return 0;
}
```
This example prints the counter, offset and drift in each INFO packet, and the raw counter and calculated timestamp of the first and last points in each Point Data packet, feel free to modify the output to best fulfil your requirement. 

PTP Implementation and Point Timestamp Calculation


## Introduction
Cepton's perception team is actively developing a system to classify point cloud objects acquired from their LiDAR sensor.
Neural networks are widely known as the state-of-the-art approach to tackle the task of classification,
and Cepton LiDAR's high-resolution point cloud can be used with them, although efficient and accurate featurization has to be completed first.

In my internship, I integrated a voxel featurization into their system and tested its capability.
In this post, I will go over how point cloud data are featurized as voxel data and how the neural network architecture is tailored for voxel featurization.
In the end, I briefly report that my voxel featurization showed high efficiency and accuracy for the object classification task.


## Featurization

Data collected from our LiDAR sensors takes the shape of a point cloud, with each point having position information in the form of $x, y, z$ coordinates.
However, these unstructured points can’t easily be used in a neural network, so they must be converted to some structured form first, i.e. the input data has to be featurized.
There are many different ways of doing this, with the one I chose being a 3D voxel grid.


I start with normalizing the points to the origin.
I compute the minimum value for each dimension across the entire point cloud and then translate all points by subtracting these minimum values, aligning the point cloud with the origin.
I use cubical voxels of size $0.1m$, each of which is set to be 1.0 if it contains a point in it, otherwise 0.0.
I set the number of voxels as $24^3$ (2.4m each), to well capture both pedestrians and vehicles.

Once this was done for all of the points, it resulted in a collection of voxels that represented the point cloud.
Finally, I converted the grid into a tensor, which could then be fed into a neural network, which I will explain next. 

```python
import numpy as np
import torch

def featurize(point_cloud):
  # Find min and normalize
  minX = min(data, key=lambda x: x[0])[0]
  minY = min(data, key=lambda x: x[1])[1]
  minZ = min(data, key=lambda x: x[2])[2]
  data = [(x - minX, y - minY, z - minZ) for [x, y, z] in data]
  width = 0.1
  # Create grid and fill with zeros
  grid_size = 24
  voxelGrid = np.zeros((min_grid_size, min_grid_size, min_grid_size), dtype=float)
  # Fill in grid
  for [x, y, z] in data:
      indexX, indexY, indexZ = int(x / width), int(y / width), int(z / width)
      indexX = min(indexX, grid_size - 1)
      indexY = min(indexY, grid_size - 1)
      indexZ = min(indexZ, grid_size - 1)
      voxelGrid[indexX, indexY, indexZ] = 1

  # Convert a NumPy array to a Tensor
  voxelTensor = torch.from_numpy(voxelGrid)
  return voxelTensor
```
<div style="text-align: center;">
<img src="/images/voxel_classifier/pointcloud_pedestrian.png" width="350" height="350">
<img src="/images/voxel_classifier/voxelized_pedestrian.png" width="350" height="350">
</div>

<div style="text-align: center;">
<img src="/images/voxel_classifier/pointcloud_vehicle.png" width="350" height="350">
<img src="/images/voxel_classifier/voxelized_vehicle.png" width="350" height="350">
</div>


## Neural Network Architecture

After featurizing the point cloud data, the next step is to feed them to a convolutional neural network (CNN) to build a classifier.
The reason that this has to take place after featurization is that the point cloud data lacks a clear structure and features to learn from. However, featurizing the points as voxels gives CNN a structure to work with, which greatly benefits the learning process.

This type of neural network consists of multiple convolutional layers with max pooling and activation functions,
followed by fully connected layers and a final output layer.
For my classifier, which I implemented in PyTorch Lightning, I pretty much followed the <a href="https://cs231n.github.io/convolutional-networks/#architectures" target="_blank">standard structure</a>
: I had three convolutional layers, each followed by a max pool and ReLU, then two fully connected layers using a linear transformation, with the first one followed by ReLU, and finally a softmax and output layer.
Because the data was in three dimensions, I used the three-dimensional versions of the convolution and max pool functions.

Throughout this process, the dimensions go from $24 \rightarrow 22 \rightarrow 11 \rightarrow 9 \rightarrow 4 \rightarrow 2 \rightarrow 1$ while the number of filters goes from $1 \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 32 \rightarrow 3$ and the kernel size stayed constant at $3 \times 3 \times 3$. See the figure below for the details.  
<div style="text-align: center;">
<img src="/images/voxel_classifier/nn.png">

Note: this figure isn't completely accurate because the visualization tool doesn't support three-dimensional convolutions, but it's a fair approximation.
</div>

## Results and Revisions
The classifier achieved good accuracy, with > 99% on pedestrians and > 90% on vehicles.
The speed was also reasonably fast, although not the fastest out of all the classifiers available.
| True/Predicted | Unknown | Pedestrian | Vehicle |
| -------------- | ------- | ---------- | ------- |
| Unknown        | 9.69%   | 0.0%       | 0.0%    |
| Pedestrian     | 0.0%    | 65.39%     | 0.06%   |
| Vehicle        | 0.0%    | 2.71%      | 22.15%  |

| Inference Batch | Object Count | Time (ms) |
| --------------- | ------------ | --------- |
| Single          | 1            | 0.83      |
| Small Batch     | 10           | 7.28      |
| Large Batch     | 100          | 89.47     |

### Further Extension
As an extension, I experimented with dropping the final convolutional layer and its associated max pool and ReLU, while reducing the initial dimensions of the grid to a $10 \times10 \times 10$. This resulted in the number of filters topping out at $32$, and the dimensions going from $10 \rightarrow 8 \rightarrow 4 \rightarrow 2 \rightarrow 1$. These changes were effective, with the classifier's speed improving significantly ($\times 35$ faster) while accuracy remained the same.

| Inference Batch | Object Count | Time (ms) |
| --------------- | ------------ | --------- |
| Single          | 1            | 0.26      |
| Small Batch     | 10           | 0.81      |
| Large Batch     | 100          | 2.59      |


<div style="text-align: center;">
<img src="/images/voxel_classifier/nn2.png">
</div>

## Conclusion
In this post, I went over the voxel featurization and the CNN architectures to featurize and classify point cloud data. 
The classifier I implemented was able to achieve high accuracy and speed. In the future, the accuracy could be improved with greater amounts of training data, and the speed could likely be increased with further refinements to the neural net architecture.

## References
<a href="https://cs231n.github.io/convolutional-networks/#architectures" target="_blank">Convolutional Neural Network Structure from Stanford</a>


Point Cloud Object Classification with Voxel Featurization


## Introduction
With Cepton Inc.'s Lidar sensor,
you can collect high-resolution point cloud data in real-time as below.

<p align="center">
  <img width="460" height="300" src="/images/lyft_dataset/cepton_lidar.gif">
 </p>

As you can see, it captures pedestrians and cars.
If you take a close look, you can also see how fast the cars are moving or even how tall the pedestrians are.
Our Lidar provides various informative data to us.

However, this data is only comprehensible to us, humans, not to computers.
That's where our team, the Perception team, comes into play.
Our team's goal is to figure out what information we can extract from the point cloud data captured by our Lidar.

In this post, I go over a public dataset,
[Lyft 3D Object Detection for Autonomous Vehicles](https://www.kaggle.com/code/tarunpaparaju/lyft-competition-understanding-the-data),
release from Lyft to see what information they handle.

### Purpose
Find out the basic usage of data properties of [Lyft 3D Object Detection for Autonomous Vehicles](https://www.kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles/data).

## Procedures and Results 

### Downloading Data

First thing first, go to the page of [Lyft 3D Object Detection for Autonomous Vehicles](https://www.kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles/data) and click **Download All** located at the bottom right of the page.
Here, I note that the total data size is around 90GB.
Make sure you have enough disk space in your environment.


### Read and Write Data
With the `3d-object-detection-for-autonomous-vehicles` directory ready, let's read the dataset.
Kindly enough, Lyft provides their own SDK in Python, `lyft_dataset_sdk` to play with the dataset.

The following code is to read the dataset and export the point cloud data and the bounding boxes data to JSON files.
(Note: Strangely, you need to rename directories for training data as follows: `train_images` -> `images`, `train_lidar` -> `lidar`, `train_maps` -> `maps`, and `train_data` -> `data`)

```python
from lyft_dataset_sdk.lyftdataset import LyftDataset
from lyft_dataset_sdk.utils.geometry_utils import (
    BoxVisibility,
)
from lyft_dataset_sdk.utils.data_classes import LidarPointCloud
import json

def generate_next_token(scene):
    scene = lyft_dataset.scene[scene]
    sample_token = scene["first_sample_token"]
    sample_record = lyft_dataset.get("sample", sample_token)

    while sample_record["next"]:
        sample_token = sample_record["next"]
        sample_record = lyft_dataset.get("sample", sample_token)
        yield sample_token

DATA_PATH = "./3d-object-detection-for-autonomous-vehicles/"
lyft_dataset = LyftDataset(data_path=DATA_PATH, json_path=DATA_PATH + "data")

for scene_id in range(len(lyft_dataset.scene)):
    for frame, sample_token in enumerate(generate_next_token(scene_id)):
        print(f"Processing scene {scene_id}, frame {frame}")
        sample = lyft_dataset.get("sample", sample_token)
        sd_record = lyft_dataset.get("sample_data", sample["data"]["LIDAR_TOP"])

        _, boxes, _ = lyft_dataset.get_sample_data(
            sample["data"]["LIDAR_TOP"],
            box_vis_level=BoxVisibility.ANY,
            flat_vehicle_coordinates=False,
        )

        pc, _ = LidarPointCloud.from_file_multisweep(
            lyft_dataset,
            lyft_dataset.get("sample", sd_record["sample_token"]),
            sd_record["channel"],
            "LIDAR_TOP",
            num_sweeps=5,
        )

        points = pc.points[:3, :].T.tolist()

        box_data = [{"corners": box.corners().T.tolist(), "label": box.name} for box in boxes]

        data = {
            "points": points,
            "boxes": box_data,
        }
        with open(f"exported/scene_{scene_id}_frame_{frame}.json", "w") as f:
            json.dump(data, f)
```

Here I visualize one of the scenes on our Cepton Viewer.
It seems like one scene is about 6-seconds long.
This scene particularly is captured by a car taking a left turn at an intersection.

<p align="center">
  <img width="460" height="300" src="/images/lyft_dataset/cepton_view.gif">
 </p>

### What objects are in it?
To know more about the dataset, let's see what types of objects it contains.
Following is the code to show the demographics of the objects in the whole dataset.

```python
# Overlapped code omitted for brevity.  ...

import matplotlib.pyplot as plt
from collections import Counter

# Overlapped code omitted for brevity.  ...
all_box_names = []
for scene_id in range(len(lyft_dataset.scene)):
    for frame, sample_token in enumerate(generate_next_token(scene_id)):
        print(f"Processing scene {scene_id}, frame {frame}")
        sample = lyft_dataset.get("sample", sample_token)

        _, boxes, _ = lyft_dataset.get_sample_data(
            sample["data"]["LIDAR_TOP"],
            box_vis_level=BoxVisibility.ANY,
            flat_vehicle_coordinates=False,
        )

        all_box_names.extend([box.name for box in boxes])

# Create the histogram.
counts = Counter(all_box_names)
labels, values = zip(*counts.items())
plt.figure(figsize=(10,5))
plt.bar(labels, values)
plt.xlabel('Object Names')
plt.ylabel('Count')
plt.title('Distribution of Object Names')
plt.xticks(rotation=90)
plt.tight_layout()
plt.show()
```

![Distribution](/images/lyft_dataset/object_demographics.png)

As it shows, the large majority consists of cars.
Unexpectedly, this dataset has a small population of pedestrians because detecting people tends to be a critical task in many perception applications.


## Conclusion

In this post, I briefly investigated the [Lyft 3D Object Detection for Autonomous Vehicles](https://www.kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles/data).
The dataset was very handy to use, thanks to the SDK provided by Lyft.
But it is important for us to be aware that there is a large imbalance between the population of cars and pedestrians.

## References
- [Lyft 3D Object Detection for Autonomous Vehicles](https://www.kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles/data)
- [Lyft Dataset SDK](https://pypi.org/project/lyft-dataset-sdk/)


---

About the author: Hikaru Ibayashi is a Software Engineer at Cepton Inc. Perception team. 
He holds a Ph.D. degree in Computer Science from the University of Southern California, where his research focused on the foundation of deep neural networks and physic-informed machine learning.

---

Lyft Dataset Investigation

<p>Firmware should only be updated when it is necessary to fix a bug or access a new feature. Customer should consult with a Cepton field support engineer before undertaking an update. The tool to update firmware and latest firmware image will be provided by field support engineers upon request. Publicly available update tool and current firmware image may be provided for download in the future.</p>

<p>Currently 802.1AS is supported, with full support planned for later.</p>

<p>Yes. Full support for Python (Python 3.6 or later) and Python specific APIs are part of our SDK release.</p>

<p>Yes. Javascript bindings to SDK is currently in pre-release beta, available upon request.</p>

<p>Yes. C# bindings to SDK is currently in pre-release beta, available upon request.</p>

Developer Portal

Downloads

Documentations

Showcase

Blogs

Videos

FAQs

Join Our Team