C++ Sample

Two samples are provided in the C++ version of the Cochl.Sense SDK: sense-file and sense-stream.

  • sense-file performs a prediction on an audio file (wav or mp3).

  • sense-stream performs a prediction on an audio buffer.

1. Check the Requirements


Follow our Getting started section to set up the environment.


2. C++ Requirements


Install the packages below to use our Cochl.Sense C++ SDK.

sudo apt update
sudo apt install unzip
sudo apt install libpulse-dev pulseaudio pulseaudio-utils

3. Prepare the Sample


The samples can be found here

git clone https://github.com/cochlearai/sense-sdk-cpp-tutorials.git

Unzip the SDK

unzip path/to/sdk/sense-sdk-<version>-cpp.zip -d path/to/sample/sense-sdk-cpp-tutorials/

Your sample directory should now look like this

└── sense-sdk-cpp-tutorials
    ├── audio_files          # audio samples
    ├── examples
    |   ├── sense-file.cc    # sense-file sample
    │   └── sense-stream.cc  # sense-stream sample
    └── sense                # Sense SDK
        ├── include
        ├── lib
        └── license

4. Install license manager for SDK


To use above v1.4.0 Cochl.Sense SDK, you should prepare the license manager before installing the SDK. This license manager is used to manage Cochl.Sense SDK license when devices are on the network or not.

# folder tree in C++ SDK 1.4.0 for x86_64 Ubuntu Linux
└── sense-sdk-cpp-tutorials
    ├── audio_files          # audio samples
    ├── examples
    └── sense                # Sense SDK
          ├── include
          ├── lib
          └── license
                ├── bin
                ├── dinst
                ├── dunst
                ├── haspvlib_25011.so
                ├── haspvlib_arm64_25011.so
                ├── haspvlib_armhf_25011.so
                ├── haspvlib_x86_64_25011.so
                └── pkg

To install the license manager, run the following commands.

cd sense/license
sudo ./dinst

Uninstall license manager for SDK

(Warning) Uninstallation may affect other applications using the license manager service. If you want to remove the license manager, please check whether the system uses this service. To remove the license manager, run the following commands.

# check the service
ps -aux | grep hasplmd

# remove the license manager
cd sense/license
sudo ./dunst

5. Parameters


There are some parameters to set details of Cochl.Sense SDK behaviours.

// sense.hpp

...
struct Parameters {
  /// The name you want to set for this device in order to find it
  /// on your Dashboard.
  std::string device_name = "";
  
  // This list depends on your platform. Cf sense.hpp.
  ModelDelegate model_delegate = ModelDelegate::Default;
  
  /// Number of threads available to the tflite interpreter.
  int num_threads = 0;

  /// See below.
  Metrics metrics;

  /// Specify the desired log level for this device:
  /// 0: Debug
  /// 1: Information
  /// 2: Warning
  /// 3: Error
  /// The default log level is 1
  int log_level = 1;

  /// New features
  HopSizeControl hop_size_control;
  SensitivityControl sensitivity_control;
  ResultAbbreviation result_abbreviation;
  LabelHiding label_hiding;
...


Every time a frame (a unit of audio inference) is processed by the SDK, a metric is created. A metric represents the result of a frame inference. Metrics are meant to be sent to the server and visible on your project dashboard.
Only one device_name can be used per device. If you want to change the device_name you’re using now, it can be changed on Project page of of the Dashboard.

Possible Metrics options

// sense.hpp
struct Metrics {
  size_t retention_period = 0;   // range, 1 to 31 days
  size_t free_disk_space = 100;  // range, 0 to 1,000,000 MB
  size_t push_period = 30;       // range, 1 to 3,600 seconds
};

retention_period

  • The sense SDK tries to push the metrics to our servers right after it was created. If the push fails, the SDK stores the data locally and will try again later.
  • retention_period determines the number of days that the SDK will keep the metrics data locally.
  • Default value is 0 (metrics data is not saved locally).
  • If an invalid value is passed (e.g., greater than 31 days), it will be set to 31 (days).
  • If a user has set this period, metrics that are older than the retention period will be removed immediately.
  • It is project-specific; e.g., retention period of project X can be 7 days, while the retention period of project Y is 14 days.

free_disk_space

  • When the available disk space is less than free_disk_space (unit MB), the SDK will stop storing the metrics locally. In that case, the retention period set by the user will be overridden and set to 0 (days).
  • If an invalid value is passed (e.g., greater than 100 MB), it will be set to 100 (MB).
  • If the retention period is 0 and there are metrics stored locally, the SDK will attempt to push them one more time before removing them permanently.

push_period

  • push_period determines how often metrics are pushed to the server.
  • If an invalid value is passed (0 or greater than 3,600 seconds), it will be set to 30 (seconds).

New Features

New features have been added starting from the version 1.4.0

Default Hop Size

  • Inference interval choice: 0.5 seconds or 1 second
  • Inference interval means the time gap between one frame (unit of inference, 1-sec long) and another.
  • For example, the frames are constructed as represented below, with an inference interval of 0.5 seconds:
    • (0.0s ~ 1.0s)
    • (0.5s ~ 1.5s)
    • (1.0s ~ 2.0s)
  • When enabled, a 0.5-second interval is applied. It makes frames overlap, offering less possibility of missing sounds between frames.
  • When disabled, 1 second is applied so no overlap happens.
  • You can turn off this feature if your device lacks computing power and takes more than 500 milliseconds to make an inference (e.g., Raspberry Pi 2).
struct HopSizeControl {
  bool enable = true;
};

Sensitivity Control

  • The detection sensitivity of all tags or each tag can be adjusted by enabling this feature.
  • High sensitivity means a lower threshold, so the inference result is more likely to include the detections.
  • Low sensitivity means a higher threshold, so only the tags detected with high probability are present in the model output.
struct SensitivityControl {
  bool enable = true;

  /// You have 5 options:
  /// - Very high sensitivity (-2)
  /// - High sensitivity      (-1)
  /// - Normal (default)      ( 0)
  /// - Low sensitivity       ( 1)
  /// - Very low sensitivity  ( 2)
  int default_sensitivity = 0;

  /// Set the sensitivity for specific tags.
  /// For example, sensitivity["Gunshot"] = -2;
  std::map<std::string, int> sensitivity;
};

Result Summary

  • By default, Cochl.Sense SDK provides detailed results for each inference.
  • When enabled, inference results are displayed less verbose.
    • It returns a more concise output showing when each sound tag was detected and how long it was present.
  • Its behavior is different in file mode and stream mode:
    • In stream mode, it groups consecutive same tags.
    • In file mode, it uses two parameters: im (interval margin) and default_im.
      • im controls the output format, allowing for intervals to be grouped when a tag is detected within a specified margin.
struct ResultAbbreviation {
  bool enable = true;

  /// The unit of the im is seconds, and it can be any.
  size_t default_im = 1;

  /// Set the im for specific tags.
  std::map<std::string, int> im;
};

Loudness Filtering

  • This feature is exclusive to stream mode.
  • When enabled, it allows you to filter out unreliable results.
  • It stores a buffer of audio data as energy values for a specified duration (buffer_size).
  • During inference, it compares the audio data’s energy to the percentile energy value from the buffer.
  • If the audio data’s energy is lower than the percentile value, the Cochl.Sense disregards it as noise, even if meaningful tags are detected.
struct LabelHiding {
  bool enable = true;

  /// Values between 60 and 600, otherwise the default value 300
  size_t buffer_size = 300;

  /// Values between 0 and 100, otherwise the default value 50
  size_t percentile = 50;
};

6. Build


Put your Project Key into the sample code. Note that it’s important to invoke the sense::Terminate() function at the end if sense::Init() was successful. Otherwise, an undefined behavior may occur while cleaning up the memory allocated by the sense during sense::init().

// examples/sense-file.cc

...
if (sense::Init("Your project key",
                sense_params) < 0) {
  return -1;
}
...
sense::Terminate();
...

We are now ready to build the sense-file.

g++ -fopenmp examples/sense-file.cc -I./sense/include/ -lsense-core -L./sense/lib -o sense-file -lm -std=c++11 -ldl -lstdc++ -Wl,-rpath -Wl,./sense/lib

Put your Project Key into the sample code. Note that it’s important to invoke the sense::Terminate() function at the end if sense::Init() was successful. Otherwise, an undefined behavior may occur while cleaning up the memory allocated by the sense during sense::init().

// examples/sense-stream.cc

...
if (sense::Init("Your project key",
                sense_params) < 0) {
  return -1;
}
...
sense::Terminate();
...

We are now ready to build the sense-stream.

g++ -fopenmp examples/sense-stream.cc -I./sense/include/ -lsense-core -L./sense/lib -o sense-stream -lm -std=c++11 -ldl -lstdc++ -lpulse -lpulse-simple -Wl,-rpath -Wl,./sense/lib

7. Run


LD_LIBRARY_PATH=. ./sense-file <PATH_TO_AUDIO_FILE>

The repository contains audio files that can be used as well.

LD_LIBRARY_PATH=. ./sense-file audio_files/babycry.wav

Make sure the input device is properly connected before you run sense-stream

LD_LIBRARY_PATH=. ./sense-stream

(Note) sense-stream troubleshooting


Try the steps below when you have trouble running sense-stream

(1) Check connection

  • Check if the input device is connected

(2) Check power

  • The board may lack of power when an external device is connected
  • Check if the board is stable and has enough power

(3) Check pulseaudio

# start pulseaudio
pulseaudio -D

# check your device index from the pulseaudio command result
pacmd list-sources | grep -e 'index:' -e device.string -e 'name:'

# set default source with the index above
pacmd set-default-source <DEVICE_INDEX>

(Note) The Cochl.Sense CPU and memory usage in stream mode


Test environment specification:

  • Sense SDK v1.2.0

    • Parameters:
      sense::Parameters sense_params;
      sense_params.metrics.retention_period = 0;
      sense_params.metrics.free_disk_space = 100;
      sense_params.metrics.push_period = 30;
      
  • Raspberry Pi 3 Model B with TensorFlow Lite

    • OS(Raspberry Pi OS)
      • Release date: April 4th 2022
      • System: 64-bit
      • Kernel version: 5.15
      • Debian version: 11 (bullseye)
    • Quad Core 1.2GHz Broadcom BCM2837 64bit CPU(ARM Cortex A53)
    • 1GB RAM
    • BCM43438 wireless LAN
    • Storage(SanDisk 16GB Ultra Micro SD HC Class 10)

Result:

  • In idle, CPU usage: 2.2526 (%), memory usage: 229,578 (kB)

CPU usage (%):

CPU usage result

Memory usage (kB):

Memory usage result