C++ Sample

Two samples are provided in C++ version of Cochl.Sense SDK: sense-file and sense-stream.

sense-flie makes prediction out of the input file, and sense-stream makes prediction out of the input stream such as a microphone.

This guide walks through how to run the samples.

1. Check Requirements

Follow Getting started and set up environments.

2. Prepare Sample

Prepare sample

$ git clone https://github.com/cochlearai/sense-sdk-cpp-tutorials.git

Unzip SDK

$ unzip path/to/sdk/sense-sdk-<version>-cpp.zip -d path/to/sample/sense-sdk-cpp-tutorials/

The prepared sample directory

└── sense-sdk-cpp-tutorials
    ├── audio_files          # sample audio files
    ├── examples
    |   ├── AudioFile.h
    |   ├── sense-file.cc    # example for file input
    │   └── sense-stream.cc  # example for stream input
    └── sense                # SDK
        ├── include
        └── lib

3. Set Up Parameters

Set up parameters with values of your choice before you build and run the C++ sample.

It can be set in the SDK file sense/include/sense.hpp. Check possible options and set your parameters.

For example, change the value of service to Service::Home_Context if you want to detect baby crying or dog barking. In this link, you can check lists of detectable events for each service.

And you can also set the metric parameters to change how SDK manages metric data.

`retention_period`

SDK tries to push the metrics to ours servers right after it is created. If the push fails, the SDK stores the data locally and will try again later.
retention_period determines the number of days that the SDK will keep the metrics data locally.
Default value is 0 (metrics data not saved locally).
If an invalid value is passed (e.g. greater than 31 days), it will be set to 31 (days).
If a user has set this period, metrics that are older than the retention period will be removed immediately.
It is project-specific; e.g. retention period of projext X can be 7 days, while there retention period of project Y is 14 days.

`free_disk_space`

When the available disk space is less then free_disk_space (unit MB), the SDK will stop storing the metrics locally. In that case, the retention period set by the user will be overridden and set to 0 (days).
If an invalid value is passed (e.g. greater than 100 MB), it will be set to 100 (MB).
If the retention period is 0 and there are metrics stored locally, the SDK will attempt to push them one more time before removing them permanently.

`push_period`

push_period determines how often metrics are pushed to the server.
If an invalid value is passed (0 or greater than 3,600 seconds), it will be set to 30 (seconds).

Parameters

// sense.hpp

...
struct Parameters {
  AudioFormat audio_format = AudioFormat::AF_FLOAT32;
  ModelDelegate model_delegate = ModelDelegate::Default;
  Service service = Service::Emergency;
  int num_threads = 0;
  Metrics metrics;
...

Possible `Service` options

// sense.hpp
enum Service {
  Emergency = 0,
  Human_Interaction = 1,
  Human_Status = 2,
  Home_Context = 3
};

Possible `AudioFormat` options

// sense.hpp
enum AudioFormat {
  AF_UINT8,
  AF_INT8,
  AF_INT16,
  AF_INT24,
  AF_INT32,
  AF_DOUBLE,
  AF_FLOAT32,
  AF_UNKNOWN
};

Possible `Metrics` options

// sense.hpp
struct Metrics {
  size_t retention_period = 0;   // range, 1 to 31 days
  size_t free_disk_space = 100;  // range, 0 to 1,000,000 MB
  size_t push_period = 30;       // range, 1 to 3,600 seconds
};

4. Build

sense-file
sense-stream

Set your Project Key into the sample code. Note that it’s important to invoke the sense::Terminate() function at the end if sense::Init() was successful. Otherwise, an undefined behavior may occur while cleaning up the memory allocated by the sense during sense::init().

// examples/sense-file.cc

...
if (sense::Init("YOUR_PROJECT_KEY_HERE", sense_params) < 0) {
  return -1;
}
...
sense::Terminate();
...

Then it’s ready to build sense-file.

$ g++ -fopenmp examples/sense-file.cc -I./sense/include/ -lsense-core -L./sense/lib -o sense-file -lm -std=c++11 -ldl -lstdc++ -Wl,-rpath -Wl,./sense/lib

// examples/sense-stream.cc

...
if (sense::Init("YOUR_PROJECT_KEY_HERE", sense_params) < 0) {
  return -1;
}
...
sense::Terminate();
...

Then it’s ready to build sense-stream.

$ g++ -fopenmp examples/sense-stream.cc -I./sense/include/ -lsense-core -L./sense/lib -o sense-stream -lm -std=c++11 -ldl -lstdc++ -lpulse -lpulse-simple -Wl,-rpath -Wl,./sense/lib

5. Run

sense-file
sense-stream

$ LD_LIBRARY_PATH=. ./sense-file <PATH_TO_AUDIO_FILE>

Sample audio files in the SDK directory can be used as well.

$ LD_LIBRARY_PATH=. ./sense-file audio_files/babycry.wav
$ LD_LIBRARY_PATH=. ./sense-file audio_files/carhorn.wav
$ LD_LIBRARY_PATH=. ./sense-file audio_files/cough.wav
$ LD_LIBRARY_PATH=. ./sense-file audio_files/dogbark.wav
$ LD_LIBRARY_PATH=. ./sense-file audio_files/glassbreak.wav
$ LD_LIBRARY_PATH=. ./sense-file audio_files/siren.wav
$ LD_LIBRARY_PATH=. ./sense-file audio_files/snoring.wav
$ LD_LIBRARY_PATH=. ./sense-file audio_files/toilet_flush.wav
$ LD_LIBRARY_PATH=. ./sense-file audio_files/whistle.wav

Make sure the input device is properly connected before you run sense-stream

$ LD_LIBRARY_PATH=. ./sense-stream

(Note) `sense-stream` troubleshooting

Try the steps below when you have a trouble running sense-stream

(1) Check connection

Check if the input device is connected

(2) Check power

The board may lack of power when an external device is connected
Check if the board gets stable and enough power

(3) Check `pulseaudio`

# start pulseaudio
$ pulseaudio -D

# check your device index from the pulseaudio command result
$ pacmd list-sources | grep -e 'index:' -e device.string -e 'name:'

# set default source with the index above
$ pacmd set-default-source <DEVICE_INDEX>

(Note) PulseAudio resampling

If Cochl.Sense SDK runs in stream mode and the input stream(e.g. microphone) uses an audio format different from what SDK uses, PulseAudio automatically converts sample format, sample rate, and channel map to be comaptible with Cochl.Sense SDK’s format. In this case, PulseAudio uses a resampler. For information on setting the resampler, please refer to this article.

(Note) CPU and memory usage in Cochl.Sense stream mode

Test environment specification:

Raspberry Pi 3 Model B with TensorFlow Lite
- OS(Raspberry Pi OS)
  - Release date: April 4th 2022
  - System: 64-bit
  - Kernel version: 5.15
  - Debian version: 11 (bullseye)
- Quad Core 1.2GHz Broadcom BCM2837 64bit CPU(ARM Cortex A53)
- 1GB RAM
- BCM43438 wireless LAN
- Storage(SanDisk 16GB Ultra Micro SD HC Class 10)
- Parameters:
```
sense::Parameters params;
params.audio_format = sense::AF_FLOAT32;
params.model_delegate = ModelDelegate::Default;
params.service = Service::Human_Interaction;
params.metrics.retention_period = 0;  // days
params.metrics.free_disk_space = 100;  // MB
```
Result:
- In idle, CPU usage: 2.2526 (%), memory usage: 229,578 (kB)
- CPU usage (%):

Memory usage (kB):