1. Check the Requirements

Follow our Getting started section to set up the environment.

2. Android Requirements

Cochl.Sense Edge SDK for Android supports Android API 26 (Version 8.0 ‘Oreo’), or later. Additionally, Android Studio is required, and the following packages need to be installed.

sudo apt update
sudo apt install -y git unzip

3. Prepare the Sample

(1) Clone the Tutorial

Clone the tutorial repository from here:

git clone https://github.com/cochlearai/sense-sdk-android-tutorials.git

(2) Unzip the SDK

# sense-file
unzip path/to/sdk/sense-sdk-<version>-android.zip \
  -d path/to/sample/sense-sdk-android-tutorials/sense-file/app

# sense-stream
unzip path/to/sdk/sense-sdk-<version>-android.zip \
  -d path/to/sample/sense-sdk-android-tutorials/sense-stream/app

(3) Android Studio Setup

(a) Edit app/build.gradle

dependencies {
    implementation files('libs/sense-sdk-v<version>.aar')  // Cochl.Sense Edge SDK
}

(b) Edit app/src/main/AndroidManifest.xml

<uses-permission android:name="android.permission.INTERNET"/>

<!-- sense-stream -->
<uses-permission android:name="android.permission.RECORD_AUDIO"/>

<!-- sense-file -->
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>

4. How to Use Cochl.Sense Edge SDK for Android

(1) Cochl.Sense Edge SDK Initialization

You can initialize the Cochl.Sense Edge SDK by calling init(projectKey, senseParams) in the ai.cochl.sensesdk.Sense package. Initialization may take some time if the process involves downloading a model. The Cochl.Sense Edge SDK will download a model if none is found on the device or if a newer version is available.

import ai.cochl.sensesdk.Sense;

public class MainActivity extends AppCompatActivity {
    private final String projectKey = "Write your project key here";
    private Sense sense = null;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        Sense sense = Sense.getInstance();
        
        Sense.Parameters senseParams = new Sense.Parameters();

        senseParams.metrics.retentionPeriod = 0;  // days
        senseParams.metrics.freeDiskSpace = 100;  // MB
        senseParams.metrics.pushPeriod = 30;      // seconds

        senseParams.deviceName = "Android device.";

        senseParams.logLevel = 0;

        sense.init(projectKey, senseParams);
    }
}

• Parameters

public static class Parameters {
    /// The name you want to set for this device in order to find it
    /// on your Dashboard.
    public String deviceName = "";

    public ModelDelegate modelDelegate = ModelDelegate.DEFAULT;

    /// Number of threads available to the tflite interpreter.
    /// NOTE: num_threads should be >= 0
    /// You may pass 0 to let sense select all the processors available.
    public int numThreads = 0;

    public Metrics metrics = new Metrics();

    /// Specify the desired log level for this device:
    /// 0: Debug
    /// 1: Information
    /// 2: Warning
    /// 3: Error
    /// The default log level is 1
    public int logLevel = 1;

    /// Features
    public SensitivityControl sensitivityControl = new SensitivityControl();
    public ResultAbbreviation resultAbbreviation = new ResultAbbreviation();

    /// Path to the directory which holds the
    /// shared libraries for the Hexagon NN libraries on the device.
    /// Must be set if you use the Hexagon Model delegate.
    public String hexagonSharedLibsFolderPath = "";
}

For more information about these parameters, refer to the parameter on C++.

• Possible ModelDelegate Options

enum ModelDelegate {
    DEFAULT(0), HEXAGON(1), NNAPI(2), GPU(3);

    public final int value;

    ModelDelegate(int value) {
        this.value = value;
    }
}

(2) Audio Input

Cochl.Sense Edge SDK receives audio data and returns a list of detected sound tags in JSON format. You can pass either the file path as a string or an array containing the audio data.

import java.io.File;

// Storage which contains the audio file
File sdcard;
if (android.os.Build.VERSION.SDK_INT < 29) {
    sdcard = Environment.getExternalStorageDirectory();
} else {  // android.os.Build.VERSION.SDK_INT >= 29
    sdcard = this.getExternalFilesDir(null);
}

File file = new File(sdcard, "some_audio_file.wav");
String filePath = file.getAbsolutePath();
JSONObject result = Sense.getInstance().predict(filePath);

import android.media.AudioFormat;
import android.media.AudioRecord;
import android.media.MediaRecorder;

private final int AUDIO_SOURCE = MediaRecorder.AudioSource.UNPROCESSED;
private final int SAMPLE_RATE = 22050;
private final int CHANNEL_CONFIG = AudioFormat.CHANNEL_IN_MONO;
// Supported audio formats: PCM_16BIT, PCM_FLOAT;
private final int AUDIO_FORMAT = AudioFormat.ENCODING_PCM_FLOAT;
private final int RECORD_BUF_SIZE = AudioRecord.getMinBufferSize(SAMPLE_RATE,
                                                                 CHANNEL_CONFIG,
                                                                 AUDIO_FORMAT);

AudioRecord recorder = new AudioRecord(AUDIO_SOURCE,
                                       SAMPLE_RATE,
                                       CHANNEL_CONFIG,
                                       AUDIO_FORMAT,
                                       RECORD_BUF_SIZE);
recorder.startRecording();

// If you have recorded audio to a 'buffer', pass it to the predict method.
JSONObject frameResult = Sense.getInstance().predict(buffer, SAMPLE_RATE);

5. Predict on Device

(1) NOTE

The sample rate of the input audio must be 22,050,Hz or higher. Audio with a sample rate lower than 22,050 Hz cannot be used. If the sample rate is higher than 22.050 Hz, the Cochl.Sense Edge SDK will downsample the audio internally.

private static final int SAMPLE_RATE = 22050;

(2) JSON result format

{
  "tags": [
    {
      "name"        : <string>, The name of the predicted tag (e.g. "Siren")
      "probability" : <float>,  Probability of the predicted tag
    }
  ],
  "start_time"      : <float>,  Starting time of this prediction window
  "end_time"        : <float>,  Ending time of this prediction window
  "prediction_time" : <double>, The amount of time it took to process this window
                                (in milliseconds)
}

private static final String TAG = "MyActivity";

File file = new File(sdcard, "some_audio_file.wav");
String filePath = file.getAbsolutePath();
JSONObject result = Sense.getInstance().predict(filePath);
Log.i(TAG, result.getJSONObject("result").toString(2));

private static final String TAG = "MyActivity";

private static final int SAMPLE_RATE = 22050;

JSONObject frameResult = Sense.getInstance().predict(buffer, SAMPLE_RATE);
Log.i(TAG, frameResult.toString(2));

6. Terminate

Cochl.Sense Edge SDK allocates several resources during initialization. To ensure these resources are released safely, call the terminate() method before exiting.

Sense.getInstance().terminate();

7. Reference

Sense

java.lang.Object
    ai.cochl.sensesdk.Sense;

The Cochl.Sense Edge SDK is available as a singleton class and operates with a project key. After successful initialization, you can add audio input to the Cochl.Sense Edge SDK for prediction.

public static Sense getInstance()

• Returns an instance of the Sense singleton class.

public void init(String projectKey, Parameters senseParams)

• Authenticates the user with the project key.
• Throws CochlException if the authentication fails.

public JSONObject predict(String filePath)

• Predicts the file located at the given path.
• Throws CochlException if the prediction fails.

public JSONObject predict(byte[] byteArr, int sampleRate)

• Predicts the audio data in the given array.
• Throws CochlException if the prediction fails.

public JSONObject predict(short[] shortArr, int sampleRate)

• Predicts the audio data in the given array.
• Throws CochlException if the prediction fails.

public JSONObject predict(float[] floatArr, int sampleRate)

• Predicts the audio data in the given array.
• Throws CochlException if the prediction fails.

public void terminate()

• Releases the resources allocated by the Cochl.Sense Edge SDK after successful initialization. Custom exceptions may be thrown by the Cochl.Sense Edge SDK.

public Parameters getParameters()

• Returns the parameters set during initialization.

public float getWindowSize()

• Returns the window size of the current model in use.
• Throws CochlException if used before initialization.

public float getHopSize()

• Returns the hop size of the current model in use.
• Throws CochlException if used before initialization.

(Deprecated) public static String getSdkVersion()

• Returns the version of the Cochl.Sense Edge SDK.

(Deprecated) public void addInput(AudioRecord audioRecord)

• Adds an AudioRecord object to perform prediction on an audio stream.

(Deprecated) public void addInput(File file)

• Adds a File object to perform prediction on an audio file.

(Deprecated) public void predict(Sense.OnPredictListener listener)

• Starts prediction with a callback function to be called after an audio frame is processed.

(Deprecated) public void pause()

• Pauses the prediction for the audio stream.

(Deprecated) public void resume()

• Resumes the prediction for the audio stream.

(Deprecated) public void stopPredict()

• Stops the prediction and allows you to add new audio input.

CochlException

java.lang.Object
    java.lang.Throwable
        java.lang.Exception
            java.lang.RuntimeException
                ai.cochl.sensesdk.CochlException

1. Check the Requirements

Follow our Getting started section to set up the environment.

2. C++ Requirements

Install the packages below to use our Cochl.Sense Edge SDK for C++.

sudo apt update
sudo apt install -y git unzip
sudo apt install -y libpulse-dev pulseaudio pulseaudio-utils

3. Prepare the Sample

(1) Clone the Tutorial

Clone the tutorial repository from here:

git clone https://github.com/cochlearai/sense-sdk-cpp-tutorials.git

(2) Unzip the SDK

unzip path/to/sdk/sense-sdk-<version>-cpp.zip -d path/to/sample/sense-sdk-cpp-tutorials/

After unzipping the zip file, your sample directory should now look like this.

└── sense-sdk-cpp-tutorials
    ├── audio_files          # audio samples
    ├── examples
    |   ├── sense-file.cc    # sense-file sample
    │   └── sense-stream.cc  # sense-stream sample
    └── sense                # Cochl.Sense Edge SDK
        ├── include
        ├── lib
        └── license

4. Install License Manager for SDK

Before installing the Cochl.Sense Edge SDK version 1.4.0, you should prepare the license manager. This manager is responsible for handling the Cochl.Sense Edge SDK license, whether the devices are connected to the network or not.

# folder tree in C++ SDK 1.4.0 for x86_64 Ubuntu Linux
└── sense-sdk-cpp-tutorials
    ├── audio_files          # audio samples
    ├── examples
    └── sense                # Cochl.Sense Edge SDK
          ├── include
          ├── lib
          └── license
                ├── bin
                ├── dinst
                ├── dunst
                ├── haspvlib_25011.so
                ├── haspvlib_arm64_25011.so
                ├── haspvlib_armhf_25011.so
                ├── haspvlib_x86_64_25011.so
                └── pkg

To install the license manager, run the following commands.

cd sense/license
sudo ./dinst

Uninstall license manager for SDK

(Warning) Uninstalling the license manager may affect other applications that rely on this service. If you want to proceed with the removal, please first check if your system uses the license manager. To uninstall the license manager, run the following commands.

# check the service
ps -aux | grep hasplmd

# remove the license manager
cd sense/license
sudo ./dunst

5. Parameters

There are several parameters available to configure the behavior of the Cochl.Sense Edge SDK.

struct Parameters {
  /// The name you want to set for this device in order to find it
  /// on your Dashboard.
  std::string device_name;

  ModelDelegate model_delegate = ModelDelegate::Default;
  /// Number of threads available to the tflite interpreter.
  /// NOTE: num_threads should be >= 0
  /// You may pass 0 to let sense select all the processors available.
  int num_threads = 0;

  Metrics metrics;

  /// Specify the desired log level for this device:
  /// 0: Debug
  /// 1: Information
  /// 2: Warning
  /// 3: Error
  /// The default log level is 1
  int log_level = 1;

  /// Features
  SensitivityControl sensitivity_control;
  ResultAbbreviation result_abbreviation;
};

Each time a frame (a unit of audio inference) is processed by the Edge SDK, a metric is generated. This metric represents the result of the frame’s inference and is intended to be sent to the server, where it can be viewed on your project dashboard. Only one device_name can be assigned per device. If you need to change the device_name you are currently using, it can be updated on the Edge SDK tab of the Project page of the dashboard.

Possible Metrics options

struct Metrics {
  /// Retention period decides how many days to keep the metrics locally.
  /// If retention period is 0, metrics aren't stored locally.
  size_t retention_period = 0;

  /// When the remaining disk space is less then `free_disk_space` (MB),
  /// metrics are not stored locally. And in this situation, even if the
  /// retention period is set, retention period is considered 0.
  size_t free_disk_space = 100;

  /// The stored metrics will be pushed to the server every `push_period`
  /// (seconds).
  size_t push_period = 30;
};

retention_period

• The Cochl.Sense Edge SDK attempts to push metrics to our servers immediately after they are created. If the push fails, the SDK stores the data locally and retries later.
• retention_period parameter determines the number of days the Cochl.Sense Edge SDK will keep the metrics data locally.
• The default value is 0, meaning the data is not saved locally.
• If an invalid value is provided (e.g., greater than 31 days), it will default to 31 days.
• If a user sets this period, any metrics older than the retention period will be removed immediately.
• This setting is project-specific; for example, project X can have a retention period of 7 days, while project Y can have a retention period of 14 days.

free_disk_space

• When the available disk space falls below the free_disk_space threshold (measured in MB), the Cochl.Sense Edge SDK will stop storing metrics locally. In this situation, the user-defined retention period will be overridden and set to 0 days.
• If an invalid free_disk_space value is provided (e.g., greater than 100 MB), it will default to 100 MB.
• If the retention period is set to 0 and there are metrics still stored locally, the Cochl.Sense Edge SDK will attempt to push them one last time before permanently deleting them.

push_period

• push_period determines how often metrics are pushed to the server.
• If an invalid value is provided (0 o r greater than 3,600 seconds), it will default to 30 seconds.

6. Build

if (sense::Init("Your project key",
                sense_params) < 0) {
  return -1;
}

sense::Terminate();

g++ examples/sense-file.cc \
  -o sense-file \
  -I./sense/include/ \
  -L./sense/lib -lsense-core \
  -Wl,-rpath -Wl,./sense/lib \
  -lm -ldl -lstdc++ \
  -fopenmp \
  -std=c++14

if (sense::Init("Your project key",
                sense_params) < 0) {
  return -1;
}

sense::Terminate();

g++ examples/sense-stream.cc \
  -o sense-stream \
  -I./sense/include/ \
  -L./sense/lib -lsense-core \
  -Wl,-rpath -Wl,./sense/lib \
  -lm -ldl -lstdc++ -lpulse -lpulse-simple \
  -fopenmp \
  -std=c++14

7. Run

LD_LIBRARY_PATH=. ./sense-file <PATH_TO_AUDIO_FILE>

LD_LIBRARY_PATH=. ./sense-file audio_files/babycry.wav

LD_LIBRARY_PATH=. ./sense-stream

1. Check the Requirements

Follow our Getting started section to set up the environment.

(1) Supported Python Versions

Cochl.Sense Edge SDK for Python currently supports.

• Python3.6
• Python3.7
• Python3.8

If you wish to use a different version, please contact support@cochl.ai.

2. Python Requirements

Install the packages below to use our Cochl.Sense Edge SDK for Python.

sudo apt update
sudo apt install -y git unzip curl
sudo apt install -y virtualenv portaudio19-dev python3-pyaudio

Install additional packages if you installed Python using a package manager instead of building if from source.

# for python3.6
sudo apt install python3.6-dev

# for python3.7
sudo apt install python3.7-dev

# for python3.8
sudo apt install python3.8-dev

3. Python Virtual Environment

Create a new virtual environment by selecting a Python interpreter and creating a ./venv directory to hold it.

virtualenv -p python3.6 --no-site-packages ./venv

# NOTE
# - If there's an error with `--no-site-packages`, then remove it and try again with 
virtualenv -p python3.6 ./venv

virtualenv -p python3.7 --no-site-packages ./venv

# NOTE
# - If there's an error with `--no-site-packages`, then remove it and try again with 
virtualenv -p python3.7 ./venv

virtualenv -p python3.8 --no-site-packages ./venv

# NOTE
# - If there's an error with `--no-site-packages`, then remove it and try again with 
virtualenv -p python3.8 ./venv

Activate the virtual environment using the appropriate shell-specific command.

source ./venv/bin/activate

When the virtualenv is active, your shell prompt will be prefixed with (venv). Installing packages within this virtual environment will not affect the host system’s setup.

Let’s start by upgrading pip:

# (venv)
curl https://bootstrap.pypa.io/pip/3.6/get-pip.py -o get-pip.py
python3 get-pip.py
pip3 install --upgrade pip

# (venv)
curl https://bootstrap.pypa.io/pip/3.7/get-pip.py -o get-pip.py
python3 get-pip.py
pip3 install --upgrade pip

# (venv)
curl https://bootstrap.pypa.io/pip/get-pip.py -o get-pip.py
python3 get-pip.py
pip3 install --upgrade pip

4. Install License Manager for SDK

Before installing the Cochl.Sense Edge SDK version 1.4.0, you should prepare the license manager. This manager is responsible for handling the Cochl.Sense Edge SDK license, whether the devices are connected to the network or not.

# Folder tree in Python 3.8 SDK 1.4.0 for x86_64 Ubuntu Linux 
sense
  ├── sense-1.4-py38-none-linux_x86_64.whl
  └── license
        ├── bin
        ├── dinst
        ├── dunst
        ├── haspvlib_25011.so
        ├── haspvlib_arm64_25011.so
        ├── haspvlib_armhf_25011.so
        ├── haspvlib_x86_64_25011.so
        └── pkg

To install the license manager, run the following commands.

cd sense/license
sudo ./dinst

Uninstall license manager for SDK

(Warning) Uninstalling the license manager may affect other applications that rely on this service. If you want to proceed with the removal, please first check if your system uses the license manager. To uninstall the license manager, run the following commands.

# check the service
ps -aux | grep hasplmd

# remove the license manager
cd sense/license
sudo ./dunst

5. Launch Examples

(1) Install Some Python Packages

# (venv)
pip3 install numpy==1.21.4 || pip3 install numpy==1.19.4 && pip3 install pyaudio

(2) Install the Cochl.Sense Edge SDK

You can now install the Cochl.Sense wheel file that you have downloaded.

# (venv)
unzip path/to/sdk/sense-<version>-<python_version>-<platform>.zip -d /path/to/sdk
pip3 install path/to/sdk/sense-1.x-xxx-xxx.whl

(3) Retrieve the Samples

git clone https://github.com/cochlearai/sense-sdk-python-tutorials

(4) Parameters

You can refer to the c++ sample page for more information about those parameters.

sense_params = Parameters()

# if <= 0. will use all the threads available on the machine
sense_params.num_threads = -1

# Metrics
sense_params.metrics.retention_period = 0   # range, 1 to 31 days
sense_params.metrics.free_disk_space = 100  # range, 0 to 1,000,000 MB
sense_params.metrics.push_period = 30       # range, 1 to 3,600 seconds
sense_params.log_level = 0

sense_params.device_name = "Testing device"

sense_params.sensitivity_control.enable = True
sense_params.result_abbreviation.enable = True
...

if SenseInit("Your project key",
             sense_params) < 0:

SenseTerminate()
...

sense_params = Parameters()

# if <= 0. will use all the threads available on the machine
sense_params.num_threads = -1

# Metrics
sense_params.metrics.retention_period = 0   # range, 1 to 31 days
sense_params.metrics.free_disk_space = 100  # range, 0 to 1,000,000 MB
sense_params.metrics.push_period = 30       # range, 1 to 3,600 seconds
sense_params.log_level = 0

sense_params.device_name = "Testing device"

sense_params.sensitivity_control.enable = True
sense_params.result_abbreviation.enable = True
...

# audio_stream_example.py

...
if SenseInit("Your project key",
             sense_params) < 0:
...
SenseTerminate()
...

(5) Run

# (venv)
python3 audio_file_example.py <PATH_TO_AUDIO_FILE>

# (venv)
python3 audio_file_example.py audio_files/babycry.wav

# (venv)
python3 audio_stream_example.py

(6) Troubleshooting

If the process ends with an error message like Illegal instruction (core dumped), try using a different version of numpy.

# (venv)
pip3 uninstall numpy
pip3 install numpy==1.19.4