audio-works

WIP

Library meant for Electron (ver. 11.x - 13.x) to determine note based on the signal received from mic or file.

Currently, tested determining sounds down to C1 ~ 32.7Hz (Less than 2Hz difference between C1 and B0) so at the moment it's accurate enough down to at least 2Hz differences.

• Added volume measuring
• Still a lot a little of garbage left in methods waiting for removal
• Added possibility to change input audio device (+ automatically changes when current device gets disconnected)
• Still a lot a little bit Almost none of garbage left in methods waiting for removal
• Added possibility to change input audio device (+ automatically updates list of devices on change / when current device gets disconnected)
• Separated most micSetup and Renderer methods into modules
• Changed objects into classes
• Fixed bug with enabling mic after disabling it
• Fixed bug with repeatedly changing input device while the mic is enabled resulting in problems with audioContext
• Untangled logic, so it's a bit more simple and less convoluted now imo I was so wrong
• Added A-, B- and C-weighting classes
• Changed audio volume measurement using weighting classes
• Added methods returning nyquist frequency and band range of current audioHandler setup
• Added possibility to change output device

TODO right now:

• [x] Add methods to frequencyMath
• [x] Untangle deviceHandler and other redundant methods etc.
• [ ] Adding possibility to automatically switch to default available device if currently used one gets disconnected
• [x] General code refactor
• [x] Output audio (the latency is/will be +- 1 second so not great, but it's Node + Chromium ¯\_(ツ)_/¯
• [x] Changes in soundStorage module for storing and determining frequencies (in progress)
• [ ] Anything else that will pop up later

ChangeLog:

• v0.6.8
• v0.6.7
• v0.6.6
• v0.6.5
• v0.6.4
• v0.6.3
• v0.6.2

Classes:

• AudioSetup
• AudioHandler
• AudioFileHandler
• Correlation
• DeviceHandler
• Device
• SoundStorage
• SoundStorageEvent
• FrequencyMath
• AudioEvents

Test coverage

• coverage

Setup, sample initialization

Electron's browser window should've contextIsolation set to false as well as nodeIntegration set to true.

window = new browserWindow({
  webPreferences: {
    contextIsolation: false,
    nodeIntegration: true,
  },
});

Then in rendering process sample initialization logging value of a correlated buffer from the default device could look like this:

const { AudioHandler, AudioEvents } = require("audio-works");

let mic = new AudioHandler();

mic.on(AudioEvents.audioProcessUpdate, (evt) => {
  console.log(evt.correlate());
}); // Event called from ScriptProcessor on new buffer chunk

await mic.setupStream(); // Start the mediaStreamSource

setTimeout(1000, () => {
  mic.end();
}); // close the stream after 1 second

It is also possible to output received signal by creating the html Audio object and setting it's srcObject property to the stream hold by the AudioHandler.

const { AudioHandler, AudioEvents } = require("audio-works");

let mic = new AudioHandler();
let audio = new Audio();

mic.on(AudioEvents.setupDone, (evt) => {
  audio.srcObject = evt.stream;
}); // Event emitted after setupStream()

await mic.setupStream(); // setupStream() is asynchronus but all the following
// actions can be done on emission of the "SetupDone" event
// so that await in such a case could be omitted

To change the device it's enough to pass an id of said device to the changeInput methods of the AudioHandler. List of devices can be accessed through DeviceHandler hold by the AudioHandler as deviceHandler property.

const { AudioHandler, Device } = require("audio-works");

let mic = new AudioHandler();

// if for some reason the device list is empty, which shouldn't really happen,
// then it can be easily fixed by calling `await mic.deviceHandler.updateDeviceList();`

// Retrieves a list of available devices
let inputs = mic.getDeviceList(Device.direction.input);
// Change default ('first available') input to the third one
mic.changeInput(inputs[2].id);

await mic.setupStream();

Classes

AudioSetup

File in GitHub
Main class responsible for setting up AudioHandler and AudioFileHandler holding two main obligatory nodes used by AudioContext which are AnalyserNode and GainNode. This class extends EventEmitter as after various steps instance dispatches related to them events.

AudioHandler

File in GitHub
Extends AudioSetup as AudioContext is crucial for all the functionalities provided by this class. Handles live audio inputs like microphones or instruments connected to audio interfaces as well as output to any available devices.

Sample object initialization

const { AudioHandler, Gain, Analyser } = require("audio-works");

let mic = new AudioHandler({ // All the values are optional.
    general: {               // Omitting some values in objects
        buflen: 8192,        // containing more properties will result
        curveAlgorithm: 'A'  // in assigment of default value
    },                       // only to the missing properties
    gainNode: new Gain({value: 1.5}),
    analyserNode: new Analyser({
        smoothingTimeConstant: 0.9,
        fftSize: 32768,
        minDecibels: -90,
        maxDecibels: -10
    }),
    correlation: {
        rmsThreshold: 0.01,
        correlationThreshold: 0.01,
        correlationDegree: 0.98
    }
});

Sample usage of getWeightedVolume

const vol = mic.getWeightedVolume(2); 
let volume = (vol / 200) * (vol / 2); // Let's take everything over 200dB as maximally "loud"
                                      // (alternatively can be written as vol^2 / 400)
volume = volume < 100 ? volume : 100; 

AudioFileHandler

File in GitHub
Extends AudioHandler class therefore retains possibility to handle live audio input but adds methods meant for audio file decoding, creating standard BufferSources with primary goal of audio output, or obtaining pulse-code modulation data.

Example of logging correlated data and playing the audio from a file:

const { AudioFileHandler, AudioEvents } = require("audio-works");

const fileHandler = new AudioFileHandler({}, "./audioFiles/sample.wav");
await fileHandler.initCorrelation(); // this call is needed as we don't
// call setupStream() method

// -- Event driven approach --
fileHandler.on(AudioEvents.processedFileChunk, (evt) => {
  // perform() is called directly on the correlation object stored
  // in fileHandler, unlike calling "correlate()" in AudioHandler,
  // as there's no mediaStream stored in the "stream" property,
  // therefore it requires to manually push the data chunk passed
  // to the listener in evt data to be correlated.
  console.log(fileHandler.correlation.perform(evt));
});

const audioSource = await fileHandler.createSource();
audioSource.start(0);

fileHandler.processEvent(); // start processing

// -- Callback approach --
const audioSource = await fileHandler.createSource();
audioSource.start(0);

fileHandler.processCallback((data) => {
  console.log(fileHandler.correlation.perform(data));
});
// While using callback processing starts immediately so there's
// no call like "processEvent()" in this case

Correlation

File in GitHub
Sole purpose of this class is performing autocorrelation on audio buffer, allowing a set-up of custom thresholds. The output of perform method is supposed to be a frequency of the sound (the fundamental frequency). This means it processes the signal in monophonic context.

DeviceHandler

File in GitHub
Main purpose of this class is interaction with navigator.mediaDevices and for that reason it uses a private helper class Device. Device class instances returned from methods of this class are only the copies of actual stored objects to keep the data stored by the instance consistent regardless of user actions on obtained device data.

Device

File in GitHub
A class representing navigators mediaDevices. It has no methods, holding only
values: id: device id, label: device label, and dir: device direction
Array of instances of this class is returned from the getDeviceList method of DeviceHandler.
Along the device direction there are also two boolean flags related to it: isOutput and isInput for more convenient array checks and filtering.
Class contains copy method for more convenient deep copies.
For more convenient direction description instead of raw strings class contains a static object serving as enum which can be accessed as Device.direction.(input|output).
For more convenient device type description instead of raw strings class contains a static object serving as enum which can be accessed as Device.type.(audio|video), nonetheless only "audio" option is used/checked in the whole implementation.

SoundStorage

File in GitHub
Class supposed to serve as a storage for outputs of the Correlation class holding methods helping correct sound frequency estimations in short periods of time.

SoundStorageEvent

File in GitHub
This class has the same purpose as SoundStorage extending it with a difference of utilizing EventEmitter allowing more diverse interactions with the storage.

FrequencyMath

File in GitHub
Class responsible for frequency calculations, as well as translating those to musical notation. It operates based on frequency in Hz or a distance of a note from sound A4. Class performs calculations in a context of equal tempered scale. It holds values about specific sound by holding data in members:

sound: string

Sound symbol of the tone [C - B] with only sharp notes in case of a half tone.

octave: int

Octave of the tone

flatNote: Optional[string]

If sound can be represented as flat note than this member hold a string of it.

flatOctave: Optional[int]

If sound has a flat note representation that has different octave (only C/Bb) it holds octave of the flat note.

initialFrequency: double

Frequency used to initialize class instance. In case of static constructor usage the frequency hold by this member is the perfect pitch of the sound passed to the constructor.

distance: [int]

Distance of given sound from A4.

AudioEvents

File in GitHub
Class serving as an enum for events emitted from components.
The sole purpose of it is to diminish required changes in case of changes in
event string values, as well as more transparent place to find all the events.
All the members are static so that no class initialization is required.
Members:

• audioContextStarted
• audioProcessUpdate
• processedFileChunk
• deviceChange
• setupDone
• streamEnd
• streamPause
• streamResume
• sampleLimit
• sampleTarget

Default setup values

File in GitHub
Fields needed to construct crucial objects:

• AudioSetup
• Correlation
• Gain node File in GitHub
• Analyser node File in GitHub
• Properties of AudioHandler and AudioFileHandler (buffer length: buflen, and curve algorithm: curveAlgorithm)

if not specified by user are loaded from this file.

Current test coverage

ChangeLog

v0.6.8

• SoundStorageEvent Uses base class empty method instead of custom implementation
• convertToArrayBuffer function from utils has default value maxSmallContainerSize set to 35000
• Small optimization of base class of A/B/C-Weight classes
• getMediaStream method of AudioHandler now accepts device ID for which it should be created
• setupStream method of AudioHandler now accepts device ID for which it should be created. If none is specified, as usual, the first available device will be used
• Added optional navigator to AudioHandler (mainly for more convenient testing) it defaults to window.navigator
• Added optional navigator to DeviceHandler (mainly for more convenient testing) it defaults to window.navigator

v0.6.7

• DeviceHandler caches devices obtained through navigator in "cachedDevices_" property
• DeviceHandler no longer sets input/output device to undefined if its ID is not found
• Methods of DeviceHandler returning Device lists and IDs now operate on copies of stored objects to ensure consistency of stored data regardless of modifications performed on returned objects
• Device class has new method "copy" that creates a new identical instance of the object
• Correlation now has the value "returnOnThreshold" set to true by default (default audio values)

v0.6.6

• ReadMe update

v0.6.5

• D-weighting algorithm has been removed as it was unreliable and unsupported by ISO
• FrequencyMath has additional static constructor symbolConstructor that can create a new instance of the class simply by a string like "C4" passed to it as an argument.
• getMediaStream method of AudioHandler now doesn't accept custom constraint as it was unreliable, although it should be back in future

v0.6.4

• No changes really ¯\_(ツ)_/¯

v0.6.3

• Shorter execution time of perform method of Correlation class
• Shorter execution time of getVolume method of AudioHandler class
• Slightly shorter execution time of getWeightedVolume method of AudioHandler class
• FTDFloat32(buflen: uint) method of AudioSetup class now takes this.buflen by default
• perform method of Correlation class now takes additional argument defaultCorrelationSampleStep that defaults to 1 (for smaller buffers < 8192) or 2 (for larger buffers >= 8192) to minimize latency. Larger buffers can still work the same way as smaller ones by explicitly passing value 1 as second argument to the method.

v0.6.2

• Added distance class member to FrequencyMath to limit recalculation of this value in other class methods
• Removed non static getDistanceFromNote method from FrequencyMath as "distance" is now a class member
• Renamed getVolume method of AudioHandler class to getWeightedVolume.
• Added getVolume method to AudioHandler class that counts average of ByteFrequencyData stored in Analyser node and casts it into a double in <0, 1) range
• getWeightedVolume returns Number instead of string