Audio input, Audio optimisation

README.md

@@ -1,7 +1,9 @@
 # FastCarPlay
-This is C++ implementation of carplay receiver for "Autobox" dongles. 
+This is C++ implementation of Carplay receiver and Android Auto receiver for "Autobox" dongles. 
 The purpose of the project is to make application lightweight to run on Raspberry PI Zero 2W using hardware decoding.
 
+![Logo](docs/images/screenshot.png)
+
 ## Dongles
 The dongles are readily available from Amazon or Aliexpress labeled by Carlinkit. They also seems to have official web site https://www.carlinkit.com/.
 Devices might have different vendor and product id's. Check your with lsusb and update settings if necessary.
@@ -18,7 +20,7 @@ sudo apt install ffmpeg libsdl2-2.0-0 libsdl2-ttf-2.0-0 libusb-1.0-0 libssl3
 ```
 
 ### USB Permissions and device id
-On linux app may not have permisiions to read USB device. You need to create udev rule to grant persmissions for dongle.
+On linux app may not have permissions to read USB device. You need to create udev rule to grant permissions for dongle.
 First you need to figure out your idVendor and idProduct. 
 ```
 lsusb
@@ -52,7 +54,8 @@ cp ./settings.txt /conf
 ### Customisation
 You can change font and background images by replacing files in ./src/resource
 - background.bmp for background image. Use BMP format only.
-- font.ttf for font. Use TTF fdrmat only
+- font.ttf for font. Use TTF format only
+- you can edit colours.h to modify text colours in RGBA format
 
 The names of the file need to be exactly same. Resources are embedded in executable, remake the project to regenerate resources
 ```
@@ -67,55 +70,56 @@ The following keys have been mapped:
 - Enter - select active item
 - Backspace - Go back
 - f - toggle fullscreen mode
+- q - exit
+- r - force refresh
 
 ## Status
 What is working:
+- Carplay and Android Auto
 - Video
 - Audio (multiple channels)
 - Key navigation
 - Simple touch
+- Microphone (calls, siri)
 
 What is not working:
-- Multi touch - i have no means to test it
-- Microphone - that's next step for me to figure out how to feed sound
-- Telephone - the listening part will work, but because there is no mic implementation you can't speak
+- Multi touch - i have no means to test it and no idea how it should work
+- Some android keys are not mapped
+- No methods to edit autoconnect list or switch wireless devices
 
 ### Notes
-Regardless the resolution there are 2 types of settings
-- source-width source-height source-fps - defines what video parameters will be requested from device
-- width height fps - defines video drawing resolution and fps
-
-The SDL will anyway scale the image to your screen/window size using internal HW scaling, so ideally you should use same values for width and height.
-If the source parameters will be different the scaling will be applied, however that scaling is not hardware accelerated and can consume a lot of CPU.
-You can set the 'scaler' in setting to define scaling algorithm. On my Raspbery Pi Zero 2W even easiest algorithm loads CPU to 100% and cause fram drop.
-
-Increasing FPS above Source-FPS will cause app to run UI loop with less delays and do more event polling. This can increase responsivenes of the system, but also will make X11 to use more resources.
+- Do not try to run debug builds if you do not need to debug application. They consume a lot of memory for address sanitising thats grow over time.
+- Increasing FPS above Source-FPS will cause app to run UI loop with less delays and do more event polling. This can increase responsivenes of the system, but also will make X11 to use more resources.
+- For multichannel audion (driving guidance over music) you need to have multichannel driver in the system (puslsaudio, pipewire). If you only have ALSA backend the second channel will not work
+- Android has it's own video resolution system which is fixed for 480p 720p 1080p. So set up resolution that is closest to what you want and adjust DPI for UI scale. If you do not have full screen Android Auto, you might need to enable resolution negotiation. Go to Android Auto app info and search for "Additional settings in the app" option. Scrol down and tap fast 5 times on version. Now you can use top right three dots menu to go to "Developer settings". Tap "Video Resolution" and select "Allow to car and phone to negotiate".
 
 ### Progress and plans
 Done
-- ✓ Implement direct buffer transfer from video decoder to renderer (should reduce amount of memory copies and CPU load)
-- ✓ Control audio buffers better (now system use 3 decoding threads but in reality only 2 required)
-- ✓ Reduce music volume when there is navigation messages
-- ✓ Add abilities to run script on device connect and device disconnect
-- ✓ Add encrypted USB communication option with magic code 0x55bb55bb for new firmware
-- ✓ Improve touch responsiveness
-- ✓ Protocol debugging option + python script to decode usb dumps
-- ✓ Android Auto (have not tested much, but seems to work for me)
+- [x] Implement direct buffer transfer from video decoder to renderer (should reduce amount of memory copies and CPU load)
+- [x] Control audio buffers better (now system use 3 decoding threads but in reality only 2 required)
+- [x] Reduce music volume when there is navigation messages
+- [x] Add abilities to run script on device connect and device disconnect
+- [x] Add encrypted USB communication option with magic code 0x55bb55bb for new firmware
+- [x] Improve touch responsiveness
+- [x] Protocol debugging option + python script to decode usb dumps
+- [x] Android Auto (have not tested much, but seems to work for me)
+- [x] Microphone support (Calls, Siri)
 
 Next 
-- Add microphone support (Calls, Siri)
-- CAR menu with status, settings and options to run custom scripts
-
+- [ ] Car menu with status, settings and options to run custom scripts
+- [ ] Canbus script communication or some sort of side key press handling
+- [ ] Better android navigation
+- [ ] Switch between wireless devices
 
 ## Acknowledgement
 The project is inspired and based on great work done by other developers:
-- ![pycarplay by electric-monk](https://github.com/electric-monk/pycarplay)
-- ![carplay-receiver by harrylepotter](https://github.com/harrylepotter/carplay-receiver)
-- ![react-carplay by rhysmorgan134](https://github.com/rhysmorgan134/react-carplay)
-- ![carplay-client by rayphee](https://github.com/rayphee/carplay-client)
+- [pycarplay by electric-monk](https://github.com/electric-monk/pycarplay)
+- [carplay-receiver by harrylepotter](https://github.com/harrylepotter/carplay-receiver)
+- [react-carplay by rhysmorgan134](https://github.com/rhysmorgan134/react-carplay)
+- [carplay-client by rayphee](https://github.com/rayphee/carplay-client)
 
 The project is licenced under GPL-3 licence. See LICENCE for details.
-The project is using Open Sans font (https://fonts.google.com/specimen/Open+Sans). See FONT_LICENCE for details.
+The project is using [Open Sans font](https://fonts.google.com/specimen/Open+Sans). See FONT_LICENCE for details.
 
 ## Finally
-If you have any questions, suggestions or you find problems running this feel free to open issue.
+If you have any questions, suggestions or you find problems running this feel free to open issue.

settings.txt

@@ -98,11 +98,12 @@
 
 # Size of video and audio buffers. Increase if you see artifacts
 #video-buffer-size = 32
-#audio-buffer-size = 16
+#audio-buffer-size = 32
 
-# Audio delay. Fill the buffer to this value before start playing. Increase if you hear audio artifacts.
-# Should be less than audio-buffer-size 
+# Audio delay for music and calls. Fill the buffer to this value before start playing. 
+# Increase if you hear audio artifacts or tearing. Should be less than audio-buffer-size 
 #audio-buffer-wait = 2
+#audio-buffer-wait-call = 8
 
 # Reduce volume of main audio source when there is higher priority audio stream.
 # This will reduce volume of the music if there is navigation instructions

src/connector.cpp

@@ -29,7 +29,10 @@ Connector::Connector(uint16_t videoPadding)
 
 Connector::~Connector()
 {
-    stop();
+    _active = false;
+
+    if (_write_thread.joinable())
+        _write_thread.join();    
 
     if (_cipher)
     {

src/helper/iaudio_sender.h

@@ -0,0 +1,12 @@
+#ifndef SRC_HELPER_IAUDIO_SENDER
+#define SRC_HELPER_IAUDIO_SENDER
+
+#include <cstdint>
+
+class IAudioSender {
+public:
+    virtual ~IAudioSender() = default; 
+    virtual void sendAudio(uint8_t* data, uint32_t length) = 0;
+};
+
+#endif /* SRC_HELPER_IAUDIO_SENDER */

src/helper/protocol_const.h

@@ -35,6 +35,9 @@
 #define CMD_VERSION 204
 #define CMD_ENCRYPTION 240
 
+#define AUDIO_BUFFER_SIZE 2560
+#define AUDIO_BUFFER_OFFSET 12
+
 struct ProtocolCmdEntry
 {
     int cmd;

src/interface.cpp

@@ -1,7 +1,7 @@
 #include "interface.h"
 #include "resource/background.h"
 #include "resource/font.h"
-#include "resource/colors.h"
+#include "resource/colours.h"
 #include "settings.h"
 #include "helper/protocol_const.h"
 #include <iostream>

src/main.cpp

@@ -23,7 +23,7 @@ extern "C"
 
 #define FRAME_DELAY_INACTIVE 200
 
-static const char *title = "Fast Car Play v0.4";
+static const char *title = "Fast Car Play v0.5";
 static SDL_Window *window = nullptr;
 static SDL_Renderer *renderer = nullptr;
 Uint32 evtStatus = (Uint32)-1;

src/pcm_audio.cpp

@@ -24,6 +24,8 @@ PcmAudio::PcmAudio(const char *name)
     _faded = false;
     _volume = 1.0;
     _fadeVolume = Settings::audioFade;
+    _config.channels = 0;
+    _config.rate = 0;
     if (_fadeVolume < 0)
         _fadeVolume = 0;
     if (_fadeVolume > 1)
@@ -64,17 +66,31 @@ void PcmAudio::callback(void *userdata, Uint8 *stream, int len)
     PcmAudio *self = static_cast<PcmAudio *>(userdata);
     const Message *segment = self->_data->peek();
     if (segment && self->_offset == 0 && getConfig(segment->getInt(OFFSET_AUDIO_FORMAT)) != self->_config)
-    {
         self->_reconfig = true;
-        self->_cv.notify_one();
+
+    bool underflow;
+    if (self->_data->has(self->_underflowSize))
+    {
+        underflow = false;
+        self->_underflowCount = 0;
     }
+    else
+    {
+        underflow = true;
+    }
+
     while (len > 0)
     {
-        if (segment == nullptr || self->_reconfig)
+        if (segment == nullptr || self->_reconfig || underflow)
         {
             std::fill_n(stream, len, 0);
             self->_faded = self->_fade;
             self->_volume = self->_faded ? Settings::audioFade : 1;
+            if (underflow && self->_underflowCount++ < 20)
+            {
+                self->_data->clear();
+                return;
+            }
             self->_reconfig = true;
             self->_cv.notify_one();
             return;
@@ -155,33 +171,40 @@ void PcmAudio::runner()
         if (!segment)
             continue;
 
-        _config = getConfig(segment->getInt(OFFSET_AUDIO_FORMAT));
-        if (device != 0)
+        ChannelConfig config = getConfig(segment->getInt(OFFSET_AUDIO_FORMAT));
+        if (_config != config)
         {
-            SDL_PauseAudioDevice(device, 1);
-            SDL_CloseAudioDevice(device);
-            device = 0;
+            if (device != 0)
+            {
+                SDL_PauseAudioDevice(device, 1);
+                SDL_CloseAudioDevice(device);
+                device = 0;
+            }
+
+            // Configure new spec
+            SDL_zero(spec);
+            spec.freq = config.rate;
+            spec.format = AUDIO_S16SYS;
+            spec.channels = config.channels;
+            spec.samples = 4096;
+            spec.callback = callback;
+            spec.userdata = this;
+
+            device = SDL_OpenAudioDevice(nullptr, 0, &spec, nullptr, 0);
+            if (device == 0)
+            {
+                std::cerr << _name << " Failed to open audio: " << SDL_GetError() << std::endl;
+                std::this_thread::sleep_for(std::chrono::milliseconds(100));
+                continue;
+            }
+            _config = config;
         }
 
-        // Configure new spec
-        SDL_zero(spec);
-        spec.freq = _config.rate;
-        spec.format = AUDIO_S16SYS;
-        spec.channels = _config.channels;
-        spec.samples = 4096;
-        spec.callback = callback;
-        spec.userdata = this;
-
-        _reconfig = false;
         _offset = 0;
+        _reconfig = false;
+        _underflowCount = 0;
+        _underflowSize = spec.channels == 1 ? Settings::audioDelayCall : Settings::audioDelay;
 
-        device = SDL_OpenAudioDevice(nullptr, 0, &spec, nullptr, 0);
-        if (device == 0)
-        {
-            std::cerr << _name << " Failed to open audio: " << SDL_GetError() << std::endl;
-            std::this_thread::sleep_for(std::chrono::milliseconds(100));
-            continue;
-        }
         SDL_PauseAudioDevice(device, 0);
         std::cout << _name << " Start playing " << _config.rate << "kHz " << (_config.channels == 2 ? "stereo" : "mono") << std::endl;
         if (_fader)

src/pcm_audio.h

@@ -58,6 +58,8 @@ private:
     bool _faded;
     float _volume;
     float _fadeVolume;
+    int _underflowCount;
+    int _underflowSize;
 
     std::thread _thread;
     std::mutex _mtx;

src/protocol.cpp

@@ -11,11 +11,11 @@ Protocol::Protocol(uint16_t width, uint16_t height, uint16_t fps, uint16_t paddi
       videoData(Settings::videoQueue),
       audioStreamMain(Settings::audioQueue),
       audioStreamAux(Settings::audioQueue),
+      _recorder(Settings::audioQueue),
       _width(width),
       _height(height),
       _fps(fps),
       _phoneConnected(false)
-
 {
 }
 
@@ -163,6 +163,14 @@ void Protocol::sendMove(float dx, float dy)
     send(CMD_TOUCH, false, buf, 16);
 }
 
+void Protocol::sendAudio(uint8_t *data, uint32_t length)
+{
+    write_uint32_le(data, 5);
+    write_uint32_le(data + 4, 0);
+    write_uint32_le(data + 8, 3);
+    send(CMD_AUDIO_DATA, false, data, length);
+}
+
 void Protocol::sendFile(const char *filename, const uint8_t *data, uint32_t length)
 {
     // filename is assumed null‑terminated, so strlen + 1 to include the '\0'
@@ -248,6 +256,9 @@ void Protocol::onPhone(bool connected)
 
     std::cout << (connected ? "[Protocol] Phone connected" : "[Protocol] Phone disconnected") << std::endl;
 
+    if (!connected)
+        _recorder.stop();
+
     pushEvent(_evtPhoneId, connected ? 1 : 0);
 
     if (connected && Settings::onConnect.value.length() > 1)
@@ -257,11 +268,36 @@ void Protocol::onPhone(bool connected)
         execute(Settings::onDisconnect.value.c_str());
 }
 
+void Protocol::onControl(int cmd)
+{
+    switch (cmd)
+    {
+    case 1:
+        _recorder.start(this);
+        break;
+
+    case 2:
+        _recorder.stop();
+        break;
+    }
+}
+
 void Protocol::onData(uint32_t cmd, uint32_t length, uint8_t *data)
 {
     bool dispose = true;
     switch (cmd)
     {
+
+    case CMD_CONTROL:
+        if (length == 4)
+        {
+            int cmd = 0;
+            memcpy(&cmd, data, sizeof(int));
+            onControl(cmd);
+        }
+
+        break;
+
     case CMD_PLUGGED:
         onPhone(true);
         break;

src/protocol.h

@@ -3,11 +3,12 @@
 
 #include "struct/atomic_queue.h"
 #include "struct/message.h"
-
+#include "helper/iaudio_sender.h"
 #include "settings.h"
 #include "connector.h"
+#include "recorder.h"
 
-class Protocol : private Connector
+class Protocol : private Connector, public IAudioSender
 {
 
 public:
@@ -26,6 +27,7 @@ public:
     void sendFile(const char *filename, int value);
     void sendClick(float x, float y, bool down);
     void sendMove(float dx, float dy);
+    void sendAudio(uint8_t *data, uint32_t length) override;
 
     AtomicQueue<Message> videoData;
     AtomicQueue<Message> audioStreamMain;
@@ -40,15 +42,17 @@ private:
 
     void onStatus(uint8_t status) override;
     void onDevice(bool connected) override;
+    void onControl(int cmd);
     void onData(uint32_t cmd, uint32_t length, uint8_t *data) override;
     void onPhone(bool connected);
 
-    static const char *cmdString(int cmd);    
+    static const char *cmdString(int cmd);
 
+    Recorder _recorder;
     uint16_t _width;
     uint16_t _height;
     uint16_t _fps;
-    bool _phoneConnected;    
+    bool _phoneConnected;
 
     uint32_t _evtStatusId = (uint32_t)-1;
     uint32_t _evtPhoneId = (uint32_t)-1;

src/recorder.cpp

@@ -0,0 +1,89 @@
+#include "recorder.h"
+
+#include <iostream>
+#include <cstring>
+
+#include "helper/protocol_const.h"
+#include "helper/functions.h"
+#include "settings.h"
+
+
+Recorder::Recorder(uint16_t buffSize)
+    : _sender(nullptr), _active(false), _device(0), _data(buffSize)
+{
+}
+
+Recorder::~Recorder()
+{
+    stop();
+    if (_thread.joinable())
+        _thread.join();
+}
+
+void Recorder::start(IAudioSender *sender)
+{
+    if (_active)
+        return;
+
+    if (_thread.joinable())
+        _thread.join();
+
+    _sender = sender;
+    _active = true;
+    _thread = std::thread(&Recorder::runner, this);
+}
+
+void Recorder::stop()
+{
+    if (!_active)
+        return;
+    _active = false;
+    _data.notify();
+}
+
+void Recorder::AudioCallback(void *userdata, Uint8 *stream, int len)
+{
+    Recorder *self = static_cast<Recorder *>(userdata);
+    std::unique_ptr<AudioChunk> frame(new AudioChunk(AUDIO_BUFFER_OFFSET + len));
+    std::memcpy(frame.get()->data + AUDIO_BUFFER_OFFSET, stream, len);
+    self->_data.pushDiscard(std::move(frame));
+}
+
+void Recorder::runner()
+{
+    setThreadName("recorder");
+
+    SDL_AudioDeviceID device = 0;
+    SDL_AudioSpec spec;
+
+    SDL_zero(spec);
+    spec.freq = 16000;
+    spec.format = AUDIO_S16LSB;
+    spec.channels = 1;
+    spec.samples = AUDIO_BUFFER_SIZE / 2; // = 2560 bytes (1280 samples * 2 bytes)
+    spec.callback = AudioCallback;
+    spec.userdata = this;
+
+    device = SDL_OpenAudioDevice(nullptr, SDL_TRUE, &spec, nullptr, 0);
+    if (device == 0)
+    {
+        std::cerr << "[Recording] Failed to open audio: " << SDL_GetError() << std::endl;
+        _active = false;
+        return;
+    }
+
+    SDL_PauseAudioDevice(device, 0);
+
+    while (_active)
+    {
+        std::unique_ptr<AudioChunk> buffer = _data.pop();
+        if (buffer && _sender)
+            _sender->sendAudio(buffer.get()->data, buffer.get()->size);
+        else if (_active)
+            std::this_thread::sleep_for(std::chrono::milliseconds(5));
+
+    }
+
+    SDL_PauseAudioDevice(device, 1);
+    SDL_CloseAudioDevice(device);
+}

src/recorder.h

@@ -0,0 +1,53 @@
+#ifndef SRC_RECORDER
+#define SRC_RECORDER
+
+#include <thread>
+#include <atomic>
+
+#include <SDL2/SDL.h>
+
+#include "helper/iaudio_sender.h"
+#include "struct/atomic_queue.h"
+
+class AudioChunk
+{
+public:
+    AudioChunk(uint16_t size)
+        : data(new uint8_t[size]), size(size)
+    {
+    }
+
+    ~AudioChunk()
+    {
+        delete[] data;
+    }
+
+    // Deleted copy constructor/assignment
+    AudioChunk(const AudioChunk &) = delete;
+    AudioChunk &operator=(const AudioChunk &) = delete;
+
+    uint8_t *data;
+    size_t size;
+};
+
+class Recorder
+{
+public:
+    Recorder(uint16_t buffSize);
+    ~Recorder();
+
+    void start(IAudioSender *sender);
+    void stop();
+
+private:
+    static void AudioCallback(void *userdata, Uint8 *stream, int len);
+    void runner();
+
+    IAudioSender *_sender;
+    std::atomic<bool> _active;
+    std::thread _thread;
+    SDL_AudioDeviceID _device;
+    AtomicQueue<AudioChunk> _data;
+};
+
+#endif /* SRC_RECORDER */

src/settings.h

@@ -40,6 +40,7 @@ public:
     static inline Setting<int> videoQueue{"video-buffer-size", 32};
     static inline Setting<int> audioQueue{"audio-buffer-size", 16};
     static inline Setting<int> audioDelay{"audio-buffer-wait", 2};
+    static inline Setting<int> audioDelayCall{"audio-buffer-wait-call", 8};    
     static inline Setting<float> audioFade{"audio-fade", 0.3};
     static inline Setting<std::string> audioDriver{"audio-driver", ""};
     static inline Setting<std::string> onConnect{"on-connect-script", ""};

src/struct/atomic_queue.h

@@ -68,6 +68,11 @@ public:
         return item;
     }
 
+    bool has(uint8_t count)
+    {
+        return _count >= count;
+    }
+
     bool wait(atomic<bool> &waitFlag, uint8_t count = 0)
     {
         unique_lock<std::mutex> lock(_mtx);