Last time I worked on the shrimp was in late 2023. I reached a point where I could control all motors from my cell phone, while displaying the video feed coming from the onboard camera.
I left the project there while work got the best of my energy in anticipation of the Xmas break. Then came a round of layoffs, and even if my job was spared, I got overwhelmed with work and the perspective of building the frame in my garage by 10° late at night didn't bring me joy, so I started another project.
Fast forward to early 2025. I was laid off early january, and I have a few weeks of time off before starting my next gig. Great timing to get back on the shrimp with the accumulated knowledge from the past year.
Introducing Godot Shrimp Deck
Last time I checked, there was no ffmpeg plugin to pull a camera stream into godot. Then last year I heard about the Gozen video editor on the Godot subreddit and found out they had their own integration of ffmpeg. I was ecstatic to try it out (th integration)! After a year I eventually did, got an error setting the source URL to a stream, asked the author and got told it would require a whole new implementation. Ok too bad for ffmpeg.
Something else that came out recently, is linux support for camera in Godot. This is ideal, since I wanted to try out Godot for the a UI on the Steam Deck, but I wasn't sure how to pull a video feed into Godot (without writing my own integration), especially given how tight the video codecs restrictions are with that engine (TLDR: Ogg Theora or nothing). Now if Godot reads from the camera itself, I leave it to it to decide on the right format, as long as I can take a frame and shove it down the network for the client to pick and assign back to a texture.
flowchart LR
cam[Camera] -- frames --> server[Shrimp Server]
server -- frames as jpeg --> client[Shrimp UI]
3 caveats:
- the raspberry pi zero 1 sports a 32 bits processor, and I couldn't bother building Godot in 32 bits. Honestly I'm not even sure it's still supported / would be possible. Instead I got a pi zero 2, which runs a 64 bits OS. Compiling Godot for arm64 was easy enough.
- the Steam Deck doesn't come with an ethernet connection - I'll use an usbC-to-ethernet adapter.
- Raspberri ships a wealth of python libraries to interact with GPIO, but if I am to run a godot server, I'm left to my own device... or not. PiGPIO would certainly be useful for my use case here.
Godot camera support
A sweet rabbit hole. Camera support worked great on desktop. I could get a basic client/server setup going in a minute. Once pushed to the Pi, it was something else. Listing camera feeds on desktop looked like this:
$v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'YUYV' (YUYV 4:2:2)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
...
Size: Discrete 160x120
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
...
Size: Discrete 176x144
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
...
but on the Pi, somehow v4l2 (the lib used by godot to deal with cameras on linux) would show improbable resolutions like this:
$v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'YUYV' (YUYV 4:2:2)
Size: Stepwise 16x16 - 16376x16376 with step 1/1 <--- 16376x16376 is not
[1]: 'UYVY' (UYVY 4:2:2) a valid resolution
Size: Stepwise 16x16 - 16376x16376 with step 1/1
...
[4]: 'RGBP' (16-bit RGB 5-6-5)
Size: Stepwise 16x16 - 16376x16376 with step 1/1
[5]: 'RGBR' (16-bit RGB 5-6-5 BE)
Size: Stepwise 16x16 - 16376x16376 with step 1/1
[6]: 'RGBO' (16-bit A/XRGB 1-5-5-5)
Size: Stepwise 16x16 - 16376x16376 with step 1/1
...
[12]: 'GBRG' (8-bit Bayer GBGB/RGRG)
Size: Stepwise 16x16 - 16376x16376 with step 1/1
[13]: 'GRBG' (8-bit Bayer GRGR/BGBG)
Size: Stepwise 16x16 - 16376x16376 with step 1/1
...
Some googling lead me to libcamerify.
TLDR Pis' cameras used to be accessible through as a single device (eg /dev/video0)
and this is what v4l2 is expecting. Now Pi's have changed this to a new stack where
it takes both the /dev/ device, but also some setup through a media controller API. Or in the OP's words:
The deprecated legacy camera stack would expose processed formats such as YUV420, RGB (of various flavours), or having passed it through either H264 or MJPEG encoders. Bullseye has switched to libcamera where /dev/video0 presents the raw typically Bayer frames from the image sensor, and libcamera then passes those through the ISP to create YUV or RGB images. Also note that /dev/video0 is expecting all the subdevices to be configured through the Media Controller API, so just configuring the format is not enough. Again libcamera deals with all of this for you.
V4l2 is lost and doesnt get the camera props right. The suggested way is to use libcamera, but it's not the one
integrated in Godot and I'm not doing this. Fortunately that's where libcamerify comes into play:
libcamerify $my_app $args
will setup the env (~most likely LD_LIBRARY_PATH~ it's LD_PRELOAD) and exec into $my_app $args. Now this
would be great,
but
libcamerify crashes with a bad assert.
$libcamerify v4l2-ctl -d /dev/video0 --list-formats-ext
[12:34:43.635312148] [4965] ERROR IPAModule ipa_module.cpp:171 Symbol ipaModuleInfo not found
[12:34:43.635465011] [4965] ERROR IPAModule ipa_module.cpp:291 v4l2-compat.so: IPA module has no valid info
[12:34:43.635552771] [4965] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+50-83cb8101
[12:34:43.682828657] [4968] WARN CameraSensorProperties camera_sensor_properties.cpp:473 No static properties available for 'imx708_wide'
[12:34:43.682923813] [4968] WARN CameraSensorProperties camera_sensor_properties.cpp:475 Please consider updating the camera sensor properties database
[12:34:43.736671041] [4968] WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[12:34:43.740573451] [4968] WARN CameraSensor camera_sensor_legacy.cpp:501 'imx708_wide': No sensor delays found in static properties. Assuming unverified defaults.
[12:34:43.742337912] [4968] INFO RPI vc4.cpp:447 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media1 and ISP device /dev/media0
v4l2-ctl: ../include/libcamera/controls.h:189: T libcamera::ControlValue::get() const [with T = long int; typename std::enable_if<((! libcamera::details::is_span<U>::value) && (! std::is_same<std::__cxx11::basic_string<char>, typename std::remove_cv< <template-parameter-1-1> >::type>::value)), std::nullptr_t>::type <anonymous> = nullptr]: Assertion `type_ == details::control_type<std::remove_cv_t<T>>::value' failed.
Aborted (core dumped)
Looking into the code itself lands us in some c++ templated getter that isn't telling much so anyone not familiar with the codebase. Someone reported my issue last week though, a fix has been found so in the meantime it seems fine to hold onto previous versions of this stack:
sudo apt-get purge libpisp-common
wget https://archive.raspberrypi.com/debian/pool/main/libp/libpisp/libpisp-common_1.0.7-1_all.deb
wget https://archive.raspberrypi.com/debian/pool/main/libp/libpisp/libpisp1_1.0.7-1_arm64.deb
wget https://archive.raspberrypi.com/debian/pool/main/libc/libcamera/libcamera-ipa_0.3.2+rpt20241119-1_arm64.deb
wget https://archive.raspberrypi.com/debian/pool/main/libc/libcamera/libcamera0.3_0.3.2+rpt20241119-1_arm64.deb
wget https://archive.raspberrypi.com/debian/pool/main/libc/libcamera/libcamera-tools_0.3.2+rpt20241119-1_arm64.deb
wget https://archive.raspberrypi.com/debian/pool/main/libc/libcamera/libcamera-v4l2_0.3.2+rpt20241119-1_arm64.deb
sudo dpkg -i ./libpisp-common_1.0.7-1_all.deb ./libpisp1_1.0.7-1_arm64.deb ./libcamera-ipa_0.3.2+rpt20241119-1_arm64.deb ./libcamera0.3_0.3.2+rpt20241119-1_arm64.deb ./libcamera-tools_0.3.2+rpt20241119-1_arm64.deb ./libcamera-v4l2_0.3.2+rpt20241119-1_arm64.deb
sudo apt-mark hold libpisp-common
BTW I couldn't find where libcamerify is defined (the project maintains their own
git server where we can see the buggy release
happened 7 weeks ago, and the lack of UI
impedes the google-iness) but it can be found in libcamera-tools.
Now v4l2 can list devices as expected:
libcamerify v4l2-ctl -d /dev/video0 --list-formats-ext
[13:08:01.226721621] [5121] ERROR IPAModule ipa_module.cpp:171 Symbol ipaModuleInfo not found
[13:08:01.226910947] [5121] ERROR IPAModule ipa_module.cpp:291 v4l2-compat.so: IPA module has no valid info
[13:08:01.227003083] [5121] INFO Camera camera_manager.cpp:325 libcamera v0.3.2+99-1230f78d
[13:08:01.300568089] [5124] WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[13:08:01.306123937] [5124] INFO RPI vc4.cpp:447 Registered camera /base/soc/i2c0mux/i2c@1/imx708@1a to Unicam device /dev/media1 and ISP device /dev/media0
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'NV21' (Y/CrCb 4:2:0)
Size: Discrete 160x120
Size: Discrete 240x160
Size: Discrete 320x2400
...
[1]: 'YU12' (Planar YUV 4:2:0)
Size: Discrete 160x120
Size: Discrete 240x160
Size: Discrete 320x2400
...
[2]: 'NV12' (Y/CbCr 4:2:0)
Size: Discrete 160x120
Size: Discrete 240x160
Size: Discrete 320x240
...
They released a fix the day after I found out, so the pinning wasn't required after all.
I spent a few hours trying to understand why Godot would crash when pulling frames from the camera, but crashes we inconsistent, yet always related to memory allocations:
ERROR: Caller thread can't call this function in this node (/root). Use call_deferred() or call_thread_group() instead.
at: propagate_notification (scene/main/node.cpp:2523)
================================================================
handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.4.beta.custom_build (36d90c73a843afa2807a0b8dcbfbf52bdb8a759c)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
malloc(): unaligned fastbin chunk detected
Aborted (core dumped)
With malloc() line sometimes switching to one of those:
malloc(): corrupted top size
realloc(): invalid old size
malloc(): unaligned tcache chunk detected
I gave up on leveraging Godot's new camera support on arm64, and fell back on plan B: using libcamera itself.
Reading Libcamera-vid Video stream
Raspberri Pi has renamed libcamera-* apps as rpicam-* (no comment :vomit:). So we open a socket where a
400x300 video stream will be written as mjpeg, 24 frames a second:
rpicam-vid -n -t0 --inline --listen --width 400 --height 300 --framerate 24 --level 4.2 --codec mjpeg -o tcp://0.0.0.0:6001
And since Godot don't provide tools to read streams, decoding mjpeg is left as an exercise. Turns out it's fairly simple
since the stream is actually an sequence of jpeg images. A jpeg image itself
starts with a specific marker (0xffd8) and ends with another one (0xddf9). Reading a stream
consists of slicing it in chunks bounded by these markers, and feeding the latest-received chunk into a texture. Pretty
straightforward.
So we'll end up with this type of sequence:
sequenceDiagram
autonumber
participant client as SteamDeck Client
participant server as Rpi Server
participant libcamera as Libcamera-vid Process
server ->> server: listen for client socket
activate client
client ->> server: connect
client ->> server: request feed
activate server
server -->> libcamera: start
activate libcamera
server ->> client: libcamera feed url
deactivate server
client ->> libcamera: connect
libcamera ->> client: video feed
deactivate client
deactivate libcamera
Which got me really happy once it ran off the steamdeck!
Off to getting these GPIOs fired up!