AMB82-mini http jepg streaming though browser example and optimize suggestions

I’ve written a multi-client video-JPEG stream web server (a little one) for the AMB82-MINI. It works pretty well, but the video feed gets pretty jerky.

client.write(buf, len); where client is of type WiFiClient seems to randomly insert 1 to 4 second pauses in the stream where it should be writing JPEG frames.

I’ve found this by profiling my code, and traced it down to client.write(buf,len), but I don’t have visibility into this code to further diagnose.

I’m seeing errors of type:
[Driver]: skb_unavailable=6 in last 2 seconds
but they don’t appear closely correlated to these pauses. The errors occasionally show up after the pauses.

Can anyone tell me what would make the stream choke like this? All the frames before and after seem to be delivered rapidly and correctly.

Further investigation shows that these pauses only occur when what’s being written is the image from the camera (not the IMG_HEADER or the STREAM_BOUNDARY text items).

When these pauses occur, I’m sending sendChunk(client, (uint8_t*)img_addr, img_len); from the camera, where img_addr is the address of the frame to send, and img_len is the length of the frame. The paused frames are not notably long. The pauses don’t corrolate with the skb_unavailable errors.

void CameraView::sendChunk(WiFiClient& client, uint8_t* buf, uint32_t len) {
    uint8_t ch_buf[64] = {0};
    uint8_t ch_len = snprintf((char*)ch_buf, 64, "%lX\r\n", len);
    client.write(ch_buf, ch_len);
    client.write(buf, len);
    client.print("\r\n");
}

I’ve removed a lot of junk data from the serial stream below - the frames that go right, the endless "ERROR] get_available Accept connection failed"messages that come when the server is not set to blocking mode - and an example log of the long pauses is below.

prof_b is the time in milliseconds it took `client.write(buf, len); to execute, and

prof_c is the size of the JPEG frame in bytes.

typically (in “working” frames without an extra pause), prof_b takes 4 milliseconds, in rare cases up to 10. Unless it’s in an extra pause, shown below. The pauses can reach into the 4,000 milliseconds. During these extra pauses, there’s a noticeable hiccup in the video stream.

I’ve forced the AMB82-MINI to 5GHz WiFi (by filtering the MAC address off my 2.4GHz - verified by looking at the device connections in my router). Still seeing pauses upon frame write.

prof_b: 1299, prof_c: 39807
[Driver]: skb_unavailable=2 in last 2 seconds
[Driver]: skb_unavailable=24 in last 2 seconds
[Driver]: skb_unavailable=28 in last 2 seconds
[Driver]: skb_unavailable=29 in last 2 seconds
[Driver]: skb_unavailable=30 in last 2 seconds
[Driver]: skb_unavailable=24 in last 2 seconds
[Driver]: skb_unavailable=21 in last 2 seconds
[Driver]: skb_unavailable=9 in last 2 seconds
prof_b: 931, prof_c: 34075
[Driver]: skb_unavailable=21 in last 2 seconds
[Driver]: skb_unavailable=26 in last 2 seconds
[Driver]: skb_unavailable=23 in last 2 seconds
prof_b: 1104, prof_c: 39920
[Driver]: skb_unavailable=5 in last 2 seconds
[Driver]: skb_unavailable=3 in last 2 seconds
[Driver]: skb_unavailable=6 in last 2 seconds
prof_b: 1277, prof_c: 31087
[Driver]: skb_unavailable=1 in last 2 seconds
[Driver]: skb_unavailable=21 in last 2 seconds
[Driver]: skb_unavailable=17 in last 2 seconds
[Driver]: skb_unavailable=12 in last 2 seconds
[Driver]: skb_unavailable=12 in last 2 seconds
prof_b: 1122, prof_c: 31790
[Driver]: skb_unavailable=8 in last 2 seconds
[Driver]: skb_unavailable=14 in last 2 seconds
prof_b: 996, prof_c: 43989

Tried yesterday on a different network: a 5GHz phone hotspot. Same result: random pauses, potentially more than on the local wifi network.

The web service looks tied to the WiFi client - were there a way to use the OTG port as a wired bridge to network, I suppose I still couldn’t eliminate WiFi as a culprit. Am I thinking straight there?

Hi @jimmiedave ,

Can I check whether your video streams have random pauses when running ameba-arduino-pro2/Arduino_package/hardware/libraries/Multimedia/examples/CaptureJPEG/HTTPDisplayJPEGContinuous at dev · Ameba-AIoT/ameba-arduino-pro2 · GitHub or any examples in ameba-arduino-pro2/Arduino_package/hardware/libraries/Multimedia/examples/StreamRTSP at dev · Ameba-AIoT/ameba-arduino-pro2 · GitHub ?

WiFi could be one of the factors that is causing this issue. Consider reducing the resolution of the video if FHD is not needed.

May I check if you are using UDP?

I can try and re-flash the camera/processor with the demos in a few days, but it’s currently in regular use, dropped-frames and all.

I’m assuming the demos on github will be the same as the latest Arduino update for the AMB82-mini. If not, please let me know what difference I’d be able to expect.

I’m not intentionally using UDP - the code is pseudo-streaming motion JPEG over http, in a manner similar to the example HTTPDisplyJPEGContinuous, though not the same: my application can serve multiple http endpoints simultaneously (though I see these pauses with only a single endpoint connected, as well as with multiple endpoints).

If UDP is in use, it’s nothing I’ve set up; I expect that the http service is using TCP/IP. The internals of the WiFiServer are opaque to me.

very pseudocode:

  // pick up WiFi Credentials
  getWiFiCreds(path, ssid, wifipw);

  // Connect to WiFi
  WiFi.begin(ssid, wifipw);
  // Start the server
  server.setNonBlockingMode();
  server.begin();
  // parse http request
  // serve http camera image HEAD
 void CameraView::serveChunk(WiFiClient& client)  {  
  // get the location and length of this frame data
  myCamera.getImage(CHANNEL, &img_addr, &img_len);
  // clear the buffer
  memset(CameraView::chunk_buf,0,sizeof(CameraView::chunk_buf));
  chunk_len = snprintf((char*)CameraView::chunk_buf, 64, IMG_HEADER, img_len);
  sendChunk(client, CameraView::chunk_buf, chunk_len);    // Content-Type: image/jpeg\r\nContent-len: 123456\r\n\r\n, 49
  sendChunk(client, (uint8_t*)img_addr, img_len);    // {image data (a frame)}, 123456
  sendChunk(client, (uint8_t*)STREAM_BOUNDARY, strlen(STREAM_BOUNDARY));    // \r\n--123456789000000000000987654321\r\n, 36
  delay(2);
}

void CameraView::sendChunk(WiFiClient& client, uint8_t* buf, uint32_t len) {
    uint8_t ch_buf[64] = {0};
    uint8_t ch_len = snprintf((char*)ch_buf, 64, "%lX\r\n", len);
    client.write(ch_buf, ch_len);
    client.write(buf, len);
    client.print("\r\n");
}

It’s been a couple months, but recalling how I cobbled things together, some of the code would’ve been cribbed directly from the JPEG over HTTP example code.

OK, I’ve spent hours on this today, running the HTTPDisplayJPEGContinuous example (same results, same pauses with smaller frame size, though fewer in number) and optimizing, testing, optimizing some more. Using the 4.10 SDK, updated today in the Arduino IDE

Result: Only a very small improvement. I’ve tested on 2.4GHz and 5GHz wifi, have reduced JPEG quality, have reduced frame rate, have tried a variety of frame sizes (Ideally, I’d like to do FHD, and not go below 720p HD) and cannot get this to go away, and do not see an error message in the serial stream that indicates that it’s happening.

If this is either a WiFi interface capacity or a CPU capacity issue on the AMB82-MINI, I have to ask: what good is a (pretty nice) camera that can do FHD if I can’t get the video off the device?

I’m not using RTSP because I am looking for a solution that can stream directly to a browser, without an intermediate streaming or transcoding server.

Now that Firefox is going to start supporting MKV, it’d be nice to just send an HTML5 video, but doing so as an open-ended stream isn’t something I see documented here; only continuous JPEG.

Is there an OTG interface that can make a wired Ethernet connection to the AMB82-Mini? It seems like there ought to be some way to get video out of it without all the choking.

I tried HTTPDisplayJPEGContinuous with “WiFiServer server(80, TCP_MODE, NON_BLOCKING_MODE);“

error disappeared [Driver]: skb_unavailable=6 in last 2 seconds

left the chock/lag issue. let me do more search about this issue. It seems not related with wifi env

For this further investigate, lets following files

I don’t have deep knowledge of the Arduino ecosystem - are these files I can instrument and compile and put on my dev machine, or are you proposing that you’ll look at them?

If I’m to put them on my machine, I’ll need a starting point for getting them from the github repo (I know how to use git clone, etc…) into Arduino and compiled and part of the build code.

Hi @jimmiedave what is your stalls frequency (eg, once every 1min)?
I have some optimizations update for api tested under demo HTTPDisplayJPEGContinuous

Testing a few minutes ago: 1 to 4 seconds or so. Makes for very choppy video. This is with my code, which is on the board right now (as I said, it’s in regular use). I can try and put the HTTPDisplayJPEGContinuous example back on sometime soon (seemed to act about the same before).

I have PR some optimize here github PR.

In my experience, for http jpag some stalls may not 100% avoidable.

There is a new early release, you may want to try this example HTTPDisplayJPEGContinuousMultitask

For standard setup of HTTP JEPG as following, then depends the internet can increase the resolution and fps.

VideoSetting config(640, 480, 15, VIDEO_JPEG, 1);
#define FRAME_INTERVAL (1000 / 15/ portTICK_PERIOD_MS)

Small tips of testing internet. Can start both browser and VLC to test. If VLC is ok, it most likely, browser cause this stall issue.

Some more information

Thanks very much for digging in to this. I’ll try the new version shortly!

1 Like

Let me adjust the title to make it more relevant, so that more people can see it and learn from it.