Boot failure after OTA update

Hi,

We integrated OTA feature in our application and tested several times without any issues. But we observed a different issue recently. After OTA update, the device not booting at all and below is the captured log. The boot ROM failed to detect NOR flash.

== Rtl8735b IoT Platform ==

[Normal mode]
BootFromNORFlash
[SPIF Err]Invalid ID
[SPIF Err]Invalid ID
[BOOT Err]Flash init error (io_mod=0, pin_sel=0)
Fallback to NAND Flash
Boot from NAND Flash
=== Load SNAND INFO ===
[BOOT Err]No valid snand ctrl info block found
[BOOT Err]snafc init error
Fallback NOR Flash
BootFromNORFlash
[SPIF Err]Invalid ID
[SPIF Err]Invalid ID
[BOOT Err]Flash init error (io_mod=0, pin_sel=0)
Fallback to NAND Flash
Boot from NAND Flash
=== Load SNAND INFO ===
[BOOT Err]No valid snand ctrl info block found
[BOOT Err]snafc init error
Fallback NOR Flash
BootFromNORFlash
[SPIF Err]Invalid ID
[SPIF Err]Invalid ID
[BOOT Err]Flash init error (io_mod=0, pin_sel=0)
Fallback to NAND Flash
Boot from NAND Flash
=== Load SNAND INFO ===
[BOOT Err]No valid snand ctrl info block found
[BOOT Err]snafc init error
Fallback NOR Flash
BFU
Image Load over UART1 baud=115200

We thought the NOR falsh chip failed and replaced new one and tried to flash the firmware, but the downloading firmware also failed. Below is the captured log.

/dev/ttyUSB0 opened
ping ok
ucfg ok
flash loader path /home/yugandhar/office-repos/nuraeye-rt/ameba-rtos-pro2/tools/Pro2_PG_tool_v1.4.3/flash_loader_nor.bin
Uart boot
programing [========================================] 100% 81920/ 81920 bytes
programing done.
uboot ok
reset device
/dev/ttyUSB0 opened
ping ok
ucfg ok
download to offset 0x0
masked area [0x0, 0x0]
dowload 0, [0x0, 0x4cf000]
fail for download 0
download fail

We connected the suspected NOR flash chip to new SoC board and verified, that is working without any issues.

What might be the issue in old SoC board? Why it is not detecting, booting and writing the firmware to NOR flash?

Thanks,

Yugandhar

Hi @yugandhar ,

Based on the description provided, we can deduce that the NOR Flash chip itself is not the problem, as you have already verified that the “suspect” NOR Flash works normally when soldered onto another SoC board. We can also assume that the issue is not caused by OTA corrupting flash contents, because even if the firmware image were corrupted, the ROM should still be able to read a valid JEDEC ID. It would not report Invalid ID.

Would you mind to carefully inspect the solder joints, traces, and vias between the old SoC and the NOR Flash? if possible, please use an oscilloscope or logic analyzer to observe the waveforms during ROM ID read.

Thank you.

Dear @KevinKL @Pammy

We were using the failed board without any issues for several weeks before performing the OTA update. This problem only started appearing after OTA.

Additionally, we have another setup where, after OTA, we are able to read from the NOR Flash but unable to write to it.

We attempted to capture signals using a logic analyzer, but the code seems to execute before we can properly capture the behavior, making debugging difficult.

Since we are currently in field testing, this issue is very critical for us. We would really appreciate your support in identifying the root cause and resolving this as soon as possible

Regards,
Amit Jain

Hi @Amit_Jain @yugandhar ,

Could you share with us the steps of the OTA process that you have executed? Did you perform OTA for bootloader? Would you mind to perform chip erase first then try to download firmware again?

You may perform chip erase using command ./uartfwburn.linux -p /dev/ttyUSB0 -b 3000000 -e chip -U -x 32

OR using Arduino IDE: Tools → Erase All Flash Memory (16MB) → Erase only

Thank you.

Hi @KevinKL ,

We are flashing only ota.bin using OTA procedure, below is the wrapper code we created which uses ameba API.

#include "bsp_ota.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// Ameba SDK headers
#include "platform_stdlib.h"
#include "fwfs.h"
#include "ota_8735b.h"
#include "sys_api.h"
#include "hal_sys_ctrl.h"

// -------------------------------------------------------------------------
// Internal SDK Declarations
// -------------------------------------------------------------------------
// From ota_8735b.c
extern int ota_flash_NOR(uint32_t target_fw_idx, uint32_t total_blocks, uint32_t cur_block, uint8_t *buf, uint32_t data_len, _file_checksum file_checksum);

// -------------------------------------------------------------------------
// OTA Context
// -------------------------------------------------------------------------

#define OTA_BUF_SIZE    4096 

// Global variables to match SDK example style
// NOTE: Global static buffer instead of malloc as per requirement
static uint8_t g_ota_buf[OTA_BUF_SIZE]; 

static uint32_t g_buf_offset = 0;
// static uint32_t g_total_bytes_processed = 0; // Not strictly used for logic, derived from block + offset
static uint32_t g_expected_size = 0;

static uint32_t g_cur_block = 0;
static uint32_t g_total_blocks = 0;
static uint32_t g_target_fw_idx = 0;
static _file_checksum g_chksum = {0};

static char g_ota_token[64] = {0};

static bsp_ota_state_t g_ota_state = OTA_IDLE;

// -------------------------------------------------------------------------
// Implementation
// -------------------------------------------------------------------------

bsp_ota_state_t bsp_ota_get_state(void)
{
    return g_ota_state;
}

bool bsp_ota_init(uint32_t firmware_size)
{
    // Require waiting state to initialize (prevents random inits)
    if (g_ota_state != OTA_WAITING_FOR_UPLOAD) {
        printf("[%s] Error: Not in WAITING_FOR_UPLOAD state (State: %d)\n", __FUNCTION__, g_ota_state);
        return false;
    }

    // 1. Boot Selection Check
    if (sys_get_boot_sel() != 0) {
        printf("[%s] Error: OTA only supports NOR Flash.\n", __FUNCTION__);
        return false;
    }

    g_ota_state = OTA_IN_PROGRESS;

    // 2. Setup Target Index (Toggle)
    uint8_t cur_idx = hal_sys_get_ld_fw_idx();
    if (cur_idx == 1) {
        g_target_fw_idx = OTA_FW2;
    } else if (cur_idx == 2) {
        g_target_fw_idx = OTA_FW1;
    } else {
        printf("[%s] Error: Invalid current firmware index: %d\n", __FUNCTION__, cur_idx);
        return false;
    }

    // 3. Initialize Variables
    g_expected_size = firmware_size;
    g_buf_offset = 0;
    // g_total_bytes_processed = 0;
    g_cur_block = 0;
    g_chksum.u = 0;

    // 4. Calculate Total Blocks
    // SDK Logic: total_blocks = (file_size + buf_size - 1) / buf_size
    g_total_blocks = (firmware_size + OTA_BUF_SIZE - 1) / OTA_BUF_SIZE;

    // 5. Initialize Buffer
    memset(g_ota_buf, 0xFF, OTA_BUF_SIZE);

    printf("[%s] Init: Size=%d, Blocks=%d, Target=%d\n", 
           __FUNCTION__, firmware_size, g_total_blocks, g_target_fw_idx);

    return true;
}

void bsp_ota_set_token(const char *token)
{
    if (token) {
        snprintf(g_ota_token, sizeof(g_ota_token), "%s", token);
        g_ota_state = OTA_WAITING_FOR_UPLOAD;
    } else {
        g_ota_token[0] = 0;
        g_ota_state = OTA_IDLE;
    }
}

bool bsp_ota_validate_token(const char *token)
{
    // If no token was set, maybe we expect no token? 
    // For security, if no token set, we should probably fail if we strictly require StartFirmwareUpgrade.
    // However, if the token is empty in storage, we deny all.
    if (g_ota_token[0] == 0) {
        return false;
    }
    
    if (token && strcmp(g_ota_token, token) == 0) {
        return true;
    }
    return false;
}

static bool prv_process_and_flash_buffer(uint32_t payload_len)
{
    int ret = 0;
    uint32_t current_block_idx = g_cur_block; // Copy for calculations

    // 1. Extract Checksum if we are at the end of the file
    // Logic from http_update_ota: if ((idx + read_bytes) > (ota_len - 4))
    
    // Calculate total bytes processed so far (including this payload)
    uint32_t total_processed_so_far = (g_cur_block * OTA_BUF_SIZE) + payload_len;
    
    if (total_processed_so_far > (g_expected_size - 4)) {
        // We have the checksum in this buffer (at the end)
        // Checksum is at the end of valid data
        
        // Safety check to ensure we have enough data in buffer
        if (payload_len >= 4) {
            g_chksum.c[0] = g_ota_buf[payload_len - 4];
            g_chksum.c[1] = g_ota_buf[payload_len - 3];
            g_chksum.c[2] = g_ota_buf[payload_len - 2];
            g_chksum.c[3] = g_ota_buf[payload_len - 1];
        }
    }

    // 2. Adjust Data Length for Last Block
    uint32_t write_len = payload_len;
    if (current_block_idx == (g_total_blocks - 1)) {
        // Last block handling from SDK
        if (write_len >= 4) {
            write_len -= 4; // Remove checksum from flash write
        } else {
            // Should not happen if size > 4 and logic is correct
            write_len = 0; 
        }
        
        // Pad remainder with 0xFF
        if (write_len < OTA_BUF_SIZE) {
            memset(g_ota_buf + write_len, 0xFF, OTA_BUF_SIZE - write_len);
        }
    }

    // 3. Write to Flash
    ret = ota_flash_NOR(g_target_fw_idx, g_total_blocks, current_block_idx, g_ota_buf, write_len, g_chksum);
    
    if (ret < 0) {
        printf("[%s] Error: Flash write failed at block %d\n", __FUNCTION__, current_block_idx);
        return false;
    }

    // 4. Prepare for next
    g_cur_block++;
    g_buf_offset = 0;
    // Note: ota_flash_NOR might modify buffer (readback), so we reset for next usage
    memset(g_ota_buf, 0xFF, OTA_BUF_SIZE);
    
    return true;
}

bool bsp_ota_write(uint8_t *p_data, uint32_t data_len)
{
    // No need to check g_ota_buf as it is static now

    uint32_t src_idx = 0;
    while (src_idx < data_len) {
        uint32_t space = OTA_BUF_SIZE - g_buf_offset;
        uint32_t copylen = (data_len - src_idx) < space ? (data_len - src_idx) : space;
        
        memcpy(&g_ota_buf[g_buf_offset], &p_data[src_idx], copylen);
        g_buf_offset += copylen;
        src_idx += copylen;

        // If buffer full, flush it
        if (g_buf_offset == OTA_BUF_SIZE) {
            if (!prv_process_and_flash_buffer(OTA_BUF_SIZE)) return false;
        }
    }
    return true;
}

bool bsp_ota_finalize(void)
{
    // Check if we have partial data pending
    // OR if we are at the last block and haven't flushed it.
    
    if (g_cur_block < g_total_blocks) {
        // Flush the final partial block
        if (!prv_process_and_flash_buffer(g_buf_offset)) return false;
    }

    printf("[%s] OTA Success. Ready for reboot.\n", __FUNCTION__);
    
    // No free needed for static buffer
    
    g_ota_state = OTA_IDLE;

    // ota_platform_reset(); // Reboot is handled by upper layer command
    return true;
}

void bsp_ota_abort(void)
{
    // No free needed for static buffer
    // Could reset variables if needed
    g_buf_offset = 0;
    g_cur_block = 0;
    g_ota_state = OTA_IDLE;
}

We are calling as below from our application.

  • bsp_ota_set_token
  • bsp_ota_validate_token
  • bsp_ota_init
  • bsp_ota_write - several times
  • bsp_ota_finalize

As I mentioned earlier, the flash chip is working perfectly with another SoC board and OTA procedure also working (we tried several times).

We tried erase command as you suggested with new flash chip on failed SoC board and erase also failed. See below log messages.

xmodem tx 32K mode
/dev/ttyUSB0 opened
ping ok
ucfg ok
flash loader path /home/yugandhar/office-repos/nuraeye-rt/ameba-rtos-pro2/tools/Pro2_PG_tool_v1.4.3/flash_loader_nor.bin
Uart boot
programing [========================================] 100% 81920/ 81920 bytes
programing done.
uboot ok
reset device
/dev/ttyUSB0 opened
ceras cmd fail
erase fail

Hi @yugandhar ,

Would you mind to provide your ota.bin for us to replicate your issue? Please modify amebapro2_firmware_ntz.json line 19 “VERSION” value to “EFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF” and recompile a ota.bin firmware for verification on our end.

Thank you.