Release v2.0.0 08/08/2024 **** Moves to Ubuntu 22.04 for VM, Raspberry Pi (RPi) and Beaglebone Black (BBB) **** **** New image available for the TI AM64B Starter Kit which is Debian 12 **** **** NOTE: Full system updates are necessary (including the VM) **** **** From this point on, the Ubuntu 20.04 platform will no longer be updated **** **** (v1.1.23 is the last release for the Ubuntu 20.04 platform) **** Release Date: 08/8/2024 ====================================================================================================== New VMs and Remote node images are available on the download page (https://riaps.isis.vanderbilt.edu/rdownloads.html). This release includes SD card images for TI Beaglebone Black, Raspberry Pi 4 and TI AM64B Starter Kit. The VM also includes a user friendly setup for using Node RED GUI and MQTT communications. Information on setting up the new VM can be found at https://github.com/RIAPS/riaps-integration/tree/master/riaps-x86runtime. NOTE: If you have an existing VM from previous releases, you will need to create a new VM and move your development efforts to the v2.0.0 VM. This release updates the system OS and many third party package updates that are coordinated with the remote node image releases. Tips for moving your development environment to the new platform are included in a section below. The RIAPS platform consists of 2 packages: riaps-pycom and riaps-timesync. The riaps-pycom package is architecture independent, there is one for the development environment (riaps-pycom-dev) and the other for remote nodes (riaps-pycom). Architecture specific packages are available for riaps-timesync(-). IMPORTANT NOTE ON REMOTE NODE IMAGES: The remote node images are basic platform prepared image. It includes the needed third-party packages already installed and the environment configurations available for the RIAPS platform. It DOES NOT INCLUDE the RIAPS platform packages pre-installed. The remote nodes images no longer contain default security keys. Once the image is copied to an SD card, the user can connect the nodes with the development system using the instructions found at https://github.com/RIAPS/riaps-integration/blob/master/riaps-x86runtime/README.md#4-connect-the-vm-to-the-remote-nodes. When all the nodes are connected, the RIAPS Platform can be install using "riaps_fab riaps.update". Note: The remote nodes need to have access to the network during this installation step. Note for CAN1 interface users: Most of our BBB boards have eMMC flash images that have an older U-Boot version (2019). To enable the CAN1 interface which is used in several applications, you need to physically update the eMMC flash image to a newer image (2023). Instructions are included in a section below. ===================== V2.0.0 Release Notes ===================== New Features ================= In `riaps-pycom`: • System platform updates o Capnproto library `from_bytes` must be wrapped in a context manager (`with` `as` `:`) due to an API change with version update - NOTE: This is a breaking change for applications that use Capnp Proto messaging, see note below on how to adjust your code o Move python version to 3.10 - Set apparmor python version using script and systemd service based on system python version o Netifaces library version update • Changes for Debian Bookworm systems o Added `--break-system-packages` to pip installs o Added softlink to `/dev/root` for quota service Improvements ================== In `riaps-pycom`: • Update tmux api calls due to library version update • `riaps_fab` redesigned to use Fabric2.X and break into CLI-based and API o Improve command helpstrings and error messages o Output execution information only on error (except for `sys.arch`) - Note: you can use `-v` to get verbose feedback on any command o Reference RIAPSHOME for loading hosts o Provide role hints on invalid role o CLI hostlists now usurp role checks o `sys.get -n` flag renames local file, prints results nicely o `sys.put` checks for file before sending o `sys.put`/`sys.get` outputs are more readable o Simplify `riaps.reset` o Add flag to pass hostfile path o ctrlgui: Update fabric usage to rfab o deplo.status exit-code acceptance - systemctl-status will exit 3 if a service is inactive, but rfab treats non-zero exits as "Failed". Adding this test accepts exit (3) as a success. o Adding Task class o Add logging files o Change `riaps_deplo` launch backgrounding to disowning o Consolidate result types to enable more hints o RFabGroupResult class always hints for socket.gaierror Fixes ================= In `riaps-pycom`: • Remove `StandardOutput=syslog` from `riaps-deplo.service` • Localhost not first interface, leading to invalid connection on locals • Opendht crash on 600 sec t/o due to not cleaning listener • Change needed to Jenkins deployment due to function deprecation • Workaround to spdlog problem -- spdlog (python) must be imported first to avoid signal handler issues • Set RIAPS* envvars for SSH sessions in `/etc/environment` using Debian `preinst` for remote nodes o Commands executed over SSH such as `ssh ` or those via Fabric occur in non-login non-interactive shells, which don't hit any of our .bash* or .profile* config files. This config sets `RIAPSHOME` and `RIAPSAPPS` for SSH sessions matching the riaps user. • Remove `setuptools_scm` from the pip install requirements o This caused an error related to git meta data, but did not affect the installation of the package. • Found extra removal of conf files for riaps-pycom • Better formatting for log message • Prohibit MulticastDNS for systemd-resolved o We use Avahi for .local domain resolution, but the default systemd-resolved configuration also runs mDNS, which conflicts. This file and option disables mDNS. • Fixes to 100% utilization in deplo (due to `zmqdevice`) • Better logging for dc • DC w/ redis works for rapid group membership changes • Added deplo/relay code • Fixes for rapid group membership changes o Note: opendht's `cancelListen` does not work (from Python) In riaps-timesync: • timesyncctl: send logger to stdout instead of stderr • Change needed to Jenkins deployment due to function deprecation =========================================================== Tips for Transferring Development VM Setup to the New Image =========================================================== • Note any additional software installed on current development system, the software will need to be added to the new VM setup • Save any application projects to a location off the current development system o It is beneficial to keep your applications in server hosted repositories such as Github or Gitlab. Changes could then be committed and pushed to the repository, then cloned on the new VM setup. o You could also save the information to a shared drive on the host machine by setting up a shared folder for the current VM using the VM settings under the “Shared Folders” section. - Setup the desired location on the host machine (Folder Path) and select “Auto-mount” - Then start the VM - You can copy the files to that shared folder location from within the VM using `sudo cp -r /media/sf_shared_folder/.’ • Suggestions for Transferring Eclipse Workspaces to a new VM o Easiest approach: create a new workspace and import the code cloned from the desired repositories o Export an archive file of the current workspace: Use Eclipse to export an archive file that can be moved to a transfer site (such as Dropbox or any other file sharing tool). Then import that file into a new VM eclipse workspace. Here are some tips for doing this: 1) In the old VM, choose "File" --> "Export" from the drop-down menus and select the "General - Archive Files" option. 2) Select all the files you want to include in the archive. You can unselect compiled files or even use the "Filter Types" tool to help with this. Indicate the file name and location to place this archive ("To archive file:" box). 3) Copy the file to a transfer site (such as Dropbox) and then retrieve it on the new VM. 4) On the new VM, choose "File" --> "Import" from the drop-down menus and select the "General - Existing Projects into Workspace" option. 5) "Select the archive file" option and find the transferred file. 6) This should have the workspace back in place in the new VM. ================================================ Capnproto Library “from_bytes” API Change Notes ================================================ Updates to pycapnp library now requires `from_bytes()` functions to be wrapped in a context manager. Below is an example of changes needed to applications using this function. This example is from the app.IMCP code for `SYSTEM_OPERATOR_MQTT.py` (line 55): v1.1.23 code: ``` def on_state_sub(self): msg_bytes = self.state_sub.recv() msg = imcp_capnp.StateMsg.from_bytes(msg_bytes) if debugMode: self.logger.info(f"{helper.Yellow}" f"Operator\n" f"on_state_sub:{msg.sender} \n" f"recv state sub msgcounter: {msg.msgcounter} \n" f"state: {msg.currentState} \n" f" breaker: {msg.breaker} and action: {msg.action} \n" f" group: {msg.group} and reconfigcontrol {msg.reconfigcontrol}\n" f"{helper.RESET}") self.node_states[msg.sender] = msg.currentState ``` Modification for v2.0.0: ``` def on_state_sub(self): msg_bytes = self.state_sub.recv() with imcp_capnp.StateMsg.from_bytes(msg_bytes) as msg: if debugMode: self.logger.info(f"{helper.Yellow}" f"Operator\n" f"on_state_sub:{msg.sender} \n" f"recv state sub msgcounter: {msg.msgcounter} \n" f"state: {msg.currentState} \n" f" breaker: {msg.breaker} and action: {msg.action} \n" f" group: {msg.group} and reconfigcontrol {msg.reconfigcontrol}\n" f"{helper.RESET}") self.node_states[msg.sender] = msg.currentState ``` Here are the list of Apps, interfaces and example code with capnproto `from_bytes` calls: • Currently used application: o app.IMCP • Older code base that might be reused: o app.MgManage o app.MgManage_SingleMg o app.OpalRT-Microgrid-Monitor-Demo o app.RensycnFullFeeder1Banshee o interface.PyDSS o interface.gridlabd o BansheeFSM o example.varcontrol ================================== eMMC Update Process for BBB boards ================================== 1) Download the am335x-eMMC-flasher-debian-12.2-minimal-armhf-2023-10-7-2gb.img.xz image on https://forum.beagleboard.org/t/debian-12-x-bookworm-monthly-snapshot-2023-10-07/36175. 2) Put the image on a SD card using balenaEtcher. 3) Hold down the button by the SD card slot on the BBB board and power on the board 4) Keep holding the button until writing begins which is signaled by a cycling of the 4 LEDs. Note: If you get a solid light only, try again making sure to hold down the button until it starts writing (which takes a couple of seconds). 5) After the eMMC write finishes, the BBB will turn off. 6) Then remove the SD card and power back up the BBB with the downloaded RIAPS image inserted.