The idea for this project sparked in this thread:
https://www.diystompboxes.com/smfforum/index.php?topic=126087.0
In short words, what it does is instead of having to write the new hex file to the onboard EEPROM everytime something has changed in the code you can just plug in a remote emulator/programmer having the following features:
- web interface for uploading the hex files and switching between the patches with 3MB disk space
- for even faster upload - access through an FTP server
- clicking on "Enable" parses the hex file and makes it available for the FV-1
- once the hex enabled, clicking on any of the 8 buttons will upload corresponding patch to the FV-1
- if the work on the code is done you can burn that firmware to the EEPROM
- as final test you can enable the onboard EEPROM to test if everything works fine
- works up to 44.1kHz sample rate (i2c clock depends on the sample rate).
(https://i.imgur.com/i0reAn1.png)
Schematic, code and instructions are on github:
https://github.com/hexeguitar/FV-1_devRemote
Project is written using Platformio + VScode.
You're a hero!
[edit]
Question! You state that one might have to update platformio.ini:
; change accordingly to your operating system (ie COMx for Windows)
upload_port = /dev/ttyUSB0
upload_speed = 230400
; change accordingly to your operating system (ie COMx for Windows)
monitor_port = /dev/ttyUSB0
monitor_speed = 115200
Normally, I only set the monitor_speed line to match whatever the serial port is opened with, I never have to set the port nor the upload speed. Have you tried without?
Yes, i tried. This is only necessary if you have more USB/UART dongles plugged in or boards that use them.
Normally platformio attemps to probe and detect the esp8266 board by looking at the serial port list.
In my case i have other similar dongles plugged, platformio decided the other one is the esp8266. Hence i hardcoded the correct upload port in the ini file.
If there is only one board at a time in the system you can comment out that line.
If the upload starts, but can't connect to the esp - that is usually indicates a wrong upload serial port.
Btw, i tried to use curlftpfs to mount the disk a directory, but it won't work. The problem is in the ESP's ftp implementation and the limit of one connection only.
The mount and listing works, but any file operation failed. Apparently curlftpfs tries to open up a new connection for file operations, something that the ESP will not handle.
Filezilla works fine.
You are the boss!!
I see that at the end you are using T0 to trigger the program reload, right? So switching to internal and then external memory again?
I will scroll through your code and reuse some of it if you don't mind. I'm planning to build something similar but slightly different as I want to have the ESP included in the pedal.
I already have part of the code done, but I might want to optimize it if yours looks better. :)
Thanks!
Yes, i used the T0 as it's separate from whatever might be switching the programs on the target board. Otherwise some kind of ORing would have to be added. I think this way it will be easier to adapt existing builds.
Sure, use the code.
It should be fairly easy to adapt it for the ESP32 in case you need more IOs, pheripherals or to make it use less power.
An SD card for the filesystem could be another possible option.
I have managed to replace the Flash on a few esp8266 boards installing an 8MB one. The partitioning is set in the platformio.ini file:
; 4MB Flash chip, 1MB for firmware, 3MB disk partition
board_build.ldscript = eagle.flash.4m3m.ld
The list of available options (linker scripts) is here:
https://github.com/esp8266/Arduino/tree/master/tools/sdk/ld
IE, for an 8MB flash chip with 1MB reserved for the firmware and almost 7MB for the filesystem that line would look like this:
board_build.ldscript = eagle.flash.8m7m.ld
I was just peeking at the code. May I ask why you used the inline NOPs as delay routine instead of the standard delay()?
Interrupts are turned off for the time of sending the data over to the FV-1 to ensure lowest latency. Delay uses timer/interrupts, so it can't be used here.
I was also getting a watchdog triggered occasionally, hence i'm resetting it inside that function.
The fact that FV1 always requests 512 bytes + we do not have to care about the initial EEPROM address decoding simplifies the i2c slave handling to the point it can do the 250-300KHz clock rate.
Within limits of course, clocking the FV-1 with 48KHz will not work with the ESP board.
I see, I didn't notice you were disabling interrupts BEFORE toggling T0. Makes sense.
Quote from: free electron on January 22, 2021, 03:24:33 AM
Btw, i tried to use curlftpfs to mount the disk a directory, but it won't work. The problem is in the ESP's ftp implementation and the limit of one connection only.
The mount and listing works, but any file operation failed. Apparently curlftpfs tries to open up a new connection for file operations, something that the ESP will not handle.
Filezilla works fine.
I wanted to try this FUSE ftpfs: https://github.com/rr-/ftpfs (http://ftpfs:%20https://github.com/rr-/ftpfs) to see if it would work with one connection, but I'm running into problems compiling.
I didn't have the SparkFun External EEPROM Arduino Library, so I put this in the platformio.ini:
lib_deps =
sparkfun/SparkFun External EEPROM Arduino Library @ ^1.0.3
however now I get the following error:
.pio/libdeps/esp12e/SparkFun External EEPROM Arduino Library/src/SparkFun_External_EEPROM.h:59:19: error: This platform doesn't have a wire buffer size defined. Please contribute to this library!
#pragma GCC error "This platform doesn't have a wire buffer size defined. Please contribute to this library!"
Looking at SparkFun_External_EEPROM.h, it turns out it cannot be used on ESP8266? Did you modify the library (add a wire buffer size definition I guess) to get it to compile?
Followup. I added the following to SparkFun_External_EEPROM.h, according to some shady websites 64 should be the correct size.
#elif defined(ESP8266)
#define I2C_BUFFER_LENGTH_RX 64
#define I2C_BUFFER_LENGTH_TX 64
Compiled & uploaded, wrote the data/ folder to the ESP and it seems it works (I can access the webpage, haven't connected it to any FV-1 pedal yet).
Installed that Python ftpfs but I cannot get it to work. However, testing with a public ftp server (ftp.freebsd.org) it doesn't work either *sigh*
Would be awesome to get the filesystem mounted so that SpinCAD hex files can be written directly to the ESP!
I see i forgot to copy the eeprom library when i was setting up the repo. Yes, i did some changes to it.
Updated, clone it again and it should work.
The FTP lib has this "limitations" section:
Quote
Server only allows one ftp control and one data connection at a time. You need to setup Filezilla (or other clients) to respect that, i.e. only allow **1** connection.
(In FileZilla go to File/Site Manager then select your site. In Transfer Settings, check "Limit number of simultaneous connections" and set the maximum to 1.)
This limitation is also the reason why FuseFS based clients (e.g. curlftpfs) seem to work (i.e. listing directories) but will fail on file operations as they try to open a second control connection for that.
I don't know what the hell I'm doing wrong but I'm getting an error
"expected class-name before '{' token class WiFiServer : public Server {"
when trying to compile. It's like it cannot recognize Server as a class.
I've traced all the includes, and it looks ok... Server.h is there as part of the core arduino libraries.
Are you on Windows?
Windows file names are not case sensitive, unix/linux are.
I named the file server.h, but apparently there is a Server.h in the Arduino core.
On Linux these are two different files, on windows not.
Solution:
change the name of the server.h + server.cpp (the ones in my repo) to something else, like fv1_server.h, fv1_server.cpp + all the includes.
I've just updated the repo, please try the most recent commit.
oh shit, this might be the issue.... I did not think on the file name. I was aware of the server vs Server, but I was thinking on the definitions, not the files.
I'll be back in 10 minutes....
ok, that was it... changing names of server.h and server.cpp fixed the error.
Thanks!
Have you JUST changed the repo? I downloaded it 3 hours ago...
Nevermind, I just checked and see you modified the names. That was a fast reaction :icon_mrgreen:
Yes, that was a small and quick change :)
Glad it worked out!
I've run into similar problems a few times in the past. Usually working on customized local for project libraries in platformio.
Btw, have you tried the WinSCP for the FTP/mount as disk thing? Apparently it does what we need, can sync folders over ftp.
Quote from: pruttelherrie on January 24, 2021, 03:00:08 PM
Would be awesome to get the filesystem mounted so that SpinCAD hex files can be written directly to the ESP!
Since I'm on Linux (Debian), WinSCP won't do, but I found another convoluted way to get this to work:
- populate ~/.netrc with the hostname/username/passwd of the ftp server (192.168.4.1/fv1/fv1)
- make a systemd unit file that watches a folder for changes, then call this:
- echo 'put path/filename.hex' | ftp 192.168.4.1
I tried it 'by hand' and it works, it uploads the hexfile to the ESP. So when the unitfile works, a hex-export to that folder from within SpinCAD should be enough to get in onto the ESP.
Hey Piotr,
I was playing with the code on a NodeMCU D1 mini I had lying around, on my Linux laptop. I noticed the following: the hexfiles that SpinCAD generated in the past has lines which were only 0x0A terminated. The FV1-devRemote would not enable these files, complaining it's not a valid FV1 fle. After using unix2dos <filename>, evrything would work fine. Could you modify the function FV1::load_file(path) so that it accepts all possible line-terminations? (I believe it is Windows=CRLF, Unix=LF, Mac=CR). I believe the code checks for \n which is an LF so my story above does not make sense?!? :icon_confused:
Hey,
i'll look into this. Generally it's a mess with OS dependant line endings.
\n is 0x0A and it's a standard line ending on Linux, we have three options:
\n - linux
\r\n - windows
\r - mac
\n should work for both linux and windows, since the data length is encoded in the record. The \r will be ignored.
Mac will fail. I'll try to implement a check for the used line ending before the file is processed. First look for '\n' and if found use it as the line ending. If failed - look for '\r'.
However in your case the problem might be somewhere else. What is the file size in bytes generates by the SpinCad?
The program checks for the length of a file generated by the SpinAsm (21517 bytes) even before it starts to read the lines.
It happens here:
if (hexfile.size() != FV1_HEXFILE_SIZE)
{
hexfile.close();
boot_complete = 1;
return FV1_INPUT_FILE_WRONG;
}
FV1_HEXFILE_SIZE is defined at the top of the fv1.cpp file. Try to put the length of the SpinCad hex output and see if it works.
Length matching will have to be also updated, depending on the detected OS.
Actually, i have been thinking about changing that part of the code to allow writing shorter files (minimum 1 patch of 512 bytes).
My line endings are 0x0A (verified with a hex editor, GHex)
The hexfiles are 20492 bytes long, while hexfiles with 0x0D 0x0A are 21517 bytes long, which makes sense since there are 1024 opcode lines plus the ending line ":00000001FF" makes 1025 bytes difference.
Commenting out the file length check yields the "Not a valid FV-1 hex file!"
Actually: the serial monitor prints "Hex file checksum error", so it's not the ending that causes it *directly* ???
Not sure how the checksum calculation can be thrown off by the line ending?
Indeed, line endings *are* messy.
I found out the following:
* A line is like this on windows ":040000004000028436" + \r\n, or on unix ":040000004000028436" + \n
* "data.length()" is then in windows 20, or on unix it is 19.
* "data.getBytes(buffer, data.length());" will put 1 less byte in the buffer in the case of unix?!?!
I mean: the getBytes() will chop off the last character (the 6 in the example line above) off the line. buffer[18] will return a 0, even though buffer[17] is the 3 preceding the 6 ...
But when I hardcode it to data.getBytes(buffer, 20); it reads the shorter lines just fine and the checksum calculates correctly. What is going on with getBytes() and data.length?
For now (with my limited testing), uncommenting the filelength check and hardcoding the buffercopy like below, the code works for both encodings. Mac encoding is a different story of course.
// each line is one FV1 instruction,
data.getBytes(buffer, 20);
[edit]
I think I found it: string.getBytes(char *buf, int len);
where len is the size of the buffer, which of course will be 0-terminated.
data.length() includes the \r if present, which will then be overwritten with a 0. If there is no \r, the last character before the \n will be overwritten with a 0.
So the correct code for variable length lines would be data.getBytes(buffer, data.length()+1);
It's no problem that a \r will end up in the buffer since it's not processed anyway (as you already noted).
Makes sense?
Totally makes sense. Good job!
I have generated a test file using SpinCAD on Linux, implemented the length check for both types of files and checking for all 3 possible variations of line endings: CR, LF, CRLF
Apparently, the modern OSX uses the unix LF line ending, the CR applies to the older pre OSX systems. I guess checking for LF only would be sufficient, but i'm going to leave the check for all three just in case.
I'll update the github repo soon.
Now on to the next challenge: automatic upload of a hexfile saved from SpinCAD (or any other hexfile generator).
The problem using an ftp-client is that the esp-ftp-server can only handle one connection at a time, and most clients open more than one connection. The 'builtin' ftp command on linux works, but I didn't manage to get that scripted.
BUT! Using 'curl' we can upload a hexfile to the web-form!
The command to upload a file using http-form-upload is as follows:
curl -X POST -F up[]=@path/to/hexfile http://IP.AD.DR.ESS/uploadhex?f=
curl http://IP.AD.DR.ESS/enable?file=/$FILE
That second line will 'enable' the newly uploaded hexfile.
Now we need something to watch a folder and call this command for every new file that appears or gets rewritten. There's a number of options for that, depending on OS: inotify, fswatch, Folder Actions, systemd path files, ... ?
I managed to get it to work on Linux using system path files. Here's a short recipe, not very flexible but it works (replace YOUR_USERNAME and IP.AD.DR.ESS):
/usr/local/bin/upload_fv1.sh
#!/bin/bash
FILE=`ls -c /home/YOUR_USERNAME/fv1/ | head -n 1`
echo `date +"%Y-%m-%d %H:%M:%S $FILE"` >> ~/home/YOUR_USERNAME/fv1_out.log
curl -X POST -F up[]=@/home/YOUR_USERNAME/fv1/$FILE http://IP.AD.DR.ESS/uploadhex?f=
curl http://IP.AD.DR.ESS/enable?file=/$FILE
/etc/systemd/system/fv1.service
[Unit]
Description="Upload changed hex file to FV1 DevRemote"
[Service]
ExecStart=/usr/local/bin/upload_fv1.sh
/etc/systemd/system/fv1.path
[Unit]
Description="Monitor /home/YOUR_USERNAME/fv1 folder for changes"
[Path]
PathChanged=/home/YOUR_USERNAME/fv1
Unit=fv1.service
[Install]
WantedBy=multi-user.target
Then start the path/service with: sudo systemctl start fv1
You can check if it started ok with: systemctl status fv1
Now any file copied or written to the /home/YOUR_USERNAME/fv1 folder will automatically be uploaded to the DevRemote and 'enabled' in the UI :icon_cool:
Other solutions/platforms are left as an exercise for the reader.
NB: Please note that I modified the DevRemote code to have a static IP.
Nice work with the auto upload!
I started to cobble a python script together to do the same, hopefully will be cross platform and easy to use for less experienced in linux-fu users, also not requiring any root privileges.
Uses watchdog to monitor events, pycurl for upload + a few checks to ensure the uploaded file is a valid fv-1 hex.
The plan is to put it into ~/home/bin/ (or add to PATH) and make it accessible from any folder. Simply invoking the program inside a hex output directory will start it and watch that directory for any valid changes like a new hex file created or an old one overwritten. Ctrl+C to quit.
Going to include it in the repo once finished.
btw. with mdns the address of the board will be just fv1.local - no hassle with setting up or looking for the ip.
-- EDIT --
Got the 1st version working on Linux.
(https://i.imgur.com/DHdxe4Y.png)
added to PATH, watcher.py can be invoked with no arguments from any directory where the hex files will be written to. Script starts to watch for any valid fv1 hex file changes and if detected, validates the file name (ie, replaces the whitespaces with underscores), uploads the file to the DevRemote and enables it.
One thing i had to add to the esp8266 code is the ability to refresh the site once a file is uploaded via curl. Normally, each time a new file is written from the script the site had to be refreshed to see the changes. I solved this by adding a new command for the server: "/trigrefresh" which sets a refresh flag as true. That flag is periodically (1 sec) polled from the main page. If true, the site is refreshed once.
The script takes two optional arguments :
- url of the board, default being fv1.local. In case the mdns is not working the ip address can be provided, ie http://192.168.4.1
- directory path to watch, if not provided it will use the current working directory from where the script is invoked. If the board is available at fv1.local, the easiest way to use is to simply go into the hex file dir and execute the script from there.
Tested on Linux only for now. If everything works fine, i'll try to make an executable file for windows.
Repo will be updated soon.
New update:
https://github.com/hexeguitar/FV-1_devRemote
I think the new auto uploading script works fine, successfully tested it on Mint19 and Windows 10.
There is a precompiled binary version for win10 64bit included in the scripts folder. Should work without installing anything. Simply run the file from the working directory, alternatively add it to PATH to be accessible from anywhere and without copying the file to each working folder.
Script checks for a valid FV1 hex file before upload, so there can be other files in that directory.
I had problems with resolving the fv1.local using the pycurl on Win10, if it doesn't work simply provide the ip address of the board as input parameter:
watcher.exe --url 192.168.4.1
Optionally turn on the pycurl verbose to see more details:
watcher.exe --url 192.168.4.1 --verbose
To see the opeartion/help use:
watcher.exe --help
Naturally, the same works with direct python version watcher.py
Awesome work!
About the fv1.local: how does that work when I want the ESP in my home network? I'd like to keep my laptop connected the way it is instead of connecting to the ESP hotspot.
It doesn't matter if the wifi is working in AP or STA mode If the mdns responder is started. You should be able to access the board via fv1.local if it's connected to your network.
I have a 2nd small wifi usb dongle plugged in my laptop for all those IOT thingies which are on a separate network or use their own APs.
You could also use the WifiManager library:
https://github.com/tzapu/WiFiManager
[edit]
Nevermind, I was being stupid. Builds fine now!
Why did you decide to emulate EEPROM instead of just flashing the EEPROM chip via the ESP? What is the advantage for the extra complexity?
Also, how long does it take to flash the EEPROM. Is this why you decided to emulate? Thanks :)
- Speed is the factor, esp. when developing the firmware.
- Reducing the number of EEPROM write cycles.
- Reducing the number of wires if the ESP is used as an external programmer. ESP would need to access the S0-2 address lines to set the program number via web interface.
- Give the ESP enough disk space (SD card) and the number of patches can be practically unlimited.
- If the ESP is an integral part of the device/pedal, i'd simply not use EEPROM at all.
- Extra complexity is fun :)
Thanks for the info. I'm working on a school project using the ESP-32 and FV-1 to make a wirelessly controlled effect pedal. I'd like to simply write the EEPROM over the ESP32 for less complexity but I am worried about the speed. Can you estimate how long it takes to transfer and flash 1 effect or all 8? Is it like 5 seconds or 2 minutes?
My very very rough estimate now is ~2.5 seconds per effect to transfer over BLE and 0.8 seconds to flash. What do you think of this?
Also, I'm unsure what to do for the enclosure. Metal is best to block noise for pedals but it will also reduce the wireless range of the ESP. What material did you select?
Thanks :)
I haven't built a pedal using the FV1+ESP, it was meant as a development tool, something that makes the programming/testing the firmware faster and more comfortable. My test rig was more like a few dev boards connected with wires.
I also haven't tested the upload speed. The point of using a filesystem on the ESP and an emulated EEPROM is the speed at which the firmware is available for the FV1. Once uploaded, switching between the bin files is pretty quick. I think most of the time was not the bin file processing, but the whole web server stuff. I haven't used BLE, but with WiFi AP the file upload took about a second.
Is your plan to have all the firmware bin files stored locally on the computer and then upload/burn it to the EEPROM when the user selects it?
For the wifi range - you could use one of the ESP32 modules with an ipex connector + an external antenna.
If the final device will have the ESP32 onboard i'd try to make the most of it, use an SD card + file system to store the bin files, use emulated EEPROM for fast firmware access, PWM outputs + RC filters to control the POT0-2 inputs on the FV1.
Probably even forget the FV-1 and use the two 240MHz cores for audio DSP :)
Quote from: free electron on October 30, 2021, 10:42:50 AM
Probably even forget the FV-1 and use the two 240MHz cores for audio DSP :)
I don't know, are there some good examples of ESP32 based audio guitar effects gadgets? I know my experiments used Faust, which may not be the way to generate the most efficient code, but other than more delay time I think I could get more out of an FV-1 than the ESP32.
Frankly, i have no idea if such pedals exist, never looked for one. Just designed my own to evaluate the platform. I'm not really interested in these highly abstracted dsp engines like Faust, will probably adapt/port the Teensy library or use similar approach.
Boards arrived today, so i haven't built it yet. I stripped down everything WiFi related, added PCM3010 codec (just because i have them waiting to be used), 4 Pots, 2 footswitches, electronics bypass with 4053, analog pass-through and a small SPI display. USB-C and programmer are built in. Oh, and a proper 4 layer layout with separate supplies for analogue and digital part :)
The chip used is the Pico v.3.02, 8MB of flash and (!) 2MB of PSRAM already inside the package. Seemed very appropriate for DSP, although PSRAM is not as fast as built in RAM since it's accessed via QSPI.
(https://i.postimg.cc/8JS5wM4J/DESP-2-freecad.png) (https://postimg.cc/8JS5wM4J)
(https://i.postimg.cc/nj2Hk5Mg/DESP1.jpg) (https://postimg.cc/nj2Hk5Mg)
Quote from: free electron on November 04, 2021, 01:01:57 PMOh, and a proper 4 layer layout with separate supplies for analogue and digital part :)
Good going there: I tried that EPS32-A1S with the on-board codec and you can hear the MCU in the audio whenever it tries to calculate something. So an external codec or ADC/DACs with prober layout is a must.
Quote from: free electron on November 04, 2021, 01:01:57 PM
Frankly, i have no idea if such pedals exist, never looked for one. Just designed my own to evaluate the platform. I'm not really interested in these highly abstracted dsp engines like Faust, will probably adapt/port the Teensy library or use similar approach.
However you go about it, I'll be interested to see what you come up with.
Quote from: Digital Larry on November 06, 2021, 01:53:25 PM
However you go about it, I'll be interested to see what you come up with.
I got it working. Ported my written for Teensy Plate reverb code to the PicoV3.02 using the 2MB PSRAM for all the reverb allpass and delay buffers.
Here is a short demo using a few different guitar sounds:
https://www.youtube.com/watch?v=VBzCPpaZFyk
The reverb is fully stereo in/out, original code for Teensy is here:
https://github.com/hexeguitar/t40fx/tree/main/Hx_PlateReverb
Quote from: pruttelherrie on November 05, 2021, 03:08:07 AM
Good going there: I tried that EPS32-A1S with the on-board codec and you can hear the MCU in the audio whenever it tries to calculate something. So an external codec or ADC/DACs with prober layout is a must.
I actually started with your Teensy->Esp32 audio lib repo :) Certainly saved a lot of work, thanks! Had to do a few changes to make it work with PCM3010, but all in all it's a good base platform for the esp.
Noise wise the pedal is pretty good i think. I haven't measured the noise levels yet, but after playing with it i haven't noticed any increased noise level compared to my other digital gear. Of course, i'm not even touching any radio stuff.
Quote from: free electron on November 26, 2021, 05:17:24 AM
I actually started with your Teensy->Esp32 audio lib repo :) Certainly saved a lot of work, thanks! Had to do a few changes to make it work with PCM3010, but all in all it's a good base platform for the esp.
Cool! I'm happy that people find a use for it :)
Your reverb sounds good btw.