Page 1 of 29

New feature: high speed continuous recording

Posted: Sat Oct 18, 2014 1:22 am
by Nils Roos

For ecosystem version 0.92 and 0.93: ecosystem, components, github

For OS versions of 0.94 and later: rp_remote_acquire + bitstream

For OS versions of 0.96 and later you also need: boot loader replacements from ecosystem 0.94
(replaces the logic analyzer setup, so you can not use the logic analyzer when you boot with these components)
When the kickstarter backers unlocked the 512MB stretch goal, I bet many thought "whoa, I can store a full second of 2 channel 14bit samples at 125MS/s in there". I know I did.
As it turned out, both the RedPitaya scope application and FPGA logic don't work that way.

But this is no longer true, because now there is the "DDR Dump" extension to the RedPitaya.

It comes in two parts:
  1. An extension to the FPGA design, that adds a direct channel from the ADC to main memory (while maintaining compatibility with existing apps).
  2. A kernel module that provides DMA-capable memory to the FPGA and allows user-space applications access to the samples.
The DDR Dump is designed to use the buffers as ringbuffers and, once started, will write into them over and over without gaps in the data, until stopped. From the POV of the CPU the samples just appear in memory, and there are registers available that tell it which block in the buffer is currently being updated. Interrupt notification is not yet provided, but I plan to add it at some point.

So, how about a demonstration ?

If anybody wants to take the new function for a ride, download the binaries (files with md5sums), copy them to your RedPitaya's /tmp directory and do the following:
(I hope you will understand that I can accept no responsibility for any and all consequences that arise from the use of the provided binaries, and by using said binaries you implicitly agree to this)

Edit: I just learned (thanks, PiccoLas) that the kernel module is only compatible with ecosystems 0.92-388 and later. There is now a download for earlier ecosystems available, and the pre-0.92-388_rpad.ko binary as a separate file.

Code: Select all

redpitaya> killall nginx
redpitaya> cat /tmp/ddrdump.bin > /dev/xdevcfg
redpitaya> nginx -p /opt/www
redpitaya> insmod /tmp/rpad.ko
This installs the FPGA bitstream and the driver. After the bitstream is loaded, the yellow LED0 will stop flashing. The driver installs /dev/rpad0, which you can use eg. as a source for cp:

Code: Select all

redpitaya> cp /dev/rpad0 /tmp/adcdata.txt

Code: Select all

redpitaya> cat /dev/rpad0 > /tmp/adcdata.txt
redpitaya> cat /dev/rpad0 >> /tmp/adcdata.txt
Each time you issue the copy command (to be precise, each time the device is opened anew), the driver will do a one-shot acquisition of one megasample length with the current scope settings. The samples are written as hex numbers to the target textfile, channel A data first, then channel B data.
Bear in mind that the acquisition length for the demo is always 1MS, so using the demo while running at high decimation will take a while.
By the way, it is possible to do the copy while the scope application is running (though it sometimes interferes with the scope's trigger).

If you are on a local console (via USB), you will see diagnostic output. If you are connected remotely, you can use dmesg to see the messages.

A note on buffer size:
The FPGA extension will happily use all available RAM if you tell it to do so. I chose to use smaller buffers (2*2MB) because linux does not support allocation of very large chunks of contiguous physical RAM. There are ways to use larger buffers and still play nice with linux and I will gladly discuss specifics with interested users.

This is very much a work in progress. Contributions of opinion, support or ideas are very welcome.

Current state of the project and outlook:
  • The FPGA functions have been simulated, tested, and found to work reliably, but new capabilities can and will be added over time. This is in large parts depending on community requests and contributions.
  • The kernel module is a proof-of-concept right now, and has been designed to allow testing of the logic, and not much more. My plans are to expand its scope to act as a driver for all of RedPitaya's functions. Progress in this area will also be heavily influenced by user feedback.
  • There is currently no web application to access the new functionality. I invite all interested developers to have a go at it and I will do my best to help with advice and code.
You can find the full source in the branch dev_ddrdump of my RedPitaya fork.
(Please note that the kernel module has to be built by hand for now. Go to OS/driver/rpad/src and execute "make" to do that after at least the linux kernel was built successfully.)

Have fun (and please tell me what you think)

Re: New feature: high speed continuous recording

Posted: Sat Oct 18, 2014 4:10 pm
by Matthias Weber
Great work, I need to try it when I'll be back in the lab.

Re: New feature: high speed continuous recording

Posted: Sat Oct 18, 2014 4:31 pm
by Nils Roos
Thank you.
I need to try it when I'll be back in the lab.
Please do. And if you have any suggestions, I will gladly consider them.
I am currently trying to figure out a convenient way to expose the functionality to applications (eg. ioctl or write access to configuration area). One thing I am fairly certain will be useful is to provide the ability to mmap the buffers.

Re: New feature: high speed continuous recording

Posted: Mon Oct 20, 2014 9:28 pm
by bexizuo

what about sending to LAN ?
NFS is not suitable for that CPU ...
so if it will be simple stream (maybe in jumbo frames) it could be useful ...
there is no netcat by default so i tested only NFS and the speed was good but there are some delays during CRC calc on ETH packets ... so if there big cache it will catch the time ...

Kind regards ...

Re: New feature: high speed continuous recording

Posted: Mon Oct 20, 2014 10:09 pm
by Nils Roos
Thank you.

Could you please elaborate on what exactly you did ?
If you just used the driver as I outlined, one huge cause of delay is the conversion to ASCII that happens while you read data from it. I built it that way to be able to quickly see if the transmitted data made sense. I can make you a version that delivers binary data, if you'd like. Should be much quicker.

A smaller source of delay is the overhead incurred by using a char device's most common transfer method, the read() call. Next in my queue is the ability to mmap the buffers. That way you can hand them to sockets without even copying the data in your application code.

Re: New feature: high speed continuous recording

Posted: Mon Oct 20, 2014 10:52 pm
by bexizuo
Thank You for fast reply :shock:

i was testing only LAN, not measurement, only dd from /dev/zero to nfs,

well ... i'm having development with different FPGA at different speeds,
i'm working on project where is 30MSPS processed live (not video), but i had difficulties with performance of actual ARM processors,
i decided to develop offline ... record on local disk and after session transfer it to higher performance computer using network,
i tried to transfer data live using network but there were too many IRQs on CPU and my ARM CPU was not fast enough to handle IRQs,
and back to redpitaya: if i could get data to high performance computer using network for example ... i could do some comparison tests of analog/converter part,
i'm having on my mind that sample rate 30MSPS will not be possible to get but i can have special testcase at sample rates that are available on redpitaya,

please could you have a look on it ? for better performance i suggest to send binary format and switchable only one analog channel , and probably UDP because of ACK's confirmations in TCP can load CPU, ... i think that it could achieve maximum 30MBytes/s ... so it could be 15MSPS on GIG.ETH., but if it will use jumbo frames maybe it could be faster because CRC will be calculated after more bytes ... 3 times less often,

i'm interested in implementation of LAN network, but i'm not very skilled on linux development ... if i could help you i will try to help you, please let me know ...

Kind regards,

Re: New feature: high speed continuous recording

Posted: Tue Oct 21, 2014 11:18 am
by Nils Roos
Early on I tested the speeds that are achievable over network. I got to 80MB/s with UDP using just standard sockets. So with one channel your 30MS/s will be within reach.

Adding a new decimation factor to get close to 30MS/s sample rate is pretty straight forward, too.

I will try to cobble something together that spews the data out over UDP and let you know.

Re: New feature: high speed continuous recording

Posted: Tue Oct 21, 2014 11:24 am
by bexizuo
You are GORGEOUS !!!

should i need to reflash something for adding new decimation ? ( FPGA part )
or is it just configuration from linux part ?

Re: New feature: high speed continuous recording

Posted: Wed Oct 22, 2014 5:43 pm
by BrandonKinman
Thanks for all of the hard work!

I'll be incredibly happy when I am able to just use a TCP or UDP to stream continuous samples over the network!

Let me know if there is anything I can help you with re: mmap, or anything else for that matter!

Re: New feature: high speed continuous recording

Posted: Wed Oct 22, 2014 7:38 pm
by Nils Roos
Updated to address incompatibility with ecosystems before 0.92-388 (download link for users of older ecosystems)

You only need to insert the new fpga bitstream at runtime (can be done from a script). But if you want to install it as your default setup, you will need to replace some files on the SD card.

I'll get in touch forthwith :mrgreen: