Non-stop data write on a storage device

[Nils Roos]

OK, that is pretty clear. If the pipe needs to soak up some tens of seconds worth of incoming data at ~4MB/s, it needs to be BIG.

Unfortunately, I have no idea if anything could be done to tune writing to usb stick on the side of the filesystem or the stick itself.


I find it a bit surprising that writing to an nfs share also has some huge spikes, does anybody have any insight on that ?

[melko]

Also a weird thing is that the write call as far as I know should be asynchronous, so isn't waiting for the data to be actually written on the device; in fact usually performances drastically decrease when opening a file with the O_SYNC flag or when mounting a filesystem with the sync parameter. Now I don't really get why then it's waiting up to tens of seconds there.

The next thing I'll try is to write directly on the raw device of the usb stick, performances should improve when no filesystem is involved.
I remember when I did some test with SD cards and microcontrollers, I could write on a raw block (512 bytes) of the SD with a constant speed of 600KB/s (writing a single block took around 800us) and that was done through SPI (so just one data line) and a slow clock (can't remember now exactly the value but usually it's under 16MHz). So with a real processor using SDIO (4 data bit and a clock that can go up to 25MHz) I'd expect something better.

[Nils Roos]

Also a weird thing is that the write call as far as I know should be asynchronous, so isn't waiting for the data to be actually written on the device; in fact usually performances drastically decrease when opening a file with the O_SYNC flag or when mounting a filesystem with the sync parameter. Now I don't really get why then it's waiting up to tens of seconds there.

I think, write is asynchronous only to the device itself. If the filesystem needs to buffer the data, because the device is busy, and there is no free buffer available, it's my understanding that write is allowed to block.

The next thing I'll try is to write directly on the raw device of the usb stick, performances should improve when no filesystem is involved.

I'd certainly like to know how that goes.

Happy hacking
Nils

[melko]

And here's the try on the raw block device (took 91s, so average of 11MB/s):

time-raw.png

EDIT:
just wrote this way 20GB of data at 7.45MB/s rate without any issue.

[christoph]

I tried to recunstruct your ideas, build the whole target based on your dma-branch and after stuggling for weeks I finally got the RP booting. But the c-code examples for dma and listener still don't work for me.

Compiling dma.c works, but trying to comile listener.c gives me:

Code: Select all

[chr@thinkpad dma]$ arm-xilinx-linux-gnueabi-gcc -o listener -g -std=gnu99 -Wall listener.c
listener.c: In function 'main':
listener.c:36:2: warning: implicit declaration of function 'read' [-Wimplicit-function-declaration]
  while( (size=read(fd, buffer, PACKET_SIZE)) ){
  ^
listener.c:37:3: warning: implicit declaration of function 'write' [-Wimplicit-function-declaration]
   write(fd_out, buffer, size);
   ^
listener.c:41:2: warning: implicit declaration of function 'close' [-Wimplicit-function-declaration]
  close(fd_out);
  ^
[chr@thinkpad dma]$ 

When I include <unistd.h> to listener.c it works, but on the RedPitaya I get these errors when executing:

Code: Select all

redpitaya> ./dma
0x00000000
failed to open target: No such file or directory
redpitaya> ./listener test.txt
error opening fifo
redpitaya> 

What did I forget/did I do wrong? How can I check if the FIFO was successfully build and exported to devicetree?

Thanks, Regards.
Christoph

BTW: Great job !!!

[melko]

you probably forgot to create the named pipe on the filesystem,
both listener and dma assume that the fifo is present in the directory from where you run them, you can see in the code that the fifo name is hardcoded there.
So before running them just execute "mkfifo fifo".

[christoph]

Thanks, it's working, you made my day !!! Wasn't familiar with the linux pipe system.

[melko]

For the record I've moved the development in the adc branch quite some time ago,
principles are the same, though as the branch name suggests I'm now using a real signal
taken from the ADC instead of the fake continuous one (an event is triggered when the adc counts has a peak above a chosen threshold).
Plus I've written a version of the listener/dma combo that uses tcp sockets to stream data over
the ethernet interface, with this version of dma.c/listener_tcp.c + client_daq.c I've managed to reach a ~20MB/s data rate.
Actually with an HTTP transfer so far I've seen 40MB/s can be reached, so indeed there's room for improvements, but for now I'm quite happy of this design (except for some quirks).

[christoph]

Thanks for the hint, very interesting

I see two different fifos in your design. The first one is the AXI4-stream data fifo in system block desing and the second one is the fifo64 in custom.v with native interface. I suggest they both are synthesized to BRAM? Is there a reason for using two fifos instead of just one big?

[melko]

Thanks for the hint, very interesting

I see two different fifos in your design. The first one is the AXI4-stream data fifo in system block desing and the second one is the fifo64 in custom.v with native interface. I suggest they both are synthesized to BRAM? Is there a reason for using two fifos instead of just one big?

You are right, they both are synthesized on BRAM,
the reason I've used two fifos is because the vivado interface let you choose the fifo size in powers of two. The AXI4-stream fifo is actually 16384x64bit, and the next step, which would be 32768 was to big for the fpga resources; fifo64 in the custom.v design is just 8192x64bit, so the total is 24576x64bit, which barely fits.

Another reason is that the AXI4-stream fifo needs a TLAST signal to know the boundary of a packet, so I took advantage of fifo64 using a counter between the two fifos. Indeed it can be done in the same way even with just the AXI fifo, so the main reason why I've used two fifos is just the size matter.