Data streaming via HP AXI to DDR3

Applications, development tools, FPGA, C, WEB
Post Reply
jlarksman
Posts: 4
Joined: Fri Nov 11, 2016 10:08 am

Data streaming via HP AXI to DDR3

Post by jlarksman » Fri Jan 20, 2017 4:35 pm

We have been working on a project to use the Red Pitaya as a fast, multi-channel lock-in amplifier. The user can choose which signals to route to the output DACs (e.g. ch1 X, ch2 Y etc); however we simultaneously would like to do away with our oscilloscope hardware and stream the lock-in data directly from the Red Pitaya.

The data we are collecting naturally falls into a "block"-type acquisition (i.e. a certain number of samples each time we receive a trigger), and currently we're storing each block in an area of memory over the High Performance AXI 32b Slave Port, at offset 0x30000000. A TCP socket server running on the host linux side then reads the data using mmap on /dev/mem, at the given offset.

Unfortunately, this causes the OS to be unstable (after a variable amount of time, we start to see spontaneous segfaults etc) -- we assume that this is because we're writing to a section of memory that the OS itself is using. We have modified u-boot.scr to include memmap=64M$384M as a boot argument, and changing the offset to match in Vivado (0x1800_0000); unfortunately this only causes the OS to freeze more quickly.

Any pointers on how to solve this problem would be greatly appreciated! On the plus side, the project works very well until it locks up the OS!

pavel
Posts: 790
Joined: Sat May 23, 2015 5:22 pm

Re: Data streaming via HP AXI to DDR3

Post by pavel » Fri Jan 20, 2017 4:59 pm

jlarksman wrote:we're storing each block in an area of memory over the High Performance AXI 32b Slave Port, at offset 0x30000000.
I'm not sure if I can help but I'm curious to know what happens when you write to an address that's outside of the DDR3 RAM range. If I'm not mistaken DDR3 RAM range is 0x0 - 0x20000000 and 0x30000000 is outside this range.

Here is a short description of how I do something similar in my projects:
  • I limit the amount of RAM accessible to OS by applying patches to devicetree and to u-boot. For example, if you need to limit OS to the first 256 MB of RAM then memory/reg should be set to <0x0 0x10000000> and fdt_high should be set to 0x10000000.
  • then I use the value from the previous step as the starting address for my direct memory access from FPGA.
By default I set this limit to 0x18000000 and I think that it's set to the same value in the configuration currently distributed by Red Pitaya. I think that 0x18000000 can be replaced with to 0x08000000. This way, OS would have access to the first 128 MB and your FPGA configuration could safely write to the remaining 384 MB.

jlarksman
Posts: 4
Joined: Fri Nov 11, 2016 10:08 am

Re: Data streaming via HP AXI to DDR3

Post by jlarksman » Fri Jan 20, 2017 5:55 pm

Dear pavel,

Thank you for your quick reply! In our experience, writing outside the 0x0 -> 0x2 range of the DDR3 results in an unstable OS, but somehow the FPGA and host OS are able to communicate. We did wonder if it wrapped around to somewhere else in the memory space? Nevertheless, data we wrote to that address with the FPGA would reliably read on the host using mmap of /dev/mem at the same offset!

I have made the changes that you have suggested:
  • modified u-boot.script, run

    Code: Select all

    mkimage -A arm -T script -C none -d ./u-boot.script ./u-boot.scr
  • used dtc to get the devicetree file, and modified reg from

    Code: Select all

    <0x0 0x20000000>
    to

    Code: Select all

    <0x0 0x18000000>
The files at the root of the SD card were then replaced with my updated versions. Unfortunately, I can't see any changes from the linux side - it still reports itself as having 480 MB of memory etc. Are there further changes I need to make to ensure the OS notices these updated files?

pavel
Posts: 790
Joined: Sat May 23, 2015 5:22 pm

Re: Data streaming via HP AXI to DDR3

Post by pavel » Fri Jan 20, 2017 10:18 pm

I'd suggest to try just three modifications:

jlarksman
Posts: 4
Joined: Fri Nov 11, 2016 10:08 am

Re: Data streaming via HP AXI to DDR3

Post by jlarksman » Mon Jan 23, 2017 11:13 am

Dear pavel,

Thanks for your continued help. I have tried as you suggested, but to no avail! I've tried it on clean versions of both the stable and beta OS, with the same outcome each time -- kernel panic.

I'd like to isolate whether the problem is on the linux end, or the FPGA end. When I make the modifications to devicetree.dtb and u-boot.scr, linux doesn't report any changes in memory tables. If I run

Code: Select all

dmesg | less
I get the following near the top:

Code: Select all

[    0.000000] Virtual kernel memory layout:
vector  : 0xffff0000 - 0xffff1000   (   4 kB)
fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
vmalloc : 0xde800000 - 0xff800000   ( 528 MB)
lowmem  : 0xc0000000 - 0xde000000   ( 480 MB)
pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
modules : 0xbf000000 - 0xbfe00000   (  14 MB)
.text : 0xc0008000 - 0xc0841544   (8422 kB)
.init : 0xc0842000 - 0xc0886000   ( 272 kB)
.data : 0xc0886000 - 0xc08e99c0   ( 399 kB)
.bss : 0xc08e99c0 - 0xc092a014   ( 258 kB)
and if I run

Code: Select all

head /proc/meminfo
I get

Code: Select all

MemTotal:         445168 kB
MemFree:          378064 kB
MemAvailable:     412872 kB
Have you ever noticed whether these two outputs change when the memory restriction is working correctly? I'm surprised that lowmem is still reporting 480 MB after the change, but possibly I'm looking in the wrong place?

Thanks again!

jlarksman
Posts: 4
Joined: Fri Nov 11, 2016 10:08 am

Re: Data streaming via HP AXI to DDR3

Post by jlarksman » Mon Jan 23, 2017 1:54 pm

Just to follow up, I have solved this by appending

Code: Select all

mem=384M
to my bootargs in u-boot.scr. I am unsure why the other approaches didn't work. However, now /proc/iomem, /proc/meminfo and dmesg all report the change in memory size, and my segfaults / kernel panics have disappeared.

Thanks for all the help!

Post Reply
jadalnie klasyczne ekskluzywne meble wypoczynkowe do salonu ekskluzywne meble tapicerowane ekskluzywne meble do sypialni ekskluzywne meble włoskie

Who is online

Users browsing this forum: No registered users and 17 guests