Ethernet connection dies after too many connection attempts

[lene85]

We are observing a 'bug' with our RedPitayas:

Our software establishes an ssh session with the redpitaya and then launches a server (written in C, listening on port 2222) and connects to it. In certain situations, our software rapidly attempts to kill, restart and connect to the server on port 2222, e.g. 10 times per second, indefinitely (this is somewhat a bug on our side). It can also happen that we attempt several ssh connections per second.
In this situation, we find that the redpitaya completely stops accepting connections of any sort:
- none on port 2222
- no ssh
- even ping stops working.

In this situation, only rebooting the redpitaya by disconnecting the power supply helps to get it working again. This is obviously impractical if the RedPitaya is used remotely, as in our case. And apart from a firewall, I don't see how this could be an intended behavior. The problem is present both in ecosystem 0.92 and 0.95.

I suspect the problem is a software issue, but it could well be hardware-related as well imho.

Does anyone have any hints on how to debug this?

[pavel]

Does anyone have any hints on how to debug this?

• use the latest Linux kernel provided by Xilinx (4.4.0-xilinx) (for example from http://pavel-demin.github.io/red-pitaya ... -ecosystem)
• connect to Red Pitaya via the USB/serial console and check what happens with the system when it stops accepting connections. I'd try to check the output of the top, ps aux, dmesg, lsof commands and I'd look at the logs in /var/log. If system hangs completely, there could be some error messages output directly to the USB/serial console.

[lene85]

Hi Pavel,

thanks for your suggestion. I tried it, but unfortunately the USB console also stops responding when the RedPitaya stops reacting to ethernet requests. I have therefore no console output available. Looking at the logfiles, it seems the RedPitaya doesn't have time to log anything before going crazy. The last information i see is this:

last message before restarting:
Oct 18 15:18:55 redpitaya sshd[1546]: pam_unix(sshd:session): session opened for user root by (uid=0)
first message after restart (the clock always goes back to the same value for some reason):
Oct 18 15:17:03 redpitaya sshd[1202]: Server listening on 0.0.0.0 port 22.

No other logfile has entries later than 15:18:55.

Furthermore, I realized that this problem only occurrs in this scenario:

1) Server process 1 reads more or less permanently data from the FPGA memory and sends it to Client 1
2) Client 2 demands to reflash the FPGA via cat bitfile > /dev/xdevcfg

So somehow, reading from FPGA memory while reflashing the FPGA kills the RedPitaya to the point that it doesn't respond to any kind of request. This problem is somewhat more acceptable since I can write a workaround (prevent reading explicitely while fpga is flashed through a message posted in a logfile), but I would still be grateful for any suggestion on how to avoid the total shutdown of the RedPitaya in this case, if possible.

Btw, I can reproduce this problem by continuously reading from the FPGA memory with my code and starting the official scope application, which - I guess - reflashes the FPGA as well.

[pavel]

So somehow, reading from FPGA memory while reflashing the FPGA kills the RedPitaya to the point that it doesn't respond to any kind of request.

I also stumble over this problem from time to time. It even happens when I try to read/write from/to a wrong memory address or when the FPGA clock is off.

I've just found a good summary of this problem on stackoverflow:
http://stackoverflow.com/a/36576693

The CPU will hang if it issues memory transactions against the programmable logic and there is not a response. There is no built-in bus error or timeout mechanism.

Here are some of the reasons I have had this problem:

Programmable logic is not programmed
AXI slave is not responding to the right address
Bug in AXI slave

So, looks like the only solution is to carefully configure the programmable logic and to carefully write programs that communicate with the programmable logic.

[lene85]

Thanks for the info. At least we somewhat understand the problem. Preventing read/write access during flashing proved a sufficient solution for me at the moment. The only time my RedPitayas crash now is when I charge a corrupted bitfile. Thanks again!

[xyefa]

I think I may have encountered this problem today - The Ethernet died. I figured the board was bad, and I had a spare. I inserted my image and I could not ping, though the Ethernet seemed to be alive. I will investigate it carefully during the week.

[xyefa]

I have the issue where Ethernet is completely dead. I connect two computers directly assign private IPs and try to ping and get 100% packet loss. What I have found that fixes this if I ifconfig down and then ifconfig up then the link comes back. I have to tinker with this but I was bitten by this on Friday and this weekend and was finally found a fix (it seemed simple). It could be the auto-negotiation ... who knows.