NETFPGA
NETFPGA
NETFPGA
NetFPGA
Contents
Introduction 1. Network Hardware and Its Components 1.1 Implementing a New Design of NetFPGA 1.2 PC Components 1.2.1 MotherBoard 1.2.2 CPU 1.2.3 Host Memory 1.2.4 DVD Reader/Writer 1.2.5 MicroATX Chassis 1.2.6 Intel Pro/1000 Dual-port Gigabit PCI-Express & PCI-express x4 NIC 1.2.7 Hard Disk 1.2.8 Cat5e Or Cat6 Ethernet Cables 1.2.9 Other Misc. Parts 2. Network Software and Its Configuration 2.1 Register to download the NetFPGA Package (NFP) 2.2 Download the NetFPGA Package (NFP) 2.3 Installing an Operating System on the Host PC 2.4 Software Installation 2.5 Create NF2 Directory in your User Account 2.6 Reboot Your Machine 2.7 Install CAD Tools 2.8 Install Memory Modules for Simulation 2.9 Micron DDR2 SDRAM 2.10 Cypress SRAM 3. Network Hardware Software Design and Configuration
2
NetFPGA
3.1 NETFPGA Directory Structure 3.2 Creating New Project 3.3 Build System 3.3.1 Build Process 3.3.2 Projects and Reusable Modules 3.3.3 Register System 3.3.4 Performing Register Memory Allocation 3.4 Compile and Load Driver 3.4.1 Compile driver and tools 3.5 Run Selftest A1. Annexure I A2. Annexure II A3. Annexure III NetFPGA Handouts NetFPGA Technical Specification 1G (4x1G) NetFPGA Technical Specification 10G (4x10G)
F1. Fig 1 F2. Fig 2 F3. Fig 3 F4. Fig 4 F5. Fig 5
NetFPGA HARDWARE KIT Connect Loopback Cables Installing Loopback1 Installing Loopback2 Ethernet Test Cable
NetFPGA
NetFPGA
four Gigabit Ethernet Media Access Controllers (MACs) instantiated as a soft core on the FPGA. The NetFPGA also includes two interfaces with Serial ATA (SATA) connectors that enable multiple NetFPGA boards in a system to exchange traffic directly without use of the PCI bus. 1.1 Implementing A New Design Of NetFPGA:1. Use the hardware as is as an accelerator and modify the software to implement new protocols. 2. Start with the provided hardware from the official NFP (or from a third-party NFP), modify it by using modules from the NFP's library or by writing your own Verilog code, then compile the source code using industry standard design tools. 3. Then, implemented a bitfile can be downloaded to the FPGA. 4. Implement a new design from scratch: The design can use modules from the official NFP's library or third party modules to implement the needed functionality or can use completely new source code. 1.2 PC COMPONENTS:1.2.1 MotherBoard: Use Micro ATX (uATX) for small case Option 1: Gigabyte MA78GM-US2H mATX MB
AMD 780G Chipset / SB 700 / Rev 1.0 Includes one port of GigE on the motherboard Includes ATI Radeon HE3200 Video (leaves PCI-E slot open) DDR2 1200 DRAM support (supports RAM faster than DDR2 800) 2PCI+PCIe x1+PCIe x16 AM2+ Phenom II Support (allows for use of quad-core CPU)
Availability
NetFPGA
Fig 1 NetFPGA HARDWARE KIT Option 2: ASUS M2N-VM DVI - Micro ATX Motherboard
Item=N82E16813131214 from NewEgg.com : $59.99 Set the BIOS to use the on-board Video. Use the PCI-express bus for the NIC We built a dozen nf-test cube machines at Stanford using this motherboard in 20072008 combined with the dual-core CPU. If you can't still locate this (now) older board, use option (1)
1.2.2 CPU
We use fast dual or quad-core systems to maximize efficiency of developers Option 1: AMD X4 940 Quad-Core (3 GHz) AM2+ CPU
For use in AM2+ Motherboard, (1) above Item=N82E16819103471 from NewEgg.com: $189, or Item CP1-AM2-940 from TigerDirect.com, Model HDZ940XCGIBOX- $199
For use in AM2 Motherboard, (2) above Item=N82E16819103228 from NewEgg.com : $169
6
NetFPGA
Use the fastest clock frequency to minimize time to place and route the FPGA 1.2.3 HOST MEMORY DDR2 DRAM: 2GB to 4GB Minimum of 2GB is needed to efficiently run Xilinx CAD tools Item=N82E16820220144 from NewEgg.com : $28.99
Or larger
1.2.8 CAT5E OR CAT6 ETHERNET CABLES C: 74|&Sort=0&Recs=10 Category 5e or c:74|&Sort=0&Recs=10 Category 6 Ethernet Cables
Short-length: 1 foot ~= 30 cm, Blue (for host) Short-length: 1 foot ~= 30 cm, Orange (for host) Medium-length: 6 foot ~= 2m, White (for neighbor machine) Medium-length: 6 foot ~= 2m, Red (for neighbor machine) Long-length: 12 foot ~= 4m, Blue (for Internet)
NetFPGA
Power Strip for PC and monitor with localized plug: Use existing or purchase for $5 SATA Cable Typically included in a new motherboard box Item=N82E16812105911 from NewEgg.com : $2.49 $700 USD
TOTAL ESTIMATED COST TO BUILD A CUBE 2. NETWORK SOFTWARE AND ITS CONFIGURATION
The Beta release of the NetFPGA Package (NFP) contains the source code for gateware, system software, and regression tests. The NFP includes an IPv4 Router, a four-port NIC, an IPv4 Router with Output Queues Monitoring System, the PW-OSPF software that interacts with the IPv4 Router (SCONE), and the Router Kit which is a daemon that reflects the routing table and ARP cache from the Linux host to the IPv4 router on NetFPGA. 2.1 Register to download the NetFPGA Package (NFP)
NetFPGA
netfpga_full_x_x.tar.gz, which includes regression scripts and binary versions of the reference projects. Replace 'x' with the latest version. netfpga_lib.tar.gz, which includes all external java code needed by the router gui. 2.3 INSTALLING AN OPERATING SYSTEM ON THE HOST PC It supports of the popular Linux distribution Fedora as the operating system for the Host PC. In the past, it has supported of CentOS (a free variation of RedHat); however, it has now discontinued support for CentOS. To install Fedora 14:- (It supports all the version of NetFPGA) Citation As Follows
http://docs.fedoraproject.org/en-US/Fedora/14/html/Installation_Guide/
To install CentOS: - (Upto NetFPGA Version 3.0.0 and It doesnt support New Version of NetFPGA) Citation As Follows
http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/CentOSInstall
2.4 SOFTWARE INSTALLATION Packages are found at given website for downloading.
http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/InstallSoftware10 http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/InstallSoftware12 - Version 1.0 - Version 1.2
Log in as root Log in as root or 'su -' to root Install Java If running the command: java version Version of Java should be of at least 6.0. Otherwise, Java version should be upgraded for future inconvenience. Install NetFPGA Base Package Install NetFPGA yum repository and GPG Key - there are two different versions for CentOS 4 and 5. To determine your version, run the command:
9
NetFPGA
For CentOS 5:
rpm -Uhv http://netfpga.org/yum/el5/RPMS/noarch/netfpga-repo-1-1_CentOS5.noarch.rpm
Next, for both versions, run the following command to install the NetFPGA base package yum install netfpga-base Note that there may be some dependencies. Select 'y' to install these dependent packages. 2.5 CREATE NF2 DIRECTORY IN YOUR USER ACCOUNT Run the following script to copy the entire NF2 directory into your account (typically: /root/NF2). WARNING: Running this command WILL overwrite any existing NF2 directory or files in your user account! If you have files that you want to preserve, 'mv' your NF2 directory to another location, such as NF2_backup. To copy the NetFPGA directory and set the environment variables run the following command /usr/local/NF2/lib/scripts/user_account_setup/user_account_setup.pl It also adds the following environment variables to your .bashrc file. NF2_ROOT NF2_DESIGN_DIR NF2_WORK_DIR PYTHONPATH PERL5LIB 2.6 REBOOT YOUR MACHINE Reboot your machine in order to finalize the installation.
10
NetFPGA
2.7 INSTALL CAD TOOLS Tool provides the Verilog source code modules so that users can compile, simulate, and synthesize gateware for the NetFPGA. Developers have tested simulation and synthesis using a specific version of the Xilinx tools (as described below). Use of other versions of the tools (older or newer) is not supported. If you do not plan to rebuild the hardware circuits, you can skip installation of CAD tools. Install Xilinx ISE Xilinx: ISE Foundation, Version: 10.1 SP3
Install Service Pack 3 Install IP Update 3 Use of other versions of the tools (older or newer) is not supported. Obtain a license for the V2Pro TEMAC core from Xilinx.
Request "Full System hardware Evaluation" Allows use of the TEMAC for 30 days, 8 hour run-time
Academic users can request a donation of the core and CAD tools
Commercial users can purchase the core through their local sales representative.
Version SE 6.2G (Also tested with ModelSim DE v6.6). But ModelSim PE and the student version are windows-only so not so good for the NetFPGA Unix environment.
Allows simulation of circuits and viewing of simulated waveforms. Testbench software assumes use of this version of ModelSim.
11
NetFPGA
Debug with ChipScope To debug signals on the FPGA using an on-chip logic analyzer, install:
Version 9.1.02i or 10.1 SP3 (ChipScope 10.1 license comes with Xilinx 10.1 these days)
Allows visibility of verilog signals in synthesized verilog on the NetFPGA Requires use of a Xilinx USB Platform Cable or this compatible cable
2.8 INSTALL MEMORY MODULES FOR SIMULATION Tools have been provided of a script to automatically install the memory modules. This script is located in netfpga/lib/scripts/fetch_mem_models Run the script called "fech_mem_models.pl". You can also manually download and install the memory modules as show below. 2.9 MICRON DDR2 SDRAM Download the model from Micron
http://download.micron.com/downloads/models/verilog/sdram/ddr2/256Mb_ddr2.zip
Extract and copy ddr2_parameters.vh, and ddr2.v to netfpga/lib/verilog/core/common/src 2.10 CYPRESS SRAM Download the model from Cypress http://www.cypress.com/?docID=25033 Extract and copy cy7c1370d.v to netfpga/lib/verilog/core/common/src Rename cy7c1370d.v to cy7c1370.v 3. NETWORK HARDWARE SOFTWARE DESIGN AND CONFIGURATION The NetFPGA platform consists of many elements including the physical hardware, hardware designs that are downloaded to the FPGA, software associated with a particular hardware design, general software tools for interacting with the hardware, and the simulation and synthesis
12
NetFPGA
environment for building new designs. Developers will most frequently develop new hardware designs to run on the FPGA and software for use with particular hardware projects. Projects are complete designs, consisting of a hardware component, tests (simulations and hardware regression tests), and associated software components. Modules are small reusable hardware units that are incorporated into projects. The hardware component for a project is typically built by interconnecting a number of reusable modules and some project-specific HDL code. Some projects, notably some of the reference designs, are built entirely by assembling the reusable modules. The simulation and regression tests associated with a project are used to specify the features of the design and demonstrate the correctness of that design (as explained in Regression Tests). The software associated with a project allows the user to interact with the hardware in a meaningful way; for example in the Reference Router project; the software provides a commandline interface for manipulating the ARP and routing tables inside the hardware. Modules consists of HDL code, specification of IP blocks built using the Xilinx Core Generator, and a specification of any registers that the module exposes. Registers and associated information are specified using an XML-based system; this system takes care of allocating memory for the registers within each module when the modules are integrated into projects. 3.1 NETFPGA DIRECTORY STRUCTURE The directory structure of the NetFPGA source tree is as follows:
NetFPGA bin bitfiles lib C java Makefiles Perl5 python release scripts {Base directory} {Scripts for simulation/synthesis/register gen} {Compiled hardware bitfiles} {Libraries and software tools} {C libraries/programs} {Java libraries} {Makefile templates used for sim/synth} {Perl libraries} {Python libraries} {XML files for packaging} {Utility scripts} 13
NetFPGA
verilog contributed core xml projects <project> doc include lib src sw synth test {Contributed Verilog modules} {Official Verilog modules} {XML schemas} {project directory} {contributed project} {documentation} {project.xml, project specific module XML} {Perl and C headers} {non-library verilog} {Project-specific software} {Synthesis directory (contains all .xco files)} {Unified (Hw/Sw) tests}
3.2 CREATING A NEW PROJECT A new project is created by following these steps: 1. Create a new project directory inside netfpga/projects 2. Update the NF_DESIGN_DIR directory to point to the new project directory. Example (Bourne shell syntax): export NF_DESIGN_DIR=$(HOME) / netfpga/projects/my_first_project (See the guide for more information.) 3. Create the following directories inside the project directory: include, src, synth, and test. Optionally create doc, lib, and sw. (Note: lib will be automatically created when the register generation tool is run.) 4. Create a project.xml file inside the include directory. See the Register System section below for more information. 5. Add any library modules to the project.xml file. 6. Write any project-specific Verilog and place inside the src directory.
14
NetFPGA
7. Write module-specific XML files for any new modules you have written and place in the include directory. 8. In the new python testing infrastructure, simulation and hardware (previously regression) tests have been unified, so a test can be written once and run as either a simulation or hardware test, unless hardware specific functions are needed. Tests should be placed in a project's test directory. Test directories should be namedboth_<major>_<minor> if they can be run in both simulation and hardware, hw_<major>_<minor> if the test can only be run as a hardware test, andsim_<major>_<minor>; if the test can only be run as a simulation test. Neither major or minor can have underscores in the name, nor can they be blank. 9. Copy the Makefile from the reference router synth directory into your synth directory. 10. Synthesize the design by running make inside the synth directory. 11. Write hardware tests and place them in the test directory.Run
with nf_test.py (See Verification for more information.) 12. Write any software and place inside the sw directory. 13. Add documentation to the doc directory. 14. Contribute your project if you think your project may be useful to others. See the Develop page for more information. Note: The act of running simulations or synthesizing the project causes the registers to be regenerated. Use the nf_register_gen.pl tool if you want to regenerate registers without running simulation/synthesis. 3.3 BUILD SYSTEM The build system manages the process of compiling projects for simulation and synthesizing projects for download to the FPGA. 3.3.1 BUILD PROCESS The following steps are undertaken during a build: 1. Query the register system for a list of all modules used by the project. 2. Instruct the register system to generate the registers. This involves reading the register descriptions for each module, allocating memory for all registers, and outputting that resultant allocation for use in Verilog, C, and Perl. 3. Generate cores associated with each module and the project.
15
NetFPGA
4. Compile/synthesize the code associated with each module and the project. 3.3.2 PROJECTS AND REUSABLE MODULES As previously mentioned, the NetFPGA platform supports reusable hardware modules that can be incorporated into projects. The build system makes using modules extremely simple: a developer simply specifies which modules that they'd like included in their project and the build system allocates addresses for registers within each module, generates any necessary IP blocks, and includes the necessary source files during compilation. Many new developers do not realize that modules are included as part of the compilation process; this causes confusion to many new developers when they look at the src directory of a project and see very few files, or even none at all. For example, reference router's src directory is completely empty; all of the functionality of the reference router is provided by the reusable modules. The set of modules that a project uses is specified in the project's project.xml file. The nf:use_modules section of the XML files specifies which modules to include. Part of the use_modules section from the reference router is shown below: core/io_queues/cpu_dma_queue core/io_queues/ethernet_macore/input_arbiter/rr_input_arbiter core/nf2/generic_top core/nf2/reference_core ... This section of code instructs the build system to use these modules: CPU DMA queues, Ethernet MAC, the generic top-level Verilog file ( generic_top), and the reference core. The reusable modules are located in the netfpga/lib/verilog directory. For example, the CPU DMA queues (specified as core/io_queues/cpu_dma_queue in the above code snippet), is located in netfpga/lib/verilog/core/io_queues/cpu_dma_queue. 3.3.3 REGISTER SYSTEM This section provides an overview of the register system. Detailed information about the register system can be found on the Register System page. The register system provides a mechanism for:
specifying the registers provided by each module specifying the modules used by each project generating a register map/memory allocation for each project
16
NetFPGA
Information for each project (eg. name, list of modules, location of modules in memory space) and each module (eg. name, list of registers) is specified in an XML file. The register generation tool (=nf2_register_gen.pl) reads the XML file of the project and the XML files of the included modules, performs memory allocation, and then outputs a set of files with the memory allocation/register map to files for use in Verilog, C, and Perl. 3.3.4 PERFORMING REGISTER MEMORY ALLOCATION The register system tool to perform register memory allocation is invoked automatically as part of every simulation and synthesis run. The tool can be invoked manually at any time with the nf_register_gen.pl command. Common command line options include:
--project <project> -- specify a particular project (instead of using the current project specified via the NF_DESIGN_DIR variable).
3.4 COMPILE AND LOAD DRIVER 3.4.1 Compile driver and tools Step1: Compile
Install the driver and reboot. The driver will be stored in /lib/modules/`uname r`/kernel/drivers/nf2.ko make install Step3: reboot After reboot log in as root. Verify that the driver loaded: lsmod | grep nf2 Step4: Verify NetFPGA Interfaces
17
Reboot
NetFPGA
Verify that four nf2cX interfaces have successfully loaded: ifconfig -a | grep nf2 Step5: Reprogram the CPCI
Run the cpci reprogramming script usr/local/sbin/cpci_reprogram.pl --all (to reprogram all NetFPGAs in a system) Every time you restart the computer, you need to reload the CPCI! To have the CPCI reprogrammed when the computer boots add the following line to /etc/rc.local /usr/local/netfpga/lib/scripts/cpci_reprogram/cpci_reprogram.pl --all If the NetFPGA refuses to send packets, and the regression or selftest is failing, make sure you've reprogrammed the cpci. 3.5 RUN SELFTEST The NetFPGA self-test is an FPGA bitfile and software that ensures that all of the components on your platform are fully functional. The self-test consists of both an FPGA bitfile that contains logic and interfaces to external components as well the software that displays the results. The self-test excercises all of the hardware in parallel. The test continues to run repeatedly until terminated by the user. The self-test was run at the factory just after the cards were manufactured. Cards are not distributed unless they completely pass all functions of the self-test process. The self-test bitfile performs rigorous testing of the SRAM and DDR2 DRAM to ensure that all memory lines can be properly written to and read back with the same data. Multiple data patterns are used to ensure that no address or data lines have faults. The network test sends bursts of packets on the Ethernet interfaces and the loopback cables are put in place so that packets can be read and compared to the data that was transmitted. The SATA loopback test transmits data using the Multi-Gigabit I/O lines (MGIOs) to ensure that data can be reliably transmitted on the high-speed I/O interfaces. The DMA test exercises the
18
NetFPGA
PCI Controller (CPCI), the VirtexII, and the PCI bus to ensure that large blocks of data can be sent between the NetFPGA the host computer's memories. The selftest bitfile runs all of the tests above in parallel and continuously runs until it is terminated. The self-test software displays the results of testing on a console. Step1: Connect loopback cables
NetFPGA
Fig 4 Installing Loopback2 Step2: Type for i in `seq 0 3`; do ifconfig nf2c$i up; done Step3: Type nf_download ~/netfpga/bitfiles/selftest.bit Step4: Run Selftest Load Self-Test Bitfile Bring nf2cX interfaces up
If you have connected a SATA cable to the NetFPGA, type the following command. ~/netfpga/projects/selftest/sw/selftest Otherwise, type the following command. ~/netfpga/projects/selftest/sw/selftest -n Step5: Run Regression Tests
20
NetFPGA
The regression test suite is a set of tests that exercise the functionality of the released gateware and software. On a fast machine, this test should take approximately 10 minutes. The features exercised by regression test suite are the only features we will try to provide support for. Additional features might be available and functional in the released gateware, but they are not supported. For more information on the features we support, as defined by tests, see the following: The NIC supports a set of features The details of how each feature is tested are described in the NIC Regression test document available both on the Wiki and Web. The Reference Router (RR) supports a set of features The details of how each feature is tested are described in the RR Regression test, a large document available both on the Wiki and Web. Step6: Connect Ethernet Test Cables
Connect 'eth1' to 'nf2c0' (c0 is the port closest to the mainboard) Connect 'eth2' to 'nf2c1' (c1 is the port one away from the mainboard) The location of your eth1 and eth2 ports may vary depending on your NIC The photo below shows the configuration of a nf-test machine
Log in as root or 'su -' to root using an X session, because we will be testing the GUI Scone.
21
NetFPGA
Step8:
Download the reference bitfile to the NetFPGA board: nf_download ~/netfpga/bitfiles/reference_router.bit Step9: Run Regression Test Suite
Run the regression test suite. The tests should take about 10 minutes total. Note: Prior to release 2.0.0 the command was nf21_regress_test.pl ~/netfpga/bin/nf_regress_test.pl Step10: Run regression scripts on new bitfile
If you installed the CAD tools, you should run this test to verify that you can build a new circuit. Skip this step if you do not plan to modify hardware. Step11: Synthesize reference_router bitfile, from source
Note: This step will take about 45-60 mins. This can be used to verify the setup of the machine for synthesis. You will need to have the NetFPGA Beta Plus package. The Beta (not Plus) package does not include the sources for this step. If you are a hardware developer and would like to synthesize your own NetFPGA Router hardware using the Verilog source code, follow the steps below. To synthesize FPGA hardware, you will need to have all of the FPGA Development tools installed. Login, either direct in an X session or via ssh -X. This step causes ~/nf2_profile, plus environment variables, to be sourced. You may say "But I'm not running anything graphical!" and you'd be right. Unfortunately, even when called with no gui, the Xilinx tools require X to be running. A bugreport on this issue has been filed to Xilinx. Set up the Xilinx ISE tools (see Xilinx's website for instructions). Make sure the Xilnx tools are in your path and that the XILINX environment variable is set. Go to the synthesis directory for the reference_nic and run make. This step should take under an hour on a well-endowed machine. cd ~/netfpga/projects/reference_router/synth
22
NetFPGA
time make Verify the reference_router bitfile (nf2_top_par.bit) has been created. ls | grep nf2_top_par.bit Step12: Load new bitfile
Download the fresh bitfile to the NetFPGA board: nf_download nf2_top_par.bit Step13: Run regression-test suite on new bitfile
Re-run the regression test suite. Note: Prior to release 2.0.0 the command was nf21_regress_test.pl ~/netfpga/bin/nf2_regress_test.pl
23