DARPA Spectrum Challenge Q&A

Spectrum Challenge Q&A

Questions regarding the Challenge rules should be emailed to spectrumchallenge@darpa.mil. Answers to questions that are deemed of potential interest to other entrants will be posted here. Your question may be modified slightly for generality prior to posting. Do not send questions about specific designs or strategies unless you want the answers to be publicized.

Questions regarding the ORBIT testbed should be emailed to dsc-questions@orbit-lab.org. ORBIT’s Spectrum Challenge Q&A page is http://www.orbit-lab.org/wiki/DSC/QandA.

This Q&A page will be updated periodically throughout the Challenge. Please check this page before sending any questions to the Challenge mailbox.

Last updated: 2/15/2014

Q. Does scoring occur over the entire duration of a match?

A. For Competitive matches, the scoring server stops as soon as one team completes the file transfer. Otherwise, scoring continues for both teams until the time limit is reached. For Cooperative matches, scoring continues for all teams until the time limit is reached.

Q. Are the scoring metrics the same as those in the Preliminary Tournament Event?

A. No. For the Competitive tournament, a team's score is the total packets delivered, accumulated for all matches played within a grouping (3 matches in the preliminary round, 6 matches in the semifinal round). Scores do not carry forward between the preliminary, semifinal and final rounds. For the Cooperative tournament, the group score is the total number of packets transmitted by the 3 cooperating teams. An individual team score is the sum of the 3 group scores from the matches it participated in. Scores do not carry forward between the preliminary, semifinal and final rounds. Time to complete the transfer is not a factor.

Q. Are the center frequency, match duration and data file sizes the same as those in the Preliminary Tournament Event?

A. The center frequency will be 600 MHz and matches will run for 3 minutes or until one team completes the transfer in a Competitive match or all teams complete the transfer in a Cooperative match. The file size (number of packets) may be different than what was used in the Preliminary tournaments.

Previously posted

Q: Have you decided and fixed the frequency band (e.g 600 MHz vs 2.4 GHz)

A: The frequency band will likely be in the range of 600-700 MHz.

Q: Will the nodes (PC) hardware be the same as in the Preliminary Event?

A: The node processor boards have been upgraded. The specs for the new boards are at http://www.orbit-lab.org/wiki/Hardware/hNodes/cNodeVer3/aMain/cLV67J.

Q: For the final tournaments, scoring is based only on the number of packets delivered and not on the time to deliver. Is this correct?

A: Yes. An exception is that time is a factor for settling ties the Cooperative tournament. See the document "Notes for Final Tournament Preparation" for how this is implemented.

Q: Will the final matches still be 3 minutes and 15K packets?

A: Match duration and file size has not yet been specified.

Q: Will there be additional interference bots?

A: Additional interference sources may be employed in the final tournaments.

Q: Do you still plan to release playbacks from the Preliminary Tournaments?

A: Yes, playbacks in the form of video screen captures of the visualization data will be provided for each match a team participated in.

Q: Will we be able to get some RX samples from the scrimmages?

A: No, I/Q samples are not recorded.

Q: Can we schedule travel now for the March 19-20 Final Tournament Event?

A: Yes. The dates for the Spectrum Challenge Final Event are firm and therefore teams can begin planning travel as appropriate.

Q: We might have a broken USRP/SBX (from the RDK). Is there any option to have it replaced?

A: The components of the RDK were provided by DARPA without warranty. Please contact the manufacturer, Ettus Research, for any service or repair issues.

Q: What form of results will be provided from the Scrimmages?

A: Results will be provided in the form of video screen captures of the visualization data for each match a team played. The names of the opponent teams will not be identified.

Q: To get a feel for our overall performance, could an anonymous list of all teams’ scores from the scrimmages be released?

A: We will provide some means to assess overall performance such as histograms or quartiles rather than rankings of specific scoring results, in order to preserve anonymity of scores and teams.

Q: Is there a mechanism for all teams to discuss and share information about the use of ORBIT and the USRPs?

A: ORBIT maintains mailing list forum that can provide this function. This will allow contestants to not only share information among themselves but to also tap into the experience of the larger ORBIT community. Information regarding ORBIT mailing lists can be accessed via: http://www.orbit-lab.org/wiki/Documentation/About/eWhoToContact

Q: What is the entire frequency range that could be used for the tournaments?

A: The frequency range has not been specified, other than the via the limitations of the SBX transceiver card.

Q: Will the validation process use the same resources that will be used for the Preliminary competition?

A: Yes, except that for the validation tests, House radios are used on the other nodes instead of competitor radios.

Q: Will there be other interference signals during a collaborative match?

A: The use of interference signals during the competitions is undecided at this time.

Q: Will a team's results of the Prelim Challenge have any effect on the Final Challenge?

A: No, other than for seeding the tournament ladders.

Q: Will teams switch nodes for the matches or always use a particular node?

A: Assignments will be balanced among the "Team A", "Team B" and "Team C" nodes as fairly as possible. Care has been taken to calibrate these nodes to provide equal power and path losses. Any differences due to their slight asymmetry are very small.

Q: Are there any publication restrictions for work done for the Challenge?

A: No - all IP and rights belong to the team.

Q: Will you provide data logs for the matches?

A: We will provide some limited log data, but not a comprehensive data capture sufficient to re-play the entire match. No raw data or IQ samples will be provided. Data will consist of the types of things displayed during the match, such as center frequency, bandwidth, power, data Tx/Rx rates, etc.

Q: Will this Challenge lead to new DARPA programs in this area?

A: Possibly, but no follow-on programs are currently planned.

Q: For the cooperative competition do we need each of the specific data packets to be received in the calculation of the completion time or just any subset of size D packet?

A: A cooperative match ends when all teams have completed their file transfers error-free, or until the time runs out. To be counted, a packet must be delivered to the server error-free, but not in any particular order.

Q: Can a radio interfere with a competitor’s radio via the Ethernet connection?

A: This would be agains the spirit of the Challenge competition. A radio may interfere with a competitor via the RF.

Q: Can we use the radios’ Ethernet connection to provide signaling or synchronization between nodes?

A: No. All communication between nodes must be performed via RF.

Q: How does a match start with respect to packet requests and starting transmissions?

A: Once all teams request their first packet, a signal is sent to the servers to actually start delivering the packets. Note that once a packet is requested and delivered, that packet cannot be requested again.

Q: Is there anything other than normal processing overhead that will limit the rate at which we can request and receive the packets?

A: The rate is limited only by network speed and normal processing overhead.

Q: Is there some positive indicator, other than a timeout, that will tell us when we have received the last packet?

A: On the Tx (source) node, the server will close the connection after the last packet has been requested. On the Rx (sink) node, the server will close the connection once all packets have been delivered without errors.

Q: During a wildcard selection cooperative match, if the two pairs of House radios have successfully transferred their files, will their transmitters stop sending or will they send files again?

A: During a cooperative match, if one of the radio pairs finishes its file transfer, the packet server continues to serve packets for the other teams until either all three finish or 180 seconds are reached. Similarly, in a wildcard competitive match, if the House radio completes its file transfer, the packet server will continue serving packets to the other team until it completes the transfer or the time runs out. In all cases, House radios that complete their file transfer will stop transmitting.

Q: How do we know when our radios have finished the file transfer?

A: On the transmit (source) node, the packet server will close the connection after the last packet is requested, however, this does not mean the file transfer has successfully completed since the Tx node can request and transmit packets using any schedule it wants. Completion is based on successfully delivering all the packets to the server on the receive side. When all packets are successfully delivered (i.e. error-free) the socket connection on the receive side will close. At this point, both the transmit and receive nodes should stop.
     If all packets have been requested, but not successfully delivered, the receive side socket will remain open to allow for transmission or re-transmission until all packets are received and delivered error-free.

Q: The wildcard selection process mentions the "House" radios. Are these something we can run during our time slots to test our protocols?

A: Yes. The House bots that have been provided, and any additional House bots that will made available, are analogous to the House radios that will be used during the wildcard matches. The exact specification of the house radios will not be specified prior to the match since it would lead to engineering for a specific solution as opposed to a general solution.

Q: Is there a base GRC project available that also handles the ORBIT packet servers?

A: The code that handles the ORBIT packet servers is included in the scripts within the bot image files that have been provided. Similar code was also included in the hurdle images. It is the responsibility of the teams to convert the necessary portions of the example Python code to other languages or formats.

Q: What is the file size during the wildcard round? How many packages are needed to transmit the whole file?

A: The total file size for a match will not be specified in advance. The packet server will close the connection after the last packet is requested.

Q: Can the receiver radiate signal for doing feedback or other purposes?

A: Yes.

Q: What are the details of the House radio interference source, including the interference pattern, the bandwidth, the power level, and the transmission modulation and rate?

A: The house radios are designed to accomplish the same tournament goals as your radios, and are not designed to simply present interference. The specific characteristics of the House radios will not be revealed. Instead, practice radio “bots” are provided that have a number of configurable parameters which can be used to test your radio designs. The House radios are implemented using the same radio hardware and design guidelines as the team radios.

Q: Is there any other interference source besides the House radio?

A: For the wildcard selection process, there will not be interference sources other than the House radios.

Q: What is the maximum time for the competitive and cooperative match?

A: 3 minutes.

Q: Is re-transmission of a packet allowed?

A: Yes. When delivering packets to the server on the receive end, only error-free packets are counted. The same packet may be delivered multiple times, but is counted (once) only if it is error-free.

Q: Will the packet server tell you real-time whether a packet is error-free?

A: No. It is up to you to detect errors in the transmission and handle them.

Q: Can we assume the same channel condition at the Tx and Rx side?

A: No. Although the geometries are identical, ORBIT cannot control the interference conditions seen by the receiver due to the other team’s radio.

Q: For the cooperative match is "D=number of packets" known beforehand?

A: No. The packet server will terminate the connection once the last packet is requested.

Q: Can the radio support half duplex in real time, for example, to implement feedback logic?

A: Yes, within the hardware and software limitations of the USRP N210s and their associated ORBIT nodes. Use of the on-board FPGA is prohibited.

Q: The documentation suggests that "additional interference" may be added. Will this be accomplished by the House radios and/or by the interference sources? Can you expand upon this now?

A: For the wildcard selection process, there will not be additional interference sources other than the House radios acting as either competing or cooperating radios. For the actual tournaments, there may be other radios or other interference signals, in addition to the team radios, injected into the grid. The specific characteristics of these other signals will not be revealed in advance of the match.

Q: Are cooperative House bots "smart", i.e., if we give them spectrum will they use it or will they do "tricky" things.

A: The House bots are designed to accomplish the data transfer goals of the match (like another competitor’s radio). They are not designed to specifically obstruct your team’s radio, but they are trying to win.

Q: For the competitive match, are the House bots also trying to communicate?

A: Yes. They are not designed to specifically obstruct your team’s radio, but they are trying to win.

Q: Will we be fighting for ORBIT time with the qualified teams?

A: Yes. While we may reserve some time on the grid for exclusive use by the wildcard teams as we get close to the final submission deadline, teams should anticipate contention and plan accordingly. Although the main grid is the primary resource for testing and performance verification, multiple sandboxes are available with identical radio hardware are available for radio development. All teams are advised to test early and often, and not wait until the deadline.

Q: Referring to the ORBIT grid diagram, during the wildcard selection process, will teams be randomly assigned as one of teams designated as A, B, or C, or will we be assigned to a particular location?

A: The wildcard teams will be assigned a specific location (Team A in the diagram).

Q: For final submission do we need to reserve time on the grid?

A: Yes. You should plan on at least 15 minutes to run the verification test and upload your image.

Q: Can the highest performing image rather than most recent verified image be accepted?

A: No. It is up to you to verify your radio’s performance using the verification tools provided. You may run the verification using multiple radio designs and multiple bot parameters, but only the latest verified image upload will be used for final testing.

Q: The "test framework" will include parameterized House bots for competitive and cooperative rounds. For the wildcard selection, particular parameters will be selected to run these bots, but we will not know the parameter settings. Correct?

A: For the wildcard evaluation and selection, House radios similar but not identical to those that are provided for practice will be used for the matches. The exact specification of these radios and any associated parameters will not be provided prior to the match.

Q: What is the power level of the interference that is generated by the house radio transmitter?

A: The House radios, which are attempting to accomplish the match goals (i.e. not simply generating interference signals), are implemented using the same radio hardware and design guidelines as the team radios.

Q: What is the distance between the interference source and the wildcard team's receiver?

A: The wildcard team nodes will be located in the positions specified for Team A in the diagram of the Spectrum Challenge Arena. The House radios will be located at the positions labeled as Teams B and C. Adjacent nodes are spaced three feet apart (in both directions) on the grid. There will be no additional interference sources used as part of the wildcard selection process. Teams are advised that the radio effects (e.g. propagation loss) associated with distances in the ORBIT facility might not directly correspond to the same distances at their own laboratories.

Q: It is mentioned in previous documents that the interference source on ORBIT can be accessed by the teams. How can we access the interference source and how can we use it? Does it need to be programmed?

A: You can find the example script very similar to the one that was used for H3 at http://www.orbit-lab.org/wiki/DSC/dc_trial1 ; also, please check the information about the service itself at http://www.orbit-lab.org/wiki/Software/bAM/mInterf

Q: During the wildcard round and tournament round, can the receiver radiate signals for interactive transmissions, e.g., ARQ, conveying CSI, or other purposes?

A: Yes.

Q: Will the results of the wildcard selection process be distributed to all teams even those that have already selected for the Preliminary Competition?

A: The results of the selection process will be distributed to all teams.

Q: Will teams not competing in the wildcard be able to log in and observe the wildcard competition?

A: No. The wildcard evaluation testing is not open to any of the teams (including the wildcard teams).

Q: There is concern that those teams that compete in the wildcard selection process and then move on to the Preliminary competition will be at an unfair advantage since the wildcard selection process has been stated to be very similar to the Preliminary competition. Is this a valid concern?

A: All teams (wildcard and Qualified) are provided with the same resources for developing, testing and evaluating their radio designs. The same test framework and verification scripts that are provided to all teams are used for the actual wildcard selection and tournament matches. The wildcard teams do not have any additional testing resources over the qualified teams. After completion of the wildcard evaluation process, the specifics behind the House radios used in scoring and testing will be released to all teams. Qualified teams and wildcard teams may then use the test framework with those House bots to subsequently test their own radio designs. Wildcard teams will have an advantage relative to Qualified teams since the House bots used for wildcard scoring will not be released until after scoring is completed—at which point all teams will have access to the same radios for further testing.

Q: Section 4 of the posted rules mentions the criteria of moving a "designated file from one node to another node as quickly as possible"; however, to date, the ORBIT lab configuration in the competition has had network servers sourcing/sinking packets in a socket-oriented approach. Are the competitors in the tournament required to fragment and reassemble a file obtained from an external file system?

A: No. Data is obtained using network servers for sourcing and sinking fixed-size packets, very similar to the approach used in the “Hurdle-3” qualification exercise. Sample code has been provided that shows how to request and deliver packets from/to the servers. Although the packet sizes are fixed at 1440 bytes, the radio may request as many as desired, and may transmit data using any protocol desired (including fragmentation/reassembly, FEC, ARQ, etc.). Packets delivered to the server must be blocked into the same 1440 byte boundaries as was requested, however packets need not be delivered in the same order they were requested.

Previously posted

Q: Will the Challenge tournaments impose a fixed bandwidth allocated to several users or will they provide a large chunk of spectrum and allow each user to select which sub-bands to use.

A: To achieve the Challenge goals, the bandwidths used in the tournaments will be selected to force contention among the users. The imposed bandwidth might be different for the competitive and collaborative tournament variations. Simply moving to unused “whitespace” may not be an effective strategy. Use of out-of-band spectrum will be monitored by the testbed.

Q: In the cooperative tournament how does each contestant know which other radios are on their team?

A: Teams can devise any protocol they like to coordinate communication between their radios. However, in the cooperative tournament, other teams in the group, whose protocols you do not know, are doing the same thing, and the entire group is scored collectively. One of the questions that the cooperative tournament aims to address is how well can radios cohabitate without prior coordination? Therefore, no information will be provided to you as to which other radios are in the group, other than the fact that there are other radios present whose ability to communicate affects your group’s score.

Q: We're interested in fielding a team for your challenge but we are not college students. When you send out the qualification processes will it be during normal working hours?

A: The qualification process will be administered and hosted by WINLAB at Rutgers University in New Jersey. Consequently the timing of announcements will be commensurate with their business hours (Eastern Standard Time). However, access to the ORBIT facilities will be accessible 24 hours a day, subject to resource scheduling limitations.

Q: Regarding eligibility: Does a “DARPA support contractor” include any contractor that has a DARPA contract or is it just a SETA contractor that supports DARPA operations and staff directly?

A: A “DARPA support contractor” refers to individuals that directly support DARPA operations and staff, e.g., DARPA SETA contractors. It does not include individuals and companies working under DARPA R&D contracts.

Q: I would like to inquire on the Orbit user account we're supposed to request and set-up. It appears that the website linked isn't in full working order.

A: Shortly after the registration period closes, all registrants will receive detailed instructions on opening an ORBIT account. These accounts will be established specifically for the Spectrum Challenge.

Q: What is the difference between the preliminary and final events besides the award amount? Is there some reason to have two identical tournaments 6 months apart? (Time to tweak our design based upon the other competitors?)

A: The tournaments will be very similar. You are correct that the intention was to provide an opportunity to see how your design performed in a real competition, then provide time for you to tweak and optimize performance for the final tournament.

Q: Can non-US institutions and individuals participate, assuming the other members, or at least team leader/contact is a US citizen?

A: Yes. As stated in the rules document, “The person identified as the team leader must be a U.S. citizen or permanent resident (green card holder) at time of registration and throughout the competition. Other team members need not be U.S. citizens or permanent residents.” The expectation is that the team leader materially participates in the Challenge throughout its duration. Further, as stated in the rules document, “The prize money will be transmitted by electronic funds transfer to the bank account specified by the leader of the team determined by DARPA to be the winner of each tournament. If the winner is a team, it is the responsibility of the team leader, not DARPA, to determine the subsequent division of any prize money.”

Q: Can collaborating institutions participate in a single team?

A: Yes, but a single individual must be designated as the team leader.

Q: Although our team leader is a US citizen, our team would be located in three different countries and have three academic affiliations, is that an issue? If not, do we simply make a comma-separated list in the form?

A: The requirement is that the team leader be a U.S. citizen or permanent resident, and the expectation is that the team leader materially participates in the Challenge program throughout its duration. Other team members do not have to be U.S. citizens or residents and may be from non-U.S. institutions.

Q: Does the team leader have to be in Arlington, VA during the full-day competitions? Do travel expenses cover only the team leader, or can they be used for other team members?

A: It is not a requirement for the team leader to attend, however, the expectation is that the team leader would be most likely to represent the team. We do not specify which members of your team, if any, must attend the competition at DARPA. Travel expenses can cover other team members.

Q: Are we allowed to modify the content of the FPGA? (i.e. make our own version of the firmware).

A: Modification to the USRP’s FPGA is not allowed. More details regarding the USRP and host PC specifications will be made available after the qualification process.

Q: Suppose that a U.S. academic institution wishes to enter the DARPA Spectrum Challenge and that the team leader is a U.S. person affiliated with the institution. Must all team members be affiliated with the academic institution?

A: No. Teams can be composed of any mix of schools, businesses and individuals. The only requirement is that the team leader be a U.S. citizen or permanent resident.

Q: Can a group of individuals who currently work for the U.S. government and have expertise in this area compete in the spectrum challenge? From the eligibility, it sounds like they would not be eligible but I wanted to clarify (U.S. academic institution, business, or individual entity).

A: No, employees of the U.S. government are ineligible.

Previously posted

Q: What is DARPA?

A: The Defense Advanced Research Projects Agency’s (DARPA) mission is to prevent and create strategic surprise by maintaining the technological superiority of the U.S. military. The Agency sponsors revolutionary, high-payoff research bridging the gap between fundamental discoveries and their military use. You can learn more about DARPA by visiting www.darpa.mil.

Q: What is the goal of the Spectrum Challenge?

A: The DARPA Spectrum Challenge is a competition to demonstrate a radio protocol that can best use a given communication channel in the presence of other dynamic users and interfering signals. It is targeted at finding strategies for guaranteeing successful communication in the presence of other radios that may have conflicting co-existence objectives.

Q: How do I enter the Spectrum Challenge?

A: The process starts by submitting the registration form from the website by midnight EST on Jan. 31. The designated team leader will then receive further instructions by email describing a set of qualification hurdles. The qualification period will start at the same time for all registered teams. The first approximately 14 teams that successfully complete all the qualification hurdles will become Challenge contestants (see the Rules for more conditions and details). Other teams that successfully complete the hurdles may opt to continue participating as wildcard teams.

Q: When does the Challenge begin and end?

A: Registration closes midnight Eastern Time on Jan. 31, 2013. Subsequent event dates are tentative, but current plans follow. The qualification process occurs in February. Final contestants and potential wildcard teams will be announced in early March. The Preliminary Challenge tournaments are scheduled to occur in September 2013 and the Final tournaments in March 2014.

Q: Do I need to design and build a complete radio to compete?

A: No. The Challenge is focused only on the algorithms and protocols used by a software-defined radio. The actual radio platform is incorporated into the ORBIT testbed at WINLAB, and will be identical for all contestants.

Q: Can I design my radio using my own hardware platform?

A: Yes, but the actual competition will use the radio hardware specified in the Radio Development Kit. It is advisable to use the RDK components or equivalent hardware, to assure that your design performs the same on the ORBIT testbed as it does in your lab.

Q: Do I have to use the GNU radio software development tools to design my radio?

A: No. You may use other development tools provided your design runs on the radio platform specified in the RDK.

Q: Can’t I just turn up the power or use more bandwidth to get my radio to work in the presence of interference?

A: You can, but you won’t have any more power or bandwidth available than any other team because all teams are using identical radio hardware. You will need to use the available power and bandwidth in ways that make your radio outperform your competitors’ radios in each of the tournament types (competitive and collaborative).

Q: What is contained in the Radio Development Kit (RDK) and how do I get one?

A: The RDK is the radio used in the ORBIT testbed. It consists of a pair of Ettus Research USRP radios, transceiver daughter boards, interconnects, antennas, power supplies, and Linux-based GNU radio development software. You must supply your own host PC for software development and to act as host for the Universal Software Radio Peripherals (USRPs) while you are developing your radios. The specific model of USRP and daughterboard, as well as the specifications for the host PC, will be made available once the Challenge contestants are determined from the qualification process. Only the teams that become Challenge contestants are eligible to receive RDKs. Other registrants and potential wildcard teams will not receive RDKs.

Q: If I participate in the Challenge, will the government have rights to my design or intellectual property?

A: No. Neither the government nor Rutgers University / WINLAB will make any claim of ownership to intellectual property developed by Challenge contestants. Furthermore, DARPA does not authorize the incurrence of any costs associated participating in this Challenge to be charged to the Government or Rutgers University.