Solicitations Home
Section Menu
Encryption
General BAA FAQs
|
 |
|
|
Frequently Asked Questions
BAA 07-04: Bootstrapped Learning (BL)
1. The PIP (p49) says "J. Technology Transition and Technology Transfer Plans. (Note: Curriculum teams only)" We are curious whether we are allowed, as a learning team, to include this section describing a transition story. In particular, we were planning to emphasize transition to the ML/BL community, in addition to some considerations that may be relevant in a military transition.
2. On page 30 of the PIP it says "Any NI method contracts can specify student-initiation, teacher-initiation, or a mixed initiation approach. NI methods that are heavily biased toward teacher-initiated instruction will place a greater burden on the development of electronic instructors. Thus, such methods will require a proposal (from the proposing learning teams) on how such interactions can be provided using an automated scheme in a cost-effective way." Could you expand on this?
3. I am currently a Learning Team bidder, but I would also like to be a Curriculum Team contributor on the evaluation of Learning Systems with appropriate stipulations to avoid a conflict of interest. Is this arrangement compatible with DARPA’s needs? Would this dual relationship affect the evaluation of the Learning Team’s proposal?
4. The last bullet of page 44 in the PIP reads: "A description of the proposed innovative approach to semi-automating the creation of different types of curriculum materials corresponding to different NI methods" Does "creation of curriculum materials" mean the instantiation of a curriculum rung to a particular NI method or the actual construction of the rung itself?
5. The first bullet of page 45 in the PIP reads: "A description of the proposed innovative approach to the rapid creation of curriculum material generators" Do these curriculum material generators correspond to the segment generators described in Section 4 of the Interaction Language Definition document?
6. I am considering participating in two teams, and would like to know whether or not that's a good idea. In particular, it would be helpful to know the effect this would have on the teams, the selection decision, and my participation.
7. Page 28 in the PIP describes "hidden ladders," and the penultimate paragraph notes that offerors for the Curriculum Team should identify ladders that they believe "could serve as the 'hidden' test ladder at the end of phase II and III." Does this mean that there will be two hidden ladders during the program---one in phase II and a different one in phase III? Or will there only be one hidden ladder, which is used both in phase II and in phase III?
8. Is each Learning Team required (see PIP, page 36) to define and build a curriculum for Phase 1 in order to test its own learning system? Even if one assumes that this can be a mini-curriculum and that it doesn’t have to conform to the “official” NIM contracts, this would still require each learning team to include a skill set sufficient to produce a curriculum. However, such a requirement is not evident in the proposal requirements for Learning Teams in Chapter 7. What is actually required?
1. The PIP (p49) says "J. Technology Transition and Technology Transfer Plans. (Note: Curriculum teams only)" We are curious whether we are allowed, as a learning team, to include this section describing a transition story. In particular, we were planning to emphasize transition to the ML/BL community, in addition to some considerations that may be relevant in a military transition.
Military transfer is not a selection criterion for learning teams. Therefore, although it is not discouraged, including a military transfer approach will not affect selection positively or negatively.
Back to Top
2. On page 30 of the PIP it says "Any NI method contracts can specify student-initiation, teacher-initiation, or a mixed initiation approach. NI methods that are heavily biased toward teacher-initiated instruction will place a greater burden on the development of electronic instructors. Thus, such methods will require a proposal (from the proposing learning teams) on how such interactions can be provided using an automated scheme in a cost-effective way." Could you expand on this?
Interaction can be driven by the student, teacher, or both. For example, in the context of "teaching by example", the student could initiate learning by asking the teacher to provide a specific example of a target concept. A student could also direct learning by requesting the evaluation of an example generated by the student. Alternatively, the teacher could direct the interaction by providing a series of examples that are structured to facilitate learning. A mixed-initiative approach might involve the student working out examples while the teacher provides dynamic feedback that is intended to improve efficiency and efficacy.
NI method contracts that require significant instruction from the teacher will necessitate teaching agents sophisticated enough to provide that instruction. Unfortunately, the development effort for such teaching agents could be tantamount to the effort required to create an electronic student. This could incrementally inflate the burden and cost of development for the Curriculum Team, which is contraindicated on page 30 of the PIP:
"The ultimate goal for the curriculum team is to provide the most curricula at the lowest cost in a manner suitable for cross-validated testing of the BL learning systems."
Hence, the cost-benefit analysis undertaken by the Curriculum Team should include consideration for support software costs applied across curricula and amortized over program phases. In this vein, the Curriculum Team should propose automation that ultimately minimizes the total cost of building facilitating tools and using those tools over the course of the project. In fact, a proposal for the automation of teaching is specifically listed among the Curriculum Team's requirements on page 45 of the PIP:
"A description of the proposed innovative approach to the cost-effective creation of sufficient electronic teacher functionality to support interactive NI-methods."
For this reason, Learning Team proposals with NI method contracts that have the potential to increase Curriculum Team development costs must propose relevant cost mitigation strategies.
Back to Top
3. I am currently a Learning Team bidder, but I would also like to be a Curriculum Team contributor on the evaluation of Learning Systems with appropriate stipulations to avoid a conflict of interest. Is this arrangement compatible with DARPA’s needs? Would this dual relationship affect the evaluation of the Learning Team’s proposal?
This issue is addressed in the PIP in two places. Under “Requirements of the Curriculum Team” in the subsection “Mitigation Strategy for Potential Conflict of Interest” the PIP states (p.33):
“In particular, [The curriculum team] must ensure that sub-performers will not be in a position to control important aspects of the tests that they, or those with whom they are affiliated, will subsequently be receiving in a learning team role.”
“Any inputs that are provided by sources with possible conflicts must be cleansed in some way that ensures the integrity of the evaluations.”
“The curriculum team must articulate in its proposal how it will ensure that the integrity of its evaluations is maintained.”
And in under “Detailed Proposal Information” in the section “Format of the Proposal” the PIP further indicates (p.45) that Curriculum Team candidates must include a “mitigation report and plan for potential conflict of interest”.
Thus, specific configurations will not be assessed in advance. Instead, attention must be given to the considerations listed above and proposals will be evaluated based on the strength of the guarantees of impartiality.
Back to Top
4. The last bullet of page 44 in the PIP reads: "A description of the proposed innovative approach to semi-automating the creation of different types of curriculum materials corresponding to different NI methods" Does "creation of curriculum materials" mean the instantiation of a curriculum rung to a particular NI method or the actual construction of the rung itself?
The primary objective is to derive the most curricula for the least cost, given the set of curricula needed over the course of the program. Thus, one must balance the cost of sophisticated solutions amortized over the life of the program, against the cost of simply employing manual construction methods.
Some of the options we have considered involve automating the generation of lesson content from an Interlingua representation of the target concept (the context for this approach is provided at the bottom of page 30 in the PIP, in the section titled “The Electronic Instructor”.)
To get a flavor of one possible automation technique, consider the following example. Imagine that the goal state of the student’s knowledge representation for a lesson rung could be used as the instructor’s representation of the curriculum content for that rung. Taking this approach would permit the instructor’s representation to be used both to drive the instruction process and to be reused iteratively in the assessment of the student through a direct comparison to the student’s post-lesson representation of the lesson content, and then providing teacher critiquing based on the difference.
To illustrate this, consider a critiquing engine for concept-learning. Imagine that a Learning System that is learning by example proposes a concept by way of predicting the valuation of an outcome (e.g., if the result is “200” and “green”, it is “good”). The system, which knows the correct valuation of the proposed outcome, provides a corrective response: to be “good”, it must be “200” and “red”. In this way, a simple implementation that has a representation of the correct concept (which takes the form of injected knowledge) and is aware of the Learning System’s behavior can provide rich criticism.
This particular example is attractive because this simple automated method will generate all interaction for a type of concept learning given only a world, a start state, and the injected knowledge that is the “answer concept” for this rung.
Back to Top
5. The first bullet of page 45 in the PIP reads: "A description of the proposed innovative approach to the rapid creation of curriculum material generators" Do these curriculum material generators correspond to the segment generators described in Section 4 of the Interaction Language Definition document?
Yes. Each generator outputs instructional segments. Each segment specifies a single start state, an optional simulated world to interact with, an optional teacher to interact with, and an optional imperative, or goal to be achieved. Thus, a segment generator generates parcels of curriculum content.
The idea is that segments could be static object; they could be the product of interacting with an automated world/teacher. And rather than building each world/teacher agent independently, one can imagine tools that assist in this creation process.
Back to Top
6. I am considering participating in two teams, and would like to know whether or not that's a good idea. In particular, it would be helpful to know the effect this would have on the teams, the selection decision, and my participation.
This is addressed in the PIP on page 32, which states: “A primary concern with respect to teaming is to avoid a conflict of interest (as explained in FAR 9.5) that could result in contamination of the go/no-go evaluations. Thus, it is not permissible to be a prime for both the curriculum team and a learning team. Additionally it is the responsibility of the curriculum team to guarantee the integrity of the testing provided by its curriculum ladders and by its evaluation process. In particular, it must ensure that sub-performers will not be in a position to control important aspects of the tests that they, or those with whom they are affiliated, will subsequently be receiving in a learning team role. Any inputs that are provided by sources with possible conflicts must be cleansed in some way that ensures the integrity of the evaluations. The curriculum team must articulate in its proposal how it will ensure that the integrity of its evaluations is maintained.”
In short, if the teams you are considering include one learning team and one curriculum team, the onus will be on the curriculum team to indicate clearly that your respective roles will not result in contamination of the go/no-go evaluations.
If you are considering contributing to two learning teams, there will be no issues of go/no-go contamination. However, the work that you would provide to one team would likely overlap with the other. Consequently, further negotiation regarding appropriate compensation would be required in the case that both teams were ultimately selected. Obviously, if you intend to contribute to two learning teams, both teams would need to be made aware of your inclusion as part of another proposal.
Back to Top
7. Page 28 in the PIP describes "hidden ladders," and the penultimate paragraph notes that offerors for the Curriculum Team should identify ladders that they believe "could serve as the 'hidden' test ladder at the end of phase II and III." Does this mean that there will be two hidden ladders during the program---one in phase II and a different one in phase III? Or will there only be one hidden ladder, which is used both in phase II and in phase III?
The primary purpose of the hidden ladders is to test the ability of a Learning System to perform against a domain for which it was not specifically constructed and upon which it has not been previously tested. Thus, in order to measure incremental improvements in the learning system between phases II and III that are not due to domain-specific enhancements, it is necessary to have a different hidden ladder and, consequently, a new domain for phase III than the one used in phase II (see Table 1 on page 37).
A special case is very general architectures such as the General Game Playing (GGP) platform. To be clear, the relevant distinction here is between the platform and the details of the curriculum (e.g., actions, percepts, etc.). Thus, if the architecture were sufficiently general, even if the performers were to have a priori knowledge of the platform, the curriculum would still be considered hidden. Such general platforms could be reused across phases provided that the actual curricula developed upon those platforms were sufficiently distinct.
Back to Top
8. Is each Learning Team required (see PIP, page 36) to define and build a curriculum for Phase 1 in order to test its own learning system? Even if one assumes that this can be a mini-curriculum and that it doesn’t have to conform to the “official” NIM contracts, this would still require each learning team to include a skill set sufficient to produce a curriculum. However, such a requirement is not evident in the proposal requirements for Learning Teams in Chapter 7. What is actually required?
The resolution of this go/no-go requirement is entirely the responsibility of the Learning Team for two primary reasons: 1) DARPA wishes to avoid timing issues between the curriculum team and the learning teams; and 2) DARPA wishes to ensure that the learning team’s efforts are unaffected by fluctuations in the API that may occur during Phase I. Learning Teams will have the leeway to diverge from any formal standard in Phase I, and latitude to develop tests that are more closely matched to the algorithms they are developing. However, DARPA anticipates that the teams will not need to exercise these liberties because 1) DARPA is attempting to clarify the Phase I languages prior to the start of the program, and 2) the Curriculum Team is required to provide quarterly releases of their curricula under development. DARPA anticipates these curricula will provide sufficient curricular complexity early enough that they can be relied upon by the Learning Teams. Nonetheless, there are many reasons why such a hand-off could be problematic, as indicated above. For these reasons, the Learning Teams will have the full responsibility (and associated latitude) of achieving the phase one learning milestones.
Back to Top
|
