KBA SSF Assessor Guidelines
===========================

To the extent feasible, attribute slot fills should be specific not
generic.  Use current attributes only, not former or past attributes.
For example, do not make equivclasses for "past president of the
Mayfair Garden Club" or "former board member of the Podunk Lions
Club."

Do not capture general religious or political affiliations,
e.g. ignore "Republican" or "Muslim".

Elected officials are not EmployeeOf their districts but they are
Affiliate of their districts.

Assessors should record all key attributes for an entity except for
Affiliate.  For example, if John Q. Berry is both a FoundedBy and in
TopMembers of the entity "Raspberry Company," record him as both.  Do
not, however, also record him as an Affiliate.  If you initially
identified Mr. Berry as an Affiliate, and later find that he is a
Founder and/or in TopMembers, record the more specific information and
delete the Affiliate slot fill.


Strings to Gather
-----------------

Each slot fill has one or more equivalent strings, called "aliases"
for the slot fill.  They form an equivalence class.  The assessors
gather these strings from the documents in the corpus.  

An alias is an alternate or different name for a given slot fill,
which is often an entity.  A name or title and the acronym for that
name or title are good examples of an alias.  Example: If North
Atlantic Treaty Organization is the entity, NATO would be an alias.
If a person's full name is a slot fill, the last name would be an
alias if it appears separately in the document.  Example: "Founder
John Q. Berry spoke to our correspondent at the Berry Growers
Convention in Las Vegas.  Berry said yada, yada, yada."  The latter
reference to "Berry" would an alias.  Web addresses (URLs) and email
addresses can be listed as aliases.


Begin/End State
---------------

For each slot fill, assessors enter begin and end dates, which refer
to when the relationship with the entity occurs or occurred.  All date
entries are relative to the "corpus," the time-span that encompasses
the documents in the database.  Date choices include any combination
of the following: began before the corpus (denoted "<"), continues
after the corpus (">"), begins within the corpus ("|"), ends within
the corpus ("|"), or I don't know ("?").

Some slot fills will always end after the corpus.  For example, an
award, once given, remains an award in perpetuity. An organization's
founder is always the founder, even if his/her association with the
organization terminates.  Whether a title terminates or continues is a
matter of judgement.

An affiliation or association may begin or end within the corpus, but
it is often impossible to know exact dates for those equivclasses, in
which case the assessor should select "I don't know."

Scoring for KBA SSF 2013 focuses on the slot fills that begin within
the corpus.  Nuances and uncertainty around end state make it complex
to use slots that end during the corpus.


General Notes
-------------
Some entities had NO slots filled, e.g. Redmond Music, Sandra Friend,
Lorenzo Williams, Ritz Apartments, Bob Plaisted, and others.

Some slots were rarely filled: date of death, cause of death,
children, significant other.

Some entities yielded only Affiliate and AssociateOf slot fills,
e.g. Sean Hampton

There were a lot of Contact_Meet_PlaceTime and Contact_Meet_Entity
that occurred within the corpus time frame.

A lot of awards won were won within the corpus, and extended beyond
the corpus, since awards tend to stick to an entity forever.



Guidelines on Specific Slots
----------------------------

EmployeeOf: The entity attribute means that the entity is an employee
of another person or an organization, e.g., "The entity is an employee
of ______________."  This attribute category is not meant to capture
persons who are employees of the entity; there is no slot for the
latter.

Title: descriptive words or phrases that characterize a person-entity.
A title should be specific, not just "board member" or "engineer."
Examples of title slot-fills: "Chair, Friends of the Smallville Park
Association," "Chairman and CEO, Cloud Computing Corp."  Titles are
recorded for the entity only, not for associates of the entity or top
members or founders of ORG-type entities.

Four slots are special in the sense that they are catch-all categories
for a broad range of events that happen to entities:

1.1) "Affiliate" is any type of relation that *directly* connects the
target entity to another entity.  Affiliate must have a FAC or ORG on
at least one side of the relation.  Affiliate is its own inverse.

1.2) "AssociateOf" is the PER-PER analog of Affiliate.  Both sides of
this relation must have a person-type entity.

The challenge with both Affiliate and AssociateOf is judging what
level of "closeness" distinguishes these relations from two entities
that merely co-occurring in the same sentence or passage.  By
attempting these new slots in KBA 2013, we will be able to measure
inter-assessor agreement and work toward more refined guidelines than
the few examples below.

Affiliate is the union of all
close relations, such as StudentOf, EmployeeOf, MemberOf, and their
inverses from ACE and KBP, and similar relations in which the relation
is of a simple unambiguous type.  We propose to use this generalized
notion of "close" relation as the base class for all the ORG-ORG, ORG-FAC,
PER-FAC and PER-ORG relations.  Instead of enumerating all possible
such close relations, we intend to allow KBA systems to generate many
such examples, which assessors will then judge in the post-hoc pooled
assessment scheduled for June 17-July 17 2013.

Guidelines to SSF assessors for AssociateOf: To judge who is an
associate, consider the degree of symmetry of the relationship and the
degree of control the entity has over the relationship.  The entity's
name on a list of names or membership in an organization does not in
and of itself mean that that entity is an associate of the other
people on the list.  Consider how much control the entity would have
over his/her inclusion with the others (i.e., was the list created by
a third party) or whether the relationship is asymmetrical (i.e.,
would both parties know each other?  Are they on a first-name basis?)
The degree of control and/or the relationship symmetry determines the
judgement about whether the entity and another person are associates.

Guidelines to SSF assessors for Affiliate: living in a geographic
area, city, etc., is not an affiliation.  In general, competitors are
not affiliates except if the circumstance is significant, such as a
lawsuit or patent infringement or hostile takeover/buyout, etc.

While this is somewhat in the spirit of open IE, our aim is
specifically to enable upstream filtering systems to down-select the
stream for feeding KB population systems operating with fixed
inventories of slot types, which will typically be more specific than
Affiliate or AssociateOf.  By filtering the stream, KBA systems allow
such KBP systems to focus compute power on a smaller substream.
Further, the less strongly typed graph of affiliated/associated
entities may serve valuable purposes for humans directly exploring the
data.

Since little annotation has been performed for "Affiliate", we are
concerned that it could generate a lot of noise.  We welcome feedback
and ideas on how to refine this.


Example 1: "Matthew DeLorenzo and Josiah Vega, both 14 years old and
students at Elysian Charter School, were honored Friday morning by
C-SPAN and received $1,500 as well as an iPod Touch after winning a
nationwide video contest."

target_id:  http://en.wikipedia.org/wiki/Elysian_Charter_School

Affiliate:  "Matthew DeLorenzo"
Affiliate:  "Josiah Vega"

NOT an affiliate:  "C-SPAN"
NOT an affiliate:  "iPod Touch"


Example 2: "Allen Curtis, administrator, Citrus Health and Rehab, and
Inverness Mayor Bob Plaisted, standing from left in rear, joined the
Joyner family in celebrating the 104th birthday of Edith Joyner on
Wednesday at Citrus Health and Rehab."

target_id: https://twitter.com/bobplaisted

Affiliate:  "Allen Curtis"
Affiliate:  "the Joyner family"
Affiliate:  "Edith Joyner"

NOT an Affiliate:  "Citrus Health and Rehab"

That is, merely being present at a place once does not substantiate an
Affiliation.  Affiliation is closer than Meeting(place-time).


Example 3: "Veteran songwriters and performers Ben Mason, Jeff
Severson and Jeff Smith will perform on Saturday, April 14 at 7:30 pm
at Creative Cauldron at ArtSpace, 410 S. Maple Avenue."

target_id: http://en.wikipedia.org/wiki/Jeff_Severson

AssociateOf: "Ben Mason"
AssociateOf: "Jeff Severson"
AssociateOf: "Jeff Smith"

NOT an Affiliate: "Creative Caldron"
NOT an Affiliate: "Art Space"


Example 4: "Driftwagon has teamed up with Peruvian designer
Dunkelvolk."

target_id:  http://en.wikipedia.org/wiki/Dunkelvolk

Affiliate:  "Driftwagon"


2) Contact_Meet_Entity is a catch-all slot for Facilities.  It is a super
set including any event in which one or more entities (of any type) are present
at the target facility.  

Example: "The Senior Wellness Coalition of Fargo-Moorhead will host a
wellness seminar from 1 to 3 p.m. March 28 at the Hjemkomst Center,
202 1st Ave."

target_id:  http://en.wikipedia.org/wiki/Hjemkomst_Center

Contact_Meet_Entity:  "The Senior Wellness Coalition of Fargo-Moorhead"


3) Contact_Meet_PlaceTime is a catch-all slot for Persons.  It is a super
set including any event in which the target entity is present at a
particular place at a particular time.  For SSF, we want short
passages that a human or downstream algorithm can digest to generate
structured location/date-time values.

The place or time might not specified in the text.  When this happens,
just emit the best available string.

Example:  "Lt. Gov. Drew Wrigley and Robert Wefald, a retired North Dakota
district judge and former state attorney general, unveiled the crest Friday
during a ceremony at the North Dakota Capitol."

target_id: http://en.wikipedia.org/wiki/Hjemkomst_Center

Contact_Meet_PlaceTime: "Friday during a ceremony at the North Dakota Capitol"



More Examples of Slot Fills
---------------------------

http://en.wikipedia.org/wiki/Gretchen_Hoffman

  Slot: Affiliate

  Fill: 7th Congressional District

  Notes: Hoffman, a MN state senator, decided to run for the 7th
  Congressional District in early 2012 after her district was
  realigned.


http://en.wikipedia.org/wiki/Gretchen_Hoffman

  Slot: Title

  Fill: Senator

  Notes: Hoffman was a MN state senator before the corpus; she
  announced her retirement from the Senate when she decided to run for
  the US Congress.  2012-04-13-16 is the earliest reference in the
  corpus to Hoffman being a state senator (she was elected before the
  corpus) 2012-11-20-14 is the last reference in the corpus to
  Hoffman; this document announced that she would not be returning to
  the senate in 2013. I know from an earlier document that she
  announced her retirement from the senate when she decided to run for
  Congress. Based on these two documents and world knowledge that the
  new senate session began in Jan 2013, I marked the end of her
  affiliation with Senate things as occurring within the corpus.


http://en.wikipedia.org/wiki/Joshua_Boschee

  Slot: both Affiliate and Title

  Fill: district 44, North Dakota House

  Notes: You see him become associated with District 44 by filing
  papers to run for the house, he campaigns, he wins, and he takes his
  seat thus becoming affiliated with all the ND House and Legislature.


http://en.wikipedia.org/wiki/Paul_Marquart

  Slot: Title

  Fill: minority lead on the House Property and Local Sales Tax Committee


  Slot: title

  Fill: chairman of the House Education Finance Committee

  Notes: Marquart has been a long-time MN House representative.  In
  2012, he ran for and won reelection for his district seat.  We know
  from the corpus that in the same election the democrats, formerly
  the minority party, took the majority in both the House and Senate.
  Obviously, Marquart is no longer the minority lead on the House
  Property and Local Sales Tax Committee.  In the newly seated 2013
  House, he does become the chairman of the House Education Finance
  Committee.


https://twitter.com/CorbinSpeedway 

  Slot: Affiliate and also Contact_Meet_Entity

  Fill: United Super Dirt Cars


https://twitter.com/tonyg203 

  Slot: Title

  Fill: Assistant Chief

  Fill: Interim Chief

  Fill: Police Chief


http://en.wikipedia.org/wiki/Barbara_Liskov 
  
  Slot: Contact_Meet_PlaceTime

  Fill: ACM conference on High Integrity Language Technology in Boston, Dec. 2012


http://en.wikipedia.org/wiki/Yann_LeCun 

  Slot: Contact_Meet_PlaceTime

  Fill: late January at NYU

  Notes:  might have only one document for substantiation


http://en.wikipedia.org/wiki/Intergroup_Financial_Services

  Slot:  TopMembers

  Fill:  Ernesto Ferrero
  

http://en.wikipedia.org/wiki/Agroindustrial_Pomalca

  Slot: Affiliate

  Fill: Peruvian government

  Notes: affiliation ended, so the state change is the dissappearance of the equivclass


http://en.wikipedia.org/wiki/Blair_Thoreson 

  Slot: AffiliateOf

  Fill: PSC

  Notes: Thoreson ran for PSC and wasn't chosen.  Run announced
  2011-12-20-15, 1st indication he might run 2011-12-13-02; Gov chose
  someone else 2012-11-30-19.


http://en.wikipedia.org/wiki/Carey_McWilliams_(marksman)

  Slot: AssociateOf

  Fill: Dawn Ziegler

  Notes: McWilliams met Ziegler for what appears to be the first time
  2011-10-08-03.  No indication that the relationship continues.


http://en.wikipedia.org/wiki/Clark_Blaise

  Slot: AwardsWon

  Fill: Rogers Writer's Trust Fiction Prize

  Notes: Blaise was up for the Rogers Writer's Trust Prize in 2011.
  List announced 2011-10-13-10, Blaise shortlisted 2011-10-18-22,
  Blaise won $2500 as a finalist for the Prize 2011-11-01-09



http://en.wikipedia.org/wiki/Jamie_Parsley

  Slot:Contact_Meet_PlaceTime

  Fill: ( too many to list )

  Notes: This man could be tracked hourly almost everyday.


http://en.wikipedia.org/wiki/Scotiabank_Per%C3%BA 

  Slot: Affiliate

  Fill: Fitch Ratings

  Notes: Fitch announced its financial rating for entity.


https://twitter.com/FrankandOak 

  Slot: Affiliate

  Fill: C2-MTL / Claudine and Stephen Bronfman Family Foundation contest for emerging entrepreneurs

  Notes: Entity among content winners announced by Bronfman Foundation.


https://twitter.com/BlossomCoffee

  Slot: Affiliate

  Fill: TEDxBayArea at Hult Business School http://www.ted.com/tedx/events/5985In

  Notes: Entity participated in ref'ed tech expo.


https://twitter.com/BlossomCoffee

  Slot: Affiliate

  Fill: Tech Crunch’s Disrupt Conference

  Notes:  Entity attended conference.


http://en.wikipedia.org/wiki/James_McCartney

  Slot: Contact_Meet_PlaceTime

  Fill: Jan. 30 "Late Show With David Letterman" performance, McCartney




http://en.wikipedia.org/wiki/Travis_Mays 

  Slot: Title

  Fill: Assistant (Associate) Head Coach of the University of Texas women’s team

  Context: He has accepted a job this week to become the Assistant (Associate) Head Coach of the University of Texas women's team.

  Notes: The full text states the date he was hired, effective
  immediately.  This one text gave a wealth of information: it tells
  when he was hired by UT, which showed when he left his previous job,
  so it yielded five entries: two Titles and three EmployeeOf


https://twitter.com/KentGuinn4Mayor 

  Slot: Title

  Fill: Mayor-elect

  Context: ... will cease to operate in the first quarter next year unless Ocala Mayor-Elect Kent Guinn and others can derail the plan.

  Notes: within the corpus, he ran for mayor, won, and became mayor of Ocala.


http://en.wikipedia.org/wiki/Haven_Denney

  Slot: Contact_Meet_PlaceTime

  Fill: 2012 World Junior Championships

  Context: 2012 World Junior Championships February 27 through March 4 in Minsk, Belarus

  Notes: the 2012 chanpionships were planned and held within the corpus

