to select the list of Web pages that was forwarded to Ryan and Lipow is problematic, Ryan’s and Lipow’s testimony established that many of the erroneously blocked sites that Edelman identified would be useful and appropriate sources of information for library patrons. 
        All of the reviewers that Janes recruited had some
relevant experience in library reference services or library collection development.  Janes divided the reviewers into two groups, a group of 11 less experienced reviewers, and a group of five more experienced reviewers.  Janes assigned the less experienced group to do a first-round review with the purpose of identifying the most obviously overblocked sites.  The more experienced group was to review the remaining sites (i.e., those that were not obviously overblocked) and to make final decisions regarding these sites.  In the first round, each person evaluated two sets of around 80 sites, and each group was evaluated by two different people.  Each set of sites included the following instructions: 

Look carefully at each of the Web sites on the list.  Please make a notation of any site that appears to meet any of the following criteria: 

a.  Contains information similar to that already found in libraries,

or

b.  Contains information a librarian would want in the library if s/he had unlimited funds to purchase information and unlimited shelf space,

or

c.  You would be willing to refer a patron (of any age) to the site if the patron appeared at a reference desk seeking information about the subject of the site.  For this last criterion, we recognize that you might not refer a young child to a Calculus site just because it would not be useful to that child, but you should ignore that factor.  Informational sites, such as a Calculus site, should be noted.  A site that is purely erotica should not be noted.

Sites that received “Yes” votes from both reviewers were determined to be of sufficient interest in a library context and removed from further analysis.  Sites receiving one or two “No” votes would go to the next round.  In the first round, 243 sites received “Yes” votes from both reviewers, while 456 sites received one or more “No” votes or could not be found.  These 456 sites were sent forward to the second round of judging.  The instructions for the second-round reviewers were the same as those given to the first-round reviewers, except that in section c, the following sentence was added:  “Sites that have a commercial purpose should be included here if they might be of use or interest to someone wishing to buy the product or service or doing research on commercial behavior on the Internet, much as most libraries include the Yellow Pages in their collections.”  The second round of review produced the following results:  60 sites could not be found (due to broken links, 404 “not found” errors, domain for sale messages, etc.),