Perspectives of East Tennessee’s Rural Public Librarians about the Extent of Need for Professional Library Education: A Pilot Study by Bharat Mehra, Kimberly Black and Shu-Yueh Lee, p. 142

From Metadata Creation to Metadata Quality Control: Continuing Education Needs Among Cataloging and Metadata Professionals by Jung-Ran Park, Yuji Tosaka, Susan Maszaros and Caimei Lu, p. 158

Barriers and Challenges to Teaching Reference in Today’s Electronic Information Environment by Denise E. Agosto, Lily Rozaklis, Craig MacDonald and Eileen G. Abels, p. 177

Teaching Information Retrieval With Web-based Interactive Visualization by Peter Brusilovsky, Jae-wook Ahn and Edie Rasmussen, p. 187

Brief Communications & Research in Progress
The first Workshop on the Future of iSchool Doctoral Education: Issues, Challenges, and Aspirations by Paul T. Jaeger, Jennifer Golbeck, Allison Druin and Kenneth R. Fleischmann, p. 201

Editors’ Notes
The Breadth and Depth of LIS Education by Michelle M. Kazmer and Kathleen Burnett, p. 129

Keywords: information retrieval, education, visualization, evaluation, user study

Introduction

Interactive visualization is a powerful educational tool. Visualization can provide a clear visual metaphor for un­derstanding complicated concepts and uncovering the dynamics of important processes that are usually hidden from the student’s eye (Gordin & Pea, 1995). Visualization has been used to enhance the teaching of various subjects ranging from chemistry (Evans, Yaron, & Leinhardt, 2008) to biology (McClean, et al., 2005) to physics (Perkins, et al., 2006). Computer science is one of the most active application areas for educa­tional visualization research (Naps, et al., 2003). In computer and information science (CIS) education, visualization is used almost exclusively in program­ming and data structure courses. We can name dozens of papers devoted to visu­alization of program execution on sev­eral levels from machine-level languages (Butler & Brockman, 2001) to high-level languages (Domingue & Mulholland, 1998; Haajanen, et al., 1997; Levy, Ben-Ari, & Uronen, 2003; Tung, 1998) to algorithms and data structures (Hundhausen, Douglas, & Stasko, 2002; Rößling, Schüer, & Freisleben, 2000). Our claim is that In­formation Science courses could benefit from this powerful technology.
This paper presents our research on us­ing interactive visualization in the non-traditional context of information re­trieval courses. Information retrieval has been in the curriculum of many computer, information, and library science depart­ments for more than 30 years. With the maturity of the World Wide Web, infor­mation retrieval became an important practical subject. Elements of information retrieval are now taught to students of many different specialties. We think that information retrieval provides an interest­ing and important application area for ex­ploring the power of interactive visualization. Over the last six years we developed and made publicly available a range of Web-based interactive visualiza­tion tools supporting various concepts taught in information retrieval courses. These tools were used for several years by faculty members at the University of Pitts­burgh to teach both graduate and under­graduate information retrieval courses. To evaluate the impact of our visualization tools and to solicit student feedback, we ran several classroom studies. The devel­oped set of tools is available on the project home page (http://ir.exp.sis.pitt.edu/res2/ resources.php). The tools, which are run­ning on our servers, could be used by any­one interested in teaching or learning information retrieval. This paper summa­rizes the results of our multi-year effort. To demonstrate our ideas, we present two examples of these interactive Web-based visualization tools for information re­trieval. We also report the results of class­room studies evaluating them. To conclude, we discuss the results and pros­pects for using interactive visualization in the context of information retrieval courses.

Interactive Visualization for Information Retrieval

One of the secrets of the power of inter­active visualization is its ability to un­cover and present in detail processes that are typically hidden from students’ eyes. Being interactive, visualization pro­grams allow the students to explore these processes step-by-step, with different pa­rameters, gaining as a result a deep under­standing of the processes and the con­cepts behind them. Every field has its critical processes and concepts, which could be better understood with the use of visualization.
The core of a traditional information retrieval (IR) course is a set of models, al­gorithms and technologies for process­ing, storing and retrieving textual information. Traditional presentation of this core usually starts with several IR models (such as the Boolean, vector, and probabilistic models and several varia­tions of them) and then follows by ex­plaining how the information is organized and retrieved in each of these models (Baeza-Yates & Ribeiro-Neto, 1999; Korfhage, 1997). The process by which information is retrieved in differ­ent models is one of the hardest topics in an IR course for students to grasp, despite being formalized and well understood by the IR research community. We have ob­served that even Boolean information re­trieval, the simplest of the models, is difficult for many students. At the same time, traditional educational tools—re­search or commercial IR systems—offer little educational help. The process of re­trieving information has several steps, from entering the query to matching the query to the documents to prioritizing the results. In an IR system (even an educa­tionally-oriented one) all these steps are hidden from a user: the only thing that a user can observe is the final results—a list of ordered documents. This is exactly a context that could benefit greatly from the use of interactive visualization and this realization formed the starting point for our research (Brusilovsky, 2002). Over the first years of our project we de­veloped and explored interactive visual­ization tools to visualize the process of retrieving information in several known models: Boolean, fuzzy, vector, and ex­tended Boolean (see Baeza-Yates & Ribeiro-Neto, 1999 and Korfhage, 1997 for the description of these classic mod­els). Since these visualization programs were the first to developed, we had more opportunities to improve them over the years and to evaluate them in the classroom. For the purpose of this paper we chose two of these model visualization tools to demonstrate the ideas of interactive visualization in the context of IR courses. The following subsections present the most recent versions of interactive visualization programs for the Boolean and vector IR models. The next section reports the results of classroom studies with these models.

Interactive Visualization of the Boolean Information Retrieval Model

The Boolean IR model is the oldest and the simplest of the IR models. In this model, a query is formed by a set of ele­mentary queries (usually keywords)con­nected by Boolean operators such as AND, OR, and NOT. The mechanism of this model is set theoretical. Every query is associated with a set of matching docu­ments. For an elementary query such as a keyword, the set of matching documents is simply all documents indexed by this keyword. To obtain the set of matching documents for two queries connected by a Boolean operator, one has simply to per­form the corresponding set operation on their matching sets (i.e., set intersection for AND, complement for NOT, etc.). Thus in several steps, a matching set for any complex Boolean query can be found.
While it all sounds quite simple and clear, we have found that many of our stu­dents have problems understanding how Boolean matching works. Our talks with students have indicated that one of the sources of their troubles is the failure to perceive Boolean operators as operations on sets of matching documents. This is a known problem for everyday users who confuse the informal use of AND, OR, and NOT in everyday language with their formal meaning in set theory and Boolean logic. Surprisingly, we also dis­covered that some students with good programming backgrounds (i.e., those who have routinely used Boolean opera­tors for writing conditional expressions in their programs) still have problems transferring their knowledge of these operators to the set theory context.
In developing an interactive visualiza­tion environment for the Boolean IR model we were trying to achieve two goals: to provide a helpful visual meta­phor and to visualize the process of Boolean matching step by step. Figure 1 presents an interface for our environ­ment. The core of this interface is a set of all documents visualized in a table (one document per row). For a sample document set in our system we choose textbook surrogates since this is the type of document most familiar to information science students. The goal of this visual representation is to help students to understand the core principle of this model—every query is associated with a particular subset of all documents. Showing the set of all documents on the screen makes it easy to demonstrate different subsets of the whole set as sets of differently colored rows of the table.
The students explore Boolean matching by writing and executing simple Boolean queries: pairs of elementary queries (terms) connected by a single Boolean operator (OR, AND), or a pair of operators (AND NOT), which simulate set difference. The results of an executed query are shown through row and cell coloring in the document table. The final set of documents is visible as a set of green rows (documents 2, 8, 12 in Figure 1). This is what a standard Boolean search engine would return. The visualization tool, however, attempts to show more by decomposing the standard “black box” matching process into two parts: (1) the process of matching an elementary query to the set of the documents and (2) the process of obtaining a new set from contributing sets by applying different Boolean operators. To achieve this goal, the tool extends the document table with three colored columns of cells. The first and the second column show the results of matching each of the documents to the first and the second query terms correspondingly. Documents matched to each term are marked as true in the corresponding column. To make the set of matched documents more visible, documents matched to the first term are marked by a red cell background and documents matched to the second term are marked with a blue background. The third column shows the results of matching the whole query using truth values and a green color for matching documents. The color-coding makes the two steps of the Boolean query matching process more transparent. For example, the student can see that only documents 2, 8, and 12, which matched to both elementary terms, are included in the final set. In contrast, documents 4, 10, and 11, which matched only to the first term and document 7, whichmatched only to the second term, are not included (see Figure 1).

Figure 1. BooleanModel Environment. Visualization of matching for a simple Boolean AND query. Documents matching the first elementary query are highlighted.

Beyond the term-based search shown on Figure 1, the Boolean IR visualization tool has several other functionalities. In particular, to help the student transfer the understanding of Boolean IR from classic IR to the database context, we have provided a very similar exploration interface where elementary queries are constructed not from keywords as in classic IR but from restrictions on various fields of a database record (i.e., year = 2000 and publisher ! = “O’Reilly”). Technically, the tool is implemented as a Java servlet working on a dedicated server.

Interactive Visualization of Vector Information Retrieval Model

The vector IR model is different from the Boolean model in many aspects. While the query in this model is also a set of terms(keywords), the terms can be weighted, stressing their relative importance. To perform the matching process, the query is converted into a weighted vector of terms. Similarly, all documents are represented as weighted vectors of terms. The weight of a specific term in a document vector represents the collection-adjusted importance of this term in the document content. The matching process is based on vector algebra. The goal of vector matching is to produce a relevance value for each document, which reflects how similar this document is to the query. The relevance value is produced by calculating the Euclidean or angular distance between the query and document vectors. Once relevance values are calculated, all documents can be ordered by their relevance to the query. Vector matching produces a ranked list, not just a subset of documents as Boolean matching does.

Figure 2. A sample document used by the vector model visualization tool.
On Figure 4 this document is marked as D2.

The hardest component of the vector matching process to understand is the calculation of the relevance value.While the geometrical nature of the relevance measure (Euclidean or angular distance) is relatively straightforward, the actual relevance values are produced by performing many operations with components of very large vectors. Not surprisingly, the final results of these calculations are much less evident to students than the results of Boolean operations with document sets. As we observed in the classroom, it is frequently hard for students to understand why a specific document was ranked particularly high or low in the resulting list.
As in the case of the Boolean visualization tool, the goal of the vector matching visualization was to uncover the steps of the matching process, which are hidden from the users of vector-based search systems. While these systems never go further than displaying the final relevance value of retrieved documents, we wanted to show how this relevance is calculated in the vector model. To uncover this process, our tool offers a small collection of documents formed from just 7 different terms (so that query and document vectors are very short). Figure 2 shows a sample document in this collection. The students explore vector matching by writing and executing simple vector queries, which are weighed sets of terms. A new query can be formed step by step by adding one term (selected from the same set of 7 terms) with its weight at a time. The example in Figure 3 shows a query formed by two terms: term Dog with weight 3 and term Fox with weight 1. For simplicity we use integer weights when forming a query; however, after the query is formed, its vector is normalized.

Figure 3. The process of forming a query in the vector model visualization tool.

The results of the query execution are shown as two tables (on the right side of Figure 4): one table presents ranking based on Euclidean distance and the second presents ranking based on the cosine measure. The more relevant a document is to the query, the smaller the distance measure and the larger the cosine measure. To dig deeper, a student can click on any relevance value (shown in blue) and see in a popup window (bottom of Figure 4) how this value is calculated from the components of the query and document vectors. (The example in Figure 4 uses Euclidean distance.) The vectors for the query and documents are shown on the left alongside the ranking results, so the students can see how the vector components are used in the calculations. By clicking on a document number (shown in blue), students can view the content of each document in a popup window. Thus the visualization tool makes all the traditionally hidden steps of the vector matching process visible: from documents and query to their vectors, from vectors to relevance values, and from relevance values to ranked lists.
In addition to the interactive simulation presented above, the tool also offers a brief tutorial on vector matching. It is implemented using a combination of Java servlets and client-side Javascript programs.

Figure 4. The results of query execution produced by the vector model visualization tool. The popup window at the bottom shows how the Euclidean distance value of 0.71 was produced for document D2 given the document and query vectors shown above.

Classroom Evaluation of Interactive Visualization

Study Design

To examine whether our visualization tools for teaching information retrieval are effective in a real educational process, we conducted several studies with students of information retrieval courses. In each of these studies we evaluated several visualization tools. This section reports results of five classroom studies performed in the context of a graduate course, “Information Storage and Retrieval,” offered annually at the School of Information Sciences, University of Pittsburgh. These studies were performed between Fall 2001 and Fall 2005. The Boolean and vector model visualization tools, which are the focus of this paper, were evaluated in each of the five studies. Table 1 shows the number of students involved. During 5 semesters, a total of 95 students completed the study, with a mean of 19 students per class (SD = 7.58).

Table 1: Classroom Studies of Boolean and Vector Model Visualization Tools.

All studies of our visualization tools have the same multi-stage design presented in Figure 5. The goal of this design was to assess both objectively and subjectively whether the visualization tools are actually working as useful aids to help the students better understand the underlying concepts. As an objective measure, we used knowledge gain from pre-test to post-test. As a subjective measure, we used student answers to a questionnaire about the systems and their features. The studies of different tools were performed during different weeks of the semester. In a given week, the students learned the concepts of the subject (i.e., Boolean or vector model) during a lecture and the instructor introduced the corresponding visualization tool, encouraging them to utilize it for their deeper understanding of the knowledge they acquired in the classroom.

Figure 5. The classroom study procedure.

At the end of the lectures devoted to the Boolean and vector space models, the students took a pre-test and were given homework assignments which, among other things, specifically required them to use the tools introduced in the lecture for actively solving hands-on problems. The role of this homework was to engage the students in active exploration of the visualization tools in the context of problem solving. Figure 6 shows an example of the homework on the vector space model.

Figure 6. Homework assignment example.

To evaluate the results of their learning, the students took a post-test before the next lecture (exactly one week from the first lecture on the topic), and filled in a questionnaire about the tools and their impact. The post-test was identical to the pre-test. The questionnaire included 14 questions, seven for each tool (Table 2). A five point Likert scale was used for the questions ranging from 1 (Strongly Agree) to 5 (Strongly Disagree).

Table 2: Survey Questions for the Subjective Feedback Analysis.

The role of the questionnaire was to evaluate student attitudes to the visualization tools and their features. However, the questionnaire alone was not sufficient for a reliable evaluation. To make sure that the student’s positive (or negative) attitude is grounded in the student learning experience, we had to check whether student knowledge increased after using this tool, and examine the correlation between knowledge gain and attitude. Knowledge increase (positive knowledge gain) after using the tool can support the student’s positive attitude and provide additional evidence in favor of the educational effectiveness of the tool. In contrast, the lack of knowledge increase (or knowledge decrease) would cause us to question the educational value of the tools even in the presence of positive feedback. Note that the format of a classroom study does not allow us to use knowledge gain alone as a reliable indicator of the effectiveness of the tools. While we attempted to minimize student learning from other sources by placing the pre-test after the lecture presentation of the corresponding topic, we were not able to prevent students from using other sources of knowledge during their work on the homework assignment. Only a controlled lab study could assure that the registered knowledge gain was the result of student work with the tools. Given that our main goal was to explore the value of the tools in a real educational process, we accepted the shortcomings of the knowledge gain measure and applied it as a secondary indicator.
To calculate knowledge gain, which measures an increase in the student’s knowledge of the topic, we used pre- and post test scores [Equation (1)]. Because of the minor differences in the number of pre/post-test questions over the study semesters, we used normalized pre- and post-test scores (both were normalized to range from 0 to 10) to calculate the knowledge gain. This allowed us to evaluate the effect of the tools over all five semesters.

Knowledge Gain = Normalized Post Test Score – Normalized Pre Test Score (1)

With two evaluation measures engaged, we can formulate two formal hypotheses to assess whether our Web-based interactive visualization tools are educationally effective.

H1: The student will feel that the visualization tools are helpful in understanding crucial concepts of information retrieval.

More specifically,

H1-1: The subjects will answer positively to the questions asking about the effectiveness of the visualization tools.

H2: Student knowledge will increase after their work with the tools.

That is,

H2-1: The knowledge gain calculated as a difference between post- and pre-test scores will be positive.

The following subsections report the results of the data analysis performed to evaluate these two hypotheses for each of the visualization tools.

Subjective Feedback Analysis

This section analyzes students’ subjective feedback on the visualization tools. Seven questions per model were asked each semester (some semesters also included one additional question). Students’ responses indicated a value from 1 to 5, where 1 is the most positive answer to each question. The distribution of student answers is shown in Tables 3 and 4 and Figures 7 and 8.

Table 3: User Feedback on the Boolean Model Tool.

Table 4: User Feedback on the Vector Space Model Tool.

Figure 7. User feedback for the the Boolean model visualization tool.

Figure 8. User feedback for the vector space visualization tool.

Q1 differs from the rest of the questions since it is focused on the students’ prior knowledge about the corresponding information retrieval model. As the data shows, about 80% of the subjects agreed or strongly agreed that they understood the Boolean model. In contrast, only 23% agreed or strongly agreed that they understood the vector model. This result is consistent with the objective analysis in the previous section, where the average pre-test score of the Boolean model was 7.87 out of 10.0whereas that of the vector model was only 4.58.
The next two questions (Q2 and Q3) assessed the usefulness of the visual and interactive nature of the tools in the classroom. About 80% to 90% of the students gave positive answers (Strongly Agree or Agree) for both the Boolean and the vector models. Almost no negative answers were submitted. Moreover, the interactive nature of the system was valued even higher than its visual nature. This is strong evidence in favor of interactive visualization.
Question 4 asked how easy itwas to understand the interface of the visualization tools. Overall, the students showed a high level of satisfactionwith theBoolean tool interface (83%) and slightly lower satisfaction with the interface of the vector model tool (72%). This indicates that the latter interface may need further improvement.
Questions 5 to 7 asked in different words about the importance and usefulness of the tools in the context of an information retrieval course. Students’ answers confirmed that the vector model tool, which helped them with a less well known topic, wasmost valuable for them. Over 80% of the students agreed that the vector tool should become one of the key course tools and over 90% wanted to recommend the system to students taking the course next semester. Corresponding numbers for the Boolean tool (over 60% and over 80%) are lower stressing that this tool was seen as less valuable. This may be caused by the simplicity of the Boolean model, which was relatively well understood by most the students before the start of the course. Yet, the feedback for both tools is overwhelmingly positive: both tools were considered important and useful by a clear majority of students. Moreover, about 90% of the students agreed or strongly agreed that both tools should be used in teaching information retrieval courses.

Table 5: A Comparison of Test Scores for the Two Models.

Knowledge Gain Analysis

As described earlier, we measured the pre- and post-test scores for two educational visualization tools for information retrieval: one for the Boolean and one for the vector model. Table shows the basic statistics comparing the scores. We can first observe that there are increases between pre- and post-test scores for both models. Test scores for the Boolean model increased by 0.59 and scores for vector model increased by 1.80. These differences were statistically significant (paired t-test) for both the Boolean and vector model tools (p = 0.03 and p < 0.01 respectively). It means there was actual knowledge gain after the students used the visualization tools.
It is interesting to observe that the starting knowledge of the Boolean model was relatively high. In this context it was a challenge for the tools to further increase the student knowledge. Indeed, the increase was quite moderate, although statistically significant. In contrast, the starting knowledge of vector model was relatively low—less than 50%. In this context, the visualization tools helped to achieve a more impressive knowledge gain, almost a 40% increase from the level of knowledge reported by the pre-test. This allows us to hypothesize that interactive visualization tools could be especially useful for teaching hard-to-understand topics, where regular educationalmeansmay be less effective.
It was also interesting that the standard deviation of pre-test scoreswere just 2.38 and 2.23 for the Boolean and vector model respectively. These values are relatively small, suggesting the starting levels of student knowledge within each topic were quite similar. Since this data was collected over five semesters of teaching the course, it could be used as a reliable estimation of student starting knowledge when preparing educational visualizations and other pedagogical material for the course.

Subjective Feedback versus Knowledge Gain

In the previous sections, we saw that there was an increase in students’ knowledge after they took the information retrieval classes, and that they supported the use of visualization tools in the classes. That is, there was an objective effect on their knowledge and the source of the effect was assumed to be the visualization tools, based on their positive response to the questions asking them about the usefulness of the tools. To confirm this assumption, we examined the relationship between the objective knowledge gain and the subjective answer from the students. We chose questions #2 and #3 from the questionnaires, which directly asked about the usefulness of the visual (Q#2) and interactive nature (Q #3) of the tools. The answers to the questions were compared with student knowledge gain scores for the Boolean and the vector model separately. Figures 9 and 10 show the results of these comparisons.

Figure 9. Comparison of subjective user feedback against knowledge gain for Boolean tools.

Figure 10. Comparison of subjective user feedback against knowledge gain for vector tools.

Here, three out of four graphs show a positive relationship between the subjective answers of the students and their actual knowledge gain. The students with higher knowledge gain scores tend to agree more with the statement on the usefulness of the visual nature of the Boolean tool (Figure 9 left). For the statement on the interactive nature of the Boolean tools, the effect is less pronounced and reversed (Figure 9 right). In terms of the vector visualization tools, the students with higher knowledge gain scores tend to be more positive about the visual and the interactive nature of the tools, although the effect is weaker than in the previous case (see Figure 10). Overall, it can be observed that students who learned more (higher knowledge gain) reacted more positively to the visualization tools for both models—Boolean and vector.

Discussion

This paper presents our experience in developing and using a set of interactive visualization tools for teaching and learning information retrieval. By using the power of interactive visualization, our tools allow a teacher to introduce important models of information retrieval in a way that cannot be replicated by traditional whiteboard and slides, and with almost no preparation time (simply by deciding which examples to show to cover the main set of ideas). The instructor can easily accommodate very different audiences by adjusting the number of examples to show, the speed, and the granularity of presentation.
Using these interactive visualizations as learning tools further increases their value. They let the students switch from passive learning-by-reading to active and interactive exploratory learning. By exploring a number of different examples with interactive visualization tools they get a chance to achieve a better understanding of complex IR topics. The results of our classroom studies indicate that use of interactive visualization in a homework context can result in significant growth of knowledge. The vast majority of the students recognize the value of interactive visualization and recommend its use in the context of information retrieval courses. The study also demonstrated that visualization focusing on less known and harder to understand topics causes a larger growth in knowledge and is perceived as more useful. This result suggests placing higher priority on the development of visualization tools for harder to understand topics.
Our results stressed the importance of developing, collecting and sharing tools for teaching information retrieval. This is consistent with the culture of sharing in IR research. A number of groups maintain Web pages of IR resources such as search software, text processing utilities, evaluation packages and topical bibliographies. We argue that similar collections of IR educational resources should be established and maintained. The Web makes it possible for researchers and educators to make their contributions to research and teaching available to others, making it much easier for others to implement. We have contributed to this process by developing a range of interactive visualization tools for teaching and learning information retrieval. We have demonstrated that these visualization tools have value for students learning the concepts behind specific information retrieval models. We have made these freely available to any IR educators or studentswishing to use them on our IR resources page (http://ir.exp.sis.pitt.edu/ res2/resources.php) along with similar educational resources developed by other teams. We welcome readers who wish to try these tools and hope that more educational repositories like ours will be established in the near future.
In our own future work we plan to continue exploration of interactive visualization for information retrieval. We hope to increase the number of visualization systems available for use in information retrieval courses and to perform additional formal studies of these tools.

Acknowledgements

Most of the tools in the original suite of learning environments were developed under the guidance of the first author by MSIS students of the School of Information Sciences at the University of Pittsburgh. Our School is fortunate to have many bright students and the author thanks them for their devotion and creativity. Special thanks go to Olena Scherbinina, Archana Sulebele, and Anand Banvasi, the developers of Boolean and vector model visualization tools presented in this paper.

References

Baeza-Yates, R., & Ribeiro-Neto, B. (1999). Modern information retrieval. NewYork, NY:ACMPress.

Brusilovsky, P. (2002, June 24–29, 2002). Web-based interactive visualization in an information retrieval course. Paper presented at the ED-MEDIA’2002—World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, CO.

Butler, J. E., & Brockman, J. B. (2001). A Web-based learning tool that simulates a simple computer architecture. SIGCSE Bulletin—Inroads, 33(2), 47–50.

Domingue, J.,&Mulholland, P. (1998). An effective Web based software visualization learning environment. Journal of Visual Languages and Computing, 9(5), 485–508.

Evans, K., Yaron, D.,&Leinhardt, G. (2008). Learning stoichiometry: A comparison of text and multimedia formats. Chemistry Education: Research and Practice, 9, 208–218.

Gordin, D. N.,&Pea, R. D. (1995). Prospects for scientific visualization as an educational technology. Journal of the Learning Sciences, 4(3), 249–279.

Haajanen, J., Pesonius, M., Sutinen, E., Tarhio, J., Teräsvirta, T., & Vanninen, P. (1997). Animation of user algorithms on the Web. Paper presented at the VL ‘97, IEEE Symposium on Visual Languages.

Hundhausen, C. D., Douglas, S. A., & Stasko, J. T. (2002). A meta-study of algorithm visualization effectiveness. Journal of Visual Languages and Computing, 13(3), 259–290.

Korfhage, R. R. (1997). Information storage and retrieval.New York, NY: Wiley.

Levy, R. B.-B., Ben-Ari, M., & Uronen, P. A. (2003). The Jeliot 2000 program animation system. Computers and Education, 40(1), 1–15.

McClean, P., Johnson, C., Rogers, R., Daniels, L., Reber, J., Slator, B. M., et al. (2005). Molecular and cellular biology animations: Development and impact on student learning. Cell Biology Education, 4(2), 169–179.

Naps, T., Rößling, G., Anderson, J., Cooper, S., Dann, W., Fleischer, R., et al. (2003). Evaluating the educational impact of visualization. ACM SIGCSE Bulletin, 35(4), 124–136.

Perkins, K., Adams, W., Dubson, M., Finkelstein, N., Reid, S., Wieman, C., et al. (2006). PhET: Interactive
simulations for teaching and learning physics. The Physics Teacher, 44(1), 18–23.

Rößling, G., Schüer, M., & Freisleben, B. (2000). The ANIMAL algorithm animation tool. Paper presented at the 5th annual SIGCSE/SIGCUE ITiCSEconference on Innovation and technology in computer science education, Helsinki, Finland.

Tung, S.-H. S. (1998). Visualizing Evaluation in Scheme. LISP and Symbolic Computation, 10(3), 201–222.

Overall, the involvement of people of all colors at the doctoral level can be fairly characterized as minimal. The need for immediate response is acute (Turock, 2003, p. 493).

Doctoral fellows serve as the nucleus of energy for continued recruitment of a diverse doctoral student population. Attrition through graduation will extend the diversity to the LIS professoriate. Future generations of librarians are educated by the professoriate (Bonnici & Burnett, 2005, p. 125).

As is demonstrated in the larger field of librarianship, there is a serious dearth of minority scholars in the Library and Information Science (LIS) professoriate (Davis & Hall, 2006). The Spectrum Doctoral Fellowship Program is well on its way to producing 12 such candidates. This is an admirable start, but more needs to be done and more minority candidates need to be recruited, promoted, and funded through PhD programs.

During her tenure as the President of the American Library Association (ALA) (1995), Dr. Betty J. Turock, in collaboration with then ALA Executive Director Elizabeth Martinez, spurred the creation of the Spectrum Scholarship Initiative, which was designed to recruit and fund members of underrepresented minority populations through graduate programs in library science. Believing that the country was rapidly changing, Turock felt that the field of librarianship should be changing as well – libraries can only be responsive to their diverse clientele if their staffs are equally diverse. Minority populations in the United States are quickly becoming the majority, and librarianship, long known for being a primarily Caucasian and female field, has not done a good job of being reflective of and responsive to these changes. Turock credits friend and mentor E J Josey with the impetus for the Spectrum Initiative; expressing frustration and “disgust” for the lack of diversity in the library profession, Josey stated that ALA only recruits one minority librarian per year and thinks that’s progressive (personal correspondence with Dr. Turock, May 7, 2009). Determined to change this trend, Turock decided that ALA should recruit at least 50 minority librarians per year. And so Spectrum began.

Turock’s initiative was not a welcome one, and if not for her tireless effort, the Spectrum Initiative would not have come to fruition, as she faced significant resistance from the field and from members of ALA, whose support was needed to pass the initiative. Some library constituents were not interested in promoting Spectrum and its goals; this was perhaps more of a negative response to the discussion of diversity the initiative would generate, and not as much a response to the initiative itself. As Sandra Rios Balderamma (2000), the first director of ALA’s Office for Diversity, discusses, diversity inspires different reactions in different people, and instead of having difficult and revealing conversations, it is easier to stifle and ignore new ideas and initiatives, such as Spectrum.

Sometimes the definitions and visualizations are sharp and explicit: racism, white privilege, homophobia, heterosexual privilege, inequity of access, institutional racism, organizational barriers, apologies and reparation, “illegal” aliens, non-English speaking, non-white, non-user, old boys’ network, and old girls’ network. Sometimes the definitions and visualizations are easier on the senses and perhaps more elusive: celebration of difference, internationalism, intellectual diversity, global village, multiculturalism, organizational cultures, pluralism, diversity of work style, and diversity of learning styles. At times the term is simply empty and unfulfilling and has not earned its credibility (p. 195).

As long time LIS educator and University of Buffalo faculty Lorna Peterson discussed in her 1999 article, issues of diversity are often perceived as threatening by the majority in the field (in this case, White females and some males), and are conflated with issues of race and racism. All related issues, they tend to inspire passionate responses and feelings of exclusion. Peterson states,

If diversity were as non-threatening a concept as the rhetoric of difference would lead us to believe, then expressions of anger would not occur when practices to readdress past discrimination are enacted. “No one helped me to get here!” is their cry, but they don’t recognize that there was no barrier either. Expressed resentment means diversity may be about achieving equity (which means loss of privilege for some), but the scant evidence of progress may mean that diversity is not about equity at all. (p. 18)

Turock’s efforts were about “advancing social justice and human rights within organizations and the profession” and wanting minority librarians, and the populations they serve to not only survive, but thrive (personal correspondence, May 7, 2009). About Spectrum’s long journey to being, Elizabeth Martinez remembers,

It was a grand idea that we developed when I was Executive Director of ALA. At the time, I was frustrated that, after hearing for 20 years how much diversity was a priority for ALA and the profession, there still were no national scholarships for librarians of color. The ALA Council struggled with accepting the proposal, and past president Betty Turock shamed them to vote yes. It was later embraced and supported by library schools and the profession, and I am grateful that there are over 600 graduates. Today it is the largest and most prestigious ALA scholarship. (personal correspondence, May 20, 2009)

Begun with seed money from ALA, donations from every ALA division and personal donations, ALA’s Office for Diversity was created and the Spectrum Initiative launched, and now boasts a formidable number of alumni scholars who have gone through LIS masters programs and now work in the field. Ironically and sadly, despite having made a significant impact on the library profession (Whitwell, 1998; Roy, et. al, 2006; Kenney, 2005), and having benefited over 600 masters level scholars, and now 12 doctoral students, there is still resistance within in the field to support Spectrum and its work. In 2010 Turock launched a new campaign to raise $1 million dollars to continue Spectrum’s mission (to which she personally donated $100,000); when addressing ALA council members at a January 2010 meeting to announce the new initiative, she was met with lukewarm response. “It’s not clear that all Councilors will follow her example. After Councilor-at-large Pat Wand suggested that each Councilor make a donation, the applause was fairly weak” (Oder, 2010, para. 12). Despite these obstacles, the Spectrum Initiative remains “ ALA’s gift to the library education” and Turock’s bold promise to the profession continues to recruit professionals “who will act as bridges” between the understanding of cultures and knowledge and society at large (personal correspondence, May 7, 2009).

New bridges were initiated with the implementation of the Spectrum Doctoral Fellowship program in 2007. In 2007 and 2008, through the benefit of an IMLS grant, the University of Pittsburg and ALA’s Office for Diversity provided full fellowships for 12 Spectrum Doctoral Fellows to pursue advanced Library and Information Science degrees at accredited institutions around the country. The 12 Fellows represent the four underrepresented ethnic populations and are in various stages of study at Rutgers University, the University of Wisconsin-Madison, the University of Pittsburgh, Simmons College, UCLA, and the University of Tennessee at Knoxville. The ultimate goal of this program is to increase racial and ethnic diversity among the profession’s next generation of LIS faculty and leaders.

The dearth and attrition of minority PhD students has long been a topic of discussion in the higher education literature, all disciplines face this challenge (Meacham, 2002; Manzo, 1994; Pruitt & Isaac, 1985). And this issue is even more acute in library and information science (Brown-Syed, et. al, 2008; Franklin & Jaeger, 2007; Reeling, 1992). A derivative issue of diversity recruitment and retention in LIS education (typically addressed in relation to masters level degrees) (Jaeger, et. al, 2010; Dewey & Keally, 2008; Stringer-Stanback, 2008; Winston, 2008; Neely & Peterson, 2007; Barlow & Aversa, 2006; Honma, 2005; Wheeler, 2005; Alire, 2001; Gollop, 1999), minority recruitment and retention into PhD programs deserves fresh and dedicated attention and study. The latest statistics available from the Association for Library and Information Science Educators (ALISE) indicate that the LIS professoriate continues to lack diversity:

Only 3.7% of the fulltime faculty members are Latino, as compared to 14.5% of the total population, while African Americans comprise just 5.5% of the fulltime faculty as compared to 12.1% of the population. In 2002 – 2003, of the 82 LIS doctoral degrees awarded, only two were awarded to African Americans and one to a Latino (Sineath, 2005). As a result, the faculty population in LIS has remained fairly stable in its level of diversity, with the percentage of African Americans and Latinos in LIS faculties changing little since the passage of the Civil Rights Act in the 1960s (Sineath, 2005, as quoted in Jaeger & Franklin, 2007, p. 21).

These rates parallel the rates of minority librarians working as practitioners in libraries. Jaeger and Franklin (2007) propose that a cycle be strengthened and perpetuated; the rationale is that increased numbers of minorities in the LIS professoriate will shape and transform LIS graduate curricula and programs, which in turn will impact and inform the next generations of minority librarians, who will then adequately and appropriately serve the diverse communities that patronize libraries. And hopefully, these minority librarians will model and inspire up-and-coming students to pursue librarianship as a career. This “virtuous” cycle underscores the dire importance of recruiting and retaining minority LIS PhD students and emphasizes the need for librarianship (at all levels) to be representative of the communities being served.

In the circular, self cycling style of education and librarianship, minority school and public librarians who serve as role models for minority children may inspire the children to go to college. In college, minority academic librarians and library school faculty may inspire them to go to graduate school to become librarians and role models themselves.” (Totten, as quoted in Jaeger and Franklin, 2007, p. 23)

A direct response to Dr. Betty Turock’s call for action, to address the “acute need” for minority LIS PhD students (Turock, 2003, p. 493), the tagline of the Spectrum Scholarship Initiative is the future is overdue. The continued recruitment and retention of minority PhD candidates will have a direct and lasting impact on the LIS professoriate and the field of librarianship as a whole.

References

Alire, C. A. (2001). Diversity and leadership. Journal of Library Administration, 32(3), 99-114.

Balderrama, S. R. (2000). This trend called diversity. Library Trends, 49(1), 194-214.

Barlow, D. L., & Aversa, E. (2006). Library professionals for the 21st century academy. Advances in Librarianship, 30, 327-364.

Bonnici, L. & Burnett, K. (2005). A Web model of recruitment for LIS doctoral education: Weaving in diversity. In M. B. Wheeler (Ed.), Unfinished business: Race, Equity, and Diversity in Library and Information Science Education (pp. 199-130). Metuchen, NJ: Scarecrow Press.

Brown-Syed, C., Baker, L., & Wicks, D. A. (2008). Doctoral recruitment factors: Results of a survey of deans and directors. Journal of Education for Library and Information Science, 49(2), 107-115.

Davis, D.M. & Hall, T.D. (2006). Diversity Counts: Office for Research and Statistics & Office for Diversity (ALA). Retrieved March 20, 2010, from htp://www.ala.org/ala/ors/diversitycounts/DiversityCountsReport.pdf

Dewey, B. I., & Keally, J. (2008). Recruiting for diversity: strategies for 21st century research librarianship. Library Hi Tech, 26(4), 622-629.

Franklin, R. E. & Jaeger, P. T. (2007). A decade of doctorates: An examination of dissertations written by African American women in library and information studies, 1993-2003. Journal of Education for Library and Information Science, 48(3), 187-201.

Gollop, C. J. (1999). Library and information science education: Preparing librarians for a multicultural society. College & Research Libraries, 60(4), 385-395.

Honma, T. (2005). Trippin’ over the color line: The invisibility of race in library and information studies. InterActions: UCLA Journal of Education and Information Studies, 1(2), 1-26.

Jaeger, P. T., Bertot, J. C. & Franklin, R. E. (2010). Diversity, inclusion, and underrepresented populations in LIS research. The Library Quarterly, 80(2), 175-181.

Jaeger, P. T. & Franklin, R. E. (2007). The virtuous circle: Increasing diversity in LIS faculties to create more inclusive library services and outreach. Education Libraries, 30(1), 20-26.

Kenney, B. (2008). The spectrum initiative: Affirmative action in the library profession. Reason and Respect, 1(1), 1-3.

Manzo, K. K. (1994). Flaws in fellowships: Institutional support essential to boosting number of African American doctoral students. Black Issues in Higher Education, 11(10), 46-52.

Marton, F. (1981). Phenomenography: Describing conceptions of the world around us. Instructional Science, 10, 177-200.

Meacham, J. (2002). Our doctoral programs are failing our under graduate students. Liberal Education, 88(3), 22-28.

Neely, T. Y. & Peterson, L. (2007). Achieving racial and ethnic diversity among academic and research librarians: The recruitment, retention, and advancement of librarians of color. A White Paper by the ACRL Board of Directors Diversity Task Force. Chicago, IL: Association of College and Research Libraries.

Oder, N. (2010, January 17). ALA 2010 midwinter meeting: Spectrum scholarship fund gets $100,000 boost from Turock. Retrieved April 20, 2010, from Library Journal.com website: http://www.libraryjournal.com/article/CA6715423.html?industryid=47133

Peterson, L. (1999). The definition of diversity. Journal of Library Administration, 27(1), 17-26.

Pruitt, A. S., & Isaac, P. D. (1985). Discrimination in recruitment, admission, and retention of minority graduate students. Journal of Negro Education, 54(4), 526-536.

Reeling, P. G. (1992). Doctorate recipients in library science: How they compare with doctorate recipients in other disciplines. Journal of Education for Library and Information Science, 33(4), 311-329.

Roy, L., Johnson-Cooper, G., Tysick, C., & Waters, D. (2006). Bridging boundaries to create a new workforce: A survey of spectrum scholarship recipients, 1998-2003. Spectrum Survey Report. Chicago, IL: The American Library Association’s Office for Diversity.

Stringer-Stanback, K. (2008). Recruitment, retention & diversity in libraries & higher education: Why doing the right thing is easier said than done. North Carolina Libraries, 66(1), 25-27.

Turock, B. J. (2003). Developing diverse professional leaders. New Library World, 104(11/12), 491-498.

Wheeler, M. B. (Ed.). (2005). Unfinished Business: Race, Equity, and Diversity in Library and Information Science Education. Metuchen, NJ: Scarecrow Press.

Whitwell, S. (1998). News fronts-ALA-why we need the spectrum scholarship. American Libraries, 29(10), 6.

Winston, M. (2008). Diversity: the research and the lack of progress. New Library World, 109(3/4), 130-149.

Innovative Services Improvised During Disasters: Evidence-Based Education Modules to Prepare Students and Practitioners for Shifts in Community Information Needs by Lisl Zach and Michelynn McKnight, p. 76

Social Networking Websites: An Exploratory Study of Student Peer Socializing in an Online LIS Program by Lili Luo, p. 86

Job Advertisements for Recent Graduates: Advising, Curriculum, and Job-seeking Implications by Robert K. Reeves and Trudi Bellardo Hahn, p. 103

Brief Communications & Research in Progress
Digital Library Education Lab by Xia Lin and Eileen Abels, p. 120

Book Review
The Portable MLIS: Insights From the Experts (Ken Haycock & Brooke E. Sheldon, Eds.) by Robert Wedgeworth, p. 125

Editors’ Notes
Current Choices and Delivery Methods by Michelle M. Kazmer and Kathleen Burnett, p. 53

Keywords: archives employment, content analysis, job advertisements, job market, library employment, MLS students

Introduction

Job advertisement analyses have peppered the LIS research and professional literature since at least the 1980s. Some of those studies have taken a longitudinal view of trends in employer expectations and others have looked at ads within a narrow range of time. Some have looked at opportunities across the field and others have focused on positions in specific functional areas of libraries or archives. Researchers have typically employed methodologies involving quantitative analysis of data derived from a content analysis, which has yielded statistics on geographic distribution of jobs, salaries, types of experience required, and personality and technical skills necessary to be successful in the job.

The purpose of this study was to conduct a quantitative analysis of current job ads and job descriptions appropriate for recent MLS or MLIS graduates. Originally, the primary goal was to collect supporting data for a comprehensive review of the MLS program at the College of Information Studies (the iSchool) at the University of Maryland—to ensure that the MLS curriculum provides the theoretical and practical education needed for graduates as they prepare to enter the information professions. The findings however are relevant to current graduates in all ALA-accredited LIS programs, and the methodology used may be useful to other schools that wish to embark on a similar analysis in the future.

Literature Review

In designing our study we aimed to utilize as many of the methods already used as possible, but we did have to adapt them quite a bit, as will be seen. In developing a standardized list of terms for the different words used to express similar requirements, we used existing studies as much as possible, which simplified the process.

The most common methodology for job advertisement studies in the past involved the collection of a discrete sample of ads based on a number of predetermined criteria—often period of time and intended audience. All studies listed their sources for the ads and they tended to be nationally relevant publications and online sources such as Library Journal, American Libraries, Chronicle of Higher Education, and Web sites and electronic lists associated with library and information professional organizations (for example, American Library Association, Special Libraries Association, or Society of American Archivists).

The study by Sproles and Ratledge (2004) was one of the few to focus primarily on positions directed toward entry-level librarians. They analyzed ads from American Libraries over a twenty-year period to explore how required skills for entry-level academic librarian positions had changed and to see which knowledge, skills, and abilities (KSAs) would qualify new graduates for such professional positions. They analyzed a sample of nearly 1500 ads, removing those for duplicate, non-permanent, part-time, and community college positions. They classified the remaining ads according to whether the primary responsibility was reference, technical services, or systems. An advertisement met the entry-level requirement if it met any one of three criteria: the job was listed specifically as “entry-level,” there was no explicit requirement for professional experience, or the required skills and experience were such that an entry-level librarian could have reasonably gained them at the paraprofessional level. For the job ads not excluded by these criteria, Sproles and Ratledge then coded for the required qualifications by category—education, experience, knowledge, and personal attributes. Their findings upheld two of their hypotheses, that over the years of the study employers increasingly required experience and knowledge that could not always be gained through the existing library school curricula and that practical experience, either through employment or internships, was virtually mandatory. Their third hypothesis, that the number of entry-level job ads was decreasing, was not supported by the evidence (entry-level positions remained rather constant over time), though it was clear that the qualifications for those entry-level positions had increased, especially in terms of previous experience (Sproles & Ratledge, 2004, p. 19).

Sproles and Ratledge referred often to Reser and Schuneman’s study from 1992, which used job ads published in 1988. Though Reser and Schuneman did not focus entirely on entry-level ads, they did single them out through the coding done for the “work experience” qualification (p. 53). Their methodology for defining an ad as entry-level was more restrictive than that used by Sproles and Ratledge; they made no allowance for KSAs that an entry-level librarian could have reasonably acquired at the paraprofessional level.

Beile and Adams (2000) sought to update the 1992 Reser and Schuneman study. They discovered a marked decrease in the number of advertised positions from the period covered by Reser and Schuneman. These decreases disproportionately affected cataloging positions and Beile and Adams suggested that libraries were instead outsourcing cataloging work or relying on paraprofessionals for copy-cataloging in conjunction with bibliographic utilities. Another of their key findings was the “definite trend toward the use of non-MLS-degreed professionals to fill systems positions” (p. 345). Their final observation was about the increased specialization of academic library positions, as evidenced by their coding of position titles as compared with the coding of Reser and Schuneman (1992); after standardization, Beile and Adams (2000) found twenty-two distinct job titles while Reser and Schuneman (1992) had found only twelve (Beile & Adams, 2000, pp. 344-345). Both studies cautioned against relying too heavily on salary data reported in the job ads, noting that salaries are often open to negotiation and may ultimately be higher than initially announced.

Grimes and Grimes (2008) employed more advanced statistical methods than most other studies to analyze the MLS degree’s role in academic libraries over a thirty-year period. They examined how various job characteristics and requirements correlated with the MLS requirement over time. While this specificity of purpose is not relevant to our present study—the MLS being required was the initial requirement for inclusion in our content analysis—Grimes and Grimes made use of a number of categories for data collection that were useful to include in our study.

Deeken and Thomas (2006) tracked changes in technical services job ads since 1995. Like the others, their methodology provided some useful models for our study. They employed many of the same criteria for sources and data collection; notable differences in their methodology were the use of Microsoft Access to collect and organize the data and the inclusion of non-print sources for advertisements. They also pointed out a potential problem with the more current ads: many referenced a job title and institution and then referred potential applicants to a full description posted on a Web site. They found that, due to the time difference between the original posting and their analysis, many Web sites no longer had the advertisement. They also noted that should this trend continue the value of studies of this nature will be seriously questioned (Deeken & Thomas, 2006, p. 138). Starr (2004) likewise pointed out that the drastic changes in how information professionals apply for jobs has greatly affected the ability to conduct studies similar to those conducted in the 1990s and early 2000s. Starr recognized that online job ads are essential to getting a comprehensive picture of the job market; with online library job banks such as ALA JobLIST, LISJobs.com, and LibraryJournal.com Job Zone supplanting the formerly traditional print outlets for ads, retrospectively analyzing job ad data is much more difficult without “advance planning for paper or electronic archiving” (Conclusion section, para. 1). According to Jones (2003) the benefits to job seekers of timely, easy access to a wide array of ads means online posting of ads will not only continue, it likely will grow.

The advertisement studies described so far all focused on positions available in academic libraries. Studies on ads appropriate for other specializations within LIS are scarcer. Adkins and Esser (2004) examined skills required for entry-level children’s librarians. Adkins (2004) looked at job ads for public librarians with a focus specifically on youth services. Adkins’s study covered ads published between 1971 and 2001, and thus many position requirements are now obsolete, especially in regard to technology. Nevertheless, Adkins’ listing of potential KSAs present in these ads was useful when we created our own standardized list.

One study that has received significant attention over the last few years is “The Entry-Level Gap” (Holt & Strock, 2005). As in earlier studies, they analyzed a sample of ads based on certain predetermined criteria. While they presented little direct statistical evidence, Holt and Strock raised provocative questions about changes in the employment outlook for newly-minted MLS graduates due to a steady excess of graduates in proportion to available jobs, the blurring of lines between paraprofessional and professional responsibilities, and the increased need for library experience as a qualification beyond the MLS for employment in entry-level librarian positions (pp. 37-38). Conners and McCarthy (2007) challenged Holt and Strock’s methods and conclusions. They analyzed Library Journal employment surveys from 2000 through 2005 and found that “at least two-thirds of new job seekers were able to get full-time permanent professional positions, and [that] employment of any type ranged from 83 percent to 93 percent” (p. 45). In a follow-up article, Holt and Strock (2007) emphasized that employment of any kind does not disprove the “entry-level gap,” by which they meant that many MLS graduates found themselves in positions that do not require the MLS degree due to the glut of experienced professionals in the market. Maatta (2007), writing more than two years after Holt and Strock, refocused the discussion to entry-level salaries. Maatta reported that none of the institutions she queried had expressed any great difficulty in placing graduates, but the most pressing concerns among recent MLS graduates were “salary levels [which] were not competitive with other professions” (p. 30) and a lack of interesting jobs in their current geographic location. Thus, apart from the salary issue, Maatta found that a lack of jobs was not the problem so much as graduates not wanting to (or not being able to) move to where jobs were available. A more recent addition to this discussion mentioned the entry-level position outlook in public libraries; Tuck (2008) stated that “in public libraries, there really are no entry-level jobs unless there are no applicants with public library experience” (p. 10). Tuck then extended this observation to academic libraries, her point being that experience would not be such an important factor for screening applicants if such a large pool of unemployed librarians were not available.

Some job advertisement analyses have concentrated on a specific segment of the profession. For example, Frederiksen (2005) examined ads for access services librarians. Within the area of library technical services, studies by Copeland (1997), Anthony and Garbs (2005), and Zhu (2008) analyzed ads for various cataloging librarian positions ranging from serials cataloging, to general academic library cataloging, to head of cataloging services positions; Fisher (2001) analyzed positions for acquisitions librarians; and Croneis and Henderson (2002) looked at electronic resources librarian advertisements. Foote (1997) analyzed requirements for systems librarians from 1990-1994. Wu and Li (2008) focused on jobs for reference librarians in health services libraries. Jones, Lembo, Manasco, and Sandy (2002) explored recruitment for science librarians, while White (1999) looked at academic subject specialists more broadly and Zhang (2008) focused on academic librarian positions that required foreign language skills. All of these studies covered similar ground to the more general analyses; they provided insights regarding methodology and KSAs, though their narrow focus limits the ultimate usefulness of their findings.

Another group of studies analyzed their source materials for specific KSAs, either to gauge how they have changed over time or whether specific requirements are included at all. Zhou (1996) analyzed ads from 1974 to 1994 for trends in computer-related skills. Clyde (2002) looked at job ads to see what role user instruction played in the requirements. Riggs (2005) explored how the arrival of Encoded Archival Description had changed ads in the archival profession. Promís (2008) investigated how qualities described in library job ads match up against established definitions of Emotional Intelligence (EI) competencies as defined by Goleman (1998); this study in particular provides an excellent framework for standardizing terminology from job ads in the area of desired skills and personal characteristics (p. 27).

Methodology

The methodology for collection and content analysis of job ads was based on components of earlier studies as well as specific considerations needed to provide sufficient data to the MLS curriculum review. To limit the analysis to recently-available jobs, only ads published or posted online between April 15, 2006 and May 10, 2009 were included. Sources of ads included two national library publications (American Libraries and Library Journal), two electronic lists (Maryland’s iSchool Discussion list and the Archives and Archivists list sponsored by the Society of American Archivists), and two Internet job banks (USAJobs.gov and LISJobs.com). The Chronicle of Higher Education and School Library Journal were evaluated for inclusion as sources for the study, but the first tended to have more senior positions that required professional experience and the latter tended not to include job ads at all. In order to increase the corpus of relevant and recent data for analysis, requests were sent to several electronic lists for local chapters of library and archival organizations asking for position descriptions for jobs suitable for, or recently filled by, entry-level MLS graduates.

Data collection was relatively straightforward, although in certain cases it was not possible to retrieve ads as far back as April 2006, in one case because the former incarnation of the iSchool listserv was not accessible following the switch to a new discussion list name. In another case, where ads contained little information beyond a URL and a note to find more extensive information online—similar to what Deeken and Thomas (2006) found—and the URLs no longer pointed to active Web pages, the ads had to be excluded from the corpus.

Decisions had to be made regarding the data points to collect and how best to standardize similar terms for positions requirements. Following the example set by Sproles and Ratledge (2004), the following criteria were used when deciding whether or not to classify a job ad as entry-level:

• Ad says “entry-level”
• No mention of professional experience
• No experience or duties impossible for entry-level librarians to gain

The last criterion, although it created the potential for some subjectivity in the content analysis, was determined to be an important one to include; many job ads, even though they did not specifically reference professional experience, seemed inappropriate for entry-level MLS graduates. Only ads that required an MLS or MLIS degree from an ALA-accredited institution were included; those for positions where the MLS was preferred but not required were excluded. Many archives positions would accept a Master’s in History, Museum Studies, or another related field if coursework in archival science was a component of the degree; these ads were included so long as the ALA-accredited MLS was specifically mentioned as an option. Part-time or temporary positions of less than nine months were excluded.

Three major areas of content analysis were identified—basic information regarding the job, requested personal attributes, and requested knowledge and experience. The basic information to be collected included: date, source of advertisement, position title, location (by state), salary, whether the job was for a library or archives, type of institution (academic, government, public, etc.), type of position (generalist, public services, technical services, systems), and years of non-professional experience required. For the personal attributes data points, the Emotional Intelligence (EI) competency conversion table developed in Promís (2008) was utilized to allow for a high level of detail when coding for individual EI traits but also for a more aggregate reporting of EI groupings. The data points selected were divided into categories for general, library/archives, and technology. The library/archives area included functional areas of experience: administrative, instruction, public/access services, reference, systems, and technical services. The technical services functional area was further sub-divided into acquisitions, arrangement and description, cataloging, collection management, electronic resources, general, metadata, records management, and serials. Also, because job descriptions often requested more than one area of technical services experience, the data intake form was formatted so that up to three areas could be coded.

After the content analysis, certain additional data points were defined based on information already gathered. For instance, geographic location was supplemented with census region and division information (U.S. Census Bureau Geography Division, 2000). Fields were added to indicate when at least one EI trait was requested for each of the five major EI groupings. Technical services experience was summarized to indicate how many different areas within the technical services grouping were requested per advertisement.

The content analysis was completed using a form and table in a custom Microsoft Access (Access) database and then the data were exported into a Microsoft Excel (Excel) spreadsheet for further manipulation. The resulting dataset was reviewed to remove as much duplication as possible, although because of the frequent posting of positions in multiple sources (or re-posting in the same source), it is possible that some duplicates remained. The de-duplication process became more of an art than a science in cases where the same position was advertised twice, but more than three or four months apart. Quantitative data analysis was accomplished through features available in Excel, with certain calculations and Analysis of Variance (ANOVA) testing processed using SPSS. Whenever practicable, the data were analyzed in terms of whether the ad was for an archives or library position, as well as the aggregate of the two.

Results

Sources of Job Advertisements

As shown in Table 1, the final data set included 1042 ads from seven different sources (the “Direct from employer” sources resulted from inquiries to employers via local electronic mailing lists—the number is relatively small because we had already found on our own most of the position descriptions that employers sent to us). A little more than 90% came from print sources or electronic lists and less than 10% came from Web sources.

Table 1: Entry-Level Job Advertisements by Source

Given the Maryland-centric orientation of the data collection, it is not surprising that almost half of the advertisements were for positions in the South Atlantic region that includes Maryland (78), Virginia (57), and the District of Columbia (103). Over 50% of the advertisements were for positions located on the East Coast, with California (85), Texas (63), and Ohio (40) having a relatively large number of the available positions outside of the Eastern Seaboard.

Position and Institution Types

Positions available in academic libraries and archives comprised nearly two-thirds of the ads, with the next highest institution types represented being public (13.0%) and other (8.5%) libraries/archives (see Table 2). When the percentage of overall “public” and “other” ads is broken into just library or archives, though, we find that the percentage of public archives ads at 0.7% is much lower than the percentage of public library ads at 17.5%. A similarly wide variance can be seen between the percentage of “other” library ads (2.2%) to the percentage of “other” archives positions advertised (25.4%). These differences in occurrence of job ads by institution type illustrate one of the principal difficulties encountered throughout the study—agreeing on a common set of terminology for library and archives positions despite the marked differences between the two professions. In the library field, the designation “special” is fairly standard but there is no such correspondingly broad category in the archives field. To that end, the “other” institution type for archives encompasses positions at a wide variety of places such as the American Academy of Otolaryngology, the Buffalo Bill Historical Center, the Historical Society of Frederick County (Maryland), and the National Baseball Hall of Fame—institutions that likely would have been categorized as “special” were the positions library-related. Similarly, while the institution type “public library” will be fairly intuitive to the average person familiar with libraries in general, the two archives positions listed with an institution type of “public” may seem somewhat incongruous; despite that, the “public” institution type was used because the positions were, in fact, archivist positions in public libraries.

Table 2: Job Ads by Institution Type

Looking at the number of archives/library jobs as classified first by institution type and then by position type (see Table 3), we get a picture of the distribution of jobs across both categories. The two largest-represented position types in archives are generalist and technical services, with both categories covering a wide range of responsibilities. The relatively low number of entry-level public services positions for archival institutions should not be seen to imply that public service responsibilities are not common in those institutions, only that they are not the primary responsibility of the positions as described in the ads. Many positions listed as generalist may involve reference or instruction responsibilities, but, with limited information in the ad itself, we were unable to code them more specifically. The greatest number of entry-level library positions found (52.2%) were for public services positions, which, for the purposes of this study included jobs with reference, instruction, or access services as their primary responsibilities. The next highest number of library jobs (23.9%) was in the various technical services functional areas. In both position and institution type coding, the designation “other” was applied to descriptions that did not clearly fall in the other defined categories; for example, Regional Sales Manager, Blackwell’s Book Services in the master data file is a library position that has both a position and institution type of “other.”

Table 3: Number of Position Types per Library or Archives Institution Type

Figure 1. Percentage of entry-level positions by position type

Figure 2. Percentage of entry-level positions by institution type

Salaries
Salary data were available for 401, or 38.5%, of the 1042 total ads examined. Figures 3 and 4 show the variations in salary by U.S. Census Region and Division. Though data for all census divisions are still included in Figure 4, note that salary information for very few positions in the New England (8), East South Central (14), and Mountain (2) census divisions were available, an issue that should not significantly affect the interpretation of salary figures by census region, but may call into question the ability to interpolate salary data in those specific divisions.

Figure 3. Average salary distribution by census region

Figure 4. Average salary distribution by census division

Analysis of Variance tests were performed to check whether all salary means are equal (null hypothesis) vs. at least one salary mean is not equal (alternative hypothesis) across position and institution types, with a significance level of α = 0.05. A similar ANOVA test was performed to assess the significance of the variation in salaries between all publication/posting years. In all ANOVA tests performed, p ≤ 0.001, allowing us to say that the alternative hypothesis is true and that the variation between means is statistically significant. Figure 5 shows that, although average salaries have not increased dramatically over the three plus years covered by this study, they have been increased each year, with a more significant increase in the salaries for positions posted in the first five months of 2009 over the average for all 2008 salaries. Even if the salaries do not reflect the actual negotiated salary, we were comparing only the advertised salaries across the three-year span.

Figure 5. Average salary 2006-2009

A final comparison of salaries was computed by institution and position type. Table 4 presents the salary breakdowns for archives and libraries and Table 5 presents the aggregate salary across institution and position types for both archives and libraries.

Table 4: Minimum Salary Reported by Archives/Library, Institution Type and Position

Table 5: Minimum Salary Reported by Combined Archives/Library, Institution Type and Position

Personal Attributes

As stated above, information coded for personal attributes requested in job ads was done in accordance with the Emotional Intelligence (EI) rubric utilized in Promís (2008). Table 6 shows that certain EI traits—especially those under the Self Awareness grouping—were found sparingly or not at all. Service Orientation, Collaboration & Cooperation, and Team Capabilities appeared in at least 20% of the ads, and Communication appeared in over 50%.

Table 6: Emotional Intelligence Traits by Individual Trait

When viewed in terms of how often at least one EI group was coded as present in a job ad (see Table 7), traits from the Social Skills, Self Regulation, and Empathy groups are the most commonly found, with Motivation appearing in a significant proportion of the ads and Self Awareness traits not commonly found. These findings demonstrate that entry-level positions are written in such a way as to emphasize social competence traits (from E4 and E5) as well as those personal competence traits (from E1, E2 and E3) that reflect on an applicant’s ability to adapt to change, meet deadlines, innovate, and take initiative.

Table 7: Emotional Intelligence Traits – At Least One Present Per Grouping

Knowledge, Skills, and Abilities

The types of experience listed in Table 8 were divided into two categories, general and technology. Subsequent tables provide greater detail on required years of non-professional experience and specific areas of library/archives experience. Relatively few ads (5.5%) called for a second Master’s degree or fluency in a foreign language (7.7%) and, similarly, only 12% required any specific subject expertise. For archives positions, experience with preservation or conservation of physical objects was a common requirement. Supervisory experience was requested slightly more often in archives positions than in library positions.

In terms of technology KSAs, general information technology (IT) skills such as the use of Web browsers, office productivity software, and e-mail were the most commonly requested. Nearly one-fifth of positions required experience with automated library systems, with specific integrated library systems often mentioned. The 21.1% of archives positions that requested digitization experience (either in terms of digital preservation or digitization for access) is noteworthy, as is the nearly 2 out of every 5 archives descriptions that called for knowledge of programming or markup languages—generally Encoded Archival Description (EAD).

Table 8: Requested Knowledge, Skills, and Experience

Subject area knowledge, though often mentioned as desirable, was infrequently a required qualification. Only one subject area—Sciences—was requested in more than 5% of either the archives or library positions over the period covered.

Experience

The majority of archives and library job ads did not explicitly require previous experience as a measure of years (see Table 9). Combined, 30.9% asked for at least one year of non-professional experience related to the advertised position. When separated, 40.8% of archives positions and 27.2% of library positions requested at least one year of experience. Notwithstanding those figures, specific types of library experience were frequently requested, though, for the most part, this was not expressed in terms of required years.

Table 9: Minimum Experience Required (Non-Professional)

A great many job ads requested work experience in specific functional areas of libraries and archives. Technical services, reference, and instruction were the most commonly requested areas of experience requested by libraries. Technical services was by a large margin the most requested functional area in archives job ads (see Table 10).

Table 10: Requested Library Experience by Functional Area

Taking a closer look at the types of technical services experience being requested (see Table 11), cataloging and metadata experience are two functional areas often requested regardless of whether the ad is for a library or archives position. Nearly half of all archives positions (45.4%) requested experience with arrangement and description, with records management experience showing up in only 37 of the 284 (13.0%) total archives positions. The technical services experience required for library positions is more distributed across the entire list of functional areas, with arrangement and description appearing in 1.7% of advertisements, likely due to the overlapping archives/library responsibilities in special collections libraries.

Table 11: Requested Technical Services Experience by Area

Conclusions

The data provide evidence that the availability of entry-level positions is increasing year-to-year over the period covered by this study. While this should be seen as an encouraging sign for soon-to-graduate students, it is clear that without significant library or archives experience (even including in some cases supervisory experience), those graduates will have a hard time finding themselves qualified for a professional position. Thus, this study underscores the conclusion in Sproles and Ratledge (2004) that practical experience throughout the graduate program—whether through assistantships, internships at local institutions, or part- or full-time employment—should be a part of every student’s portfolio.

The fact that overall salaries are increasing, even if only slightly, can also be viewed as heartening in these difficult economic times. However, we did not address the issue that during the same period as this study, many potential positions were not filled—job freezes, outsourcing, and position consolidations at least temporarily shrunk the pool of positions actually available for new graduates.

Even with some pre-professional experience, most graduates will need to look for positions in academic settings; only a small percentage of entry-level jobs were found in public and special libraries/archives.

Graduates interested in archives will find the most opportunities as generalists or with specialties in technical services (particularly in preservation or conservation of physical objects). This contrasts with library opportunities for new graduates, where most jobs are in public services positions (however, a significant number of positions are in technical services as well). With the importance of specific technical services experience in both libraries and archives for entry-level employment prospects, it is important to ensure that those functions are woven into the curriculum, especially with respect to how they relate to the more commonly taught information access courses.

Employers are seeking to hire individuals who not only have certain skills and experience, but also certain personal attributes such as excellent communication abilities, service orientation, a predilection for collaboration and cooperation, a penchant for participating in teams, and social and personal competence traits that reflect on an applicant’s ability to adapt to change, meet deadlines, innovate, and take initiative. It is not immediately apparent how these attributes can be taught—or whether they can be taught. Nevertheless, job applicants cannot ignore the fact that employers prefer individuals who have these personal attributes.

Finally, as several recent studies found and we found to be increasingly true in the last year or so, job ads typically appear now online, which is a distinct advantage for job-seekers, but a serious problem for future studies of this type if the ads are not captured and stored before they disappear.

References

Adkins, D. (2004). Changes in public library youth services: A content analysis of youth services job advertisements. Public Library Quarterly, 23(3/4), 59-73.

Adkins, D., & Esser, L. (2004). Literature and technology skills for entry-level children’s librarians: What employers want. Children & Libraries: The Journal of the Association for Library Service to Children, 2(3), 14-21.

Anthony, P. & Garbs, J. (2005). A scarce resource? A study of academic cataloger recruitment 2000-2002. Cataloging & Classification Quarterly, 41, 45-62.

Beile, P., & Adams, M. (2000). Other duties as assigned: Emerging trends in the academic library job market. College & Research Libraries, 61, 336-347.

Clyde, L. A. (2002). An instructional role for librarians: An overview and content analysis of job advertisements. Australian Academic & Research Libraries, 33(3). Retrieved from http://alia.org.au/publishing/aarl/33.3/full.text/clyde.html

Conners, D., & McCarthy, L. (2007, September). The jobs can be found. Library Journal, 132(14), 44-45.

Copeland, A. W. (1997). The demand for serials catalogers: An analysis of job advertisements, 1980-1995. The Serials Librarian, 32, 27-37.

Croneis, K., & Henderson, P. (2002). Electronic and digital librarian positions: A content analysis of announcements from 1990 through 2000. Journal of Academic Librarianship, 28, 232-237.

Deeken, J., & Thomas, D. (2006). Technical services job ads: Changes since 1995. College & Research Libraries, 67, 136-145. 

Fisher, W. (2001). Core competencies for the acquisitions librarian. Library Collections, Acquisitions, & Technical Services, 25, 179-190.

Foote, M. (1997). The systems librarian in U.S. academic libraries: A survey of announcements from College & Research Libraries News, 1990-1994. College & Research Libraries, 58, 517-526.

Frederiksen, L. (2005). Access services librarians: A content analysis of job advertisements, 1977-2004. Journal of Access Services, 3(2), 15-27.

Goleman, D. (1998). Working with Emotional Intelligence. New York, NY: Bantam.

Grimes, M., & Grimes, P. (2008). The academic librarian labor market and the role of the Master of Library Science degree: 1975 through 2005. Journal of Academic Librarianship, 34, 332-339.

Holt, R., & Strock, A. (2005). The entry-level gap: Breaking into the profession is harder than ever—if you’re fresh out of library school. Library Journal, 130(8), 36-38.

Holt, R., & Strock, A. (2007). The entry-level gap, revisted. Library Journal, 132(16), 44.

Jones, L. L. (2003). Job searching in the new millennium. Christian Librarian, 46(2), 55-58.

Jones, M. L. B., Lembo, M. F., Manasco, J. E., & Sandy, J. H. (2002). Recruiting for entry-level sci-tech librarians: An analysis of job advertisements and outcome of searches. Sci-Tech News, 56(2), 12-16.

Maatta, S. (2007, October 15). What’s an MLIS worth? Library Journal, 123(17), 30-38.

Promís, P. (2008). Are employers asking for the right competencies? A case for emotional intelligence. Library Administration & Management, 22(1), 24-30.

Reser, D., & Schuneman, A. (1992). The academic library job market: A content analysis comparing public and technical services. College & Research Libraries, 53, 49-59.

Riggs, M. (2005). The correlation of archival education and job requirements since the advent of Encoded Archival Description. Journal of Archival Organization, 3(1), 61-79.

Sproles, C., & Ratledge, D. (2004). An analysis of entry-level librarian ads published in American Libraries, 1982-2002. Electronic Journal of Academic and Special Librarianship, 5(2/3). Retrieved from http://southernlibrarianship.icaap.org/content/v05n02/sproles_c01.htm

Starr, J. (2004). A measure of change: Comparing library job advertisements of 1983 and 2003. LIBRES: Library & Information Science Research Electronic Journal, 14(2). Retrieved from http://libres.curtin.edu.au/libres14n2/Starr_final.htm

Tuck, E. N. (2008, March 1). The experience trap. Library Journal, 133(4), 10.

U.S. Census Bureau Geography Division. (2000). Census regions and divisions of the United States. Retrieved from http://www.census.gov/geo/www/us_regdiv.pdf

White, G. W. (1999). Academic subject specialist positions in the United States: A content analysis of announcements from 1990 through 1998. Journal of Academic Librarianship, 25, 372-382.

Wu, L., & Li, P. (2008). What do they want? A content analysis of Medical Library Association reference job announcements, 2000-2005. Journal of the Medical Library Association, 96, 378-381.

Zhang, L. (2008). Foreign language skills and academic library job announcements: A survey and trends analysis, 1966-2006. Journal of Academic Librarianship, 34, 322-331.

Zhou, Y. (1996). Analysis of trends in demand for computer-related skills for academic librarians from 1974 to 1994. College & Research Libraries, 57, 259-272.

Zhu, L. (2008). Head of cataloging positions in academic libraries: An analysis of job advertisements. Technical Services Quarterly, 25(4), 49-70.

Theories-in-Use and Reflection-in-Action: Core Principles for LIS Education by Phillip M. Edwards, p. 18

Research Articles
Educating Future Academic Librarians: An Analysis of Courses in Academic Librarianship by Edgar C. Bailey, Jr., p. 30

WISE Economics: ROI of Quality and Consortiums by Bruce Kingma and Kathleen Schisa, p. 43

Editors’ Notes
Integrating Social Theories into LIS Research by Laurie J. Bonnici and Manimegalai M Subramaniam, p. 1

Also Included in This Issue by Kathleen Burnett and Michelle M. Kazmer, p. 3

JELIS 2009 Satisfaction Survey by Sung Jae Park and Janet L. Capps, p. 4

Keywords: theories-in-use, reflection-in-action; collection development, search techniques, questionnaires, small group instructional diagnosis

For decades, organizational learning theorists have engaged in critical analyses of professionals’ work practices. Through this work, they have attempted to uncover 1) how individuals develop theories to guide their practices, and (2) how individuals adapt these theories to deal with more complex professional situations. Argyris and Schön (1974/1992), for example, explore how professionals apply theories-in-use during their work, and they characterize conditions (dilemmas) under which these theories-in-use could be challenged. Schön (1983) expands upon these notions to consider reflection-in-action, a process through which professionals engage in inquiry and “reflective conversation” with a complex situation in order to address dilemmas. This article examines the extent to which these two concepts—theories-in-use and reflection-in-action—could align with typical learning outcomes associated with LIS education, using two illustrative case studies: one from an undergraduate-level course on search strategies, and one from a graduate-level course in collection development.

Background and Purpose

Published work on the professions and the nature of professional education is rich and varied. Several touchstone publications (Schein, 1972; Schön, 1987; see also, with a sharper focus on librarianship and the information professions, Abbott, 1988) have contributed to and shaped the contemporary discourse around education for the professions. More recent and overarching shifts in higher education toward learner- or learning-centered teaching have offered an expanded range of approaches for characterizing the roles of teachers and students (e.g., Brookfield, 1995; hooks, 1994; Weimer, 2002). Beyond the teacher-student relationship, the recent conceptualization of “authentic professional learning” by Webster-Wright (2009) is suggestive of promising new directions for the design of continuing education programs for the professions, implicitly reflected in recent proposals by Ball (2008) and Mehra and Robinson (2009). Teaching professional skills and values to diverse sets of students in a manner that respects their prior experiences while reflecting authentic tensions that emerge across a variety of work settings is an on-going challenge within LIS education.

 Application of two concepts from the literature on organizational learning—theories-in-use and reflection-in-action—to the LIS classroom may effectively facilitate the alignment of course content, instructors’ expectations, and students’ needs. In the following discussion of these concepts, preference has been given to two “classic” works. This choice, while sacrificing breadth, permits a deeper exploration of each of the associated models as conceived by the original authors. The current application of these models to LIS practice, as evidenced through literature searches, is relatively sparse; the specific application of these models to LIS education has yet to be fully articulated.

Theories-in-Use

The theories-in-use concept arises from the work of Chris Argyris and Donald A. Schön, most thoroughly described in their 1974 monograph, Theory in Practice: Increasing Professional Effectiveness. Theories, for these authors, serve as “vehicles for explanation, prediction, or control” (p. 5); furthermore, theories and practices are inextricably linked within the professions:

A profession, then, not only has a practice but also a theory of action in which that practice can become a reproducible, valid technique. This means that the job of professional education consists not only in teaching technique but in teaching the methods by which behavioral worlds in which techniques can work can be created. (p. 149)

Argyris and Schön distinguish between espoused theories of action (i.e., what someone says they would do under certain circumstances) and theories-in-use (i.e., what someone actually considered when acting, subject to their assumptions and prior experiences). While there may be some level of incompatibility between an individual’s espoused theories and theories-in-use, the latter serve as useful but inherently limited heuristics for managing the complexity of the professional work environment. What occurs during professional education, therefore, is “the experience-based modification of some elements of theories-in-use—governing variables, action strategies, or assumptions” (Argyris & Schön, 1974/1992, p. 18).

In order to achieve this kind of learning, Argyris and Schön focus on the exposure and resolution of dilemmas arising from students’ theories-in-use, including dilemmas of incongruity, dilemmas of inconsistency, dilemmas of effectiveness, dilemmas of value, and dilemmas of testability:

In an appropriately redesigned professional curriculum, courses would describe these inconsistencies and incongruities and perhaps begin to deal with them… Exploring these issues should have several positive consequences. First, the students would realize from the outset the extent of the gap between their academic training and practice. Second, while at school, they would begin to think about and test with each other their own theories of practice. Third, students could then press for the additional courses they need to make their budding theory of practice more effective. (p. 176)

Given this stance on professional curricula, Argyris and Schön (1974/1992) present two complementary models for student learning, which they label model I and model II. Their tone suggests a moderate preference for the second model:

In general, we believe, those strongly disposed to learn model II possess two important characteristics. First, these individuals are able to listen to feedback about the errors they make as they try to learn model II, so long as that feedback is crafted to produce minimal defensiveness, and to strive to correct their errors once these have been pointed out to them. Secondly, once these individuals have received valid and constructive feedback, they tend to “hang in” in order to learn. For example, even if it happens that every new action they try out seems wrong and they feel stuck, they realize that recognizing their stuck-ness is necessary for learning. (p. xxiii)

Thus, the ideal instructional setting would be “a learning environment that produces valid information about each participant’s espoused theories, theories-in-use, and any inconsistencies within each theory as well as among them” (Argyris & Schön, 1974/1992, p. 97). This approach “makes dilemmas recognizable, which creates tension to resolve them [and] this tension motivates learning” (p. 97). A similar shift appears in Lewin’s (1951/1997) three-step model of change in group performance—a process that requires “unfreezing” of a group’s initial understanding of a situation, “moving” through some form of intervention or dialogue, and “freezing” of the newly negotiated group standards.

Reflection-in-Action

Schön’s later work illustrates how professionals and students resolve these dilemmas through a process he terms reflection-in-action. Schön (1983) suggests:

When the phenomena at hand eludes ordinary categories of knowledge-in-practice, presenting itself as unique or unstable, the practitioner may surface and criticize his initial understanding of the phenomenon, construct a new description of it, and test the new description by an on-the-spot experiment. Sometimes he arrives at a new theory of the phenomenon by articulating a feeling he has about it… When he is confronted with demands that seem incompatible or inconsistent, he may respond by reflecting on the applications which he and others have brought to the situation. Conscious of a dilemma, he may attribute it to the way in which he has set his problem, or even to the way in which he has framed his role. He may then find a way of integrating, or choosing among, the values at stake in the situation. (p. 62-63) 

These notions of reflection are not unique to Schön’s writing, however; perhaps the most notable proponent of reflective thought and its role in education is John Dewey (1910/1997). Schön and Dewey both consider education an essential mechanism through which these habits of reflective practice are learned. Schön (1983) claims that “when a practitioner makes sense of a situation he perceives to be unique, he sees it as something already present in his repertoire [of examples, images, understandings, and actions]” (p. 138). This notion of reflective contemplation as an educative process continues to be carried forward by educational philosophers such as Greene (1988).

Potential Applications of Theories to LIS Education

There is a great deal of conceptual overlap between the notions of theories-in-use and reflection-in-action, which suggests a coherent, constructivist foundation for many graduate- and undergraduate-level courses in LIS. General principles for course design associated with these concepts might include: 1) valuing and incorporating student experiences and interests in class sessions and assignments, particularly those that express espoused theories of developing professional practice (cf. theories-in-use); 2) creating authentic exercises that allow students to test their espoused theories against situations emerging from practice (cf. exposing dilemmas); 3) focusing on the collection of evidence—including lived experience, analysis of internal processes in a particular work context, and gathering information about the various communities being served in a particular setting—in support of their choices as developing professionals (cf. resolving dilemmas); 4) offering students a non-threatening space for dialogue with their peers and with practicing professionals to refine their theories-in-use (cf. resolving dilemmas); and 5) allowing students structured and deliberate opportunities for reflection on course content and their own perceptions, attitudes, and assumptions (cf. reflection-in-action). These principles and the theoretical concepts upon which they are based may serve as elements of evaluation as well, given that these qualities may emerge from students’ reflections on their experiences during the course. In the following cases, the concepts of theories-in-use and reflection-in-action are used in both senses: as aids for course design and as lenses for understanding students’ feedback for a particular course.

Method

The School of Information and Library Science (SILS) at the University of North Carolina at Chapel Hill (UNC-CH) supports four degree programs—the B.S. in Information Science (B.S.I.S), the M.S. in Information Science (M.S.I.S.), the M.S. in Library Science (M.S.L.S.), and the Ph.D. in Information Science—and in the discussion that follows, I consider two courses from these programs as cases: the undergraduate-level INLS 200 (Retrieving and Analyzing Information) and the graduate-level INLS 513 (Resource Selection and Evaluation). Representative syllabi for these courses—including course descriptions and anticipated student learning outcomes—are available from my website as archived course materials (Edwards, n.d.).

As a standard practice in my courses, multiple forms of anonymous feedback are collected from students at various points during each semester. I collect anonymous written feedback from students near the third or fourth week of the term, based around open-ended prompts such as:

• Are there things that we’ve been doing that have been particularly helpful for how you’re learning?
• Are there things that could be modified to better match your needs?

Near the middle of the semester, I invite a consultant from the Center for Faculty Excellence at UNC-CH to visit my classes to conduct Small Group Instructional Diagnosis (SGID) sessions over two class periods. The SGID is a formative approach for instructional improvement that has been used in postsecondary education since the mid-1970s (Clark & Redmond, 1982; Diamond, 2004). During the first class period, the consultant collects written feedback from all enrolled students in response to the prompts:

• List the major strengths in the course. (What is helping you learn in the course?) Please explain briefly or give an example for each strength, and
• List changes that could be made in the course to assist you in learning. Please explain how suggested changes could be made.

In the next class session, the consultant returns to the class with a compiled list of themes that emerged from the written feedback and engages students collectively in a member-checking exercise. The final set of results—the students’ transcribed written responses, categorized into several themes, along with any themes that were confirmed, added, or modified based on member-checking—are returned to me.

The final form of anonymous feedback consists of written responses to a SILS-specific questionnaire, generated by students during the final class session of the course. Completed questionnaires are submitted to the SILS office, routed internally among the SILS administration, and returned to instructors after students’ course grades are recorded. All of the representative qualitative responses that follow were gathered anonymously from students in these courses through the above means of feedback collection during the Fall Semester 2008 (INLS 513, with 31 enrolled students; denoted as [F08] below) and the Spring Semester 2009 (INLS 200, with 42 enrolled students, and INLS 513, with 35 enrolled students; denoted as [S09] below).

Results

Theories-in-use and reflection-in-action in an undergraduate-level course

The B.S.I.S. program is designed to “prepare its graduates for a variety of careers in the information industry, including information architecture; database design and implementation; Web design and implementation; and networking support and information consulting; as well as for graduate study” (SILS, 2004a). The INLS 200 course is a prerequisite that undergraduate students complete prior to their admission to the B.S.I.S. program.

In my sections of INLS 200, students are primarily responsible for researching a self-selected topic for inclusion as an entry in Wikipedia. To assist students in their searching and writing, students complete several graded reviews of print- or digitally-based sources that relate to their topic, several rounds of peer review of their classmates’ in-progress entries, and a summative reflective essay on the role of search in daily life and their experiences while researching and writing during the semester.

One of the challenges in teaching a class such as this relates directly to students’ theories-in-use related to searching, as one student expressed during feedback collected in the fourth week of Spring Semester 2009: “We know what searching is and what the value of information is. We use it every day—we do not need to go over it every other class” [S09]. By the time written feedback was collected during the ninth week of this semester, several students made comments which suggest this reticence was melting, largely a consequence of students’ reflections on how small group work and class discussions had enhanced their learning:

“When we have class discussions, it is very interesting and helpful to hear others’ points of view and the different ways they use and think about the material” [S09];

“Openness to speak up in class; ex. I don’t feel embarrassed to throw out an answer to one of his questions when I do in other courses” [S09];

 “Working in groups to actually apply the things we have learned is very helpful because we can see how they can be used and get help and new ideas from classmates” [S09].

By the ninth week of the course, many students also commented on the helpfulness of relevant, authentic examples and assignments as a key factor in helping them learn:

“After taking this course for half a semester, I have become [a] more savvy Internet and database user” [S09];

“I learned many things that I just passed by—privacy on the Web, use of Blackboard, Sakai, etc.” [S09];

 “The material taught is helpful and applicable to daily life” [S09];

“Articles relate to programs that we as students incorporate into our daily lives [S09]; and

“On-hand [sic: hands-on] relevant work; by creating a Wikipedia page over the course of the semester, it makes topics learned in class more relevant” [S09].

Despite the above feedback, which might be seen as the application of reflection-in-action to dilemmas exposed from the course materials and activities, some students expressed frustration regarding the level at which the course was being taught:

“Assume we know nothing about computers, for example, a lot of us are not majors and need step by step directions” [S09];

“Don’t take students’ knowledge of the subjects for granted; I (plus many others) are unfamiliar with developing a user page on Wikipedia, and wish [the instructor] knew that; he was open to a question today when someone basically said they had no idea” [S09].

Another student voiced a contrary assessment:

“Take this class out of the major/minor requirement for INLS; I haven’t learned anything, most material is intuitive and doesn’t need to be taught” [S09].

By the seventeenth week of the semester, the above sentiments about the intuitiveness of course materials and activities continued to emerge, albeit less frequently, during the summative feedback collected about the course:

“We have no idea how to write Wikipedia articles. I don’t think it is a beneficial/helpful thing to learn, felt like a waste of time” [S09]; in contrast with,

“Class material was either too abstract to be worth learning or too intuitive to be taught” [S09].

Nevertheless, over half of the students (9 out of 17) who completed the summative course evaluations mentioned that the most useful learning experience of the course was the research and writing for their Wikipedia entries. One student mentioned peer review activities as being most helpful, and three mentioned the resource review assignments. The students’ final reflective essays, the analysis of which will appear in a separate report, revealed more insight into the students’ changing perceptions of themselves as not only consumers of information but also as producers of information and designers of information resources.

Theories-in-use and reflection-in-action in a graduate-level course

The two masters’ programs at the School of Information and Library Science, the M.S. in Information Science (M.S.I.S.) and the M.S. in Library Science (M.S.L.S.), are guided by complementary sets of competencies (see SILS, 2004b; SILS, 2004c). Within this framework, graduate students may enroll in classes that satisfy requirements for either program. The INLS 513 course is a requirement for the M.S.L.S. program and an elective for the M.S.I.S. program.

In my sections of INLS 513, students are primarily responsible for three projects: 1) a review of a reference resource or substantive work of fiction or nonfiction; 2) selecting resources for a collection associated with a particular community, supported by insights students gain from a community analysis; and 3) a comparative critique of two collection development policies, self-selected by individual students. As with INLS 200, students complete several rounds of peer review of their classmates’ in-progress drafts, and they write short personal reflections to accompany these assignments throughout the semester.

In contrast with INLS 200, however, the graduate students in INLS 513 appeared to be more initially aware of the importance of others’ ideas for their own learning. For example, a student commented during the fourth week of the semester that: “I really enjoyed class when you brought books in so we could evaluate them and hear from everybody their opinions of what should be considered” [S09]. This attitude—that there was a benefit to students’ learning from small group work and discussion—was reflected in the written feedback collected during the ninth week of the semester as well; students noted that strengths of the course included:

“Facilitating [discussion] so people can bring their experiences, ask questions” [F08];

 “Encouraging students to examine perspectives which may be different from their own; interaction with students and including participation in learning” [S09]; and

“Asking us to think/question” [S09].

These sentiments were not universally shared, however; not all students were as apt to see these course activities as being helpful for their learning. By the ninth week of the semester, several students had remarked:

“Learning primarily as a result of completing assigned readings – prefer more focused class discussion and involvement of teacher” [F08];

“Stop the group work; it’s unrealistic for us as graduate students for scheduling and it doesn’t reflect ‘the real world’. Group work wastes my time since I’ve already been in the real world and end up holding people’s hands to get through group projects” [S09];

“Final assignment: I understand the point of submitting drafts, providing feedback, presenting, etc. but not sure this is necessary at all (for me personally, at least) in the learning process” [S09]; and

“Group work is detrimental to my learning because people get way off topic and screw around until the professor calls us back, I learn nothing” [S09].

As with INLS 200, the authenticity of examples, activities, and assignments was an aspect of the course that, upon reflection, students’ found helpful for their learning at week nine:

“Helpful to hear how concepts from readings might appear in different library settings” [F08];

“Use[s] ‘real’ examples to explain collection development rules – online material, online policy, etc. which made it easy for me to understand” [S09];

“[The instructor’s] ‘welcome to class, do ya have any collection development news?’ is a good way to remind us that all of this is potentially relevant to our future careers” [S09];

 “The assignments feel directly relevant to what we are learning in class as they draw on topics we’ve covered without mere rote memorization” [S09]; and

“There is a lot I like about this class and I don’t have many complaints. This has been one of the most constructive classes in SILS so far. I love how interactive the class is, how it relates theory to practice and I feel the projects are relevant” [S09].

Again, attitudes toward the level of authenticity in course activities were not uniform across the class by the ninth week; several students’ reflections on hindrances to their learning at that point included:

“Don’t feel like I am learning specific things” [F08];

“Utilize more solicitation of student experience for this. Philosophical style is fun and makes class enjoyable, however, it makes the usefulness of readings and class discussion less apparent. Emphasize utility of discussions to future work” [F08];

“The content is a bit obvious; collection development is more common sense than learning skills or such, so the class itself has so far been a waste of my time. Nothing [the instructor] can do about that though since it is a required course” [S09];

“Assignments don’t seem related to real-life work; critiquing and/or deconstructing, evaluating policies is perhaps much more useful than having us write policies that in real life are done by committee” [S09]; and

“More practical interactions with materials; more short exercises of play acting as policy writer, collection developers, etc.” [S09].

By the seventeenth week of the course, students’ reflections on their learning in the course seemed to regard many of the activities as being more helpful for their learning, after the “intuitiveness” of course concepts may have been explicitly challenged to a greater extent. For example, many students noted that the opportunity to learn from others’ experiences and opinions was a benefit:

“I liked the mix of PowerPoint and student input; [the instructor] encouraged people to share their opinions and didn’t make it feel like his ideas were better than yours” [F08];

“I really enjoyed the peer reviews for Assignment 3. Very helpful!” [F08];

“Peer editing of final papers – very useful” [S09];

“I like how he asks for class members to bring up topics at the beginning of class” [S09];

“[The instructor] was respectful and encouraged student input” [S09]; and

“The group conferences were particularly helpful” [S09].

There were limits to this approach, however, several of which were expressed as:

“When someone would ask the teacher a question, instead of answering it he often threw the question out to the class to see what they thought. Sometimes you just want the teacher to answer the question” [S09];

“Group projects with young, inexperienced, highly sensitive students, and insanely busy schedules. We work!” [S09].

By the conclusion of the course, students were also generally more positive about the authenticity of activities, assignments, and examples used in the class:

“The group project [was most useful] because it imitated what we would have to do in real life” [S09];

“I liked the first assignment (evaluating a reference source) the most – it seemed to be the most relevant to what I will do as a librarian” [S09];

“I appreciate [the instructor’s] many practical examples” [S09];

 “The reading which pertained to or were actual institutions’ policies and community analyses were helpful” [S09]; and

 “In general, the practical course readings of actual policy really helped me understand the concepts” [S09].

In some cases, students mentioned being able to better understand the breadth of the topics covered and ways in which course topics related to other aspects of the profession:

“I liked the way [course organization] was done—starting with details and moving out to the bigger picture” [F08];

“Now that I’ve taken this class I’m way more aware of collection development issues and how they fit into other classes I’ve taken” [S09]; and

“The worst part for me was the fact that collections can seem to be a nebulous subject to have a conversation about.” [S09]

Nearly half of the students who completed the summative course evaluations mentioned that the most useful learning experience of the course was the collection building assignment where community analysis techniques were applied as a small group (see Table 1).

Discussion

In the early stages of these courses, at both the undergraduate level and the graduate level, a substantive instructional challenge is posed by many students’ initial perceptions that everything is intuitive—that their existing theories-in-use are sufficient to guide their practices. Argyris and Schön (1974/1992) recognize this challenge:

Especially during the early phases of learning, participants will probably feel frustrated and uncomfortable and doubt whether they can learn model-II behavior; they may even doubt whether model II is valid. (p. 97)

In terms of search techniques, many aspects of the search process are not obvious, even if students have tacitly adopted particular aspects in their existing practices: to identify the “best” sources for addressing a particular information need; to evaluate sources based on their quality, authoritativeness, bias, authorship, credibility, currency, and intended audience; to develop skills for formulating effective search queries (cf. Taylor’s (1968) “compromised need”); and to synthesize the results of searches for a particular audience. In terms of collection development, tensions emerge in professional practice: balancing demands to acquire materials based on patron demand versus professional judgment; managing an effective portfolio of order types (approval plans, standing orders, blanket orders, and firm orders) across multiple vendors; accommodating requests for reconsideration of materials; remaining aware of current trends within the publishing industry (e.g., bundling and licensing); and making selection decisions subject to budgetary constraints.

Introducing authentic materials from practice that confront the “obviousness” of students’ theories-in-use—how they approach the decisions they make as searchers and selectors—appear to provide a space in which students can learn from their peers, reflect on their attitudes and practices, and modify their ways of thinking about their roles in their future careers. In-class discussions and other opportunities for students to work in small groups around authentic examples, as an instructional design strategy, seem to be effective mechanisms for aligning the content of the course with students’ experiences and professional goals.

Within LIS, Bell and Shank (2007) propose the application of an inquiry-/design-based paradigm to librarianship. There is evidence elsewhere in the literature (e.g., Watson-Boone, 2000) that librarianship and the information professions are increasingly incorporating inquiry-, design-, and evidence-based strategies for approaching the complexity of modern practice. One of the more complete characterizations of the role that design plays in the information professions comes from Bell and Shank (2007), who build upon the stages of the ADDIE model—analyze, design, develop, implement, and evaluate—and characterize success of a program or service by the impact it has upon user learning. While not all library services or programs may have learning at their core, it is possible to think of the aim of successful design as producing excellent user experiences, which may include learning as well as, say, positive affective responses. This general model bears some fine-level distinctions with other approaches, such as action research (e.g., Wilson, 2000), evidence-based librarianship (e.g., Booth & Brice, 2004; Eldredge, 2000), information needs assessment (e.g., Davis, 2004; Evans & Saponaro, 2005), community analysis (e.g., Consulting Librarians Group, n.d.), and contextual design (Beyer & Holtzblatt, 1998) as well as emerging approaches in positive design (Faust, 2009) and studio-based design (Snyder, Heckman, & Scialdone, 2009), among others.

Interestingly, the two assignments that students noted as being most helpful for their learning across both of these courses—in INLS 200, the Wikipedia entry, and in INLS 513, building a small core collection for a particular community, were arguably the assignments that were most design-based. While working in Wikipedia, undergraduate students interacted with their classmates as well as Wikipedia administrators and editors, and this process helped shape their understanding of how their final entries should be written, researched, and structured. While working with their community analyses as lenses through which materials could be identified and selected, graduate students grappled with trying to understand the needs, capabilities, and constraints of a particular community when building their collections. Both of these projects involved the collection of some evidence about what “acceptable” products would look like, the reflection on this evidence, and some amount of dialogue about how to design the final product.

Conclusions

The preceding discussion is intended to be persuasive rather than prescriptive, and, at times, it may appear more personal in nature than readers typically observe in JELIS articles. There are many different approaches to teaching within LIS, as evident from the pages of JELIS over the years, and as an author, I am hard-pressed to claim that my application of organizational learning concepts to LIS education represents the single, unassailable way forward. My personal stance on LIS education strives for a balance between behavioral and progressive philosophies of adult education (cf. Elias & Merriam, 2005). Many readers may see glimmers of their own approaches to teaching in these concepts and cases without drawing explicit support from the models discussed above; others may use cognate models—for example, the taxonomy of educational objectives (Anderson, Krathwohl, Airasian, Cruikshank, Mayer, Pintrich, Raths, & Wittrock, 2001), case study approaches (Galvin, 1973), experiential learning (Kolb, 1976), communities of practice (Wenger, 1998), or transformative learning (Mezirow & Associates, 2000)—to guide their teaching.

That said, the five principles derived above from the concepts of theories-in-use and reflection-in-action may be broadly applicable to the design and assessment of the effectiveness of classroom activities, exercises, and assignments. Exposing and resolving dilemmas emerging from professional practice, while 1) not universally perceived by students as being useful educational activities, and 2) seemingly more influential among graduate students than among undergraduate students, appears to benefit many students. Deriving shared yet flexible conceptual frameworks for guiding course design and assessment, driven by the best available evidence on how adult students learn, are clearly on-going and evolving processes. What remains essential for continuing this conversation—the importance of which is highlighted by Wallace (2009)—is a community of educators who are willing to share, openly and transparently, the benefits and challenges of their various approaches.

References

Abbott, A. (1988). The system of professions: An essay on the division of expert labor. Chicago: University of Chicago Press.

Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Complete ed.). New York: Longman.

Argyris, C., & Schön, D. A. (1992). Theory in practice: Increasing professional effectiveness. San Francisco: Jossey-Bass. (Original work published 1974).

Ball, M. A. (2008). Practicums and service learning in LIS education. Journal of Education for Library and Information Science, 49(1), 70-82.

Bell, S. J., & Shank, J. D. (2007). Academic librarianship by design: A blended librarian’s guide to the tools and techniques. Chicago: American Library Association.

Beyer, H., & Holtzblatt, K. (1998). Contextual design: Defining customer-centered systems. San Francisco: Morgan Kaufmann.

Brookfield, S. D. (1995). Becoming a critically reflective teacher. San Francisco: Jossey-Bass.

Booth, A., & Brice, A. (2004). Evidence-based practice for information professionals: A handbook. London: Facet.

Clark, D. J., & Redmond, M. V. (1982). Small group instructional diagnosis: Final report. Washington, D.C.: Fund for the Improvement of Postsecondary Education. (ERIC Document Reproduction Service No. ED 217 954)

Consulting Librarians Group. (n.d.). Community analysis methods and evaluative options: The CAMEO handbook. Retrieved October 6, 2009, from http://skyways.lib.ks.us/pathway/cameo/

Davis, D. M. (2004). Needs assessment. In J. C. Bertot & D. M. Davis (Eds.), Planning and evaluating library networked services and resources (pp. 187-221). Westport, CT: Libraries Unlimited.

Dewey, J. (1997). How we think. Mineola, NY: Dover. (Original work published 1910).

Diamond, M. R. (2004). The usefulness of structured mid-term feedback as a catalyst for change in higher education classes. Active Learning in Higher Education, 5(3), 217-231.

Edwards, P. M. (n.d.). www.unc.edu/home/pmedward/. Retrieved October 6, 2009, from http://www.unc.edu/home/pmedward/

Eldredge, J. D. (2000). Evidence-based librarianship: An overview. Bulletin of the Medical Library Association, 88(4), 289-302.

Elias, J. L., & S. B. Merriam. (2005). Philosophical foundations of adult education (3rd ed.). Malabar, FL: Krieger.

Evans, G. E., & Saponaro, M. Z. (2005). Information needs assessment. In Developing library and information center collections (5th ed.). Westport, CT: Libraries Unlimited.

Faust, J. (2009). Positive design. Journal of the American Society for Information Science & Technology, 60(9), 1887-1894.

Galvin, T. J. (1973). The case method in library education and in-service training. Metuchen, NJ: Scarecrow.

Greene, M. (1988). The dialectic of freedom. New York: Teachers College Press.

hooks, b. (1994). Teaching to transgress: Education as the practice of freedom. New York: Routledge.

Kolb, D. A. (1976). The learning style inventory technical manual. Boston, MA: McBer and Company.

Lewin, K. (1997). Frontiers in group dynamics. In Resolving social conflicts & field theory in social science (pp. 301-336). Washington, D.C.: American Psychological Association. (Original work published 1951).

Mehra, B., & Robinson, W. C. (2009). The Community Engagement Model in library and information science education: A case study of a collection development and management course. Journal of Education for Library and Information Science, 50(1), 15-38.

Mezirow, J., & Associates. (2000). Learning as transformation: Critical perspectives on a theory in progress. San Francisco: Jossey-Bass.

Schein, E. H. (1972). Professional education: Some new directions. Berkeley, CA: Carnegie Commission on Higher Education.

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco: Jossey-Bass.

School of Information and Library Science (SILS), University of North Carolina at Chapel Hill. (2004a). Degrees and programs: BSIS overview. Retrieved October 6, 2009, from http://sils.unc.edu/programs/bsis/

School of Information and Library Science (SILS), University of North Carolina at Chapel Hill. (2004b). Degrees and programs: MSIS overview. Retrieved October 6, 2009, from http://sils.unc.edu/programs/msis/

School of Information and Library Science (SILS), University of North Carolina at Chapel Hill. (2004c). Degrees and programs: MSLS overview. Retrieved October 6, 2009, from http://sils.unc.edu/programs/msls/

Snyder, J., Heckman, R., & Scialdone, M. J. (2009). Information studios: Integrating arts-based learning into education of information professionals. Journal of the American Society for Information Science & Technology, 60(3), 1923-1932.

Taylor, R. S. (1968). Question-negotiation and information seeking in libraries. College & Research Libraries, 29(3), 178-194.

Wallace, D. P. (2009). The iSchools, education for librarianship, and the voice of doom and gloom. Journal of Academic Librarianship, 35(5), 405-409.

Watson-Boone, R. (2000). Academic librarians as practitioner-researchers. Journal of Academic Librarianship, 26(2), 85-93.

Webster-Wright, A. (2009). Reframing professional development through understanding authentic professional learning. Review of Educational Research, 79(2), 702-739.

Weimer, M. (2002). Learner-centered teaching: Five key changes to practice. San Francisco: Jossey-Bass.

Wenger, E. (1988). Communities of practice: Learning, meaning, and identity. Cambridge, UK: Cambridge University Press.

Wilson, T. D. (2000). Recent trends in user studies: Action research and qualitative methods. Information Research, 5(3). Retrieved on October 6, 2009, from http://informationr.net/ir/5-3/paper76.html

JELIS is a scholarly double-blind peer-reviewed publication of ALISE. As such, the goals of ALISE are foundational to the vision statements of the JELIS editors. Capps andHinson noted that, “both co-editors [Burnett and Kazmer] strongly agree with the need to address how the changing needs of libraries affect the education of librarians. Other current goals they hold for JELIS include: (a) what is taught, (b) how to teach the topics, (c) what is known about research, and (d) how the research is used” (2009, p. 64). These goals are in alignment with the ALISE goals; however, if JELIS is to be a venue for research dissemination, it is important to measure the perceptions of how well the journal is serving the needs of the research community.

The last systematic measurement of JELIS subscriber satisfaction was over 25 years ago. In 1984, the Readership Survey was conducted at the urging of the ALISE Board of Directors to the Editorial Board (Patterson, 1985). Patterson, who served as the JELIS editor from 1980 to 1988, stated, “a study had never been done and both boards were concerned about the status of the Journal among other refereed publications in the profession” (1985, p. 302). Patterson highlighted the 31-item questionnaire results in the JELIS 25(4) article titled, “An Assessment of the Status of the Journal” (Patterson, 1985). Based on the opinions gathered through the analysis of 177 usable surveys, Patterson concluded that the journal’s place was “firmly established among the scholarly refereed publications in our profession” (1985, p. 311).

Twenty-two percent of the ALISE membership participated in the 2008 ALISE survey (Cloonan, 2008). Part of this questionnaire was a design to solicit feedback on the value of various member services and benefits. The top three member benefits were identified as networking (64 members), conference (62 members), and JELIS (62 members). With just over half of the respondents placing JELIS as an important component of ALISE membership, the journal serves a vital role in the organization, and as such, JELIS must continue to solicit feedback from its readers.

Research Design

The JELIS Satisfaction survey was designed to assess the satisfaction of readers and to inform the editorial decision processes. Five major categories were devised: (1) JELIS use, (2) Degree of satisfaction, (3) Topics on demand, (4) Loyalty, and (5)Demographic questions. The instrument, hosted through an online survey system, was pre-tested for readability, reliability, and questionnaire completion-time. After a few minor adjustments, the survey was announced on the ALISE/JELIS web site and through other professional venues, such as an ALISE conference announcement, and the jESSE listserv.

The sample was divided into two categories: (1) ALISE members and (2) Non-members. The ALISE member category included institutions and individuals who receive the journal as part of their membership. The non-member category included individuals and institutions (primarily libraries) that obtain the journal through volume-based subscriptions, and other individuals who read or cite articles published in the journal.

Results and Discussion

The survey was conducted from January 21st to February 16th, 2009, and had 108 respondents with 95 usable responses. The questionnaire data was exported to an Excel spreadsheet from the online survey system and analyzed with SPSS. Open-ended question responses were manually coded. As results of this study, respondent profile, JELIS use, and JELIS satisfaction are discussed below. An expanded discussion related to the open-ended responses and the loyalty factor is planned.

Respondent Profile

The demographic questions gathered occupational position, organizational affiliation, and geographic region information. Seventy-four percent of the respondents described their occupational title as faculty and 2% as librarians. The faculty category encompassed deans/department chairs, professors including associate and assistant professors, and lecturers. Most of the respondents (n = 70) stated they were affiliated with doctorate-granting universities as classified by the Carnegie Classification of Institutions of Higher Education (Carnegie Foundation for the Advancement of Teaching, 2009) while 14 people were from master’s colleges and universities.

The majority of the respondents (91%) stated they were from regions within the US, while 6% selected Canada and 3% Asia and Pacific. This appears to be in keeping with the results of the earlier Readership Survey response of 90% of the 177 respondents reporting the US as the region of residents (Patterson, 1985).

JELIS Use

Respondents were asked how they use JELIS in terms of reading, citation, publication, length of subscription, sharing, and retention of the journal. The majority of the respondents (89%) indicated that in the past year they have either read or looked through an issue. Taken independently, this appears respectable; however, 37% (n = 33) state they spent between 16–30 minutes looking through an issue upon receipt. Twenty eight respondents (31%) stated they spent less than 15 minutes, and 72% of the participants reported skimming the titles to be aware of the contents and reading the articles of interest. This strongly indicates that past JELIS articles were not holding the reader’s attention for an extended period.

When the respondents were asked if they had cited JELIS articles in the past year, 70% responded negatively. When asked how often they cite articles in JELIS, 21 percent responded ‘never,’ 42% responded ‘less than once a year,’ and 28% skipped this question entirely. Thirty-two percent indicated they had previously published in JELIS.

The length of subscription has changed since the 1984 survey. Patterson wrote that “fifty-four percent of the respondents subscribed to JELIS for 10 or more years, 35% from 10–15 years, 27% from 4 to 9 years, 19% for more than 15 years, 18% from 1 to 3 years, and one person made no response to this question” (1985, p. 303). The 2009 survey indicates 33% have received JELIS for 10 or more years and 39% for three years and under. Understanding the needs of the latter large group of new subscribers may improve the journal’s citation rate.

When asked if they shared their copy of JELIS with others, three fourths of the respondents reported that they do not share JELIS with anyone. Asked how long they keep a copy of JELIS, 58% responded ‘indefinitely.’ With almost three fourths of the respondents self-selecting a faculty occupational classification, future administrations of this survey may need to investigate the extent to which JELIS articles are shared or used in the classroom as required readings.

                                                                Figure 1. Satisfaction with JELIS

Satisfaction

Respondents were asked to rate their satisfaction with JELIS. The degree of subscriber satisfaction was assessed using a 5-level Likert scale (extremely satisfied, somewhat satisfied, somewhat dissatisfied, extremely dissatisfied, or undecided). As Figure 1 shows, 8% of the respondents were ‘extremely’ satisfied, 52% were ‘somewhat’ satisfied, 23% were ‘somewhat’ dissatisfied, 10% were ‘extremely’ dissatisfied, 7% were undecided, and 18 respondents skipped the question. Overall, 60% of the respondents (n = 54) were satisfied, while 33% (n = 30) were dissatisfied. The overwhelming majority response of Somewhat Satisfied for all of the satisfaction attributes listed in Figure 1 reinforces the value of conducting JELIS Satisfaction Surveys.

Table 1 shows the evaluation of satisfaction elements by the respondents who answered “Extremely Dissatisfied” or “Somewhat Dissatisfied” on overall satisfaction. Even though the statistical significance was not tested due to the small sample size, relevance of content is the major factor of dissatisfaction followed by quality of content, breadth of topics, and timeliness of content. The last two factors, quality of authors and design and layout are weakly related to the dissatisfaction, in that the percentage of satisfaction is higher than the percentage of dissatisfaction responses.

Conclusion

The major goal of this inquiry has been descriptive, with the intent that readers’ perceptions be used to inform future editorial efforts. Results from the 2009 JELIS survey indicate that relevance and quality of content are major factors contributing to dissatisfaction with JELIS. The survey outcomes provide important information to assist the current editors to obtain a closer journal alignment with the needs of its readers and subscribers. See Appendix A for additional demographic and JELIS use survey data.

Acknowledgements

The JELIS co-editors and editorial staff sincerely thank all of the respondents of the JELIS 2009 Satisfaction survey. Your participation and feedback will assist in improving the readability and reputation of JELIS. If you missed the opportunity to share your thoughts in 2009, we look forward to hearing from you in the 2010 JELIS Satisfaction survey. Special thanks to Michèle Cloonan (ALISE President) and Kathleen Combs (Executive Director) for their support in publicizing the survey.

Appendix A– Supplementary Data

References

Capps, J., & Hinson, C. (2009). JELIS connections. Journal of Education for Library and Information Science, 50(2), 59–67.

Carnegie Foundation for the Advancement of Teaching. (2009). The Carnegie classification of institutions of higher education. Retrieved August 9, 2009 from http://www.carnegiefoundation.org/classifications/

Cloonan, M. C. (2008). ALISE 2008 survey results. Retrieved August 9, 2009, from http://data.memberclicks.com/site/ali/ALISE_Survey_Results.pdf.

Patterson, C. D. (1985). An assessment of the status of the journal. Journal of Education for Library and Information Science, 25(4), 301–312.

Discussions in the higher education literature regarding online teaching, where content is delivered primarily over the Internet rather than in face-to-face classroom meetings, have shifted in recent years from the general “is it wise and is it good for learning?” focus to a quest to discover and share best practices. While certain voices still call for more careful attention to whether teaching via remote access is in the overall best interest of student learning (Shieh, 2009), we nevertheless generally acknowledge that new delivery methods are here to stay. The current argument is more often centered not on whether to deliver the curriculum online, but how. Central to the “how” discussion needs to be a focus on retooling professors who, when teaching online, are shifting to a completely different teaching and learning environment.

We tend to teach the way our favorite professor taught, which for many of us was the lecture method, or what Brent (2005) refers to as “teaching as performance” and Bain (2004) calls “the transmission model of teaching” (p. 173). But an online learning environment differs from the face-to-face classroom, where human interaction, eye contact, facial expressions, and verbal cues help faculty and students engage in the learning process.

This paper explores the mechanisms used by college professors to adapt to online teaching environments, especially regarding concurrent efforts of their institutions to offer support in the midst of these changes. The paper includes the results of a study conducted to gauge the current availability and impact of such support systems within LIS graduate schools.

To inform this study, I looked for literature that described the importance of, or need for, faculty development in teaching and learning technologies, with an emphasis on online course design and development. Most of this research comes from online course evaluation studies or from IT-focused articles which evaluate the effectiveness of university investments in IT-based teaching and learning tools. I then searched for examples of best practices in technology-based faculty development. Some best practices were true examples of structured institutional support for online course development or for teaching and learning technologies; others simply articulated the need for these institutional supports in our changing environments. As the focus of this study is LIS institutional support, I paid special attention to examples in the literature from within LIS graduate programs.

Background and Literature

Allen and Seaman (2005) report that more than 2.3 million students were enrolled in online courses in the fall of 2004; they also report an 18% per year growth in online course delivery. Within the 62 American Library Association (ALA) accredited graduate schools of LIS, 41 programs (66%) offer some of their curriculum online and 14 other institutions (23%) offer their entire program online (American Library Association, 2009). This indicates that 89% of ALA-accredited LIS programs offer courses online. In the face of such growth, the issue of educational quality has been actively discussed in the higher education literature, and training faculty is core to that discussion. “The question is no longer whether online education is as good as face-to-face instruction, but rather how to prepare and support faculty in the online environment and ensure that students achieve important learning outcomes whether they study in online or face-to-face settings or both” (Moskal et al., 2006, p. 26).

Some institutions have addressed the concern of training faculty by separating course design from the content experts, creating what Wright (2005) calls “academic sweatshops.” Wright writes about the changes in academic culture as more teaching is outsourced, but one wonders if this marginalization of core faculty is in the long term interest of the faculty, the students, or the institution. Noble (1998) suggested something similar in his much-cited article suggesting “the automation of higher education . . . is not a progressive trend towards a new era at all, but a regressive trend, towards the rather old era of mass production, standardization and purely commercial interests” (Abstract, para. 1). Noble argues that distinct lines have emerged between student and faculty desires for quality learning, and administrators’ desires for the profitable economic model seen in online course delivery.

In his critique of online education, Speck (2000) comes to a similar conclusion regarding the power of the economic model; he develops three criticisms which together he calls a breach of ethics in the academy. His criticisms are: (1) the failure of the academy to evaluate learning effectively in the online environment; (2) the concomitant failure to prepare faculty to teach online courses; and (3) an assessment that the academy has chosen online course delivery due to an economic model rather than an educational one. Speck argues that these problems can be overcome; he is not philosophically against online teaching but rather against the afore-mentioned gaps in institutional preparation to move so quickly (and basically unprepared) into new learning environments. Writing about faculty preparation, Speck argues:

The academy not only fails to provide adequate training for professors to teach online courses but also undermines professorial authority by putting them in situations where they are dependent on others to deliver subject matter content . . . in doing this, the academy violates the contract it has with students—namely, the agreement that professors are credentialed as expert teachers. (pp. 76–77)

This concern is reflected as more published literature emphasizes the critical need to train faculty to use teaching and learning technologies effectively. For example:

Colleges have spent millions on ‘smart classrooms’ packed with the latest gadgets to assist teaching computerized projection systems, Internet ports at every seat, even video cameras with motion detectors that can track the movements of a lecturer. But colleges have spent far less time and money giving professors the skills to use even the simplest technology effectively. (Young, 2004, p. A31)

Lewis and Abdul-Hamid (2006) investigated how highly-qualified faculty members incorporate effective teaching practices in their online courses. Their research demonstrates that effective online teaching “is, at best, an elusive and confusing process” (p. 95). However,

Despite differences in online course platforms, one of the expectations for effective online instruction is for structured pedagogical approaches, which evolve around interactivity and the deliberate actions of faculty willing to provide careful attention to student needs. Evident from this study is that this type of environment is not one that emerges naturally or unwittingly in online courses. (pp. 95–96)

A significant concern with online courses is that the current deliverymechanisms seem to flatten the traditional multi-sensory learning environment. How do we learn to add back those critical multidimensional elements into the learning environment? One way is to take advantage of web conferencing systems. With web-based synchronous learning environments, professors and students can begin to recapture the human touch we miss in the frequently-silent online course structure. For example, Hofmann (2004) considers the instructor’s voice to be an essential tool in an online synchronous environment (p. 52). She recommends best practices such as using a script for guidance rather than for reading, using humor and positive reinforcement, and “smiling” with your voice.

How does a faculty member learn these skills and techniques? Even in institutions where good teaching is clearly articulated as part of the core mission, support structures for online teaching need to be examined. Reder (2007) challenges us to rethink our assumptions of “good teaching”; because an institution values excellence in teaching does not necessarily mean that the structures are in place to provide support for learning effective pedagogical practices. This is true for faculty in various disciplines, at various career stages, and in relation to emerging teaching and learning technologies. Reder writes of the need for formal centers of teaching and learning excellence for ongoing faculty development. This is especially necessary, says Carey (2006), for members of the baby boom generation, who struggle to teach Net generation students “in a manner that capitalizes on the group’s technology-driven lifestyle and fosters quality liberal learning” (p. 3).

Much research has focused on current student satisfaction with the use of information technology in higher education. The EDUCAUSE Center for Applied Research conducts an annual study of undergraduate students and their uses of IT in college. Their 2008 study shows increased ownership (80%) of laptop computers, a high rate of confidence in their own use of the Internet (80% report being “very skilled” or “expert” Internet searchers), and an 85% participation in social networking sites (Caruso & Salaway, 2008). Additional results from the study (Caruso & Salaway, 2008) indicate that “students perceive that more instructors need to use IT effectively in courses,” “students are increasingly mobile,” and “students expect IT to be available” (p. 4).

There is evidence that students come to college as expert users of information technologies, and expect to find a university infrastructure to support their use. Whether their courses are fully online or their professors are incorporating more teaching and learning technologies in their face-to-face classes, these expert users set the bar high for effective use of teaching technologies in the academy.

Because of changing student expectations and because of significant institutional investments in teaching and learning technologies, there have been ongoing efforts to evaluate their effectiveness. Focusing on the efforts to evaluate online teaching, a helpful starting point is the overview of research by Tallent-Runnels, Thomas, Lan, and Cooper (2006). Key points in their review include the importance of learner-centered course design for online courses, the importance of interactivity, and the key roles of faculty in online teaching. Berge (1999) emphasized instructional design over delivery mechanism, and Hansen
and Gladfelter (1996) drew conclusions regarding the perplexing nature of online pedagogy to many faculty. Tallent-Runnels et al. (2006) provide many best-practice examples from research; they also evaluate the quality of research available in this emerging area of higher education study. Overall, they find:

Institutional aspects of these studies showed that few universities have written guidelines or policies for online courses. They also confirmed the lack of technical support for both faculty members and students engaged in online instruction. Faculty members want training and course development assistance as well as rewards for preparing courses to be taught online. Few faculty members said that they would require additional compensation for the work if they could get help developing and delivering courses . . . Appropriate and excellent course design and development may prove to be paramount to the success of students in online courses. (pp. 116–117)

Other reports from research regarding best practices in online course delivery include Barron (2003); Bouhnik and Marcus (2006); Bourne and Moore (2004); Chickering and Ehrmann (1996); Jones (1997); Kassop (2003); Pribesh, Dickinson, and Bucher (2006); and Smith, Ferguson, and Caris (2001), and on emerging centers for teaching and learning excellence in Lewis, Fino, Hungar, Wallace, and Welch (1994) and LeBarron and McFadden (2008). To maximize the potential of online course delivery, we must provide structured learning about best practices for online course design and development through formal institutional centers for teaching and learning.

One frequent method of developing quality teaching in online programs is to rely on early adopters among the faculty to provide inspiration and peer support for others. While individual leaders and innovators are essential to moving forward, dependence on these individuals is ill-advised. Harman (2008) cautions against this sort of dependence on the innovation and energy of a few, citing short-lived and spotty progress, inconsistent results, and limitations of scale across the institution (p. 25).

What is preferable is a system-wide approach to training and support as demonstrated by several programs included in Barron’s (2003) review of LIS progress in distance education. For example, when developing the fully online LEEP program at the University of Illinois, planners included a strong emphasis on faculty training and instruction:

The challenge [of moving to online teaching] was the time required to make the change from lecture in front of a classroom or a TV camera to this new design requiring change in methodology. The University of Illinois . . . had overcome this difficulty by providing faculty with a semester off before teaching and assigning the faculty member a single class the semester they taught the new class. (Woolls & Loertscher, 2003, p. 265)

Faculty training and a strong technological infrastructure were developed and maintained through deliberate regenerative funding using LEEP’s tuition (Estabrook, 2003).

Woolls and Loerscher (2003) reference the new skill sets required by the faculty in their description of San Jose State University’s preparation for online offerings. In their early days of online course delivery, San Jose received an Institute of Museum and Library Services (IMLS) grant to create a two-week intensive program for all kinds of library educators in the state (state library agencies, library staff development officers, etc.). The two week program provided instructors focused time with professional instructional design personnel.

Support centers for instructional design and technology training are described by Barron (2003) at the University of South Carolina, Kovacs (2003) at the University of North Carolina at Greensboro, and Turner (2003) at the University of North Texas. Each of these authors reports on the necessity for faculty development and support when delivering online courses.

In other situations faculty have felt a lack of support from their IT department, or an absence of instructional design support, and have adapted an attitude of learned helplessness. In a podcast about faculty technology training issues, Carlson and Arbogost (2008) say that when learning new technologies, potentially innovative faculty who are not supported by their institution may exhibit learned helplessness and tend to “simply give up.”

Unfortunately, often “the digital reform of the academy has happened so quickly and with such little consultation with the faculty that the ‘transformation’ of higher education is being conducted without due regard for faculty expertise” (Speck, 2000, p. 77). Speck recommends that administrators provide adequate training for faculty before they teach online, an opinion reflective of the best practices described above.

Faculty need to be prepared for online teaching, in terms of pedagogical shifts and technological tools. The emergence of teaching and learning centers in the academy indicates an acknowledgement of the importance of excellent teaching and of the distinction between subject expertise and pedagogical skills. Are centers for teaching and learning addressing the needs of the online learning environment? What other support systems exist for faculty in institutions offering online course delivery? These questions indicate a need for further study, beginning with the LIS education community.

Research Question

What support structures exist in LIS programs and their institutions to help faculty develop new skills in online course design, delivery, and content? An exploratory study was developed to investigate existing support systems in LIS education which assist faculty who are retooling for online teaching.

Limitations

This survey was limited to faculty in LIS programs, and did not try to survey the respondents’ university support structures for comparison to faculty responses. No component in this survey measured student satisfaction or teaching effectiveness. Rather, this study concentrated on faculty perceptions of support for their own development in online teaching and emerging course content. The study did not try to measure the effectiveness of one training method over another.

Methods: General Description

A questionnaire sent directly to faculty members was identified as the best research approach, as it was their experiences the study wanted to capture. Electronic survey tools distributed via the Internet make it possible to collect information rapidly from a large number of participants. SurveyMonkey (http://www.surveymonkey.com), a Web-based commercial software tool, was selected for its ease of use and for its built-in ability for email follow-up to non-responders. The survey (Appendix A) included multiple choice and open-ended questions. The combination of quantitative and qualitative questions provided concrete data regarding the range of support structures and an opportunity to gauge respondents’ attitudes and priorities. Data analysis was conducted using SPSS v.16 and NVivo.

Methods: Sample Selection

Using the ALA’s alphabetical list of accredited schools, LIS faculty and their email addresses were identified. All full-time faculty members of ALA-accredited master’s programs were sent an email invitation to participate in an online survey. The email included a brief description of the research project, an invitation to participate, and a link to the online survey. The email and the first screen of the survey assured the participants of anonymity and of the voluntary nature of their response.

Of the 907 emails sent to potential respondents, 883 invitations to participate in the survey were delivered, while 24 were returned due to bad email addresses or previously designated blocks of SurveyMonkey mailings by individual recipients. There were 331 total respondents, nine of whom declined to participate in the survey and did not advance further. Twenty-six respondents agreed to participate in the survey but did not respond to any of the questions; they are excluded from further analysis. The resulting number of total valid respondents is n = 296, although not all respondents answered every question. Percentages cited in this report are rounded to the nearest whole number.

Data Analysis

The survey consisted of 16 multiple choice, yes/no, and open-ended questions. Respondents were given the opportunity to provide comments on four of the questions, and a final question invited additional comments. The data from SurveyMonkey were downloaded into SPSS v.16 for analysis. NVivo software was used to identify word frequency counts in the qualitative responses. Narrative responses were further analyzed with hand-tabulation to identify common themes.

Results from Multiple Choice and Yes/No Questions

The majority of respondents (n = 243, 83%) were from institutions with over 10,000 students, with the remaining coming from smaller institutions (Table 1). The greatest number of respondents (n = 166, 58%) teach in LIS programs with more than 250 students.

Of the 297 respondents, 92% (n = 262) reported that their program offers some online courses; 36% (n = 103) reported teaching where the program is available fully online as an option. Approximately 69% (n = 198) of the respondents teach online courses or courses offered in a hybrid environment (partially face-to-face and partially online). These figures correspond well to the overall ALA statistics cited earlier in this paper; 89% of ALA accredited LIS programs offer courses online, and 23% of those are available fully online. It appears as though respondents to this survey are slightly more likely to be involved in a fully online program than the overall LIS faculty population reported by the ALA.

When asked if they had received formal and/or informal training in online course design and delivery (Table 2), 17% of respondents said little to no support was available to them. Sixty three percent rely on peers, confirming the observations expressed in the literature cited above. The next most frequent form of support came from university IT workshops (58%), and 44% cite formal university support as part of their preparation. Twenty percent cite formal training through their own LIS program as being a part of their training for online course design and delivery. (This question invited multiple responses, so the total is greater than 100%.)

Similar results were seen in the responses regarding support for faculty development in incorporating new technologies into LIS coursework, with the most support (64%) coming from peers, and the least support (18%) coming from within the LIS program.

Survey participants were asked to note incentives offered by the institution for developing online teaching skills. Ninety eight (24%) of the respondents said no incentives were available. Of the 405 total instances of institutional support cited, the most frequently cited (19%) was competitive faculty development grants, with support for conference attendance (15%), consideration in faculty review (15%), and case-by-case reimbursement for expenses incurred from outside training (14%) fairly evenly distributed. Thirteen percent of respondents cited the availability of a course release for developing new teaching skills for online course design and delivery (Table 3).

When asked about general support for new course development and professional development, 24% reported receiving financial support and 13% of respondents referred to course release time. No support for new course development and ongoing professional development was reported from 63% of respondents.

Additional analysis as done to assess if institution size made a difference with regard to support for developing new content. Table 4 shows the breakdown of incentives reported for developing new course content; respondents in the smallest and largest institutions reported the highest percentage of “no support is available” (76% and 77% respectively), with 61% of respondents from institutions of 5,000 to 10,000 students reporting no support in this area.

Whether training opportunities and incentives are offered by the institution is one issue; whether faculty take advantage of them is another. In this survey, the most frequent (n = 161, 56%) type of opportunity respondents reported taking advantage of within the past two years was informal instruction from colleagues. The next most popular (n = 126, 44%) response was conference attendance, with formal training through the university next (n = 104, 37%). Sixteen percent had not taken advantage of any opportunities during the past two years (Table 5).

Conferences attended most frequently were ALISE (51%), ALA annual and midwinter (40% and 36% respectively), and ASIS&T (39%). Respondents frequently attend more than one conference annually.

Respondents were asked to rate themselves on a scale of 1–5 for confidence in online teaching (1 = not at all confident , 5 = highly confident). Sixty five percent rated themselves at a 4 or a 5; the mean score was 3.84. A related question asked respondents about their level of confidence in adapting their LIS course content to include emerging LIS practice tools such as social software and Web 2.0, with an overall mean of 3.61.

Examining the relationship between teaching some courses fully online and the degree of confidence in teaching online, there was a significantly increased level of “confident” and “highly confident” responses from those who teach online.

When formal support for instructional design is made available by the institution (reported to be the case by 155 respondents), 63% of professors report being confident to highly confident in their online teaching. Seventy percent at those levels of confidence report having IT workshops at their university; 77% report formal training being available through their LIS programs. Of the 179 respondents reporting informal peer-to-peer support, 70% report being confident to highly confident in their online teaching. Thirty five respondents report little to no support being available at their institution; 58% of those 35 still report high levels of confidence in online teaching.

Results from Open-Ended Questions

When asked in the two final questions for additional comments, including examples of what they would like to see offered by their institutions, participants expressed many needs. Content analysis of these qualitative comments indicated the most frequently expressed desires were for time, course release, financial support, instructional design support, and training. Some of these expressions overlap; “time,” for example, sometimes refers to course release, but also seems to refer to “time” in a more cosmic sense (e.g., “Is there ever enough time?”). Financial support refers to continuing education, course release (again), and infrastructure investments. Table 8 is a breakdown of common themes from the responses to the question “What support would you like your institution to offer?”

Other less frequently occurring support mentioned included: more access to teaching assistants, peer support from open sharing sessions, Web 2.0, XML, Web development training, andmore creative cyber spaces for online teaching.

162 total responses to this question; many included more than one reference.

A frequently expressed desire was for greater structure in the support offered; respondents mentioned the need for formal programmatic structures beyond the technology, from mentors to administrative recognition and institutional rewards.

Fourteen respondents expressed a high level of satisfaction with the support provided by their institution, but there was often concern about finding time to take advantage of the learning opportunities. Respondents also expressed a need for the IT offerings to be more applicable to the LIS environment, and for instructional design support within the LIS program. These responses reflect a desire for customized services, and for support closer to home. This concern of the LIS school being underserved by the university was expressed as an outright tension in multiple responses; several comments referenced a lack of cooperation from the university’s IT department and a lack of attention to emerging technologies important in LIS (such as Web 2.0).

One faculty member mentioned an interesting problem: “As we try to be ahead of the curve in our teaching (with the virtual world Second Life, for example), we frequently find that we are ahead even of the IT support staff at our institutions.” Individual faculty members reported being ahead of some LIS peers, and the LIS program being ahead of the university. Another faculty member echoes this concern, saying: “. . . [it’s harder] to get a server or application in place and supported throughout the semester by our school IT staff . . . There is often a disincentive to introduce new tools into a class, because doing so puts a serious setup, administrative and IT support burden on the instructor.” Another respondent referred to a need for “courses aimed at higher levels of competence.” A few respondents pointed out that since online teaching is not a priority in their program, any lack of support for this kind of development is not unusual, although incorporating emerging technologies into the curriculum requires a strong IT support infrastructure.

The desire for release time for learning, reflection, and course refinement was a consistent theme in the narrative responses. Another commonly expressed need was for funding to take outside courses, including commercial continuing education opportunities. The top item on the wish-list was for release time to pursue the new learning required to adapt to online teaching environments, through a course release or simply expressed as “time.” As one respondent stated, “What I desperately need is time, even if to learn this stuff on my own, without a lot of strings attached. I don’t believe that this technology is that difficult, but there is a tremendous amount of it and it is constantly changing. . . . I can’t say this enough.” And, from another respondent, “TIME, TIME, TIME.”

Interpretation and Discussion of the Data

Connections to the Existing Literature

Several examples from this study confirmed previous findings and best practices. Speck’s (2000) call for adequate faculty preparation before online teaching was consistent with the high value this survey’s respondents placed on their preparation, whether it came from their peers or from other sources. Lewis and Abdul-Hamid’s (2006) emphasis on developing mechanisms for learning effective online pedagogies is mirrored in the respondents’ calls for LIS level and institutional level supports.

Respondents to this survey generally reinforce the observations from Tallent-Runnels et al. (2006) in that LIS faculty members understand the importance of good training in creating successful online courses; they use it when it is available and they want it where it is not. They are looking for leadership to provide support structures, but when those structures are not there, faculty forge ahead and make things work to their greatest capabilities. This would indicate that people who teach online manage to gain the needed skills to do so; it is also possible that by gaining experience in online teaching the faculty member also gains the needed skills to feel confident. One could also say that those who do not feel confident simply opt out, when possible, of teaching in online environments. As one participant responded, “Aside from assistance from colleagues and occasional university workshops, the general approach is ‘sink or swim.’ So I stay out of the pool.”

This study supports the existing literature emphasizing the need to prepare faculty to teach online (Carey, 2006; Lewis & Abdul-Hamid, 2006; Speck, 2000; Tallent-Runnels et al., 2006; Young, 2004). For programs offering online courses, a more clearly articulated, systemic approach to faculty development for online teaching is essential.

A Model for a Culture of Support for Online Teaching

Three of the items in the top six listed in the themes from the survey’s narrative responses (Table 8 ) are also the top three mechanisms listed as boosting the professor’s level of confidence for online teaching (Table 7): course release, financial support, and LIS specific help or formal training infrastructure. This finding can guide administrators looking for the most effective ways to train faculty for online teaching, particularly because these three items also frequently appear as best practices in the literature. The same three important items appear in the results of this survey as being rarely offered. Only 13% of respondents reported the availability of a course release (Table 3) and only 20% reported formal training through their LIS programs (Table 2). This study also supports the effectiveness and desirability of peer support in faculty development; it is a support structure faculty are highly likely to use (Table 5).

Components of an effective model for faculty development therefore include a course release and/or a learning stipend, LIS-specific training in instructional design and online pedagogical skills, and structured mentoring. The importance of a strong program-level support structure which accounts for the special needs of LIS emerged as a recurring thread throughout the survey responses and should be an integral component of this model. The importance of a culture of support for innovation is indicated in the best practice literature and is supported by the findings in this study.

From close attention to the themes evidenced in this literature review and the findings in this study, a model emerges that combines best practices and faculty’s expressed needs for online teaching support. The model consists of a multi-layered support system which includes formalized structures for program- level support and training, institutional-level supports, and outside continuing education opportunities. In this model, faculty members share responsibility for their ongoing professional development specific to online teaching skills. This shared responsibility is appropriate within the higher education culture. The model is presented in Figure 1 and is described below.

Figure 1. Creating a culture of support for online teaching.

A desire for discipline-specific support from the LIS program, beyond the generic offerings from IT, was another theme. When formal support was offered through the LIS department, 76% of this survey’s respondents felt confident to highly confident in their online teaching skills (Table 6). There is significant room for growth in this area among LIS programs, as only 20% of respondents in this survey report formalized LIS-level online teaching support (Table 2).

Finally, respondents acknowledged the expertise of their students. Opportunities to partner with skilled student workers who could act as support personnel in faculty’s online courses were seen as highly valuable, and are suggested as part of this formal model.

Institutional Support

The institution is responsible for strong infrastructure investments in technology and in teaching and learning. Many university IT departments are still evolving from hardware and software support systems to multi-layered departments with complex demands. There is also a strong trend toward the development of centers for teaching and learning excellence in higher education. However responsibilities for online teaching support are divided among departments and personnel, a culture of support should include instructional design support and resources for online pedagogy best practices. Technical workshops should focus on campus-wide systems (such as the dominant course management system in place) and on basic technical skills associated with changing software and hardware environments. One-on-one technical training should be available to faculty who are working on special projects or interested in developing specific skills.

Finally, the institution is responsible for broader infrastructure issues such as specific incentive and reward programs, including positive components in retention and tenure policies and financial incentives such as development grants for outside learning. The institution has a unique opportunity to make a positive difference in the culture of support by creating thoughtful policies associated with online course design and delivery. These policies, for example, could mandate faculty training and impose strict
course caps to address the need for appropriate faculty workload in online courses and to ensure quality experiences for students. Policies such as these standardize best practices across the institution and protect individual faculty members from
having to negotiate for them on a case-by-case basis.

External Continuing Education

Conference attendance was second (to peer support) among the faculty development opportunities this survey’s respondents
acknowledged using in the last year. It could be assumed that this is ranked highly because it is the most readily available, but it is also an established part of higher education culture. Faculty members traditionally use conference attendance to keep current. Financial support for these learning experiences should be part of the culture of support. In addition, many skills associated with online teaching are available though local community college courses, or from within the institution’s own degree curriculum. Commercial programs are available locally or online which address a wide variety of technical and online course development skills.

Summary

This model provides a basis for institutions’ planning for helping faculty prepare new ways of teaching in a rapidly changing environment. Taken together, the structures presented in this model create a positive culture of support for online teaching.

Suggestions for Further Study

Additional research is needed in the design and testing of such a faculty development model within LIS programs. It would be useful to broaden this study by surveying faculty in other graduate programs, and undergraduate faculty. It would be useful to include university IT departments and Centers for Teaching Excellence to measure impressions from the service providers, and to assess the percentage of schools with online courses where there are formal policies about online teaching and course delivery. Studies which correlate specific aspects of faculty training with student success will be important as part of an ongoing cycle of improvement.

Conclusion

As the number of courses delivered online continues to increase, faculty members must gain enhanced pedagogical skills for online learning environments. Universities should see their commitment to faculty development in this area as a significant investment in institutional quality. Results from research literature regarding best practices for online teaching development have been combined with a survey of LIS faculty to determine their existing structures and their perceived needs. The result is a
model for a culture of support which should be used to improve online teaching.

Appendix A: Survey Questions

1. (The first question indicated an agreement
   to participate in the survey.)
2. How many students attend your university as of Fall 2008?

          a. 1000–5000
          b. 5000–10,000
          c. Over 10,000

3. How many students attend your LIS program as of Fall 2008?

          Under 100
          100–250
          Over 250

4. Does your program offer online courses? (Yes / No)
5. Is your program available 100% online as an option? (Yes / No)
6. Do you teach any fully online or hybrid courses? (Yes / No)
7. Do you have formal and/or informal training for online course design and delivery? (check all that apply)

          a. Formal training programs through the university
          b. Informal training through university IT workshops
          c. Formal training through our own LIS program
          d. Informal peer-to-peer training among our own faculty
          e. Little to no training is offered

8. Overall in your teaching, do you have formal and/or informal instructional design support or training, such as assistance for course syllabus development, training for effective teaching strategies, and development of instructional materials? (check all that apply)

         a. Formal support at the university level via professional Instructional Design personnel on staff
          b. Informal support at the university level through IT workshops
          c. Formal training through our own LIS program
          d. Informal support among faculty for peer-to-peer training
          e. Little to no training is offered

9. What incentives does your institution offer to faculty to develop new teaching skills for online course design and delivery? (check all that apply)

          a. Funding via competitive faculty development grants
          b. Funding for faculty conference attendance
          c. Course release
          d. Positive factor in the university’s faculty retention and promotion policy
          e. Case-by-case reimbursement requests to fund outside training or courses
          f. None

10. As LIS programs face rapid changes in the profession, schools are responding with curriculum changes and additions to include such aspects as the use of social software tools. What support is available to you for course development and updates in your knowledge of today’s professional practice? (check all that apply)

          a. Course release for study and / or training
          b. Financial support for study and / or training
          c. No support is available for new course development at this time.

11. Which of these training opportunities and / or incentives have you taken advantage of in the past two years? (check all that apply)

          a. Formal training through the university
          b. Formal training through my LIS program support
          c. Informal instruction from my colleagues
          d. Formal training that I sought from outside sources, independent of my institution
          e. Course release
          f. Financial support
          g. Conference attendance
          h. None

12. Which conferences have you attended in the last 18 months? (check all that apply)

          a. ALA Annual conference
          b. ALA Midwinter
          c. ALISE
          d. ASSIS&T
          e. Internet Librarian
          f. None
          g. Other (please specify)

13. Overall, on a scale of 1–5 (1 being not at all confident, and 5 being highly confident), how confident do you feel in adapting your teaching methods to online environments?
14. Overall, on a scale of 1–5 (1 being not at all confident, and 5 being highly confident), how confident do you feel in adapting your course content to include emerging in LIS practice tools (such as social software andWeb2.0)?
15. What assistance would you like to see your university offer in relation to the development of new course design and delivery skills, and in the area of learning new LIS content? (open-ended)
16. Other comments: (open-ended)

References

Allen, I. E., & Seaman, J. (2005). Growing by degrees: Online education in the United States. Needham, MA: Sloan Consortium.

American Library Association. (2009). Directory of ALA-accredited master’s programs in library and information studies. Retrieved February 25, 2009, from http://www.ala.org/ala/educationcareers/education/accreditedprograms/directory/i dex.cfm

Bain, K. (2004). What the best college teachers do. Cambridge, MA: Harvard University Press.

Barron, D. D. (Ed.). (2003). Benchmarks in distance education: The LIS experience. Westport, CT: Libraries Unlimited.

Bates, A. W. (2000). Managing technological change: Strategies for college and university leaders. San Francisco: Jossey-Bass.

Berge, Z. L. (1999). Interaction in post-secondary Web-based learning. Educational Technology, 39(1), 5–11.

Bouhnik, D.,&Marcus, T. (2006). Interaction in distance-learning courses. Journal of the American Society for Information Science and Technology, 57(3), 299–305.

Bourne, J., & Moore, J. C. (Eds.). (2004). Elements of quality online education: Into the mainstream. Needham, MA: Sloan Consortium.

Brent, D. (2005). Teaching as performance in the electronic classroom. First Monday, 10(4). Retrieved July 1, 2009, from http://www.uic.edu/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/1221/1141

Burnett, K., Bonnici, L. J., Miksa, S. D., & Kim, J. (2007). Frequency, intensity and topicality in online learning: An exploration of the interaction dimensions that contribute to student satisfaction in online learning. Journal of Education for Library and Information Science, 48(1), 21–35.

Carey, S. J. (2006). From the Editor. Peer Review, 8(4), 3.

Carlson, S., & Arbogast, W. (2008). Professors and technology: Helpless or hopeless? Tech Therapy, [episode 20]. Retrieved July 2, 2009, from http://chronicle.com/blogPost/Tech-Therapy-Helpless-or-H/3852/

Caruso, J. B., & Salaway, G. (2008, October). The ECAR study of undergraduate students and information technology: Roadmap. Boulder, CO: EDUCAUSE Center for Applied Research. Retrieved August 20, 2009, from http://www.educause.edu/ECAR/TheECARStudyofUndergr aduateStu/163291

Chickering, A., & Ehrmann, S. C. (1996, October). Implementing the seven principles:Technology as
lever. AAHE Bulletin, 3–6.

Eastabrook, L. (2003). Distance education at the University of Illinois. In D. D. Barron (Ed.), Benchmarks in distance education: The LIS experience (pp. 63–73). Westport, CT: Libraries Unlimited.

Graves, W. H. (2005). Improving institutional performance through IT-enabled innovation. EDUCAUSE Review, 40(6), 78–99.

Hansen, N., & Gladfelter, J. (1996). Teaching graduate psychology seminars using electronic mail: Creative distance education. Teaching of Psychology, 23(4), 252–256.

Harman, J. L. (2008). Moving teaching and learning with technology from adoption to transformation. EDUCAUSE Review, 43(6), 24–25.

Hofman, J. (2004). The synchronous trainer’s survival guide: Facilitating successful live and online courses, meetings, and events. San Francisco: Pfeiffer.

Jones, M. (1998). Creating engagement in computer-based learning environments. ITFORUM Paper 30. Retrieved January 1, 2009, from http://itech1.coe.uga.edu/itforum/paper30/paper30.html

Kassop,M. (2003, May/June). Ten ways online education matches, or surpasses, face-to-face learning. The Technology Source. http://technologysource.org/article/ten_ways_online_education_matches_or_surpasses_facetoface_learning/

Kingma, B., & Keefe, S. (2006). An analysis of the virtual classroom: Does size matter? Do residencies make a difference? Should you hire that instructional designer? Journal of Education for Library and Information Science, 57(2), 127–143.

Kovacs, B. (2003). The University of North Carolina at Greensboro Master of Library and Information Studies. In D. D. Barron (Ed.), Benchmarks in distance education: The LIS experience (pp. 169–176). Westport, CT: Libraries Unlimited.

LeBaron, J., & McFadden, A. (2008). The brave new world of e-learning: A department’s response to mandated change. Interactive Learning Environments, 16(2), 143–156.

Lewis, C. C., & Abdul-Hamid, H. (2006). Implementing effective online teaching practices: Voices of exemplary faculty. Innovative Higher Education, 31(2), 83–98.

Lewis, P. H., Fino, L. M., Hungar, J. Y., Wallace, W. H., & Welch, R. F. (1994). Campus based academies, institutes, and seminar or workshop series. New Directions for Higher Education, 87, 27–39.

Mellon, C. A., & Kester,D.D. (2004). Online library education programs: Implications for rural students. Journal of Education for Library and Information Science, 45(3), 210–220.

Montague, R., & Pluzhenskaia, M. (2007). Web-based information science education (WISE): Collaboration to explore and expand quality in online education. Journal of Education for Library and Information Science, 48(1), 36–51.

Moskal, P., Dziuban, C. D., Upchurch, R., Hartman, J., & Truman, B. (2006). Assessing online learning: What one university learned about student success, persistence, and satisfaction. Peer Review, 8(4), 26–29.

Noble, D. F. (1998). Digital diploma mills: The automation of higher education. First Monday, 3(1). Retrieved July 2, 2009, from http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/569/490

Pribesh, S., Dickinson, G. K., & Bucher, K. T. (2006). A comparison of online and face-to-face cohorts in a school library media specialist graduate program:A preliminary study. Journal of Education for Library and Information Science, 47(4), 303–323.

Reder, M. (2007). Does your college really support teaching and learning? Peer Review, 9(4), 9–13.

Shieh, D. (2009, February 10). Professors regard online instruction as less effective than classroom learning. Chronicle of Higher Education. Retrieved July 2, 2009, from http://chronicle.com/free/2009/02/11232n.htm

Smith, G. G., Ferguson, D., & Caris, M. (2001, April 1). Teaching college courses online vs. face-to-face. T.H.E. Journal. Retrieved January 08, 2009, from http://thejournal.com/articles/15358

Speck, B. W. (2000). The academy, online classes, and the breach in ethics: Principles of effective teaching in the online classroom. New Directions for Teaching and Learning, 84 (Winter), 73–81.

Tallent-Runnels, M. K., Thomas, J. A, Lan, W. Y., Cooper, S., Ahern, T. C., Shaw, S. M., et al. (2006). Teaching courses online: A review of the research. Review of Educational Research, 76(1), 93–135.

Turner, P. M. (2003) What can we learn from automating the card catalog? Distributed learning at the University of North Texas. In D. D. Barron (Ed.), Benchmarks in distance education: The LIS experience (pp. 177–183). Westport, CT: Libraries Unlimited.

Weiss, R. E., Knowlton, D. S., & Speck, B. (Eds.). (2000). Principles of effective teaching in the online classroom. San Francisco: Jossey-Bass.

WISE; Web-based Information Science Education. Retrieved January 08, 2009, from http://wiseeducation.org

Woolls, B., & Loerscher, D. V. (2003). Distance education at San Jose State University. In D. D. Barron (Ed.), Benchmarks in distance education: The LIS experience (pp. 263–270). Westport, CT: Libraries Unlimited.

Wright, A. (2005, January 26). From ivory tower to academic sweatshop. Retrieved January 6, 2009, from http://www.salon.com/tech/feature/2005/01/26/distance_learning

Young, J. R. (2004, November 12). When good technology means bad teaching. Chronicle of Higher Education, p A31–A32.

Tip of the Iceberg: Meaning, Identity, and Literacy in Preteen Virtual Worlds by Eric M. Meyers, p. 226

Studying Collaborative Learning Using Name Networks by Anatoliy Gruzd, p. 237

MPACT Family Trees: Quantifying Academic Genealogy in Library and Information Science by Terrell G. Russell and Cassidy R. Sugimoto, p. 248

Everything Old is New Again: The Evolution of Library and Information Science Education from LIS to iField by Laurie J. Bonnici, Manimegalai M Subramaniam and Kathleen Burnett, p. 263

Research Articles
Learning to Teach Online: Creating a Culture of Support for Faculty by Kate Marek, p. 275

Brief Communications & Research in Progress
The Maryland Modular Method: An Approach to Doctoral Education in Information Studies by Allison Druin, Paul T. Jaeger, Jennifer Golbeck, Kenneth R. Fleischmann, Jimmy Lin, Yan Qu, Ping Wang and Bo Xie, p. 293

Editors’ Notes
Transforming LIS Education for the 21st Century: Best Conference Papers (2009) by Kathleen Burnett and Michelle M. Kazmer, p. 218