Now that the iPhone/iTouch version is finished and available, I’ll be back showing off the features at conferences. Here are a few that we’re signed up for. If you are attending, please contact me or stop by the exhibit booth.

NASP (National Association of School Psychologists) Chicago, March, 2010. Exhibit Booth

CEC (Council for Exceptional Children) Nashville, April, 2010. Presentation & Exhibit Booth

ABAI (Association of Behavior Analysis International) San Antonio, May 2010. Presentation & Exhibit Booth

COSA (Confederation of Oregon School Administrators), Seaside, OR, June, 1010. Exhibit Booth

NSDC (National Staff Development Council), Atlanta, December, 2010. Exhibit Booth

——————–

And a plea – what conferences should we consider attending? National or state level? If you have a favorite conference that you feel would fit eCOVE, PLEASE let us know (john@ecove.net). I love to talk to folks about eCOVE and data-based observations, but there are so many conferences out there….

We have developed an iPhone/iTouch version, and have submitted it to Apple for approval (1/29/10). They say that 98% take 7 days for approval. Since this is a more sophisticated program than the simple games, it may take longer. It should work on the new iPad, but we haven’t tested that yet.

As currently designed, the iPhone Editions will come with a default set of 5 tools (different for each Edition), and licensed laptop users can add additional tools by syncing. Look under the Solutions/iPhone Version to see screenshots.

We’ve searched the Apple App store and there is nothing even remotely as advanced or easy to use. You can add comments, change observees during a single observation, email reports with graphs right from the iPhone. Syncing to the computer version adds tools, more report types, uploading of classrooms, etc.

….how busy I’ve been. Most of the time has been in developing new tools, working with districts and universities, and updating the software. We now have a version on the site that works with the small netbook/mini-computers, and we will shortly have a version that works on the iPhone/iTouch. It’s exciting, but time consuming and I’ve neglected posting here.

Nevertheless, if you have a netbook download the ‘Compact version’ from the website. If you have a license for one of the advanced Editions,  your same license key will work (email if you’ve lost it). All the features are the same, but a few buttons have been moved and resized.

If you are a handheld user, I strongly recommend that you look into the iTouch as a replacement. It’s under $200 and is VERY functional on both Windows and Mac. eCOVE runs much better on it that either the Palm or PocketPC. There are some new but subtle features coming that make the iTouch (or iPhone if you need the phone capabilities) really exciting to use.

When?   a couple weeks of programming (you don’t know how many times I’ve said that) and then it goes to Apple for approval (2 weeks to many months).

If Apple approves things, you will NOT believe the price.

In a nutshell, the revised Bloom’s Taxonomy is easier to reliably use during an observation. The original Bloom’s is very useful when examining written questions, but if you’re not quite skilled with it, it’s difficult to categorize questions on the fly. The revised actually covers the same behaviors, but the terms used make it easier to identify spoken questions.

In addition, the revised Bloom’s Taxonomy includes sub-categories for each level, which can be very useful when doing more detailed observations about question-asking or answering behaviors. The original Bloom’s Taxonomy is in the Basic Edition; the new Revised Bloom’s is in the current advanced General Edition. All of the Bloom’s tools (there are 12) are available in the Build Your Own heading on the website. Look under Solutions and Buy menu headings.

Here is a comparison chart.

Comparison chart - Original Blooms versus Revised Blooms

Just a quick note about the Confederation of Oregon School Administrators conference in Seaside, OR. This is our 6th year of attending and although the attendance is down, it’s been rewarding to talk to the satisfied users and the new administrators discovering eCOVE for the first time. There seems to be less focus on ‘evaluation’ of teachers and more on helping teachers develope their skills. It very clear that the value of having objective observation data as a significant part of the process is high, not for the data itself but for the reflection and professional discussions that it produces.

There are now new ‘advanced’ Editions available. We’ve been working hard to develop new tools, toolsets, and complete Editions. Here’s a brief description of each of them:

General Edition. This is the new name for the ‘Advanced Edition’ that came out February, 2009. It has the same 40 tools but several additional features. The tools are useful for  wide variety of observation needs – regular classroom, special ed, etc. It is a free update for owners of the Advanced Edition, but you will need a replacement license key. We’ll be sending those soon, but email if you need it right away (or we miss you). Very often, our ‘contact person’ is in the business or tech support office and not the end user.

Special Education Edition. This is a set of 25 tools designed by a team of special education professionals. The tools are grouped by eligibility area (which can be changed) and deals with the full spectrum of disabilities. There is a 6 tool overlap with the General Edition, so most of the tools are brand new. There are additional tools available in the Build-Your-Own section. Search for ’special education’ for our recommendations.

Administrator Edition. This is a set of 25 tools focused on general classroom observation and is useful for building administrators, lead teachers, coaches as a way to provide data to the classroom teacher. There is a 19 tool overlap with the General Edition, so there are 6 new tools in this Edition. General Edition owners can add the 6 new tools in the website ‘Add-On Toolsets’ section. Also there are many additional tools available in the Build-Your-Own section.

Sheltered Instruction/ESL Edition. Another set of 25 tools useful to those observing classroom with second language students. These tools support the use of the Sheltered Observation Observation Protocol (SIOP) but is not limited to that use. These are a core set of tools to track best practices for teaching ESL (and other) students. Again, additional tools are available in the Build-Your-Own section – search for Sheltered Instruction.

As we continue to create new tools for the general classroom observation, special education, sheltered instruction, and implementation of a curriculum or behavior plan, it’s becoming clearer that there is major need to focus on the fidelity of implementation. So often the evaluation efforts are focused on either student outcomes (test scores or a culminating performance) or the level/type of student engagement. While those are critical pieces of data, the first data needs to on whether the teacher is implementing the curriculum or behavior plan as it was designed to be used. 

I’m not in favor of lock-step following of the directions of ‘experts’, especially textbook publishers. To expect someone not familiar with the students and the school culture to lay out a specific sequence of teaching steps is asking too much. However, good curriculum is carefully designed and reviewed, and should have a consistent delivery system that should be initially followed. It’s very important to track the fidelity of implementation if you want to determine if the new curriculum or behavior plan is effective. If the teachers are consistent in the organization and delivery, then the data on change of student behavior can be trusted. Without the fidelity of implementation data, you would have no way of determining the cause of the success or failure of the efforts. Is it because the curriculum and/or delivery system is flawed or is the actual delivery by the teacher inconsistent or significantly changed?

With eCOVE Observation Software and the ability to create tools to match the desired behaviors, it’s possible to track both in the same observation. And with the ability to see the data over time and in comparison to other groups, it’s easy to get serious about making data based decisions. I get excited thinking about the time saved and frustration avoided when you can use the data to make in-progress adjustments, schedule retraining, or engage the teachers in an objective evaluation of the intervention. Rather than waiting for the outcomes to determine that it’s not working, tracking the student behavior can indicate the effectiveness of the intervention early in the process. Where that effectiveness is lacking, the data on the fidelity of implementation can help identify the cause, and in-progress corrections made in a timely manner.

Of course, this applies to not only school wide curriculum implementation but also to an individual student’s IEP. Whatever the level, the three basic questions are the same: Is the intervention being appropriately implemented? Is it having the desired effect on the student’s behavior? Does that result in greater learning? You need all three pieces of data to make professional decisions.

Peace,   John

This is an email sent to a person who requested a copy of the powerpoint presentation I did on Observation Reliability where I presented my new idea about the sequence from research to standards to indicators to data collection to teacher support and evaluation. If you’d like to see the Powerpoint send me an email.

When I first wrote eCOVE I was focused on giving helpful feedback to student teachers. From years of working with student teachers and new teachers I knew that they needed help thinking through the problems that came up in their classrooms. Providing them with ‘my’ answers and ideas was of much less benefit that getting them to think through things and devise their own solutions.

I also knew, again from personal experiences working with them, that giving them data (pencil and paper before eCOVE) help them honestly reflect on their own actions and outcomes, and it also greatly diminished the fear factor that came with the ‘evaluator’ role of a supervisor.

When I first started working with administrators and eCOVE I was totally focused on changing their role from judge to support and staff development. I preached hard that working collaboratively would have great effects and would/could create a staff of self-directed professionals. I still strongly believe that, and have enough feedback to feel confirmed.

However, a recent conversation with an ex-student, now an administrator, has added to my perspective. He likes eCOVE and would love to use it except that his district has a 20 page (gulp!) evaluation system that he needs to complete while observing – so he doesn’t have the time to work with teachers. We agree that it’s a waste of time, and corrupts the opportunity for collaborative professionalism.

As I thought about his situation and the hours of development time that went into the creation and adoption of that ‘evaluation guide’, I realized that my approach to observation as staff development had ignored the reality of the required and necessary role of administrator as evaluator. The guide that he’s stuck with seems to me to be the main flaw in the process, and what I believe is wrong with it (and the thousands in use across the country) is that they ask the observer to make a series of poorly defined judgments based on a vaguely defined set of ’standards’. It’s an impossible task and is functionally a terrible and ineffectual burden on both administrators and teachers.

When I thought about how a standards based system might be improved, I developed the basis for the idea in the powerpoint – Standards should be based on research; the implementation of the standard should be in some way observable, if not directly then by keystone indicators; the criteria for an acceptable level of performance should be concrete and collaboratively determined. I say collaboratively since I believe that administrators, teachers, parents, and the general public all have value to add to the process of educating our youth. Setting those criterial levels in terms of observable behavior data should, again, be based on research, and confirmed by localized action research efforts. That’s not as difficult as it sounds when the systematic process already includes data collection.

For the last couple of years, whenever I presented eCOVE I made a big point of saying that I was against set data targets for all teachers, that the context played such a big part in it all that only the teacher could interpret the data. I think now that I was wrong about that, partially at least. A simple example might be wait time – the time between a question and calling on a student for an answer. There’s lots of research that shows a wait time of 3 seconds has consistent positive benefits. While I’m sure it’s not the exact time of 3 seconds that is critical, the researched recommendation is a useful concrete measure. If a teacher waits less than one second (the research on new teachers), the children are robbed of the opportunity to think, and that’s not OK. An important facet of the process I’m proposing has to do with how the data is presented and used. My experience has been that the first approach to a teacher should be “Is this what you thought was happening?” This question, honestly asked, will empower the teacher and engage him or her in the process of reflection, interpretation, and problem solving. During the ensuing professional level discussion, the criteria for the acceptable level of student engagement is a 3 second wait period should be included, and that’s the measure to be used in the final evaluation. For, in the end, a judgment does have to be made, but it should not be based on the observer’s opinion or value system, but on set measurable criteria — criteria set and confirmed by sound research.

A more complex example – class learning time. The standard illustrated in the powerpoint stated that ’students should be engaged in learning’, a commonly included standard in most systems. There is extensive research that indicates that the more time a student is engaged in learning activities, the greater will be the learning. While the research does not propose a specific percent of learning time as a recommended criteria, I believe we as a profession can at least identify the ranges for unsatisfactory, satisfactory, and exceptional. I think we’d all agree that if a class period had only 25 % of the time organized for teaching and/or student engagement in learning activities, it would be absolutely unsatisfactory. Or is that number 35%? 45%? 60%? What educator would be comfortable with a class where 40% of the time lacked any opportunity for students to learn. I don’t know what the right number is, but I am confident that it is possible to come to a consensus over a minimum level. Class Learning time is a good example of a keystone data set – something that underlies the basic concept in the standard ‘engaged students’. I know there are others.

But then my personal experience as a teacher comes into focus, and the objection “How can you evaluate me on something I don’t have full control over?” pops up. I remember my lesson plans not working out when the principal took 10 minutes with a PA announcement and there were 4 interruptions from people with important messages or requests for information or students. How could it be fair to be concerned about my 50% learning time when there were all these outside influences?

That would be a valid concern where the evaluation system is based on the observer’s perception and judgment, but less so when based on data collection. It is an easy task to set up the data collection to identify the non-learning time by sub categories – time under the teacher’s control and time when an outside event took the control away from the teacher. The time under the teacher’s control should meet the criteria for acceptable performance; the total time should be examined for needed systematic changes to provide the teacher with the full allotment of teaching/learning time. Basing the inspection of school functioning on observable behavior data will reveal many possible solutions for problems currently included in the observer’s impression of teaching effectiveness.

It’s reasonable to be suspicious of data collected and used as an external weapon, and for that reason I believe it to be critical that the identification of the keystone research and indicators, and the setting of the target level be a collaborative process. Add to that the realization that good research continues to give us new knowledge about teaching and learning, and with that the process should be in a constant state of discussion and revision. That’s my vision of how a profession works – critical self-examination and improvement.

So now my thinking has come to a point where I believe (tentatively, at least) that we have sufficient research to develop standards, or to better focus the standards we do have; that we can identify keystone indicators for those standards; that we can use our collective wisdom to determine concrete levels for acceptability in those keystone indicators; that we can train observers to accurately observe and gather data; and that that data can be used to both further the teacher’s self-directed professional growth and to ensure that the levels of effective performance as indicated by sound research are met.

I’m hoping that my colleagues in the education field (and beyond) will join in this discussion and thinking. What is your reaction? Can you give me “Yes, but what if…..?” instances? Do we really have the credible research to provide us with keystone indicators? How could a system like this be abused? How could we guard against the abuse?

Lots of good teachers, me included, work quite hard to get student to think at a ‘higher level’. In Bloom’s Taxonomy this would be in the Analysis/Synthesis level, or in some thoughtful response to a divergent question. Thinking at a higher level about the content at hand would be great, but a deeper desire is just that they exercise their brains for more than stimulus-response game playing or repeating the obvious.

After years of challenging, encouraging, praising, modeling I came to the conclusion that higher order thinking will only naturally (not forced) occur if the topic is related to the life of the student. It the broad sense, related can be as simple as having fun…solving puzzles, creating new ideas…self-directed mental challenges that end up with the intrinsic reward of a self-approved solution.

And now that I’m engrossed with data based observation, I have discovered something quite interesting – give kids data on their own behavior, either as an individual or as a group, and they go immediately to the analytic level, and love it. They will reflect, think divergently, propose and test changes, and anxiously look forward to the next round of higher order thinking.

And it’s a pretty easy step to transfer that analytic thinking to school related content — “Remember how 40% of your statements to each other were negative? How does that relate to the X versus Y conflict (take your pick)?” or “Compare your individual time-on-task rate with the campaign promises of President X (take your pick) for greater government efficiency.”

My observation is that the data collected needs to be real (not how many are wearing red, or how many pencils were dropped), and best if collaboratively identified as something of interest. Assigning a student to be the data gatherer further engages them.

Tools I’ve seen used with students include Time On Task, Positive/Negative, Verbal Tics, Bloom’s Taxonomy (levels of questions answered or asked by students), Teacher Travel (tracking what % each part of the room was engaged in a discussion), and of course, the Generic Tools. Entering the names (Generic Timer) of a small group working on a project together and then tracking the % of time each contributed to the discussion is enlightening.

Give them the data and ask “Is this what you thought was happening?” “Why/Why not?” “Is there a need for a change?”… and away you go.

There seems to be some conversation about the right term for these– rubrics, scoring guides, continuums, etc., but I’m sure we are all picturing the same table of headings describing a scale from not-good to great.

In the business world, and somewhat in education, they are also called Behaviorally Anchored Rating Scales (BARS). I’m adding one word to that, making it Data-based Behaviorally Anchored Rating Scales (D-BARS). If you ever see that somewhere else, you can say you know where it started.

If you’re adopting, amending, writing your own D-BARS, there are some errors to avoid lest the outcome be less than helpful to the observer and observee.

A very common error has to do with creating a continuum of behavior indicators. Since across the top of these documents is a scale that progresses from one extreme to the other, conceptually with no gaps or overlaps between the divisions, the D-BARS (the physical observable behavior that exemplifies each division) should also be a continuum. As I look at these documents from across the country and world, one of the most common errors is that the actual behavior being used as an indicator changes from one division/cell to another. It shouldn’t. What should be described is one behavior across the continuum, poor to great. An example…..

The target standard/behavior is “Teachers involve and guide all students in assessing their own learning.” The category headings are Unsatisfactory, Emerging, Basic, Proficient, and Distinguished.

The behavior indicator for the Unsatisfactory level is “Students do not assess their own learning.” That’s a clear statement, but there’s more to the unsatisfactory level than no assessment at all. Students might be assessing themselves once a year, unguided, inaccurately, using the wrong criteria, etc. The descriptor for Unsatisfactory should describe the range of indicators, all of which are unsatisfactory.

The next level, Emerging, has this as a behavior indicator: “Teacher checks student work and communicates progress through the report card.” This indicator is unrelated to student assessment of their own learning, and doesn’t provide the guidance for whoever would use the D-BARS to clearly be able to determine the difference between Unsatisfactory and Emerging. This statement might fit well in a target standard related to ‘communicating progress to students’, and in that standard might well fit in the ‘Emerging’ category.

Perhaps (and this is brainstorming – collaborative discussion needed)…

Emerging would be “Students are asked to state/guess what their grade on an assignment will be.” or “Students are asked to grade each other’s papers without the use of a scoring guide.” [Students are assessing their work related to grades, and with little guidance]

The Basic category could be something akin to “Students are assessed by the teacher according to a scoring guide and asked to describe why they agree/disagree with the grade.” [Students are asked to apply the scoring guide in their reflection, but do not actually self-assess]

A descriptor for the Proficient level might be “Using a teacher provided scoring guide, students are asked to assess their work before they hand it in to the teacher.” [Students assess their work according to a scoring guide]

And finally, the Distingished level could read “Using collaboratively developed (teacher and students) scoring guides, students are engaged in self and peer assessment of progress toward meeting the standards.” [Students are engaged and guided in the process of creating the criteria, and then applying that criteria to themselves and others.]

I would hope that there would be discussion about my choices and wording, as this is only to illustrate the need for a continuum in the described behavior indicators.

The next thinking might be about what are the keystone observable behaviors that should be tracked to gather data on “Involving and guiding students in assessing their own learning.” Is it the amount of time students are engaged in assessing learning? The number of references to standards made by the teacher? The number and or level of questions asked by students related to assessment and standards? Every observer who makes a determination of level is doing so on the basis of something they see. We need to come to consensus concerning what’s valid and reliable.