Friday, May 20, 2011

Question: Where Should the In-Person Assessments Have Occurred?

This is definitely something exploratory, and it's certainly not intended to be predictive. I am just musing here, partially to determine whether the in-person assessment locations were sound or not.

So the question is this: Based on what we know of the 2011 semifinalist pool, where would the most effective in-person assessment locations have been? Let's broaden that a bit to get at what we really might want to know. Which cities with a Federal presence would have made good cities in which to conduct PMF in-person assessments, based on what we know about where the semifinalists (those invited to the in-person assessments) presumably originated? I hope that's a clear question, but we can break this down into components so the criteria are better defined, and the follow-up questions are enumerated.

  • Federal presence: This can mean one of a few things. First, and fundamentally, is there any Federal agency with an office in the city? For many cities, the answer is a qualified yes. Qualified, because even if a city has some Federal presence, this does not mean that the city is at all suited to hosting in-person assessments, either because it is located too far away from the bulk of semifinalists, or because it is simply too small to provide convenient transportation options. It turns out there is another way we can measure Federal presence in a city: The Federal Executive Boards (FEB). FEBs form a nationwide network of Federal branches providing communication and collaboration solutions to agencies outside the DC area. Given their wide geographic distribution, it seems clear that FEB cities might serve as a good starting point to analyze future in-person assessments. In the graphics below, I show a summary view of how many semifinalists were listed as closest to each of these cities.
  • Semifinalists: I chose semifinalists from this year 1) because there were semifinalists for 2011, and 2) because semifinalists were the ones invited to take in-person assessments. It doesn't make much sense to me to choose nominees or finalists for this particular comparison, although choosing nominees would at least provide some information for future planning. What we know about semifinalists is the school they listed in their materials, and not much more. This is a limitation of the data set, of course, but it's all we have to work with. What we have to assume from it is that the schools in question were correctly identified for purposes of geolocation; that every semifinalists were correctly listed with their schools; and that the locations of the schools reflect the geographic origins of the semifinalists. That's a tall order, but again, what choice do we have? Some of these schools conduct extensive online programs that mean students could be widely dispersed beyond the brick-and-mortar campus. What we have, then, is close enough approximation of the truth for this analysis.
Now our question becomes this: Of the FEB cities, which are closely located around the most 2011 semifinalists? That is a question we can answer. I took the cities in which there are FEB offices and calculated the distance from that city to each of the 278 schools represented in the semifinalist data. Then I figured out, for each school, which was the closest FEB city. And finally, I aggregated the FEBs and summed up the semifinalists that were listed as closest to each FEB. That data is shown below:

One thing you'll notice, of course, is that Washington, DC, is listed here, even though it's not an FEB location. I trust you'll understand why this is the case. Regardless, what we see is that, outside Washington, DC, the top 5 FEB locations are Boston (161), Atlanta (118), New York City (110), Chicago (92), and San Francisco (76). If we were looking for validation of the 2011 in-person assessment location choices, this might suffice. What the top 5 FEB list doesn't really account for, though, is that there are significant numbers in other locations. The trick here would be to determine locations that are central to a region in some way. DC makes sense for most of the East Coast, especially given the ease of transportation between, for instance, Boston and DC. It is entirely fitting, then, to keep DC as an assessment location. Atlanta also makes sense for large portions of the South. The Midwest is well served by Chicago, and the West Coast is well served by San Francisco (although Los Angeles looks to be a good second choice). That just leaves areas like the North Plains and the Southwest less well served. But we can frame a different question that might help here. Let's eliminate all but one NE city (DC), one Southern city (Atlanta), one Midwest city (Chicago) and one Western city (San Francisco), leaving the others on the list to see what we can come up with. That leaves us with the following:

Based on this, I think we can recommend that either a city in Texas or Oklahoma City could serve as the only other location needed. I am choosing OKC because of its fairly central location compared to Denver and Albuquerque. If we do that, then the numbers look like this:

Even so, the payoff for adding OKC is much lower than other locations, and it may not ultimately be worth the effort to add the assessment location.

Next, let's see if we can determine whether there's a distance factor involved here. That is, if we take these five locations, is there an average distance we're looking for that might be ideal? The first table shows us a widely variable average distance between the school and the assessment center.

The largest average is for DC, but this can be partially explained by the large distribution of semifinalists (Boston to a point about halfway between Atlanta and DC) and the inclusion of overseas schools in this list, all of which are in excess of 2000 miles away. There aren't many, but they are enough to affect the result. Filtering those out will give us perhaps something more meaningful.

So there you have it. These are pretty good distances from what I can tell, but I am interested to know what you all think. As one final point of comparison, here are the average distances to the original in-person assessment locations. By omitting OKC, we raise the averages for Chicago, San Francisco, and Atlanta, but DC is unaffected.

So after examining the locations that might have made sense for the in-person assessment, what we found is that the original locations seemed to be about right. We could fragment the assessment centers a bit more by adding one in OKC, but doing much more than that seems to have a lower benefit. What do you all think? Were these distances doable for you? I know many of you would have preferred not to travel as far as you did, but consider the alternatives (such as all in-person assessments being held in DC). What locations do you think should be considered?


  1. Just tossing it out there and maybe planting a seed for some OPM staffer... How about using Skype or some other form of video conferencing? Invest in one of the software platforms and then allow participants to use their own computers (or a school's).

  2. I would add a location in Boston as well. We need to get more Ivy league and top tier schools to send graduates to the PMF assessment and only having one location (DC) serving the plethora of great schools in New England seems insufficient.

  3. I think the Ivy League thing is silly, DC isn't that far and flights are still cheap.

    I, too, think Skype for the international semi-finalists would be great, but unfortunately, I think it would hurt their chances in the group portion and it wouldn't be standard for the writing section. Finally, unless you are standing far away from skype, you miss out on some presentation factors-- how would they get the prompt in a fair format? Would they get to type notes up on their computer, unlike the rest of the group? I think Skype would just add too much complexity and difficulty. Maybe, for the international group, they could choose one or two locations-- one in Europe and one in Asia. Were there many of those finalists? I know people complained on the board, but I'm not sure if it was even 10 people.

  4. I'd use video conferencing and adjust the testing accordingly.

  5. Yes, because after all, students at "ivy league top tier schools" do not have enough access to career opportunities. Give us a break.

  6. this idea seems to not acknowledge that during the in-person assessment there needs to be (1) PMF + (1) PMF alumni for each of the groups. I doubt that ONF program has enough of a footpprint in OKC or even LA for these to be viable alternatives. I could of course be proven wrong by the data, but my intuition is to suggest that this isn't the case.

  7. Two comments:

    1) I'd choose Dallas over OKC just because of flight availability and cost of flights. It can be costly to fly into OKC and involve connections depending on where you come from. Dallas is a relatively inexpensive hub with many direct flights.

    2) There's an underlying premise in this analysis that the the semi-finalists will continue to come from the same geographic areas. As OPM and the Agencies begin to recruit more broadly (the FedRecruit initiative), I think (hope) we will see some changes to this. I truly believe our country's great land grant universities and certain areas are grossly underrepresented, just because they don't know about the PMF program. While it's likely that it will continue to skew the way it already does, better recruiting could result in a need for that OKC/Dallas location and even more scheduling in Chicago to serve the broader reach.

    And if OPM wants to place me as a PMF in recruiting (shameless plug), I will knock it out of the ballpark recruiting for mission critical occupations in places they have overlooked/have had limited success. I know for certainty that too many university programs who produce the needed competencies don't know about the program (mine didn't; I found it on my own and brought it to the department chair's attention--and I graduated from a university which produces many PMFs--but only from the select programs who know about it. These departments are mini universities within the system which OPM doesn't seem to fully understand.

    I have a whole rant about this where I can cite specific universities which produce the best candidates for certain growing occupations but aren't represented in the PMF but I will spare you the rest... OPM's recruiting is improving but not fast enough.

  8. Should we talk about the personal/professional qualities that the in-person assessments valued? It seems that this year's assessment valued skills such as public speaking, BS-ing vague policy discussions, and making small talk. Was there some other underlying standards actually being used to evaluate the PMF-ers?

  9. @ 9:13 I think they did have a serious matrix of apparent competencies/indicators/behaviors they were marking as we went through the assessment. My understanding is they have psychologists and org development people who put those judgment criteria together, and it's based on sound research. Still, your point is well taken and constant reassessment of how well this works, and what to assess for, is important.

    There was a guy in my group who was pretty quiet. He was obviously intelligent and just really well grounded, from talking to him between exercises. I thought they might overlook him to their loss. I was delighted when I found out he made the cut--whatever criteria they are using worked at least in his case. I'm sure it's not perfect, including possible assessors' biases etc., but what is?

    I think their larger problem is increasing the pool of candidates, and putting all candidates on a level playing ground (as much as that is ever possible). The larger schools have alum groups who coach their nominees.

    I'm a bit of a broken record on hoping they increase the regional diversity. Certainly the schools in the NE are overrepresented, and although their students may come from across the US, you can argue they fit a certain profile. Those who study in other regions perhaps have more of an outside "the Beltway/NE/Ivy League/Elite School" outlook and a needed kind of diverse experience which might help the federal government relate to their "clients"; all citizens (not knocking those schools--I went to one as well as the other kind for which I am advocating).

    It would be interesting to get a survey of PMFs which included: How many grew up in rural/exurban America? How many grew up in low income urban core environments? How many come from households with family income at or below the US median? Below the 75% or 80% level? How many are first generation college graduates? How many graduated from a community college before pursuing their four year undergraduate degree? From the types of schools and programs represented, who is retained and who is successful in federal service?

    The PMF is a blind assessment. That's good, and it should continue to look for the best qualified as the primary criteria. But does the recruiting and outreach about the program reach the full spectrum of qualified students? How can OPM do that? Those are the bigger questions in my mind. It's all about providing a best qualified pool who represents the diversity of the "customer" and can who can best relate to and serve them.

  10. This is quite off topic, but you mentioned in the previous post that you take requests for posts:

    I'm curious how many of the jobs posted on the PMF board have already been filled, but no one has bothered to take them down? It just seems unlikely that quite that many jobs are open.

    Thank you!

  11. And two days later I find out! Looks like all but 14 were taken, they took off over a hundred and fifty!!