By Chris Cwik
It's hard to forget a memorable draft bust. Despite less-than-stellar performances at the highest level of their respective sports, Ryan Leaf, Darko Milicic and Brien Taylor will always be remembered by fans for how spectuacularly they flamed out. The truth is, it's really hard to evaluate how amateur players will transition to the highest level in any sport.
But over the past few seasons, baseball has offered a glimmer of hope as top prospect lists have started to become more accurate. While analysts will never be perfect, their understanding of what comprises a future major leaguer has increased exponentially since these lists came into existence.
Since 1990, Baseball America has been the gold standard when it comes to prospect lists. While other publications and sites have entered the game, virtually no one has done it as long as Baseball America, and it's that history that makes Baseball America's lists one of the easiest to analyze. Chris St. John of Beyond the Box Score did exactly that in 2013, looking at a number of factors related to how well Baseball America performed with their annual list.
Perhaps the most interesting tidbit: The magazine had hit on a higher number of prospects in recent years as compared to their earlier work.
"It definitely does appear that more Baseball America top-100 prospects have been in the top 100 players each year," St. John said in an email interview. He sent along the following chart, which shows just how well the publication has performed over its run.
The above chart shows the number of prospects from Baseball America's lists who ranked within the top 100 by Wins Above Replacement in each season. Basically, it explains how many prospects were actually top-100 players in the majors each season. Starting in 1996, the figures start to see a slow rise. Things actually topped out from 2006 to 2010, when about 74 percent of prospects ranked by Baseball America were top-100 players in the majors according to WAR. Even if 2006 to 2010 is an outlier, the fact that Baseball America has consistently nailed about 70 percent of the best players in the majors since 1997 is astounding.
Even in the early days, "the sources were really good," according to MLB.com and MLBPipeline.com senior writer Jim Callis. Callis would know. After joining the magazine in the late 1980s, Callis contributed to his first top-100 list in 1990. He briefly left Baseball America to work for STATS, Inc. in 1997, but returned in 2000. His second stint lasted until 2013, when he left to join MLB.com. Every year he was with the publication, Callis played an integral role in the creation of each year's list.
Early on, the magazine had just four or five full-time employees. Those members would each come up with their own list of players, and meet for roughly six hours in order to determine the final product. At this point, the magazine had already cultivated a strong relationship within the game due to founder Allan Simpson. Simpson started Baseball America in 1980, and worked to establish its credibility early. Though he left to work for Perfect Game, he still remains an influential presence at Baseball America. Current Baseball America editor John Manuel believes Simpson should be in the Baseball Hall of Fame based on the impact the magazine has had on the game. "Look at how much writing and how much interest there is in prospect writing, and how many people in the game see the game with our point of view," Manuel said. "That's all because of Allan Simpson."
Due to the groundwork laid by Simpson, Callis felt Baseball America had quality sources in the early days. "When I was a 21-year-old kid starting at Baseball America, I felt like I could call almost anyone in baseball and they would be cooperative," he said. "It still baffles my mind."
The sources may have always been strong, but gains in player analysis really pushed prospect lists to another level. "The first information revolution came from statistical analysis," according to Manuel. There's a sense that the stats revolution caused analysts to have a better understanding of what makes a good ballplayer. It's not even necessarily advanced numbers, as minor-league box scores and video can now be seen instantaneously. Back in the early days of Baseball America, the magazine would print minor-league stats every two weeks. Aside from the occasional local paper located near a minor-league club, that was the only place the stats were being printed.
Stats and prospects don't usually mix. In 2008, Red Sox general manager Theo Epstein told Manuel that evaluators shouldn't get too caught up in statistics at the minor-league level. Where the stat revolution really helped was with context. "There's more attention paid to statistics, and a greater understanding of what matters the most," according to Callis. ESPN Insider's Keith Law agreed, citing the importance of ground ball rates for pitchers and the emphasis on catcher framing as newer information that has been helpful in evaluation. "It's also clear that teams know things, due to private data or superior analytical firepower, that we don't know on the outside.," according to Law. There's a chance for that understanding to improve even further in the future. If PITCHf/x or FIELDf/x numbers ever found their way into the minors, Callis believes they would be a tremendous resource.
Advances in technology have also played an important role in the accuracy of prospect lists. "To track down scouts you had to know what hotel they were staying at, or track them down through the team or track them down at home," Callis said of the pre-cell phone days. Even then, Callis might finally reach a scout only to discover the scout had not seen the player Callis was researching. It's much easier to get in touch with sources and share information today.
Technology has also ensured there are far more outlets for analysis. Fans will tweet about games, different publications and sites will do analysis and people will upload videos of prospects to YouTube. Video in particular has been important. "I can watch a few swings or a few pitches, even just on video, and start to form an opinion a player that I can then refine through further looks," said Law.
In the hands of someone experienced, like Law, this resource can be useful. When unqualified people get their hands on this freely available information, it can cause some issues. "Accessibility, with the way that the internet is now, allows people who have no f***ing clue what they're doing to put out things people think are legitimate," according to Baseball Prospectus' prospect guru Jason Parks. Parks isn't necessarily opposed to more people having a voice, but only if they employ a sound process. "I run into far too many glorified Google-scouts," he adds.
At the same time, the competition, at least among smart analysts, isn't a bad thing. "I do feel that competition made us better," said Manuel. "We are better because we have to be." Callis agreed, saying Baseball America goes much deeper now than they had to in the past. Back in the early 90s, Baseball America and John Sickels, now of MinorLeagueBall.com, were putting out lists. They've since been joined by nearly every prominent baseball site out there. Some of those sites brought in talent from Baseball America. Former Baseball Prospectus list guru Kevin Goldstein worked with Baseball America prior to joining the staff at Prospectus. He's now the Director of Professional Scouting for the Houston Astros. Matt Meyers, one of Law's editors at ESPN, was also a member of Baseball America.
Finally, there's a sense that repetition matters. All four evaluators said they believed their lists have improved in recent years. "The more you do it, you get a feel for what template works and what template might not work," according to Callis. Though Law agreed, he didn't want to take it too far. "You'll never catch me calling my lists 'accurate.' That is an impossible standard in this job."
The process used to compile these lists differs greatly among evaluators. Since the early days, Baseball America's rankings meeting has become more streamlined. Anyone in the office is allowed to participate, but it's on them to make sure they are informed. "We put that responsibility on every staff member to inform themselves in order to contribute," said Manuel. Each participating member ranks 150 prospects, which are then sorted into a weighted spreadsheet and discussed in chunks.
Manuel stressed that Baseball America tries to remain consistent with their rankings. They won't boost one player too much in a short period of time. One of their most controversial rankings in 2014 was Orioles draft pick Hunter Harvey. Though Harvey received rave reviews following the draft, he was still the 20th overall pick. Baseball America doesn't list him in their top-100 for 2014, though many other publications have. Manuel admits the strategy can burn them occasionally, but stands by the process. "The industry doesn't know what to do with those guys, and they are our sources," he said.
Law never truly starts or ends his process, saying it is "perpetual." He will typically scout players and talk to scouts and execs throughout the year. That's part of his everyday job. In about November or December, things start to heat up. He will speak to a source within each organization and that team's system. He then speaks to outside sources to compare information, and comment on his early lists. "The master list never feels done, because it really never is done," Law explained. "Players are volatile creatures."
Baseball Prospectus has done things a little differently since Parks took over. "I wanted to run this like a scouting department," Parks said. He hires people in prospect-heavy environments all around the country who have long-term ambitions to work within the game. It has worked, as four members of his team have joined major-league clubs in the past eight months. "We base the majority of our work on our own eyewitness take." Parks doesn't even contact sources within organizations to discuss their prospects until he's done with his lists. He does this in order to eliminate manipulation in his information. Certain players may be talked up by their organization because the team is trying to trade that player. Parks' approach eliminates those issues. "I try to build up a model based on our own eyewitness reports, and then I augment those, either in support or argument, with outside the org scouting opinions," he explained.
While it's been an effective strategy at BP, it has burned Manuel in the past. Manuel was working on a Yankees prospect list, and couldn't decide whether he should list Eric Duncan or Robinson Cano as the team's top prospect. Scouts outside the organization were hesitant on Cano's ability at the time, but team officials were always higher on Cano. Manuel went with Duncan first, and the decision has stuck with him for 10 years. "One guy is going to make $300 million in his career, one is Eric Duncan," he said.
Despite the varying approaches, these prospect lists tend to wind up featuring a lot of the same players. "While looking at different prospect lists, it is very apparent that there truly is a consensus among the industry," according to St. John. "In general, the same 150 players show up on everyone's list." That's not a symptom of groupthink. Instead, it speaks to the ability of these evaluators to properly identify the game's next wave of talent.
Statistical research continues to produce interesting data, and the implementation of PITCHf/x and FIELDf/x could provide the opportunity for a better understanding of the game at the amateur level. While the perfect top-100 list is unattainable, there are still opportunities for future growth. As Manuel noted, "it's just a matter of time before we have more data on these players."
* * *
Chris Cwik writes for various baseball sites on the internet, CBSSports.com and FanGraphs.com. He has also contributed to ESPN and the Hardball Times Baseball Annual. Follow him on Twitter at @Chris_Cwik.