Letters of Support.
So for the R03 I’m writing, I’ll get letters of support from all the key personnel and my department head, etc.. But I wonder about other letters. When I wrote my first R03eq grant at my former employer, I got a local famous person in a closely-related field to write me a letter that basically said, “Hi! I’m famous and I think Dr24hours is super awesome. We expect this to be a success, and when it is, we’d love to collaborate directly on the follow-up R01eq!” That grant was funded.
Now I’m writing another R03 to another agency. In this case, I’m again developing a pilot project to hopefully eventually lead to an R01 or R18. Because this project straddles research and quality improvement, and because it will directly impact several areas of MECMC, I’m thinking of asking the medical directors of those areas to write, essentially, amicus briefs. Once again, not really agreeing to participate, but saying, “Hi! We’re famous and we know about this research and its potential to impact our areas. We think those impacts will be great, and we’ve worked with Dr24hours before and he’s a total rockstar. We’d love to advise him informally throughout the process.”
Do these kinds of letters help? Do they mean anything? Do letters help at all, or are they just digital checkboxes for reviewers to say, “Yup. The dude got letters from people.”? What should a good letter of support say, if it’s from someone other than key personnel?
Infactorium 
Letters help. When specific-ish. No more than 5 in my view, my eyebrows go up past 5. Basic point is “I’m excited by your project b/c ….[insert some individualized reference]. I am happy to consult with you on X, Y and Z which I [insert something about individual’s expertise]”
No more than 5 beyond the letters from key personnel, you mean?
Wait, how many letters of support do you have already? This is starting to sound like a lot of letters….
This could totally vary by field, but IME letters of support from key personnel are unusual. In general, their biosketch statement should be enough to cover their role and appropriateness for being on the grant. Obviously they support it, since they’re on it.
IMO, non-required letters of support should be included for the purpose of anticipating possible perceived weaknesses in the proposal, not to pad it with pre-review praise. In your case, I do think the letter from the dept head might be a good idea since it sounds like it’s uncommon for people in your dept to apply for grants, and that could be seen as a reason to ding it. But if you worry that the reviewers won’t appreciate the potential impact in your grant unless someone famous says it will have impact, then that is an issue with the proposal itself.
For VA grants, there must be a LOS from all key personnel, in addition to biosketch. It’s a rule. Is that not true for NIH?
No, especially since key personnel can be “TBD.”
!! That doesn’t make any sense at all for a VA grant. All key personnel must be named, and have biosketches, CPS docs, and LOSs. By definition, if you don’t have that, they’re not key personnel.
I’m in a different field of applied science, but our practice may be relevant. I get letters from the potential beneficiaries of the work (such as the medical directors in your example). The letters establish that the need is real. Their letters say things like “We could do great thing X if only we had a way to overcome obstacle y.” and also “the proposed solution z would be an excellent way to overcome obstacle y.” The letter writers do not have to be famous, and they do not need to comment on the scientific merit of the work. If you have solved problems for them in the past, they could usefully mention that. If you have a long-established collaboration on something, they could mention that. Both of those facts speak to the likelihood of adoption. Many PIs have rather rosy assessments of how useful and relevant their work will be, so these letters establish that you are working on a realistic solution to an important problem.
Having been on review panels for NSF, NOAA and Sea Grant, I do believe that letters supporting the project do help. I thought this guide had some interesting information: http://www.cihr-irsc.gc.ca/e/45246.html