A colleague recently said to me:
There is no “they.”
There is only “we.”
There is no “them.”
There is only “us.”
A colleague recently said to me:
There is no “they.”
There is only “we.”
There is no “them.”
There is only “us.”
tl;dr: Web apps should work behind reverse-proxy servers out of the box. Use relative URLs.
Here’s my issue: Continue reading
On May 9th, 2013 it will be the 10th anniversary of the issuance of the DoD’s Net-Centric Data Strategy. In honor of this, I would like to highlight a collection of slides that were used (many, many times) to illustrate the concepts of net-centricity, the data strategy, Communities of Interest, and User-Defined Operational Pictures.
These slides are quite dated… last updated in 2006, but there are some great illustrations and concepts that remain relevant. The Data Strategy was a brilliant work that had great promise for changing the way people see IT in defense, in my opinion. It’s tragic that various personnel changes and other events shifted the focus elsewhere.
[Extra disclaimer: I am not currently involved in any deliberations with regard to AHLTA or VistA or iEHR, and have no particular knowledge of current thinking on EHR inside the Pentagon. I briefed the Hon. Beth McGrath on Open Source Software in April 2011 – a long time ago.]
As some of you probably know, Jon Stewart of the Daily Show recently ran a clip about AHLTA vs VistA. (embedded below for convenience)
One of the interesting side-stories from this is the open-source vs proprietary angle which Stewart did not address. The VA was (is?) a big supporter of Open Source Software. (It is less clear how much commitment remains, now that both Roger Baker and Peter Levin have left the VA.)
An Open Source Software model makes perfect sense for VA. They had the business problem of 133 treatment facilities with 133 custom versions of VistA. One of the beautiful things about VistA was that it was open to customization at the various installations so that it could be tailored to specific needs and this allowed innovation at point of service. The problem is that the overall operational cost increases over time as the versions diverge, and there is the possibility that interoperability decreases. The VA created OSEHRA to bring these forks back together. By using open source methodologies, they created a common “VA Enterprise VistA” baseline, gives a place for all those local improvements to be merged into a best-of-breed VistA.
This collaborative model is a huge win for innovation and continuous modernization. Allowing local innovation is both good and bad: the innovation part is good, but the loss of standardization is bad. Coupling a tolerance for local innovation with a enterprise reference standard makes the prospect win-win.
I’ve personally fixed three bugs in the Linux kernel, and successfully gotten those patches incorporated into the Torvalds-approved kernel.org kernel. I fixed the bugs because they affected me personally. I worked to get them into the upstream because I didn’t want to have to keep re-applying my patches every time a new kernel was released. There is also a non-trivial pride and ego-boost associated with that accomplishment. These same incentives will cause VA doctors and IT staff to work to improve VistA locally, and also to merge those changes into an enterprise baseline.
These arguments in favor of an open-source development model could apply to the DoD also, or to a joint DoD-VA approach.
What’s not clear to me is how much VistA should play a starring role in that future.
Here’s an issue: building a vibrant, collaborative community around a software development project is about people and process as much as it is about technology. Some great books have been written about this. It will be difficult to get bright, talented software engineers to work on VistA, because of the 30-year old tools and design practices. I have two degrees in computer science from MIT and you couldn’t pay me enough to work on VistA. Literally. Even if you aren’t a software engineer, take a look at this random sample of VistA source code, which is barely distinguishable from line noise:
Not all the VistA code is this bad; some of it is worse.
That said, the DoD alternative (AHLTA) – and in fact the most well-known proprietary alternative – all share the same obscure programming language (MUMPS) and are almost certainly equally bad – but since their code is not public, it’s harder to critique them. In fact, I would expect them to be worse since they were not written with collaborative development in mind.
Say all the nice things about MUMPS you want: In the end, the choice of an ugly, archaic technology will decrease interest in any project by prospective contributors, thus decreasing the value of the collaborative model. This is Technical Debt we may not be able to repay.
Interestingly, Philip Newcomb, CEO of The Software Revolution Inc., has asserted that his company’s technology could convert VistA from MUMPS to J2EE in about a year for $10m. If true, this would be a bargain, IMHO. I’ve met with Phil twice in the past decade, and his claims are impressive, although I have no personal experience with the results of his company’s work. Other companies perform similar services – Hatha Systems for example claims to do automated analysis of MUMPS.
Tom Munnecke, one of the original architects of VistA has eloquently defended the MUMPS database design, pointing out that medical data is rarely suited to the structured, SQL-esque approach of relational databases. He makes some fabulous points, and I actually think that he’s right in that many of the decisions that were made for MUMPS and VistA were remarkably prescient. It’s taken the rest of the IT world three decades to rediscover document-structured non-relational databases, now known collectively as NoSQL. Now that there’s a huge-amount of energy and expertise focused on large-scale non-relational data stores, maybe we should consider how to use that talent and energy for EHR?
In short, I think the VA should keep doing the OSEHRA thing to consolidate and modernize VistA. There’s two threads to that: first, they need to consolidate into an enterprise version of VistA for the VA to bring together the forks (which they are doing), and second, they should refactor and modernize the architecture and the tooling (which they claim they are doing). I am suspicious that they aren’t being bold enough, but I don’t know.
I remain skeptical that VistA can survive in the long term as a vibrant, community-driven open source project, if it continues as it is. In order to make VistA a viable project, the current MUMPS-based database need to be replaced with a modern NoSQL datastore of some kind, and the hyper-abbreviated MUMPS code needs to be replaced with something readable and maintainable. A colleague of mine (David Wheeler) once pointed out that MUMPS code doesn’t have to be unreadable, but the coding practices of VistA do not lend themselves to readability. A bold step would be to re-write the thing in Java or some other modern language; a mimimalist step would just re-write it in MUMPS that doesn’t suck so much. In David’s words:
I think you should note another alternative as well: Keep MUMPS, but translate the current MUMPS into readable code.
Yes, you’d still be working in an uncommon language. But that transformation would be especially trivial to do (and trivial to automate), and the risks from auto-translation would be far lower because the underlying environment and assumptions would be unchanged.
It seems to me that the big problem here isn’t really MUMPS, it’s the way MUMPS has been used. Developers have used MUMPS’ “you can abbreviate anything” combined with “use bad names”, which perhaps made sense 30 years ago but is a bad idea today. But you do not *HAVE* to create ugly code in MUMPS.
Using the Wikipedia MUMPS article example, here’s some line noise:hello() w "Hello, World!",! q
But here’s legal MUMPS – it’s the same code, but unencrypted:hello() write "Hello, World!",! quit
I have no idea if translating to another language would be a better trade-off than translating it to readable MUMPS. But it’d be easy to hire somebody to briefly investigate the options, pros, and cons, so that a reasonable decision (based on EVIDENCE) could be made. And I think, in fairness, that alternative should be considered.
I also have concerns about the governance model of OSEHRA, but, as this blog post is already too long, I’ll save that for another article.
As promised at the top: Jon Stewart on AHLTA vs VistA:
I have a guilty and shameful confession to make.
It happened years ago, and it’s been troubling me ever since. I didn’t understand my mistake until recently. It’s a little hard to explain, so bear with me.
Sometime around 2007, I happened to be at the old DISA Skyline-7 building, when a project called “Button Two” was unveiled. Continue reading
What the heck is CAIV?
CAIV is an idea that when buying systems, Cost should be treated As an Independent Variable (CAIV). The concept and term were popularized in the 1990s Defense Acquisition Reform efforts, and (I think) comes from the Military Operations Research community.
Traditionally, there is a waterfall-like process for buying things in the military: first, ask the “warfighter” what they need, and they come up with requirements. Then figure out what that costs, ask Congress for the money, and build it.
Wait, what? The users decide what they think they need, before considering how much it will cost? How does that work? Continue reading
[This is mirrored on Intelink-U]
In a previous post, I traced through the various policy documents that describe the certification and accreditation processes for the Department of Defense, ultimately tracing back to OMB Circular A-130. In summary, “systems” need accreditation, while “applications” do not, and the distinction (per A-130) turns on the highly subjective decision of whether it is a “major” application.
In tracing this definitional tangle, I unwittingly provided a roadmap for how to get your
system “application” on a DoD network without a full-blown Approval To Operate (ATO). I was not trying to provide an easy-out for getting operational without being accredited … although the method is a well-trodden path with a lot of history. I’m was trying to show that our C&A policy is at-least-slightly broken, and we generally don’t even understand it ourselves.
Again, to summarize, this is what not to do: Continue reading
The Mystery of the Missing Service Provider
In the Defense Department, there is a web-conferencing service called “Defense Connect Online“. Until about two weeks ago, no one was responsible for providing this service. DCO is a finalist for the 2013 Excellence.gov awards, and no one was responsible for providing the service. (i-kid-you-not) This service has 800,000+ registered users, and there were over a half-billion user-conferencing-minutes used last year across the DoD. It is a huge success, especially in an era of declining budgets and severe travel restrictions. People love it, and they hate it too. They love it because it helps them do their jobs. They hate it because they see how much more it could be. Even the people who criticise DCO, (and there are such) do so because they absolutely need the capabilities it provides them, and they wish it did more.
How could this be?
Here’s another story about buying systems without a clear concept of who is going to operate it:
I was at my neighborhood spaghetti dinner and I happened to sit across from some guy who – it turns out – also works for OSD. He’s a contractor working for OUSD(AT&L) on a system that he was quite enthusiastic about called “Contingency Acquisition Support Model (cASM)”. (I have no idea why they capitalize it that way.) This system, cASM, is a web-based application designed to assist people responsible for initiating contracting requirements in an contingency or expeditionary environment. I understand it as workflow automation for contracting in a crisis.
Since I’d been thinking about this a lot, I immediately asked him, “Who is going to run it?” and he said, “Oh, it’s hosted by DISA.”
I tried not to choke, and said something like, “That’s not what I meant. Okay, DISA hosts it, but who is going to run it?”
These “ERAM” reviews had a bunch of process and structure, but the important thing was that we had a team of smart folks do several days of on-site interviews with involved personnel: we interviewed everyone from the 3-star sponsor to the GS-9 users, the contracting officer, the gov’t Program Manager (PM), the Program Executive Officer (PEO), the chief engineer, the contractor’s PM, the software jocks writing the code, and anyone else we could think of.
Generally, when we asked this question, we would get a puzzled, head-cocked-to-the-side look, like the RCA dog.