EPA should seek to empower EPA staff to do more in-house development, as opposed to relying as extensively on contractors as we currently do. Even for routine applications, EPA staff often end up turning to contractors for a variety of reasons, for example because most EPA staff do not have access to development IDEs, tools and development platforms. Empowering EPA staff to write more code and providing access to the tools and environment would open many options in terms of rapid application development, prototyping, technology evaluations, application maintenance and other activities that could be done in-house, which could in turn yield substantial savings that could be directed toward more robust development efforts and technology acquisitions.
Currently a lot of development efforts happen at contractors' offices, on development environments that are controlled by contractors and which are only accessible to contractors. This is often not conducive to integrated Agile development efforts, except within the contractor's office. Agency staff can typically not access nightly builds, for example. Typically contractors also rely on their own configuration management tools, which are often not accessible to EPA staff, limiting EPA participation in Agile efforts. Additionally, VMs can ensure clean, consistent installs and configurations as currently there are often differences between environments, where the development environment at the contractor's office may be set up and configured differently from EPA staging and production environments, or where some EPA resources are not accessible and available to the contractor in the way that the intended production architecture will rely on, and this can lead to application misconfiguration issues on deployment as well as unanticipated issues with data and service connections and performance.
In a more ideal world, EPA should have the ability to spin up development instances and IDEs that get around this issue, for example images containing images and IDEs with most of our standard production stacks which could be instantiated as virtual machines, whether via Vagrant or other DevOps tools. For technology evaluations, Vagrant and VirtualBox could also be used to spin up a temporary VM that could then be destroyed after the evaluation is completed, freeing resources up and not cluttering other infrastructure with any leave-behinds. Use of hosted VMs, whether in NCC or cloud-hosted would also allow the VMs and IDEs to be shared by development teams, rather than having development just happening on one user's machine.
Additionally, these IDEs should be tied to a common, shared configuration management framework, ideally Github. All code developed on the taxpayer's dollar should have its source code available, documented, and accessible publicly via Github - unless there is a specific reason why, i.e. CUI, and these reasons should be clearly indicated and documented. Regarding Github, there are currently a few wonky requirements which should be re-evaluated, such as having a copy of the source code available on a static EPA site, which doesn't seem to understand the function of Github.
Any new development efforts should first look at existing reference architectures and technology stacks, and then existing applications in Github, rather than starting from scratch.
Additional thoughts relate to improving how EPA developers document their code. For example, considering swagger.io as a means of developing documentation for REST APIs, or documentation.js.org as a means of developing documentation for JavaScript.
In addition to just source code and configuration management, EPA should look toward some of the more current models and frameworks out there for managing and tracking requirements, software testing and build management, issue tracking and other parts of development, as having a common agency-wide framework for this will also help enable the agency to become more agile, as for example I currently have two projects ongoing which involve separate contractor teams building different pieces which need to interact via web services – having a common framework would improve how contractors can check on status of each piece, eliminating communications lag and generally streamlining the process overall.
With regard to APIs, it may be worthwhile for EPA to consider a more robust API strategy, potentially with some additional infrastructure that can help track usage, parameters, enable caching, load balancing and scaling, as well as providing the agency with better means of modernizing and improving backend capabilities without having to worry about our endpoints changing as a result.
EPA relies very heavily on a lot of proprietary and COTS software which has been very costly. In many cases, these licenses may want to be revisited with a more holistic, agency-wide approach, to see if savings can be realized either through renegotiation, moving to enterprise license agreements, or looking at viable Open Source alternatives. Many proprietary commercial software packages, over the years, have had their capabilities matched and often exceeded by Free/Open Source software (FOSS) alternatives. For example, EPA is spending somewhere in the vicinity of $18m to maintain Oracle licenses, in many instances to support basic RDBMS capabilities, when PostgreSQL could in many cases replace Oracle as the backend.
OEI currently has a huge amount of issues in contracting, there are significant backlogs in the contracting shop that routinely put us at risk, both project risk and financial risk. Perhaps there are ways that this could be streamlined and improved through some process improvements and some boilerplate approaches to how IT procurements are handled. It would additionally be good to have some reference language that could be referred to in IT contracts, relative to standards and best practices, IDEs and source code (i.e. use of GitHub) in order to better foster transparency, collaboration and interoperability in the contractor development space.
In addition to empowering current EPA staff, EPA should look to leverage any lessons learned via the 18F/USDS/PIF on how EPA can attract/recruit/retain top-notch technical experts in order to continue to bolster our in-house technical expertise.
In terms of training, EPA currently has various pockets of information, such as URLs for references, online training and courses that have been posted on technical resources in various fields, such as R, GIS and web development, however these are currently scattered in various pockets. It would be good to have a more robust and navigable set of training resources for use across the agency, which could be community-curated and managed. In addition to free resources, there are a number of low-cost IT trainings and certifications available via providers such as Coursera, howerver EPA currently is requiring Continuing Service Agreements for Coursera courses, even though they in many cases only last 6 weeks and only cost $49 - with the reason cited being that the Coursera courses are being offered through a university. This does not seem like a reasonable criterion in this case, and training should be more flexible to allow Coursera coursework without requiring a CSA.
Currently, developing Risk Management Framework (RMF) artifacts is very time consuming and if contractor staff are used, it can be very costly as well. A lot of this effort and cost could be minimized through development of pre-packaged information based on some of EPA's standard application architectures and production stacks. Components could be added to the RMF package as overlays. An example of this was done by Greg Elin as [GovReady] (https://github.com/GovReady), where many core application environments can be deployed quickly and easily (again, see Vagrant and other DevOps tools as a means to aid with this) but together with the reference architecture comes an RMF package that is mostly pre-built, making it easier to get to deployment, rather than each application starting off with a blank sheet of paper to develop their RMF package.
Additionally, EPA strongly needs a more direct path to deploying on commercial clouds. FedRamp seems to have failed to deliver where it comes to providing ready-to-run options for EPA that satisfy FISMA and our IT security requirements. EPA should look to develop a set of RMF-documented cloud-hosted core technology stacks, which could be deployed with minimal headaches and issues. Again, we could look to the GovReady model.
We have an entire group for research and development. We also have a lot of the tools necessary. We do lack technical folks and there is power in having contractor support. Could we accomplish this simply by insisting contractors work on space provided by EPA -- AND actually providing it?
Also, you speak of AGILE and other fabulous things, but this leaves existing IT staff behind because we don't have this experience -- so hiring new folks only furthers this gap and fractures development activities. I think we need more technical staff and we need to not stifle them, however, there are many ways of accomplishing tasks ( ie programming languages) which makes hiring full time folks less palatable than using contractors who can pivot much more rapidly.
I think part of the problem is the way "IT" is viewed -- structure for others getting work done. Our regional staff's focus is making sure the infrastructure, identity stores, and other things are in place for staff to work. They are not terribly concerned with development and in fact, "policy" almost forbids it. This equates to a few folks experimenting on the side of their regular job. They rarely have time to document it out as an API or otherwise because they're more concerned with getting their own job done.
I'm incredibly sympathetic with Open Source, but the "Free Tag" doesn't do it true justice. We also had a senior official who declared we would ONLY use COT solutions and I don't think we've recovered from that. There is a lot of value in COT solutions as well. What I would like to see the agency do is pick something and start developing it....so whats the most expensive software and how do we need to get it into something open source. DB's are the easiest target here, however, its more complicated. Our entire regional lab is built on one vendor and so is EQUIS and others. Because this is paid for, it isn't worth the Open Source curve for other smaller applications and so we need some migratory plan -- which likely will take years...at which point, the transition costs outweigh the short-term cost benefit plus its a headache to move systems because DBs can have such looong tendrils that snake many places.
Those are some thoughts to digest on. As with most things, start small.