Wouldn’t the National information exchange be better served by deferring the National Information Exchange Model (NIEM) and instead implementing some sort of Google-like search of federal, state, and municipal text data records. Most federal, state and local data resides in sophisticated databases using their information management tools but such tools all seem to support ways to create a PDF, DOC, or other text output for their information records. Once in text form, such data could easily be indexed by Google or other search engines, and thus, searched by any term in the text record.
Now this could never completely replace NIEM, e.g., it could never offer even “close-to” real-time information sharing. But true real-time sharing would be impossible even with NIEM. And whereas NIEM is still under discussion today (years after its initial draft) and will no doubt require even more time to fully implement, text based search could be available today with minimal cost and effort.
What would be missing from a text based search scheme vs. NIEM:
- “Near” realtime sharing of information
- Security constraints on information being shared
- Contextual information surrounding data records,
- Semantic information explaining data fields
Text based information sharing in operation
How would something like a Google type text search work to share government information. As discussed above government information management tools would need to convert data records into text. This could be a PDF, text file, DOC file, PPT, and more formats could be supported in the future.
Once text versions of data records were available, it would need to be uploaded to a (federally hosted) special website where a search engine could scan and index it. Indexing such a repository would be no more complex than doing the same for the web today. Even so it will take time to scan and index the data. Until this is done, searching the data will not be available. However, Google and others can scan web pages in seconds and often scan websites daily so the delay may be as little as minutes to days after data upload.
Securing text based search data
Search security could be accomplished in any number of ways, e.g., with different levels of websites or directories established at each security level. Assuming one used different websites then Google or another search engine could be directed to search any security level site at your level and below for information you requested. This may take some effort to implement but even today one can restrict a Google search to a set of websites. It’s conceivable that some script could be developed to invoke a search request based on your security level to restrict search results.
Gaining participation
Once the upload websites/repositories are up and running, getting federal, state and local government to place data into those repositories may take some persuasion. Federal funding can be used as one means to enforce compliance. Bootstrapping data loading into the searchable repository can help insure initial usage and once that is established hopefully, ease of access and search effectiveness, can help insure it’s continued use.
Interim path to NIEM
One loses all contextual and most semantic information when converting a database record into text format but that can’t be helped. What one gains by doing this is an almost immediate searchable repository of information.
For example, Google can be licensed to operate on internal sites for a fair but high fee and we’re sure Microsoft is willing to do the same for Bing/Fast. Setting up a website to do the uploads can take an hour or so by using something like WordPress and file management plugins like FileBase but other alternatives exist.
Would this support the traffic for the entire nation’s information repository, probably not. However, it would be an quick and easy proof of concept which could go a long way to getting information exchange started. Nonetheless, I wouldn’t underestimate the speed and efficiency of WordPress as it supports a number of highly active websites/blogs. Over time such a WordPress website could be optimized, if necessary, to support even higher performance.
As this takes off, perhaps the need for NIEM becomes less time sensitive and will allow it to take a more reasoned approach. Also as the web and search engines start to become more semantically aware perhaps the need for NIEM becomes less so. Even so, there may ultimately need to be something like NIEM to facilitate increased security, real-time search, database context and semantics.
In the mean time, a more primitive textual search mechanism such as described above could be up and available for download within a day or so. True, it wouldn’t provide real time search, wouldn’t provide everything NIEM could do, but it could provide viable, actionable information exchange today.
I am probably over simplifying the complexity to provide true information sharing but such a capability could go a long way to help integrate governmental information sharing needed to support national security.