Is your organization ready for a future with significantly more advanced and competitive information management strategies? Are you ready for the inevitable turbulence that accompanies changes: even those that are mostly evolutionary, but some revolutionary? It’s time for Enterprise Information Management (EIM) to grow up and assume its rightful role at the center of competitive advantage. EIM refers to policies, architectures and practices defining how data is acquired, processed, stored, mastered, enriched, cleansed, transformed, distributed, consumed and exploited. While choices about relational and “Big Data” strategies are critical in the modern enterprise, I will keep this discussion a notch higher level, discussing overall strategies and the roles of AI.
A few years back I saw a thoughtful article by Phil Fersht. I think he’s spot on with the “integrated automation” focus, and nowhere is this more important than in automating enterprise information management as a core capability, rather than a bunch of disparate info silos managed, analyzed and understood separately. There is a related opportunity that many predictions seem to miss: using EIM to build “intent models” that can make all automated processes smarter. The intent models can reflect both customer intent and user intent: the information and processes users need to succeed. These models can simplify adding AI to the mix. Approaches described as intelligent ERP and or intelligent CRM are likely to involve more siloed, narrowly-defined architectures. The truly intelligent enterprise will be much more inclusive of information assets that cross boundaries, and intent models that reflect both customers and internal workers.
Specific Predictions on Competitive EIM
We are seeing the analysts’ forecast becoming reality. Gartner’s “Predicts 2017: Artificial Intelligence”, there are three important predictions that I find salient to the question of competitiveness in EIM:
- Organizations using cognitive ergonomics and system design in new artificial intelligence projects will achieve long-term success four times more often than others.
- Why care? If projects (investments) are so much more likely to succeed with good AI, why invest in projects that don’t have it?
- Startups will overtake Amazon, Google, IBM and Microsoft in driving the artificial intelligence economy with disruptive business solutions.
- Why care? The underlying reason for this phenomenon is the innovation agility of startups. Choose your partners wisely!
- Artificial intelligence platform services will cannibalize revenues for 30% of market leading companies.
- Why care? This prediction focuses on the AI “Platform”. This is consistent with our understanding of the overall benefits of the Platform Revolution.
Careful reading of the cited articles and others point to the need for better decisions at all levels of the business (see ABCs). Better decisions want better processes and data. Semantic models of intent behind each type of decision can make this possible.
In the corresponding 2018 predictions, Gartner says: “AI and machine learning strategy development/investment is already in the top five CIO priorities.” Accenture also cites this and explains that: “Increasingly, businesses have invested in building AI-enabled applications. Examples range from business chatbots and question answering systems to more complex systems, such as self-driving cars, virtual doctors, and AI-assisted decision-making platforms.” They point out that their clients are increasingly combining search with Natural Language Processing (NLP), machine learning, and big data to add AI capabilities to information management strategies and the new and existing systems that implement them. The AI component that can serve as a foundation for NLP based interaction, machine learning and big data analytics is a robust semantic model of intent.
A Case for Holistic
Why is it so important for EIM and AI to transcend functional areas, lines of business, divisions, regions and apps to deliver more sustainable competitive advantage? Because, no matter how distant the data seems to be from the customer or the sale, it can contain clues to help good analysts find ways to reduce costs and/or increase revenues.
Phil Fersht describes “boundary-less organization where there is only one office that matters – and that is the office that caters to the customer” (ibid). He suggests that the path to this utopian vision includes:
- automation (see “More Agile than Agile” ),
- artificial intelligence (see the section on AI), and
- smart analytics (see “Anatomy of Insight: Prediction and Qualitative BI“).
May I add:
- Semantic models of intent that “know” why in addition to what.
My world view comes partly from being in the IT Architecture business for a long time and having experienced many of the trends and technologies that you talk about in your blog. I have used many architectural and programming styles and patterns. Over the years I have become convinced that most process centric architectures do not stand the test of time. Part of the early genesis of SOA came from early attempts at inter-application cross-platform communication in standards like HL7, EDI and CORBA an output of OMG. I watched CORBA collapse under its own weight and I would contend that some other weighty Web Services standards are also well down that path.
What I have seen survive is the “information” that applications exchange (I hesitate to use the word “share”). The format for that exchange or even the semantic meaning of the information is often derived from the context of its use, so most “meta data” schemes fail to capture meaning and fail to empower any but the most savvy users and technicians. This is where I am a fan of some of the concepts in “Data Lakes”. In most Data Lake models the creation of “meaning” is placed on the consumer of the information and often meaning is only derived by combining information from many sources that know nothing about one another. This phenomenon suggests that intent models for each domain in the enterprise should be interconnected.
There is much confusion and arm-waving about AI and where it can be applied to bring value in the enterprise. Whether your enterprise information management strategy is primarily offensive or defensive or both, the most important questions to ask are probably at the core capabilities level: what AI and learning processes can be applied to this core capability and will the costs justify the benefits? There are cases in which an organization has enough money and bandwidth to play around with several different ideas and see which pan out. I think those are exceptional organizations, and most budgets and teams should be focused on initiatives most likely to deliver enough business value to justify the investment. So I assembled some ideas on where to apply AI/ML to augment core EIM capabilities.
There are many processes that can be included in an Enterprise Information Management strategy beyond the ones mentioned in the table above. Notably absent are data cataloging and references to some of the finer-grained processes that uniquely apply to streaming data and big data. The TOGAF architecture standard chapter on data management services (TOGAF Ref) provides useful descriptions of these. They include:
| Data Dictionary and Repository services | These are repositories for data stewards, modelers, administrators and engineers to define and use the metadata that defines all data objects | To simplify building and managing data dictionaries AI bots should scan data sources and automatically build and periodically update the catalog |
| Data dictionaries are often vendor/technology specific, but heterogeneous data dictionaries may support multiple database types and sometimes unstructured data as well | Semantic associations between data tables and columns in the catalog will dramatically increase its usefulness and empower users | |
| Intent models describing domains, subdomains, concepts and associations are based on robust semantic associations | ||
| Database Management System (DBMS) services | Database admin for both ACID and BASE transactions and functionality for database admin are standard in commercial databases | Determine the best data platform for each type of data and carefully consider the platform’s adaptability to support AI and automatically generated queries and intent models |
| Languages like SQL, NoSQL, Cypher and SparQL provide controlled access to structured data in relational and graph databases | Tie all databases, tables and columns to a central/aggregate enterprise intent model | |
| File Management services | File and document structured databases and file systems are growing in importance as part of the analytics and data exploitation ecosystem | Scan full text and tie all documents at a section or clause level to a central/aggregate enterprise intent model |
| Query Processing functions | Queries provide interactive selection, extraction, insertion, deletion and formatting of information in files and databases | Intelligent automated query generators that use the intent model to associate the users’ questions with data / content |
| Screen Generation functions | Dashboards and other visualizations provide the capability to define and generate views from the retrieval, presentation, and update of data. | AI can be used to adapt to each individual dashboard user’s preferences both in content (including filters and sorts) and presentation (labels, types of visualizations, arrangement…) |
| Report Generation functions | Provide the capability to define and generate hardcopy reports composed of data extracted from a database | n Paper requires no special intelligence |
| Networking/Concurrent Access functions | The functions that manage concurrent user access to databases are typically not included in an EIM framework | AI monitoring scans to improve security and access control is growing exponentially as threats mature and proliferate |
| Data Warehousing functions | Functions and systems that store very large amounts of data – usually captured from other database systems | AI can be used on data in transit into the warehouse to ensure it is meaningfully cataloged and registered |
| Functions to support or perform online analytical processing on it in support of ad hoc queries | Intelligent monitoring and rules can identify inefficient queries and tune or redirect them to preferred sources | |
| Document Generic Data Typing and Conversion | Services and specifications for encoding data (text, image, video, numeric, special character…) and the logical / visual structures of electronic documents/compound documents | There are many use cases for data typing and conversion, but there fewer opportunities for automating decisions using AI – only AI for automating the conversion itself |
| Graphics Data Interchange | Services for device-independent descriptions of picture elements for vector-based and raster-based graphics | These services are standardized and AI opportunities are clustered around data cleansing and transformation |
| Specialized Data Interchange | Services and specifications like EDI that describe data for specific vertical markets such as the Medical, Library, Dental, Assurance, and Oil industries | These services are standardized and AI opportunities are clustered around data cleansing and transformation |
| Electronic Data Interchange | Services for paperless office and e-commerce to improve quality, responsiveness, and savings. Such services include vendor search and selection, contracting, product catalog, shipping, forwarding, receiving, customs, e-payment, credit card, inventory control, claims, electronic tax filing and more | This broad category has many potential applications of AI to facilitate, improve and exploit information in transit supporting the full range of transactions. Electronic Claims Filing for Health Insurance for example: Use AI to improve the coding for both diagnosis (ICD) and procedures (CPT) |
| Fax and Image Transfer | Services to create, examine, transmit, or receive fax and other images | Use AI or natural intelligence to replace faxing entirely |
Each possible enhancement should be matched with a business outcome and weighed against the cost to determine the business value. This process makes prioritization straightforward. Some experienced business leaders working with experienced technologists can analyze oppotunities and make decisions quickly. For most, formal methodologies and architecture discipline are needed. But the day will soon be upon us when not just large enterprises, but mid-sized businesses will need to integrate AI in ways that lead to competitive advantage, and information will be at the center.
