Once an organisation employs even one member of staff, maintaining data becomes challenging because data is stored in different systems.
The task of connecting these systems up, is known as system integration. This post talks about some of the reasons your organisation should integrate their data, and the terminology and processes involved in doing so.
The Information Maturity Model
Rostow's staged economic model inspired our own model.
We, created a five stage process explaining how a company evolves.
- Logbooks, receipts, email, text messages.
- Spreadsheets, Email, Website.
- Some databases, some reports, CRM systems.
- Data Warehouses, Business Intelligence, System Integration, Automation.
- Decision Support, Machine Learning, Artificial Intelligence, Data Visualisation.
Key outcomes of the Information Maturity Model;
- Organisations tend to have business functions at every stage.
- Few companies have the in-house capability to implement stages four and five.
- Every stage supports the business to increase efficiency, redeploy headcount, increase understanding of their business.
How can my company benefit from integrating data from our software systems?
A great example would be connecting a stock application's database with a billing database. Another example is in finance, where a risk system monitors the credit risk of its counterparties. Adaptors to monitor changes to files on a system can tell us activity at a point in time.
- Having a single point of access to all data, speeds up decision making.
- Connecting disparate data in reporting dashboards and data .
- Having data automatically connected stops a need for manual intervention, reducing operational risk.
- Seeing real-time/near real-time customer interactions, against company data gives a better understanding of your business.
Incorporating external data
Organisations nearly always look outwards from within. Doing so, ignores the huge number of interactions going on in the world. Here are some external data sources your organisation may look to use;
- Twitter - see what people are talking about to gauge interest.
- Online surveys, create free surveys online and ask your customers to complete them.
- News articles through scraping - see what headlines are in the paper.
- Company performance data by sector - see how your industry is doing.
- Competitor prices. See what prices competitors charge versus your own company.
- Customer sentiment data.
Challenges to using external data
This can be tricky, because external data does come with some caveats;
- It may not be regularly updated.
- Its structure can change.
- Your organisation has no guarantee over the accuracy of the external data.
- It may be harder to integrate into your company's reporting system.
- External data may be at an incorrect granularity. Your data may be at town level, but external data is at regional level.
There are many ways to overcome these issues if you engage a Data Solutions Provider like Info Rhino Limited.
What are the advantages of using external data
Curating and managing data costs money. When onboarding data from external sources, there is a cost to it - but typically, a lot of the hard work has been undertaken. Nothing is for free, but significant cost has been employed externally.
Be prepared to clean data - external or internal
A big challenge organisations overlooks, is that data may need significant enhancement or improvement to be useable. Data Quality can be measured in terms of;
Understand the importance of personal data
Often, it is important to know who said what. Or who was engaged in a specific activity. Many times, storing personal data is not only intrusive - it is illegal. Ways to overcome this include privacy policies, fair-use policies. Technical approaches include pseudonymisation, encryption, data hashing, data access control.
For example - you may have a list of Twitter Followers, do you need their Twitter username within every system? Or could you hash that data, but retain it just for mail shots?
Technical Section - How do we connect software applications to get information from them?
Feel free to skip this piece if you are non-technical.
As we saw from the Information Maturity Model, stage one to two companies will have information in multiple systems.
We see there are three main ways to connect data up.
- Application Programmable Interfaces (APIs).
- Publications including data migration.
These main three mechanisms allow the following types of applications to connect data;
- Self-service BI.
- Reporting solutions.
- ETL solutions (Extract, Transform and Load).
- Streaming services.
It is vital, that any system integration is as automated as possible. This can be undertaken using batch automation or event handling with publish and subscribe patterns.
Closing thoughts on System Integration
Large companies maintain their market dominance because they integrate their data systems. They define clear strategies on how to manage their data, to ensure it is useful in supporting effective decision making. Info Rhino has worked on large reporting implementations at financial institutions and other dominant sector entities.
Small to medium sized enterprises should be doing the same.