There are some briefly described pre-requirements and useful considerations for a ‘real-time enterprise’ in this recent William McKnight’s post. To achieve real-time data warehouses, William also suggests rescheduling ‘batch’ ETL (Extract, Transform and Load) to run more frequently.

I’d add that it’s not about rescheduling frequency only; it’s about rescheduling the very moments of ETL operations. Sometimes it is not enough to increase the frequency, it even may be in vain. First, there’s a challenge of increased queue contentions. Second, there’s no use updating data, if there were no changes in it.

Some EAI advisors recommend using alert notifications instead. It is about defining the moments of updating as well as transforming data on an event basis, rather than according to a fixed schedule. However, it is clear that the real-time event model is more complicated, while developing. But as we see, EAI/ETL/EII vendors are already being forced to aiming at it, anyway.   

 

Filed under: Business Intelligence, Data Integration, Data Quality | No Comments »

In an article about Regulatory Compliance and Data Quality, Robert Lerner mentioned watch lists, “the lists of suspect individuals, organizations, and countries with which it is unlawful for transact business with.”

“If an organization has problems in the quality of its customer data, then it will have difficulty making accurate matches against the watch lists. For example, if an organization has misspelled the name of a customer or potential customer – using Smith instead of Smyth – then the odds of correctly matching this customer against a watch list are slim.”

That reminded me of a true story I’ve heard about a man, say, Mr. Smith, who received a call from a bank one day. After ‘hellos’ and ‘how-do-you-dos’ bank officer asked that man to return to the bank and “sign the rest of the papers”.

“What papers?” Mr. Smith asked.

“The papers confirming that you transferred $1,000,000 from your account yesterday,” bank officer answered.

Mr. Smith was shocked. “Me?”

“Are you Mr. Smith?” bank officer asked.

“Yes.”

“Do you live in blah-blah-blah?”

“Yes.”

“Is your account number blah-blah-blah?”

“Yes.”

“Then it might be you who transferred $1,000,000 yesterday.”

In a few hours Mr. Smith found himself sitting in the bank watching yesterday’s video control system tapes. After additional verifications they noticed that there was a mistake in spelling Mr. Smith’s name. Everything was correct, first name, the country of residence, but the last name was misspelled in the papers. Just one letter, one silly letter was misspelled. The man who visited the bank the day before was, say, Mr. Smyth. That’s why the bank officer mixed up the clients and called the wrong number, he called Mr. Smith instead of Mr. Smyth.

That’s it. It’s all about data quality, data validation, data accuracy, etc., etc. Keep an eye on spelling your names, and avoid getting into a watch list.

Oh, yes, and install data integration software. =)

Filed under: Data Integration, Data Quality | No Comments »