Data architecture

Why it matters

Every balance, payment and loan in a bank is data, and so is every report to the regulator. A bank that cannot trace a figure in a regulatory return back to the transactions behind it cannot prove the figure is right. Good data architecture makes that tracing routine. It also lets risk models, fraud systems and management reports use the same trusted numbers instead of each building its own.

The model explained

Transaction systems (OLTPOnline transaction processing. Systems that record many small transactions quickly and reliably, such as a core banking ledger.) record individual events quickly and safely: a core ledger, a payment switch, a card system. They are optimised for many small writes. Analytical systems (OLAPOnline analytical processing. Systems designed for analysing large volumes of data, such as reporting and dashboards.) answer questions across millions of records: how many loans are overdue, which merchants are growing. They are optimised for large reads. Running heavy analysis on a transaction system slows it down for customers, so data is copied from one to the other.

Warehouses, lakes and lakehouses

A data warehouse holds cleaned, structured data with agreed definitions: one meaning of “active customer”, one meaning of “overdue”. A data lake holds raw data of every kind, cheaply, before anyone has decided how to use it. A lakehouse combines the two, adding warehouse-style tables and controls on top of lake storage. Most banks end up with some of each.

Batch and streaming

Batch pipelines move data in bulk, often overnight after end of day, and suit reports that need a complete, balanced day. Streaming pipelines move each change within seconds and suit fraud detection, alerts and live dashboards. The two often feed the same stores.

Feature stores for models

Risk and fraud models need the same inputs, called features, at training time and at the moment of a decision: “number of payments in the last hour”, “average balance over 90 days”. A feature store calculates these once and serves them consistently to both, so a model is not trained on one definition and run on another.

Lineage and quality

Lineage records where each figure came from and every step that changed it. Data quality controls check completeness, validity and consistency at each step: did every branch’s data arrive, do debits equal credits, do totals match the source. For regulatory reporting, these are not extras. They are how a bank shows that a return is right.

Retention and deletion

Keeping data forever is neither cheap nor safe. Each class of data needs a retention period, set by legal and regulatory requirements and by business need, and a process that deletes or anonymises it when that period ends. Deletion must reach every copy, including backups, analytics stores and test environments, and be recorded so it can be proved. Personal data protection law adds purpose limits: data collected for one use should not quietly be reused for another.

How it plays out in Indian banking

Where data may live is a design constraint, not only a policy. RBI's circular of 6 April 2018 requires payment system providers to store the entire data relating to their payment systems only in India; data on the foreign leg of a transaction may also be stored abroad.^{1Reserve Bank of India. Storage of Payment System Data (RBI/2017-18/153). As of 6 April 2018.} Any pipeline, copy or backup of payment data must respect that, including test and analytics copies. When data moves to a cloud platform, RBIReserve Bank of India. India's central bank. It regulates banks and payment systems, runs RTGS and NEFT, and holds the accounts in which banks settle with one another. expects the whole lifecycle to be governed: When using cloud services, a regulated entity's outsourcing policy must cover the whole lifecycle of data, from creation and entry into the cloud until it is permanently deleted.^{3Reserve Bank of India. Master Direction on Outsourcing of Information Technology Services (RBI/2023-24/102). As of 10 April 2023.} Backups count only if they work: Regulated entities must back up data and periodically restore it to check that it is usable.^{2Reserve Bank of India. Reserve Bank of India (Information Technology Governance, Risk, Controls and Assurance Practices) Directions, 2023 (full text). As of 7 November 2023.}

Trade-offs

Failure modes

• Two versions of the truth, when different teams define the same measure differently.
• Silent gaps, when one source’s feed fails and the report runs anyway on partial data.
• Copies nobody governs, such as test databases full of real customer data.
• Lineage that stops at the warehouse, so a figure can be traced back to a table but not to the transactions behind it.

When to use it, and when to avoid it

Invest in a governed warehouse, lineage and quality checks for anything that feeds a regulator, a board or a model that decides credit. For exploratory work, a lake with light controls is fine, as long as nothing leaves it for a decision or a report without passing through the governed path. Avoid copying production data into lower environments at all, unless it is masked.

Questions for a design review

• Can every figure in a regulatory return be traced to the transactions behind it?
• Which quality checks stop a report running on partial data?
• Is there one agreed definition of each key measure, and who owns it?
• Where do copies of payment data live, including backups, logs and test environments?
• When is data deleted, and how is deletion proved?