About three years ago the Financial Industry Regulatory Authority (FINRA) began plotting a migration to Amazon’s cloud. Most companies when they do so target low-hanging fruit: Applications that can be relatively easily lifted and shifted to the cloud.
Not FINRA. “We started with the migration of our most critical systems to AWS,” explains Steve Randich, executive vice president and CIO of FINRA, and a former CIO of the NASDAQ stock exchange and Citibank. “We moved our most mission critical, data-intensive services first.” Randich called them FINRA’s “crown jewels.”
+MORE FROM NETWORK WORLD: Amazon’s biggest re:Invent announcements | Inside Bank of America’s IT transformation +
FINRA records every order and quote in the New York Stock Exchange daily. That’s about 75 billion individual events per day. FINRA processes in one day the magnitude of data that Visa and Mastercard process in six months, Randich says.
FINRA stores this information so that it can analyze trends over days, weeks and months. That amounts to trillions of records and about 20 petabytes of storage. FINRA’s IT “center of gravity” is now in Amazon Web Services, he says.
Why move to the cloud
The U.S. Securities and Exchange Commission, the primary government regulator for the stock market, has rules upcoming that will require an even greater amount of data to be collected and analyzed compared to what is required today. This has been one of the driving factors toward FINRA’s use of the cloud, Randich says.
The organization’s legacy proprietary data warehouse appliances worked reasonably well, but Randich questioned if they were capable for serving FINRA’s future use cases. “When looking forward, we very quickly determined that we needed something different,” he said during a keynote state at AWS’s recent re:Invent conference in Las Vegas. (Watch Randich talk about FINRA’s cloud journey in the video below at the 1:33:20 mark of the re:Invent keynote.)
FINRA evaluated many providers. Legacy infrastructure vendors tried to convince him that a database of this scale could not run in the public cloud. After an evaluation and proof of concept process FINRA found AWS to be “several years ahead of the closest competitor,” a gap that Randich says is increasing.
He had four principles in moving to the cloud:
-Self-sufficiency: Randich didn’t want to rely on consultants or vendors. FINRA built expertise in-house to manage its AWS cloud.
-Public vs. private cloud: Many in the financial services world prefer a private cloud. “Our view is, why own, manage and support all that commodity hardware?” he asks. “Why not be even with Moore’s Law?” Randich says “infrastructure people who want to stay within their comfort zone” advocate for private cloud.
-Open source: Randich focused development efforts on HBase and Hive, open source database standards instead of proprietary databases from vendors that would require licensing.
-Don’t lift and shift: Instead, rewrite and replatform applications for the cloud so they can take full advantage of the opportunity.
FINRA uses a full gamut of Amazon cloud services: From EC2 virtual machines to the Elastic Container Service and Lambda for computing; Virtual Private Cloud, Route 53 and Direct Connect for networking; data services such as DynamoDB, Relational Database Service (RDS), Simple Storage Service (S3), Glacier and Elastic Block Storage (EBS); security tools like the Key Management Service (KMS), Identity and Access Management (IAM); and app features like Simple Que Service (SQS) and Simple Notification Service (SNS), among others.
Using the cloud has reduced costs, allowed FINRA to get rid of proprietary infrastructure and has allowed the organization to leverage massive processing and storage at large scale and commodity costs, Randich says. The system has a 400X improvement in interactive queries compared to the previous platform, he added. “It (was like) researching something and only being able to do a few Google searches a day, it’s impossible,” he says. “Now we can do these things in seconds and subseconds.”
FINRA can better absorb “flash-crashes” and other extreme market events by automatically spinning up tens of thousands of nodes momentarily and then taking them offline, “without generally being aware of it until after it's happened and we review the logs.” In 2015 Randich said he came to the realization that cybersecurity was better in the cloud than in on-premises environments. “It’s allowed us to achieve freedom from the proprietary database vendors that were warning us,” not to use the cloud, he said.