CME’s Director of Software Engineering, Roger Moore, reminisces over technologies that have impacted the sector and how data will be adopted to transform the industry.
Q. At what stage does Data officially become termed as Big Data?
There are various measures of what constitutes 'Big Data' at a particular company. Big Data, at its core, is a collection of data from a mixture of sources. One of the best ways to explain this is using the 3 V's, Volume, Velocity and Variety. Volume accounts for the amount of data generated from multiple sources and saved within your organisation. Velocity measures the speed of data generation, whether that be real-time or in batch or micro-batch. Variety is concerned with the different types of data that is created and processed. This can take the form of structured, semi-structured or unstructured data. All data generated by an organisation has the potential to be useful to someone and the ability to gather and analyse this data is critical to the organisation's ability to make decisions based on what is happening within their company and their wider business domain.
Q. Big data is a combination of structured, semi-structured and unstructured data – how do these definitions typically apply in the Fintech sector?
Fintech generates a lot of data. The vast majority of financial data is structured data from things such as trades or market data generated from the various markets. This is structured in predefined formats that make it understandable between different institutions and customers. It allows applications to easily consume and work with the data. Semi-Structured data encompasses data that does not fit into a relational database model but still has tags that describe the data contained. This can be things like emails, that tag things like sender, receiver, subject, or data from websites that come in a format like JSON. There is information on what the data is but the contents can vary greatly. Unstructured data can take the form of video, text voice messages and social media posts. One of the big uses of combining the different types of data in Fintech is fraud protection. The ability to understand what is happening in markets and identify trends and unusual activity allows financial services to manage the markets more effectively. It also enables companies to understand their customers better and create products that help customers and deliver the tools and features they want. Companies that are good at this can be seen to stay ahead of the curve in providing useful services to their customers.
Q. What are the most challenging and rewarding aspects of leading a team of over 70 engineers?
The most rewarding aspect of managing multiple teams is watching the success of the teams. That might sound obvious, but what success looks like for each team will be different. The obvious success of delivering stable software to the business is a given, but more rewarding across multiple teams is the success of the individuals within the teams in learning new things and progressing through their careers. Watching the teams maturing with complementary skills and personalities, growing together and enjoying the work they are doing gives me great satisfaction. Each team will be at their own point in that journey and will be structured slightly differently from the team beside them. Ensuring that every team has what they need to grow and for each individual on those teams to have the ability and opportunity to learn, and progress in their careers is hugely rewarding.
Q. Over the last two decades what have been the most transformational technologies to have impacted Fintech?
Two decades is a long time in technology. I’m showing my age as I have been in the industry that long and have seen a lot of change. When I started, a lot of financial companies were still running software on mainframes! In the last 20 years, we have moved from there to Unix and Linux servers, to server virtualization and progression of that into on-demand cloud infrastructure. The evolution of cloud computing from a beta AWS in 2002, combined with increasingly cheap storage and lower-cost computing power, propelled the data space forward as more and more data was collected, processed and put to use. With the volume of data generated yearly predicted to be around 175 Zettabytes by 2025, the value in the data has an immense value. Sitting on top of this infrastructure is the software to organise all this data. From relational Databases to NoSQL, to building Data lakes, data fabrics and data meshes, the ability to effectively store, access and query these massive amounts of data has evolved as well. With the capability to now store massive amounts of data in effective data structures, ML and AI are coming to the forefront in enabling the enhanced analytics of these datasets to give a much deeper understanding of the data and the ability to make better decisions using the data. Something that would take humans an immense amount of time.
In the financial space, the collection, monitoring and usage of this data is critical in providing good services to customers and maintaining well-functioning and well-regulated markets. Financial services continue to build better connected and informed services for customers that rely on data to provide better products based on customer needs and wants. None of this would have been possible without the evolution of computing storage and computing, the evolution of data stores on top of this, and the coming together of these with the advent of AI and ML.
Here at CME Group, we have entered into a 10-year partnership with Google that will transform derivatives markets by expanding access and creating efficiencies that will benefit market users. We are working with Google to utilise their 20+ years of AI and ML experience to gain greater insights into our own business and evolve the whole financial services sector. ML is helping us better understand how our customers are using our services and our decades of historical market data allows customers to train their models to make better decisions for their business. Our vast historical data stores allow our engineers to build intelligent data quality models to improve the data we serve to our internal and external customers, allowing us to build new and more advanced data analytic products The continued advances in AI and ML, combined with data, will open further avenues of growth, quality and customer service for many financial companies.
Q. What would be the best advice you would give to anyone just starting off their career in a Fintech company?
Fintech is a great place to work. There is a lot of innovation in the financial space and evolution of companies that have a rich history. CME Group is over 150 years old and is one of the first financial exchanges to announce a partnership to move the entire company into the cloud. It is a great place to learn technology and good practice in software development. There is lots of opportunity to learn, so if you are just starting out, try different things in the business to find the area you enjoy the most. Most financial companies have many different technology teams so there is opportunity to try new things and build your knowledge across multiple domains. This all leads to great experience and builds you into a well-rounded technologist.
This article appears in the Big Data edition of Sync NI magazine. To receive a free copy click here.
