Kx and the NASA Frontier Development Lab
By: Nicolle Eagan
It was bit daunting when I first walked through the doors of the Search for Extra-Terrestrial Intelligence Institute (SETI). I had always been casually aware of SETI; and I had always associated them with science fiction fantasies, of secret bunkers filled with some of the cast members of the movie Independence Day. You can imagine my surprise when I was told that working there would be my new 9-5.
Co-founded by Carl Sagan, the SETI Institute is a not-for-profit research organisation devoted to the search for alien life. When SETI isn’t busy living out every young science enthusiast’s dream, they also host the NASA Frontier Development Lab (FDL), an R&D accelerator which supports NASA’s operations and helps to safeguard humanity’s future.
The NASA FDL works on a variety of projects, from studying asteroids for planetary defence to identifying lunar water as an off-planet resource. Their most recent project was one of the most exciting yet: attempting to predict potentially destructive solar storms which threaten infrastructure on Earth. This was the project that I was asked to work on.
Solar storms are the end-product of the vast overheating of the supercharged protons and electrons which swirl across the surface of the sun. When the heat gets too intense these particles erupt at near light-speed across the solar system, passing massive levels of electrical charge to any unfortunate planets in their path. While these storms are relatively common, there have been few truly major events in recent history. The last truly global-level solar storm being the infamous 1859 “Carrington Event” in which telegraph services went down, operators were plagued by electric shocks and fires lit up in offices across the world as sparks flew from telegraph machines onto stacks of dry paper.
While the Carrington Event was destructive enough, it luckily took place in a world largely independent from electricity. If the same level of solar storm were to take place again today, national power grids would shut down, the global aircraft fleet forced to make mass groundings, cell phones would be knocked out, and GPS would be gone.
It was hard at first to imagine where I fit into all of this, defending the world from infrastructural collapse seemed like a major task. After researching the details of the project and discussing the problem in depth with the FDL team, I began to realise that the combination of my mathematical background with my skills in kdb+ were a perfect match to tackle their challenges. The ultimate goal was to predict storms using an LSTM (Long Short-term Memory) recurrent neural network, which is optimal for time-series predictions. With time-series analysis over massive amount of data being a main strength of kdb, I knew this would be a perfect fit.
Our mission was to do two things: load the data into kdb+, and gain insights from time-series analysis. First, I leveraged the Geomag Algorithms open source library to scrape seven years of data from the USGS Geomagnetism Program website and save the data into pickle files. Each data centre created a record for every minute of every year, and with over 500,000 minutes in a year, the amount of data added up very quickly. Then, I used Python to read in the pickle files and transpose the data from 56,000 columns to 14, and re-save it as a CSV so I could read them into kdb+ with ease. From there, it was just a task of clean up and saving my database to disk.
Finally the database was in the most optimal form to let kdb+ do its magic. True to form, kdb+ ran my analysis in nanoseconds and made my job a lot easier. This allowed me to start running queries and getting real insights into the trends and patterns behind solar storms. It is hard to believe even afterwards that those insights might yet help to protect us against something so potentially destructive.
During my time at FDL I made a presentation to the whole SETI Institute on different applications for kdb+ and gave a live demo using NASA data. A scientist using manual techniques to study long tail comets was particularly interested in the possibility of creating rich visualisation tools using kdb+. It was a privilege as a data scientist to have the opportunity to work with some of the top researchers investigating the solar system and beyond.