Computer scientists simplify deep learning

June 1 (UPI) -- Computer scientists at Rice University have developed a new technique for minimizing the amount of computations required for deep learning.

The simplification technique is similar to methods commonly used to minimize the amount of math required for data analysis.

"This applies to any deep-learning architecture, and the technique scales sublinearly, which means that the larger the deep neural network to which this is applied, the more the savings in computations there will be," lead researcher Anshumali Shrivastava, an assistant professor of computer science, said in a news release.

Deep-learning networks hold tremendous potential in a variety of fields, from healthcare to communications. The networks are still cumbersome, requiring significant amounts of computing power.

Scientists regularly use a data-indexing method called "hashing" to slim-down large amounts of computation. Hashing converts larger numbers and computations into smaller datasets and then organizes the information into indexes.

"Our approach blends two techniques -- a clever variant of locality-sensitive hashing and sparse backpropagation -- to reduce computational requirements without significant loss of accuracy," said Rice graduate student Ryan Spring. "For example, in small-scale tests we found we could reduce computation by as much as 95 percent and still be within 1 percent of the accuracy obtained with standard approaches."

Deep learning relies on artificial neurons, mathematical functions that turn one or more received inputs into an output. Neurons in deep learning networks begin as blank slates but become more specialized over time as they learn from the inflow of data. As neurons become more specialized, they form a hierarchy of functions.

Low-level neurons perform simple functions, while higher-level neurons are responsible for more sophisticated computations. More layers can yield more sophisticated and powerful results. But more layers require more time, space and energy.

"With 'big data,' there are fundamental limits on resources like compute cycles, energy and memory. Our lab focuses on addressing those limitations," Shrivastava said.

Researchers believe their latest efforts will allow computer scientists to more efficiently deploy large deep learning networks.

"The savings increase with scale because we are exploiting the inherent sparsity in big data," Spring said. "For instance, let's say a deep net has a billion neurons. For any given input -- like a picture of a dog -- only a few of those will become excited. In data parlance, we refer to that as sparsity, and because of sparsity our method will save more as the network grows in size. So while we've shown a 95 percent savings with 1,000 neurons, the mathematics suggests we can save more than 99 percent with a billion neurons."

Shrivastava and Spring are scheduled to share their work at the SIGKDD Conference on Knowledge Discovery and Data Mining, to be held in August in Nova Scotia.