The fact that the brain is not encrypted and that there would be huge risks of hacking for basically any neurally-directed activity (from electromagnetic interference and/or direct stimulation of the brain) could be an important concern for having a calculator powered directly by neural activity.
Unlike farming, where most do not have knowledge that was basic for survival but all retain the ability to independently learn the skills again if we please, it would be a problem if we were unable to independently perform mathematical activities without being restricted to whatever style of thinking process was automated. You can learn to farm in minutes or hours as a labourers, and months or a few years under tutelage. But if you cannot do basic thinking without computer assistance? That would be a huge threat to freedom.
It is would be naive to think that those who could put such products to market would not be exposed to influences that would seek to direct usage and/or development in a way that would enable further control, for example via more-standardized or more-easily-proliferated mass-brain-hacking.
Aside from more obvious risks of governments brainwashing entire populations from arithmetic up, this would then be a hackable system where domestic and/or foreign operations could brainwash, influence and/or otherwise control significant shares of a population with little advance warning for voters and various institutions to amass a suitable response.
Say, it only provides the result of the mathematical query if in some way you can transmit an estimated-as-genuine pro-regime attitude alongside the request.