I'm not sure about timescales here, but I know that a stick with regular etchings on it was found that is around 13,000 years old. Did this come before or after language? I don't know - but then, you can't say either.
But Maths a branch of language? Absolutely not. You can say "one", "un", "eins", "uno", "bat", or "two, "deux", "dos", zwei", "bi"... and all you mean is "one" or "two". The mathematical meaning of "one"? It's way beyond language.
And, again, "x" doesn't means "multiply by" any more than "=" means "equal to". We have assigned those symbols to mathematics, not mathematics to those symbols. The maths came first, you don't need language to describe it and what you do use is so ridiculously arbitray. Take "=" again. This means "equal to" for most people yet in computer codes "=" often means "assign this new value to this variable", while "x" just as often means "column vector" or "1xn matrix" or something.
The mathematics is already there and all we have to do is discover it. Once we do, we attach labels randomly to what we found. There is no, NO, dependence on language. Because you can do maths equally well in Swahili, French, English, Mandarin. Indeed, language is more dependent on maths because every time a new mathematical area is made we have to either invent a new word for it (maths coming first again) or invent a new definition for a word (maths coming first again). Either way the maths comes first.
Language, similarly, isn't much of an achievement because it's arbitrary and doesn't take much work to start off really. Since, after all, every other animal has their own language. And how many of them can do maths, I wonder?
And since, as you say, language is limiting the brain, doesn't that suggest that once again language is just a block, holding us back, whereas maths is the exact opposite, moving us ahead in greater strides?
Someone who stopped studying maths before they left high school is in a bit of a problem really. The maths you are talking about is high-school maths. I just earlier showed you an example of something that, quite clearly, shows that maths doesn't just describe the physical world but also IS the truth of the physical world. There are hundreds and hundreds, thousands of such cases, where the mathematics is true before you even think of hlow this aplies to reality.
A simply stunning example of this, I think, is the fact that in 1926, two physicsists, Schrodinger and Heisenberg, decided that they would look at the world of the atom in two different ways. Schrodinger chose to think of it in terms of a physical model (waves) and Heisenberg decided to focus purely on the numbers and not even try to think of what it all meant physically. And they developed theories that explained what was going on. Both of them were right, but more importantly, both of their approaches, so radically different, led to the same answers. Again, the mathematics is true before you try to match it to the real world.
The fact is that whether you choose "1", "2", "3" or "sausages" to descibe the quantity of "1", that quantity exsists independent of what you call it. A bit like something called a "linear operator" that exists independent of what co-ordinates you choose to work with.